vector database - Compaction Plan Conflict After Cluster Restart in Milvus - Stack Overflow

admin管理员组

文章数量:1313372

After restarting the Milvus cluster, I noticed that many compaction tasks are facing conflicts. Before the restart, there were a large number of ongoing compaction tasks, which seem to have remained in the task queue. As a result, many segments have their metadata state set to inCompacting.

However, after restarting the cluster, the task queue becomes empty, and the coordinator starts creating new compaction plans. Most of these newly generated compaction plans fail with the following error:

[2025/01/03 01:14:22.100 +00:00] [WARN] [datacoord/compaction_trigger.go:490] ["failed to execute compaction task"] 
[collection=455007134314668300] [triggerID=455046071905748780] [planID=455046071905748783] 
[inputSegments="[455007134350525077,455007134316823700,455007134381307513,455046071898345279]"] 
[error="segment is compacting: compaction plan conflict"]

I suspect that these input segments are still marked as inCompacting in etcd, preventing Milvus from executing the new compaction tasks. If these segments remain inCompacting, does that mean they can never be compacted again? Does Milvus have a mechanism to restore such metadata after a failure?

After restarting the Milvus cluster, I noticed that many compaction tasks are facing conflicts. Before the restart, there were a large number of ongoing compaction tasks, which seem to have remained in the task queue. As a result, many segments have their metadata state set to inCompacting.

However, after restarting the cluster, the task queue becomes empty, and the coordinator starts creating new compaction plans. Most of these newly generated compaction plans fail with the following error:

[2025/01/03 01:14:22.100 +00:00] [WARN] [datacoord/compaction_trigger.go:490] ["failed to execute compaction task"] 
[collection=455007134314668300] [triggerID=455046071905748780] [planID=455046071905748783] 
[inputSegments="[455007134350525077,455007134316823700,455007134381307513,455046071898345279]"] 
[error="segment is compacting: compaction plan conflict"]

I suspect that these input segments are still marked as inCompacting in etcd, preventing Milvus from executing the new compaction tasks. If these segments remain inCompacting, does that mean they can never be compacted again? Does Milvus have a mechanism to restore such metadata after a failure?

• vector-database
• milvus

Share Improve this question Follow asked Jan 30 at 20:51 tmandyaitmandyai 8311 silver badge44 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

After investigating the issue, I found that the core logic related to compaction is mainly handled in datacoord, and compaction tasks are recorded in metadata. Upon restarting, any unfinished compaction tasks are read from the metadata and executed.

The relevant logic can be found in:

milvus/internal/datacoord/compaction.go

Line 90 in e6bf141

meta             CompactionMeta 

When tasks are loaded from metadata at startup, the corresponding segments are marked as isCompacting. However, compaction_trigger and the recovery mechanism operate independently. The compaction_trigger scans for segments that can be compacted, but when it encounters segments already marked as isCompacting, it logs the following warning:

segment is compacting: compaction plan conflict

This conflict can be safely ignored. The compaction_trigger will skip those isCompacting segments. Once the previously recorded tasks from metadata are completed, compaction_trigger will resume normal operation, and compaction will proceed as expected.

In summary:

The warning message does not indicate a failure but a temporary state. The system will not attempt to compact isCompacting segments again until the previously unfinished compaction tasks are completed. Once those tasks finish, the new compaction plans will execute normally.

本文标签： vector databaseCompaction Plan Conflict After Cluster Restart in MilvusStack Overflow