vmagent: Potential drop of incorrect persistence queue when removing remotewrite URL about victoriametrics HOT 3 CLOSED

Some appendix after re-test: It may be unsafe to remove any remotewrite configuration except the last one.

To reproduce:

1. setup multiple remotewrite configurations with different URL. The folder structure should be:

└── persistent-queue
    ├── 1_A
    ├── 2_B
    ├── 3_C
    ├── 4_D
    ├── 5_E
    └── 6_F

2. remove 1st remotewrite configuration. The expected folder structure becomes:

└── persistent-queue
    ├── 1_B
    ├── 2_C
    ├── 3_D
    ├── 4_E
    └── 5_F

3. vmagent compare the expect result and current folder structure, and decided to:

• remove 6 folders.
• create 5 folders.

Logs:

2024-04-18T09:00:30.493Z        info    app/vmagent/remotewrite/remotewrite.go:266      removing dangling queue "1_B6EB6B7CD67EE018"
2024-04-18T09:00:30.500Z        info    app/vmagent/remotewrite/remotewrite.go:266      removing dangling queue "2_7671E3D57693E853"
2024-04-18T09:00:30.506Z        info    app/vmagent/remotewrite/remotewrite.go:266      removing dangling queue "3_ABAF801ED13C9AA1"
2024-04-18T09:00:30.512Z        info    app/vmagent/remotewrite/remotewrite.go:266      removing dangling queue "4_0FA401F50943511E"
2024-04-18T09:00:30.521Z        info    app/vmagent/remotewrite/remotewrite.go:266      removing dangling queue "5_39D21E2644A67225"
2024-04-18T09:00:30.528Z        info    app/vmagent/remotewrite/remotewrite.go:266      removing dangling queue "6_ED4C191D78378D55"
2024-04-18T09:00:30.534Z        info    app/vmagent/remotewrite/remotewrite.go:273      removed 6 dangling queues from "vmagent-remotewrite-data", active queues: 5

from victoriametrics.

Hello @jiekun!
Thanks for such detailed report! Really appreciate it!

Potential solutions would be:

I don't think we need to fix it, at least for now. Please see my reasoning in this commit 4251292

from victoriametrics.

Sure. Let me close the issue. Thanks for your attention.

It's still necessary to mention the case again as in the 1st comment:

• if we have n configs (with different URLs), removing the configs[i] will result in potential data loss for config i+1 to n remote write.

Instead of just covering the case with identical urls, it would be good to cover the more general one (also a more risky one) in code comment. And I tried to ensure that this case is also noticed by our maintainers, rather than the original bug report.

from victoriametrics.