Badger Compaction Issue about sloop HOT 5 CLOSED

Local repro was as easy as you described. Treating this as a high pri issue.

from sloop.

I'm hopeful that an updated badger version fixes this. There's an issue referenced in dgraph-io/badger#1062 that was fixed in the latest release. This is a data-store breaking change, unfortunately. But better now than later on down the line.

Interestingly, this won't actually fix the max-disk-mb=1 method of reproducing the issue. What's happening there is that our compaction runs in a tight loop, since it doesn't have the disk space to store the current state of all the k8s resources. We should have a better (really, any at all) error message in the UI declaring that you need to allocate at least enough disk space to store the current state. (That would be a good first issue!)

The upgrade to badger v2 should fix the compaction oom crash, though. I'm still testing.

from sloop.

Ive been looking into this a bit. Sloop has a background job to detect when there is too much data on disk and clean up old keys with Badger DropPrefix. It appears that is not resulting in any actual cleanup on the disk, so it goes into a fairly tight loop. While DropPrefix is running all reads and writes to Badger are blocked. Still researching a fix.

from sloop.

Thanks - we're looking at this internally as well. I will do some investigation and get back to you.

from sloop.

Give the latest build a try. In is after the badger v2 upgrade, so you will need to wipe your old data unfortunately. But supposedly badger is going to stop making breaking changes.

from sloop.