[badgerdb] vlog growing unbounded - consider adding GC and exposing options about server HOT 6 OPEN

I've checked the official documentation of Badger regarding garbage collection. @woodzy , you're right.

Since neither of us has experience using BadgerDB in a production environment, if you could, it would be great if you could submit a PR. This would be very helpful for most users who use BadgerDB as a persistent database.

Garbage Collection

Badger values need to be garbage collected, because of two reasons:

• Badger keeps values separately from the LSM tree. This means that the compaction operations that clean up the LSM tree do not touch the values at all. Values need to be cleaned up separately.

• Concurrent read/write transactions could leave behind multiple values for a single key, because they are stored with different versions. These could accumulate, and take up unneeded space beyond the time these older versions are needed.

DB.RunValueLogGC(): This method is designed to do garbage collection while Badger is online. Along with randomly picking a file, it uses statistics generated by the LSM-tree compactions to pick files that are likely to lead to maximum space reclamation. It is recommended to be called during periods of low activity in your system, or periodically. One call would only result in removal of at max one log file. As an optimization, you could also immediately re-run it whenever it returns nil error (indicating a successful value log GC), as shown below.

ticker := time.NewTicker(5 * time.Minute)
defer ticker.Stop()
for range ticker.C {
again:
 err := db.RunValueLogGC(0.7)
 if err == nil {
       goto again
 }
}

Memory usage

Badger’s memory usage can be managed by tweaking several options available in the Options struct that is passed in when opening the database using DB.Open.

• Number of memtables (Options.NumMemtables) If you modify Options.NumMemtables, also adjust Options.NumLevelZeroTables and Options.NumLevelZeroTablesStall accordingly.
• Number of concurrent compactions (Options.NumCompactors)
• Size of table (Options.BaseTableSize)
• Size of value log file (Options.ValueLogFileSize)
  If you want to decrease the memory usage of Badger instance, tweak these options (ideally one at a time) until you achieve the desired memory usage.

from server.

@woodzy Can this #371 fix the issue?

from server.

Thanks! These look like better implementations of my local workarounds during experimentation.

from server.