Excessive memory consumption? about sloop HOT 14 CLOSED

@looztra, I can help with that. Please look at https://github.com/salesforce/sloop#build-from-source . Follow all the steps mentioned over there but before you run make, you need to make two changes.

1. Run

go get -v -u github.com/dgraph-io/badger/v2@0edfe98dbc31621145f8bfe3e7af86bde04bdbb5 

This will update the badger version in sloop. If this runs successfully, you should have change in go.mod and go.sum file.

2. We have changed one of the APIs in badger so make the following change in sloop.

diff --git a/pkg/sloop/store/untyped/store.go b/pkg/sloop/store/untyped/store.go
index 7bb098e..eaa8e7a 100644
--- a/pkg/sloop/store/untyped/store.go
+++ b/pkg/sloop/store/untyped/store.go
@@ -9,11 +9,12 @@ package untyped
 
 import (
 	"fmt"
+	"os"
+	"time"
+
 	badger "github.com/dgraph-io/badger/v2"
 	"github.com/golang/glog"
 	"github.com/salesforce/sloop/pkg/sloop/store/untyped/badgerwrap"
-	"os"
-	"time"
 )
 
 type Config struct {
@@ -51,10 +52,6 @@ func OpenStore(factory badgerwrap.Factory, config *Config) (badgerwrap.DB, error
 		opts = badger.DefaultOptions(config.RootPath)
 	}
 
-	if config.BadgerEnableEventLogging {
-		opts = opts.WithEventLogging(true)
-	}
-
 	if config.BadgerMaxTableSize != 0 {
 		opts = opts.WithMaxTableSize(config.BadgerMaxTableSize)
 	}

After this, you can run make and you have a the latest sloop binary in your $GOPATH/bin.

from sloop.

Thanks @looztra for raising the issue. We know of this issue and its related to garbage collection. We are currently working in the fix which is almost ready. A PR would be coming next week with the fix.

from sloop.

For the record, the last infos in the README regarding memory tuning associated to the latest version published were really useful as we can now run sloop within the memory limits we chose (1Gi) without having to lower the maxLookBack to 1h.

from sloop.

I've restarted the pod and removed existing data.
Each 30 minutes (the default value for kube-watch-resync-interval), the container consumes 300Mo of memory

from sloop.

After some investigations, it seems to be related to Badger.

Running pprof on a local (workstation) sloop process gives :

When profiling a sloop running inside a container in the k8s instance gives :

This seems related to these Badger issues :

• Decompression uses too much memory
• Reducing memory usage is complicated

from sloop.

Hey @sana-jawad and @looztra, I work on badger and I'm trying to reduce the memory consumption. I have a PR dgraph-io/badger#1308 which I expect would reduce the memory used by decompression but I haven't been able to reproduce the high memory usage issue.

It would be very kind of you if you can test my PR in sloop and confirm if the memory usage was reduced. Or if you have some steps that I can follow to reproduce the high memory usage, I'd be happy to do that.

from sloop.

Hard for me to provide a way to reproduce without a kubernetes cluster running.

I'd be happy to test this PR inside sloop (and run it against the cluster I used previously), but as I'm not a go dev, I'm not sure how to produce a sloop binary that would integrate the badger version associated to this PR.

Any hints on the steps needed to do that?

from sloop.

Thank you very much for the instructions, I was able to build a sloop version with the fix. It is currently running.
A first memory dump 30 minutes after the start is very promising :

I will wait a few more hours and post new results after that.

from sloop.

looks good, the consumed memory stays low!

from sloop.

@jarifibrahim I have tested the PR and it has reduced the memory consumption. Thanks for the pointer. I have noticed that the memory consumption is directly proportional to the rate of incoming data. I am going to try setting the flag for badger-keep-l0-in-memory to false. Any other pointers that can help in memory reduction?

@looztra try following values for sloop flags for less memory consumption.
badger-use-lsm-only-options: false
badger-keep-l0-in-memory:false
The PR for keeping sloop disk size in check when garbage collection limit is hit is also in review. It is also a factor that helps in reducing memory consumption.

from sloop.

We are especially monitoring the value of container_memory_working_set_bytes as it is the value watched by the OOM killer.

The value was growing of 300Mo every 2 hours, up to 6Go without the patch.

Now we observe values staying around 300Mo (with the same amount of watchable update count) so we are pretty happy without having to play with the flags you mentioned.

On the graph, the usage (container_memory_usage_bytes) value is what looks like the closest to the process_resident_memory_bytes

from sloop.

Hey @looztra and @sana-jawad, thank you for testing my PR. It was definitely helpful.

However, my change, I do this
https://github.com/dgraph-io/badger/blob/0edfe98dbc31621145f8bfe3e7af86bde04bdbb5/table/table.go#L643-L651
which means I took a byte slice from pool and reduced it's length to zero (it's capacity is still the same). Now, this zero length buffer is passed to snappy. If you look at the code below, you'll notice that snappy will allocate memory if length of buffer is zero (and we've given it a zero length buffer)
https://github.com/golang/snappy/blob/ff6b7dc882cf4cfba7ee0b9f7dcc1ac096c554aa/decode.go#L62-L67

So my PR shouldn't cause any reduction in memory usage. The reduction in memory was because of commit dgraph-io/badger@c3333a5 which disabled compression by default in badger.

I noticed that the go.mod in sloop is using badger v2.0.0 . We've released v2.0.3 which disabled compression by default. So the code that @looztra and @sana-jawad tested isn't using compression and hence the low memory usage.

I would suggest you update badger in sloop and use the latest version of badger.

from sloop.

Thanks @jarifibrahim. Yes the upgrade to 2.0.2 was already in review. I will update it to move to 2.0.3.

from sloop.

Thats great to know @looztra!

from sloop.