Comments (14)
@looztra, I can help with that. Please look at https://github.com/salesforce/sloop#build-from-source . Follow all the steps mentioned over there but before you run make
, you need to make two changes.
- Run
go get -v -u github.com/dgraph-io/badger/v2@0edfe98dbc31621145f8bfe3e7af86bde04bdbb5
This will update the badger version in sloop. If this runs successfully, you should have change in go.mod and go.sum file.
- We have changed one of the APIs in badger so make the following change in sloop.
diff --git a/pkg/sloop/store/untyped/store.go b/pkg/sloop/store/untyped/store.go
index 7bb098e..eaa8e7a 100644
--- a/pkg/sloop/store/untyped/store.go
+++ b/pkg/sloop/store/untyped/store.go
@@ -9,11 +9,12 @@ package untyped
import (
"fmt"
+ "os"
+ "time"
+
badger "github.com/dgraph-io/badger/v2"
"github.com/golang/glog"
"github.com/salesforce/sloop/pkg/sloop/store/untyped/badgerwrap"
- "os"
- "time"
)
type Config struct {
@@ -51,10 +52,6 @@ func OpenStore(factory badgerwrap.Factory, config *Config) (badgerwrap.DB, error
opts = badger.DefaultOptions(config.RootPath)
}
- if config.BadgerEnableEventLogging {
- opts = opts.WithEventLogging(true)
- }
-
if config.BadgerMaxTableSize != 0 {
opts = opts.WithMaxTableSize(config.BadgerMaxTableSize)
}
After this, you can run make
and you have a the latest sloop binary in your $GOPATH/bin
.
from sloop.
Thanks @looztra for raising the issue. We know of this issue and its related to garbage collection. We are currently working in the fix which is almost ready. A PR would be coming next week with the fix.
from sloop.
For the record, the last infos in the README regarding memory tuning associated to the latest version published were really useful as we can now run sloop within the memory limits we chose (1Gi) without having to lower the maxLookBack
to 1h
.
from sloop.
I've restarted the pod and removed existing data.
Each 30 minutes (the default value for kube-watch-resync-interval
), the container consumes 300Mo of memory
from sloop.
After some investigations, it seems to be related to Badger.
Running pprof on a local (workstation) sloop process gives :
When profiling a sloop running inside a container in the k8s instance gives :
This seems related to these Badger issues :
from sloop.
Hey @sana-jawad and @looztra, I work on badger and I'm trying to reduce the memory consumption. I have a PR dgraph-io/badger#1308 which I expect would reduce the memory used by decompression but I haven't been able to reproduce the high memory usage issue.
It would be very kind of you if you can test my PR in sloop and confirm if the memory usage was reduced. Or if you have some steps that I can follow to reproduce the high memory usage, I'd be happy to do that.
from sloop.
Hard for me to provide a way to reproduce without a kubernetes cluster running.
I'd be happy to test this PR inside sloop (and run it against the cluster I used previously), but as I'm not a go dev, I'm not sure how to produce a sloop binary that would integrate the badger version associated to this PR.
Any hints on the steps needed to do that?
from sloop.
Thank you very much for the instructions, I was able to build a sloop version with the fix. It is currently running.
A first memory dump 30 minutes after the start is very promising :
I will wait a few more hours and post new results after that.
from sloop.
looks good, the consumed memory stays low!
from sloop.
@jarifibrahim I have tested the PR and it has reduced the memory consumption. Thanks for the pointer. I have noticed that the memory consumption is directly proportional to the rate of incoming data. I am going to try setting the flag for badger-keep-l0-in-memory to false. Any other pointers that can help in memory reduction?
@looztra try following values for sloop flags for less memory consumption.
badger-use-lsm-only-options: false
badger-keep-l0-in-memory:false
The PR for keeping sloop disk size in check when garbage collection limit is hit is also in review. It is also a factor that helps in reducing memory consumption.
from sloop.
We are especially monitoring the value of container_memory_working_set_bytes as it is the value watched by the OOM killer.
The value was growing of 300Mo every 2 hours, up to 6Go without the patch.
Now we observe values staying around 300Mo (with the same amount of watchable update count) so we are pretty happy without having to play with the flags you mentioned.
On the graph, the usage (container_memory_usage_bytes) value is what looks like the closest to the process_resident_memory_bytes
from sloop.
Hey @looztra and @sana-jawad, thank you for testing my PR. It was definitely helpful.
However, my change, I do this
https://github.com/dgraph-io/badger/blob/0edfe98dbc31621145f8bfe3e7af86bde04bdbb5/table/table.go#L643-L651
which means I took a byte slice from pool and reduced it's length to zero (it's capacity is still the same). Now, this zero length buffer is passed to snappy. If you look at the code below, you'll notice that snappy will allocate memory if length of buffer is zero (and we've given it a zero length buffer)
https://github.com/golang/snappy/blob/ff6b7dc882cf4cfba7ee0b9f7dcc1ac096c554aa/decode.go#L62-L67
So my PR shouldn't cause any reduction in memory usage. The reduction in memory was because of commit dgraph-io/badger@c3333a5 which disabled compression by default in badger.
I noticed that the go.mod
in sloop is using badger v2.0.0 . We've released v2.0.3 which disabled compression by default. So the code that @looztra and @sana-jawad tested isn't using compression and hence the low memory usage.
I would suggest you update badger in sloop and use the latest version of badger.
from sloop.
Thanks @jarifibrahim. Yes the upgrade to 2.0.2 was already in review. I will update it to move to 2.0.3.
from sloop.
Thats great to know @looztra!
from sloop.
Related Issues (20)
- Publish HelmChart HOT 6
- How to query real-time data
- Template missing HOT 1
- Failed to create kubernetes client: invalid configurations: no configuration has been provided
- Permissions on ghcr.io container image HOT 1
- Error: INSTALLATION FAILED: YAML parse error on sloop/templates/statefulset.yaml: error converting YAML to JSON: yaml: line 20: did not find expected key (New) HOT 1
- Im unable to get logs HOT 1
- Update helm chart for Private image Pull HOT 1
- sloop not working via ingress rule HOT 1
- while running sloop application getting violation of Content Security policy violation
- Data race in refreshStats HOT 6
- Define Go versioning policy and update version
- Helm Chart still on 0.1.0 version and image tag latest HOT 1
- Update Release Process
- sloop catβt work offline HOT 3
- Add support for policy/v1 PodDisruptionBudget
- Sloop Service Prometheus Metrics view "Not Found"
- Sloop OOMKilled
- A Toggle for pods in Compleded state
- "glibc not found" causes CrashLoopBackOff in fresh installation with helm HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sloop.