Comments (12)
Hi @m90 ,
Sorry my late answer.
Thank you so much for your investigation and your time for looking into this, it's far above my skills in Docker :)
So since last time, i did not notice any bump of memory. The last executions of backup were successful and the container never exceeded 30mb of memory (it reached this value after one week of running) but stayed for several days below 5mb.
This figures are from docker stat.
So far everything seems fine but starting from today, i will add this memory limit to prevent any issue.
I will keep you posted if i notice any restart of the container or something else.
Thank you very much
from docker-volume-backup.
and the container never exceeded 30mb of memory (it reached this value after one week of running) but stayed for several days below 5mb
I would assume this happens because Docker knows it can purge the cache entries the backup container created once other
containers build up more memory pressure. Let me know if setting the limit changed anything in your context.
from docker-volume-backup.
Hmm, I have multiple containers of the image running on multiple hosts for a long time now and none of them consumes more than 10MB of RAM (if even). Backup sizes are ~1GB as well here.
I also don't really know how a shell script could leak memory other than by hanging indefinitely. Will you see these lines:
docker-volume-backup/src/backup.sh
Lines 92 to 93 in 5fd6f66
printed in your log output? What's the cron interval you run this on?
from docker-volume-backup.
Thanks for your help.
Yes i see the right message at the end of the job:
[INFO] Backup finished
Will wait for next scheduled backup.
[INFO] Pruning old backups
Sleeping 10m before checking eligibility.
No backups found older than the configured retention period of 15 days.
Doing nothing.
This is my cron expression:
BACKUP_CRON_EXPRESSION: "0 5 * * 1,3,5"
so it's running more or less every 3 days in a docker Swarm environment (and network is of type Overlay).
I don't know if it can help but:
Docker version 20.10.7, build f0df350
Debian version 10.10
VPS: 8gb ram, 4 vCore
I just restarted the service and when it starts the memory usage is just 2mb.
from docker-volume-backup.
I currently don't have any hint about what to do here unfortunately as I can't reproduce this in any way. If you have any further hints on why this could be happening I'd be happy to look into it.
As a workaround you could always set a memory limit on the backup container and have Docker OOM kill and restart it once it reaches a threshold.
from docker-volume-backup.
It would also be interesting to know if your memory usage is growing constantly (i.e. while the container is idling), or if running the backup script increases the memory.
from docker-volume-backup.
It would also be interesting to know if your memory usage is growing constantly (i.e. while the container is idling), or if running the backup script increases the memory.
Yes i will definitively check this! i will keep you posted.
Thanks
from docker-volume-backup.
So I tried reproducing this using the test cases in here https://github.com/offen/docker-volume-backup/tree/main/test and it seems this is only happening in Swarm mode, when plain compose is unaffected.
The leak seems to happen on running the script which is very odd as it's exiting successfully, which means it should theoretically free all resources that have been allocated by the script.
What's kind of interesting though is the following: when you do not mount the host's Docker socket into the backup container, the leaking does not happen in Swarm mode anymore, just like when removing all labels that would stop containers, which means it's likely connected to this.
I wonder if this is related to Docker inside of the container erroneously keeping references to old containers that would be purged outside of the container somehow.
from docker-volume-backup.
After looking into this further I'm kind of stumped here to be honest. What I found out:
- when inspecting the processes while a backup is running in the container using
top
I can see that all processes exit cleanly and the two long running processes (the cron and the entrypoint script) never change anything about the amount of memory they consume - looking at what
free
prints I can also see no change in memory stats from inside the container at all
Looking at the memory stats of Docker for the container itself (https://docs.docker.com/config/containers/runmetrics/#metrics-from-cgroups-memory-cpu-block-io) I couldn't find out anything obvious either (it doesn't seem to accrue unevictable memory).
My assumption now is that this leak is happening on Docker level (not on container level) for reasons I do not understand but might well be related to accessing the "swarmy" docker socket inside of the container.
If anyone has any idea why this might be happening feel free to chime in.
from docker-volume-backup.
Yeah, I can confirm that the leak seems to happen at cgroup level and not inside the container as dropping caches on the host via
echo 3 | sudo tee /proc/sys/vm/drop_caches
resets the memory usage displayed by docker stats
back to where it should be.
Some things from the web about this (most of them k8s related):
- https://stackoverflow.com/questions/59922728/container-memory-keeps-growing-while-application-and-top-report-stable-memory
- https://stackoverflow.com/questions/50865763/memory-usage-discrepancy-cgroup-memory-usage-in-bytes-vs-rss-inside-docker-con
- https://stackoverflow.com/questions/46677536/incorrect-reporting-of-container-memory-usage-by-cadvisor
- https://discuss.linuxcontainers.org/t/how-to-drop-caches-inside-a-container/3200
from docker-volume-backup.
@GrandOurs35 it seems you can prevent the leak by simply applying a hard memory limit on the service. When running the test setup like this:
backup: &default_backup_service
image: offen/docker-volume-backup:${TEST_VERSION}
depends_on:
- minio
deploy:
restart_policy:
condition: on-failure
resources:
limits:
memory: 25M
environment:
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: something-something
AWS_ENDPOINT: minio:9000
AWS_S3_BUCKET_NAME: backup
BACKUP_FILENAME: test.tar.gz
BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
the container's memory usage stayed at bay even after a lot of backup runs. Docker will probably take care of purging cached entries in such case and the scripts inside the container do not require too much memory. Could you try this in your setup and see if it prevents the leak from happening so I can add it to the documentation?
from docker-volume-backup.
I added a note about this to the README and will be closing this issue now as it seems to be solved. Feel free to reopen if you run into any unforeseen behavior regarding this again.
from docker-volume-backup.
Related Issues (20)
- loadConfigsFromEnvFiles does not ignore comments in file configs HOT 3
- "Page not found" error when using with balena HOT 9
- Hard to understand error message during exec HOT 3
- archive-pre is triggered on each container backup instead of just container with exec-label HOT 3
- Health Check HOT 6
- Notifications Delay HOT 1
- Replace Log line feeds with HTML <br> tag HOT 3
- Backup the entire content without directory HOT 4
- Backup to subfolders HOT 2
- Resolve date in AWS_S3_PATH HOT 1
- Can't get stop during backup working HOT 2
- Remove local backup after upload to S3 HOT 3
- Invalid memory address crash when running a backup HOT 16
- Fail to configure cron schedule HOT 3
- Docker swarm error HOT 5
- "swarm exec" support HOT 1
- "input/output error" when copying file to NFS mount HOT 4
- Azure storage: Support access tier HOT 2
- schedule the docker-volume-backup? HOT 1
- [Feature] Automated backups of all named volumes HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-volume-backup.