Memory leak about docker-volume-backup HOT 12 CLOSED

Hi @m90 ,
Sorry my late answer.
Thank you so much for your investigation and your time for looking into this, it's far above my skills in Docker :)

So since last time, i did not notice any bump of memory. The last executions of backup were successful and the container never exceeded 30mb of memory (it reached this value after one week of running) but stayed for several days below 5mb.
This figures are from docker stat.

So far everything seems fine but starting from today, i will add this memory limit to prevent any issue.
I will keep you posted if i notice any restart of the container or something else.
Thank you very much

from docker-volume-backup.

and the container never exceeded 30mb of memory (it reached this value after one week of running) but stayed for several days below 5mb

I would assume this happens because Docker knows it can purge the cache entries the backup container created once other
containers build up more memory pressure. Let me know if setting the limit changed anything in your context.

from docker-volume-backup.

Hmm, I have multiple containers of the image running on multiple hosts for a long time now and none of them consumes more than 10MB of RAM (if even). Backup sizes are ~1GB as well here.

I also don't really know how a shell script could leak memory other than by hanging indefinitely. Will you see these lines:

printed in your log output? What's the cron interval you run this on?

from docker-volume-backup.

Thanks for your help.
Yes i see the right message at the end of the job:

[INFO] Backup finished

Will wait for next scheduled backup.

[INFO] Pruning old backups

Sleeping 10m before checking eligibility.
No backups found older than the configured retention period of 15 days.
Doing nothing.

This is my cron expression:

BACKUP_CRON_EXPRESSION: "0 5 * * 1,3,5"

so it's running more or less every 3 days in a docker Swarm environment (and network is of type Overlay).
I don't know if it can help but:

Docker version 20.10.7, build f0df350
Debian version 10.10
VPS: 8gb ram, 4 vCore

I just restarted the service and when it starts the memory usage is just 2mb.

from docker-volume-backup.

I currently don't have any hint about what to do here unfortunately as I can't reproduce this in any way. If you have any further hints on why this could be happening I'd be happy to look into it.

As a workaround you could always set a memory limit on the backup container and have Docker OOM kill and restart it once it reaches a threshold.

from docker-volume-backup.

It would also be interesting to know if your memory usage is growing constantly (i.e. while the container is idling), or if running the backup script increases the memory.

from docker-volume-backup.

It would also be interesting to know if your memory usage is growing constantly (i.e. while the container is idling), or if running the backup script increases the memory.

Yes i will definitively check this! i will keep you posted.
Thanks

from docker-volume-backup.

So I tried reproducing this using the test cases in here https://github.com/offen/docker-volume-backup/tree/main/test and it seems this is only happening in Swarm mode, when plain compose is unaffected.

The leak seems to happen on running the script which is very odd as it's exiting successfully, which means it should theoretically free all resources that have been allocated by the script.

What's kind of interesting though is the following: when you do not mount the host's Docker socket into the backup container, the leaking does not happen in Swarm mode anymore, just like when removing all labels that would stop containers, which means it's likely connected to this.

I wonder if this is related to Docker inside of the container erroneously keeping references to old containers that would be purged outside of the container somehow.

from docker-volume-backup.

After looking into this further I'm kind of stumped here to be honest. What I found out:

• when inspecting the processes while a backup is running in the container using top I can see that all processes exit cleanly and the two long running processes (the cron and the entrypoint script) never change anything about the amount of memory they consume
• looking at what free prints I can also see no change in memory stats from inside the container at all

Looking at the memory stats of Docker for the container itself (https://docs.docker.com/config/containers/runmetrics/#metrics-from-cgroups-memory-cpu-block-io) I couldn't find out anything obvious either (it doesn't seem to accrue unevictable memory).

My assumption now is that this leak is happening on Docker level (not on container level) for reasons I do not understand but might well be related to accessing the "swarmy" docker socket inside of the container.

If anyone has any idea why this might be happening feel free to chime in.

from docker-volume-backup.

Yeah, I can confirm that the leak seems to happen at cgroup level and not inside the container as dropping caches on the host via

echo 3 | sudo tee /proc/sys/vm/drop_caches

resets the memory usage displayed by docker stats back to where it should be.

Some things from the web about this (most of them k8s related):

• https://stackoverflow.com/questions/59922728/container-memory-keeps-growing-while-application-and-top-report-stable-memory
• https://stackoverflow.com/questions/50865763/memory-usage-discrepancy-cgroup-memory-usage-in-bytes-vs-rss-inside-docker-con
• https://stackoverflow.com/questions/46677536/incorrect-reporting-of-container-memory-usage-by-cadvisor
• https://discuss.linuxcontainers.org/t/how-to-drop-caches-inside-a-container/3200

from docker-volume-backup.

@GrandOurs35 it seems you can prevent the leak by simply applying a hard memory limit on the service. When running the test setup like this:

  backup: &default_backup_service
    image: offen/docker-volume-backup:${TEST_VERSION}
    depends_on:
      - minio
    deploy:
      restart_policy:
        condition: on-failure
      resources:
        limits:
          memory: 25M
    environment:
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: something-something
      AWS_ENDPOINT: minio:9000
      AWS_S3_BUCKET_NAME: backup
      BACKUP_FILENAME: test.tar.gz
      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?

the container's memory usage stayed at bay even after a lot of backup runs. Docker will probably take care of purging cached entries in such case and the scripts inside the container do not require too much memory. Could you try this in your setup and see if it prevents the leak from happening so I can add it to the documentation?

from docker-volume-backup.

I added a note about this to the README and will be closing this issue now as it seems to be solved. Feel free to reopen if you run into any unforeseen behavior regarding this again.

from docker-volume-backup.