GNU sort uses too much space on /tmp; runs out of space about distiller-nf HOT 7 CLOSED

It should be logical to put sort temp files in the work directlry perhaps, or wherever all other temp files are created.

from distiller-nf.

I had the same problem and the pipeline failed because of insufficient space. I ended up providing available space 30x the size of the input to get around the problem. I would also suggest putting tmp files somewhere in the work dir. Also, it would help to see full paths in the logs instead of relative paths.

from distiller-nf.

When distiller is running on docker images, it stores the tmp files inside /var/lib/docker
Inside /var/lib/docker, the intermediate files are stored here: /var/lib/docker/aufs

So, unless we can specify the tmp folder for GNU sort, one needs a lot of space available in the OS volume.

##########################

du -hs /var/lib/*
12K /var/lib/AccountsService
12K /var/lib/apparmor
132M /var/lib/apt
4.0K /var/lib/dbus
8.0K /var/lib/dhcp
158G /var/lib/docker

du -hs /var/lib/docker/aufs
165G /var/lib/docker/aufs

from distiller-nf.

@hakanozadam @sergpolly do you think the issues of space got completely fixed?

from distiller-nf.

I haven't experienced first hand, so I asked @hakanozadam to check

from distiller-nf.

I am still running an earlier version of distiller. But I plan to upgrade to the most recent one soon. Then I can check and provide feedback.

from distiller-nf.

this issue is solved, but i'll create a new one for a similar issue

from distiller-nf.