[Bug]: "Bad Gateway" error on homepage when servers runs out of storage about coolify HOT 5 OPEN

I had the almost same error, but only after 2 minutes from a fresh boot of my VPS (1vCore/1Gb RAM/ 10Gb storage). My VPS became very instable, barely able to even perform a simple ls -l.
I had to hard reboot it from my hosting provider's dashboard where I also found out that my instance's disk reads went through the roof as soon as Coolify started.

After doing some research, I found #1482 which hinted me to the fact that my VPS didn't have any swap file in place. After creating a 1Gb swapfile, I restarted Coolify which instantly started using the swap (195M out of 1024M as I type this). My VPS seems to be much stable now.

from coolify.

I ran into the bad gateway again when storage was full.
Reset my box to defaults, installed coolify but didn't try to setup any services or anything.
My storage currently is sitting at 3.55 GB of 5 GB Used/1.45 GB Free

I'm going to check back in a few days and see if the storage gets all filled up again without it doing anything.

from coolify.

Right now it says 3.97 GB of 5 GB Used/1.03 GBFree so about 400mb more are being used now without having installed anything.

from coolify.

Now the VPS is at 4.19 GB of 5 GB Used/826.29 MBFree

from coolify.

So I think... I figured it out somewhat.
running journalctl I see every second or so a message like this:
run-docker-runtime\x2drunc-moby-b9a4c56d5830a2d1e21bc537b535b3860d3d6ce917329f61f29f1455c51b8521-runc.XDlk09.mount: Succeeded
Running journalctl --disk-usage shows I'm using 456MB of disk for this.

Which is related to Docker flooding syslog with run-docker-runtime logs. Which ultimately links to systemd logs filled with mount unit entries if healtcheck is enabled.

Long story short, I think it'll be addressed with runc 1.2. Which then needs to be released by containerd here.

Until then, you can take your pick of temp fixes with filtering or limiting the log size mentioned in the above links such as:
set SystemMaxUse=50M in /etc/systemd/journald.conf or similar as mentioned here. SystemMaxUse defaults to 10% of the filesystem size.

This leaves an question open for me:
How did a 5GB system (where 500MB can be used for journalctl), start with 1.45GB free and currently have 826MB free? I don't think I've really installed or done anything else. Though perhaps I filled it up last time by trying to update coolify and not having enough space, which took up just enough of the remaining space.

For now, I'll leave everything with the default configs and see if the disk fills up again on its own.

from coolify.