Support for S3 storage backend about oioioi HOT 4 OPEN

I parsed through last year of logs and found no signs of corruption like that. It found it easy to crash the filetracker2 on purpose with a rejudge when number of available cores is about 200 and concurrency is high enough to utilize that. I was unable to do that with filetracker1.

from oioioi.

Is the current plan for SZKOpuł and sio2.mimuw to use a cloud s3 or a self-hosted solution?

This really depends how it will be implemented. In case no deduplication will take place, then It would be too expansive to put in in the cloud and the latency might by also an extra issue. To all to the options for selfhosting I would consider also ceph.

The main goal is to get something more mature for storage backend. As for the migration, I think a simple translation layer from Filetracker to S3 should allow to do a downtimeless upgrade.

from oioioi.

More than half of recent Szkopuł outages were due to filetracker2 instability.

Did that instability manifest, by any chance, in random DB corruption(?) requiring a restart of the filetracker server?
Like HTTP/500: (-30973, 'BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery -- BDB0060 PANIC: fatal region error detected; run recovery').

At least for my deployment, simply updating filetracker's dependencies seemingly fixed this, though checking the memory usage didn't come to my mind.
I should still have at least one problem package that allows for somewhat reproducing this error locally.

from oioioi.

Please comment Your thoughts about this below!

For me and Stowarzyszenie Talent, as long as decent options for self-hosting exist, switching to an s3-like backend shouldn't be an issue apart from the migration.
Based on my short research, MinIO and SeaweedFS seem ok. Localstack looks more tailored to local development than production deployments, but should be enough for a development environment. Additionally, it's written in python and implements a multitude of services apart from s3, so resource usage might not be perfect.

Is the current plan for SZKOpuł and sio2.mimuw to use a cloud s3 or a self-hosted solution?

from oioioi.