saremox / ocbench Goto Github PK

since glibc doesn't support the memfd feature of linux yet. We need an own abstraction layer around the needed syscalls.

• implement basic memfd calls
• implement basic wrapper calls for memory maping
• add debuging and sanity checks
• add test's

currently ocworker don't give a way to store results into ocdata objects

provide some simple tools to evaluate results of the benchmark.

• user defined evaluation constraints
• automatic diagram creation based on evaluation

to get acurate results we need to ensure the worker don't switch cpu core while compressing.

collect results in an easy to evaluate format like:

• SQL
• XML
• JSON
• CSV

in HPC it is common that jobs get aborted during execution. Implementing routines to ensure a aborted session get's resumed there it left.

Encapsulating with IPC and fork the worker processes which later do the compression benchmark from the master process which is loading files into memfd and collecting results from the worker.

• develop abstraction layer for fork and pipe communication
• develop communication scheme woker <-> master
• load files into a memfd and let a worker read it correctly

currently my cmake specifications don't correctly find squash. Still need to figure out if it's my fault or maybe a configuration problem in squash upstream.

• understanding cmake pkgconfig wrapper...
• cmake find correct header files
• linking simple test.c with squash api calls

If a process dies, the watchdog should detect that and restart that worker and retry the failed job for a given amount of tries.

Recently added message queue's and new functions need test coverage

An algorithm which choose a file to reduce the error of a compression prediction. This can speed up the process of evaluation on large datasets more than 1000 files.

• evenly choose files with diffrent sizes
• compare prediction with actual compression ratio
• user defined precision like stop by 5 % of precission.

Use https://github.com/quixdb/squash to compress memfd buffers

• working compression
• working decompression

as of 0a574f2 there is no documentation which dependencies are needed to get ocbench working.

memfd is still lacking of support in glibc. The syscall got introducd in Linux >= 3.17
Thinking of adding a shm fallback if memfd is not available on a platform like openbsd.