fluent / chunkio Goto Github PK
View Code? Open in Web Editor NEWSimple library to manage chunks of data in memory and file system
License: Apache License 2.0
Simple library to manage chunks of data in memory and file system
License: Apache License 2.0
Today, any errors in chunkio
file operations will be reported to stderr
. This is not a big deal when running an application using this library as a direct binary, since there are methods to redirect stderr
when running directly. When running as a systemd
service, you can set a StandardError
configuration to report stderr
to a file. The main situation where there is no recourse for this is when running as a Windows Service, which does not retain stderr
and does not have a standard way to reconfigure it.
Allow configuring a log file for any chunkio logs.
Since the problem really shines as a Windows Service, this could be solved on Windows specifically by using the SetStdHandle function in the Win32 Console API. However, with all the work that would need to be done to allow configuration of this file just for Windows, you might as well do the implementation in a cross platform manner that may still benefit users on other platforms even if they have other escape hatches.
This issue was discussed in person at the Fluent Bit Developer's Meeting on July 12,2023.
I met a compilation error on shared build.
CMake Error at src/CMakeLists.txt:44 (target_link_libraries):
Cannot specify link libraries for target "chunkio-static" which is not
built by this project
It seems to be caused by wrong target name on shared build.
https://github.com/fluent/chunkio/blob/v1.1.6/src/CMakeLists.txt#L44
I think it should be target_link_libraries(chunkio-shared ${libs})
.
This is just a casual proposal that (I think) would help the development.
Backgrond
We use chunkio as a library in Fluen Bit. Right now the workflow is that
(1) we do some development on chunkio and then (2) occationally sync the
result into fluent/fluent-bit.
The synchronization is done by manually copying files and adding a commit
like below:
commit 0e000ca45cd70731896008d1cf1a34c3caf5a88f
Date: Thu Sep 26 11:59:29 2019 -0600
lib: chunkio: sync changes, details below:
- validate if fallocate is supported
- new trace mode
- new api call to retrieve metadata size
- fix crash when opening a corrupted file
- new unit test issue_51
But it's kinda opaque from the commit which version of chunkio is in use.
We need to look into the history and figure out is which is corresponding
to which.
If there is a versioning, we can simply say:
commit 0e000ca45cd70731896008d1cf1a34c3caf5a88f
Date: Thu Sep 26 11:59:29 2019 -0600
lib: chunkio: update version from v1.0 to v1.1
In my opinion, this is much clearer. The commit message clearly tells us
that the synchronization is done in a well-defined point.
Proposal
Let's call 4f197e0, say, v1.0.0 and do versioning thereafter.
All elements created by chunkio.c:check_root_path and cio_stream.c:check_stream_path are created with hard-coded mode 0755. Therefore anyone can read the data; it is not confidential.
Provide a config file option allowing a user-configurable permission to be defined, eg 0700.
While porting fluent-bit (which uses chunkio), I found out that it couldn't be compiled against UClibc.
For now, I've added a guard to enforce glibc: https://github.com/buildroot/buildroot/blob/master/package/fluent-bit/Config.in#L3.
It would be nice to have UClibc and e.g. also Musl support. Then, fluent-bit can be used on all supported platforms on buildroot.
[ 16%] Building C object lib/monkey/mk_server/CMakeFiles/monkey-core-static.dir/mk_vhost.c.o
/home/thomas/br-test-pkg/bootlin-armv5-uclibc/build/fluent-bit-2.0.8/lib/chunkio/src/cio_utils.c:29:10: fatal error: fts.h: No such file or directory
29 | #include <fts.h>
| ^~~~~~~
compilation terminated.
make[4]: *** [lib/chunkio/src/CMakeFiles/chunkio-static.dir/build.make:174: lib/chunkio/src/CMakeFiles/chunkio-static.dir/cio_utils.c.o] Error 1
make[4]: *** Waiting for unfinished jobs....
In the chunkio code there are system calls to expand files by using the system call "fallocate" that limits usage to certain file system types, this is mainly due to performance reasons.
To support more file systems apart from XFS, ext4 etc, it would be good if there could be an option to use posix_fallocate and let the decision on performance be up to usage and configurable.
There are some error messages that could be more helpful with more context:
No space left on device => No space left on device. Space requested {} space available {} device/path {}
No such file or directory => No such file or directory. Path {}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.