ikorennoy / jasyncfio Goto Github PK
View Code? Open in Web Editor NEWJava asynchronous file I/O based on io_uring Linux interface
License: Apache License 2.0
Java asynchronous file I/O based on io_uring Linux interface
License: Apache License 2.0
Support a native library build for arm64. It must supply binaries for amd64 and arm64 and load the required library at startup.
At the moment the problem is that when trying to use gradle with qemu an error Unsupported setsockopt level=0 optname=49
occurs
Hi,
I think there's an issue with offset based random writes in a file:
With a FileChannel I'm getting the following hexdump (writing at offset 208 and leaving a gap, before writing a header/uberpage which is written two times (bytes 0...99 and 100..199):
With the FileChannel based implementation:
johannes@luna:/tmp/sirix/json-path1/resources/shredded/data$ hexdump sirix.data 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 00000d0 0036 0000 5382 414e 5050 0059 0000 0100
With your library:
johannes@luna:/tmp/sirix/json-path1/resources/shredded/data$ hexdump sirix.data 0000000 0036 0000 5382 414e 5050 0059 0000 0100 0000010 0000 0100 0000 2200 042b 0001 0111 010c 0000020 0100 0100 2003 0f00 0000 1f00 0080 1180
Kind regards
Johannes
Also refactor C code and add io_uring close
Hello,
I want to use io_uring for my database prototype [1]. I have interfaces for doing I/O to read and write page fragments, which are only word aligned, but do not have a predefined length, otherwise.
A parent page of the leaf data page fragments in a trie index has at most N (currently set to 4 per default) references to leaf data page fragments. In order to reconstruct a leaf data page in memory at most these N page fragments have to be read. Currently, this is the function to read the page fragments in parallel: https://github.com/sirixdb/sirix/blob/1385d96916015f9703a6ef6e34de0c268e1536a7/bundles/sirix-core/src/main/java/org/sirix/access/trx/page/NodePageReadOnlyTrx.java#L544
I guess it would make sense to switch to Java 19 and to switch to virtual threads for this. In the I/O layer I'd use your library for instance instead of a simple FileChannel
based implementation to read page fragments: https://github.com/sirixdb/sirix/blob/master/bundles/sirix-core/src/main/java/org/sirix/io/filechannel/FileChannelReader.java
Do you think it makes sense? However I'm not sure... guess I would have to change the interfaces to return Futures with the pages to do async I/O already in the I/O layer itself: https://github.com/sirixdb/sirix/blob/master/bundles/sirix-core/src/main/java/org/sirix/io/Reader.java
Do you plan a version based on the FFM API (should be faster): https://youtu.be/8sFt1_7RxGk
hi jasyncfio:
There are plans to optimize java URLs with io_uring?thanks
regard
Hi Ilya, can you add a sponsoring possibility? As SirixDB got the first "one-time" sponsor, I'd also like to sponsor great projects 👍
Hi,
after a few tests in sirix-core are successful without any failures any subsequent test fails, because it can't allocate enough memory somehow.
failed to allocate memory for io_uring ring; Cannot allocate memory
java.lang.RuntimeException: failed to allocate memory for io_uring ring; Cannot allocate memory
at one.jasyncfio.Native.setupIoUring0(Native Method)
at one.jasyncfio.Native.setupIoUring(Native.java:32)
at one.jasyncfio.Ring.<init>(Ring.java:43)
at one.jasyncfio.PollRing.<init>(PollRing.java:23)
at one.jasyncfio.EventExecutorImpl.<init>(EventExecutorImpl.java:125)
at one.jasyncfio.EventExecutor$Builder.build(EventExecutor.java:183)
at one.jasyncfio.EventExecutor.initDefault(EventExecutor.java:205)
at org.sirix.io.iouring.IOUringStorage.<init>(IOUringStorage.java:62)
at org.sirix.io.StorageType$5.getInstance(StorageType.java:87)
at org.sirix.io.StorageType.getStorage(StorageType.java:112)
However, via debugger I can see that the storage.close()
method is executed:
@Override
public void close() {
try {
if (revisionsOffsetFile != null) {
revisionsOffsetFile.close().join();
revisionsOffsetFileEventExecutor.close();
}
dataFile.close().join();
dataFileEventExecutor.close();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
The executors are initialized with the default factory method.
Implement asynchronous FileOutputStream for BufferedFile.
According to io_uring_enter man man this can probably be done via IORING_OP_WRITE with offset -1.
Implement asynchronous FileInputStream for DmaFile.
According to io_uring_enter man man this can probably be done via IORING_OP_READ with offset -1.
Implement asynchronous FileInputStream for BufferedFile.
According to io_uring_enter man this can probably be done via IORING_OP_READ with offset -1.
At the moment working with io_uring using this flag is impossible, because the API uses io_uring to open/close files, and if you create an io_uring instance with the flag IORING_SETUP_IOPOLL, you cannot make these calls through it.
When using the IORING_SETUP_IOPOLL flag, it is necessary to create a io_uring service instance that will be used for operations that are not related to writing/reading data.
Wiki documentation main page states
Jasyncfio similar to Java NIO API, but comes in two kinds:
BufferedFile - similar to Java NIO API, but asynchronous. Regular file backed by operation system page cache
DmaFile - O_DIRECT file asynchronous API
BufferedFile has been renamed to AsyncFile.
I'm curious about when you might want to use .readFixedBuffer()
.
From reading, it seems like it could be ideal if you know you're dealing with fixed buffer sizes.
Are there any drawbacks to using it?
I'm experimenting with using jasyncfio
for I/O in a database, where the usage looks something like this. Does it make sense to use that here? (Also, why is there no writeFixedBuffer()
out of curiosity?)
Thank you =)
const val PAGE_SIZE = 4096
class AsyncDiskManager(dbFile: Path) : IDiskManager {
private val eventExecutor = EventExecutor.builder().withBufRing(1024, PAGE_SIZE).build()
private val asyncFile = AsyncFile.open(dbFile, eventExecutor).get()
suspend fun readPage(pageId: Long, buffer: MemorySegment) = withContext(Dispatchers.IO) {
asyncFile.read(buffer.asByteBuffer(), pageId * PAGE_SIZE).await()
}
suspend fun readPageFixedBuffer(pageId: Long, buffer: MemorySegment) = withContext(Dispatchers.IO) {
asyncFile.readFixedBuffer(pageId * PAGE_SIZE).await().use {
buffer.asByteBuffer().put(it.buffer)
}
}
suspend fun writePage(pageId: Long, buffer: MemorySegment): Int = withContext(Dispatchers.IO) {
asyncFile.write(buffer.asByteBuffer(), pageId * PAGE_SIZE).await()
}
}
Just a tip:
private static void testWrite(Path p) throws Exception {
try (FileChannel fc = FileChannel.open(p, StandardOpenOption.WRITE,
ExtendedOpenOption.DIRECT)) {
FileStore fs = Files.getFileStore(p);
int alignment = (int)fs.getBlockSize();
ByteBuffer src = ByteBuffer.allocateDirect(PAGE_SIZE + alignment - 1)
.alignedSlice(alignment);
for (int j = 0; j < SIZE; j++) {
src.put((byte)0);
}
src.flip();
fc.write(src);
}
}
private static void testRead(Path p) throws Exception {
try (FileChannel fc = FileChannel.open(p, ExtendedOpenOption.DIRECT)) {
FileStore fs = Files.getFileStore(p);
int alignment = (int)fs.getBlockSize();
ByteBuffer dest = ByteBuffer.allocateDirect(PAGE_SIZE + alignment - 1)
.alignedSlice(alignment);
fc.read(dest);
}
}
void close() {
ring.close();
bufRings.values().forEach(IoUringBufRing::close);
}
the bufRings
instance should in either way be initialized to an empty map instead of null
if the condition returns false.
Implement asynchronous FileOutputStream for DmaFile.
According to io_uring_enter man man this can probably be done via IORING_OP_WRITE with offset -1.
-Wall -Wextra -Wpedantic -Wconversion -Wsign-conversion -Wcast-qual -Wshadow -Wformat=2 -Wundef -Werror=float-equal -Werror=strict-prototypes -Wwrite-strings
Hi Ilya,
Did you make some performance improvements? I wonder if you can provide a new version :-)
kind regards
Johannes
-Xcheck:jni
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.