Implement asynchronous FileInputStream for BufferedFile.

According to io_uring_enter man this can probably be done via IORING_OP_READ with offset -1.

Hi Ilya, can you add a sponsoring possibility? As SirixDB got the first "one-time" sponsor, I'd also like to sponsor great projects 👍

Hi,

after a few tests in sirix-core are successful without any failures any subsequent test fails, because it can't allocate enough memory somehow.

failed to allocate memory for io_uring ring; Cannot allocate memory
java.lang.RuntimeException: failed to allocate memory for io_uring ring; Cannot allocate memory
	at one.jasyncfio.Native.setupIoUring0(Native Method)
	at one.jasyncfio.Native.setupIoUring(Native.java:32)
	at one.jasyncfio.Ring.<init>(Ring.java:43)
	at one.jasyncfio.PollRing.<init>(PollRing.java:23)
	at one.jasyncfio.EventExecutorImpl.<init>(EventExecutorImpl.java:125)
	at one.jasyncfio.EventExecutor$Builder.build(EventExecutor.java:183)
	at one.jasyncfio.EventExecutor.initDefault(EventExecutor.java:205)
	at org.sirix.io.iouring.IOUringStorage.<init>(IOUringStorage.java:62)
	at org.sirix.io.StorageType$5.getInstance(StorageType.java:87)
	at org.sirix.io.StorageType.getStorage(StorageType.java:112)

However, via debugger I can see that the storage.close() method is executed:

  @Override
  public void close() {
    try {
      if (revisionsOffsetFile != null) {
        revisionsOffsetFile.close().join();
        revisionsOffsetFileEventExecutor.close();
      }
      dataFile.close().join();
      dataFileEventExecutor.close();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

The executors are initialized with the default factory method.

Just a tip:

    private static void testWrite(Path p) throws Exception {
        try (FileChannel fc = FileChannel.open(p, StandardOpenOption.WRITE,
             ExtendedOpenOption.DIRECT)) {
            FileStore fs = Files.getFileStore(p);
            int alignment = (int)fs.getBlockSize();
            ByteBuffer src = ByteBuffer.allocateDirect(PAGE_SIZE + alignment - 1)
                                       .alignedSlice(alignment);
            for (int j = 0; j < SIZE; j++) {
                src.put((byte)0);
            }
            src.flip();
            fc.write(src);
        }
    }

    private static void testRead(Path p) throws Exception {
        try (FileChannel fc = FileChannel.open(p, ExtendedOpenOption.DIRECT)) {
            FileStore fs = Files.getFileStore(p);
            int alignment = (int)fs.getBlockSize();
            ByteBuffer dest = ByteBuffer.allocateDirect(PAGE_SIZE + alignment - 1)
                                        .alignedSlice(alignment);
            fc.read(dest);
        }
    }

At the moment working with io_uring using this flag is impossible, because the API uses io_uring to open/close files, and if you create an io_uring instance with the flag IORING_SETUP_IOPOLL, you cannot make these calls through it.

When using the IORING_SETUP_IOPOLL flag, it is necessary to create a io_uring service instance that will be used for operations that are not related to writing/reading data.

I'm curious about when you might want to use .readFixedBuffer().

From reading, it seems like it could be ideal if you know you're dealing with fixed buffer sizes.

Are there any drawbacks to using it?

I'm experimenting with using jasyncfio for I/O in a database, where the usage looks something like this. Does it make sense to use that here? (Also, why is there no writeFixedBuffer() out of curiosity?)

Thank you =)

const val PAGE_SIZE = 4096

class AsyncDiskManager(dbFile: Path) : IDiskManager {
    private val eventExecutor = EventExecutor.builder().withBufRing(1024, PAGE_SIZE).build()
    private val asyncFile = AsyncFile.open(dbFile, eventExecutor).get()

    suspend fun readPage(pageId: Long, buffer: MemorySegment) = withContext(Dispatchers.IO) {
        asyncFile.read(buffer.asByteBuffer(), pageId * PAGE_SIZE).await()
    }

    suspend fun readPageFixedBuffer(pageId: Long, buffer: MemorySegment) = withContext(Dispatchers.IO) {
        asyncFile.readFixedBuffer(pageId * PAGE_SIZE).await().use {
            buffer.asByteBuffer().put(it.buffer)
        }
    }

    suspend fun writePage(pageId: Long, buffer: MemorySegment): Int = withContext(Dispatchers.IO) {
        asyncFile.write(buffer.asByteBuffer(), pageId * PAGE_SIZE).await()
    }
}

Support a native library build for arm64. It must supply binaries for amd64 and arm64 and load the required library at startup.

At the moment the problem is that when trying to use gradle with qemu an error Unsupported setsockopt level=0 optname=49 occurs

Also refactor C code and add io_uring close

-Xcheck:jni

Wiki documentation main page states
Jasyncfio similar to Java NIO API, but comes in two kinds:

BufferedFile - similar to Java NIO API, but asynchronous. Regular file backed by operation system page cache
DmaFile - O_DIRECT file asynchronous API

BufferedFile has been renamed to AsyncFile.

    void close() {
        ring.close();
        bufRings.values().forEach(IoUringBufRing::close);
    }

the bufRings instance should in either way be initialized to an empty map instead of null if the condition returns false.

Hi,

I think there's an issue with offset based random writes in a file:

With a FileChannel I'm getting the following hexdump (writing at offset 208 and leaving a gap, before writing a header/uberpage which is written two times (bytes 0...99 and 100..199):

With the FileChannel based implementation:

johannes@luna:/tmp/sirix/json-path1/resources/shredded/data$ hexdump sirix.data 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 00000d0 0036 0000 5382 414e 5050 0059 0000 0100

With your library:

johannes@luna:/tmp/sirix/json-path1/resources/shredded/data$ hexdump sirix.data 0000000 0036 0000 5382 414e 5050 0059 0000 0100 0000010 0000 0100 0000 2200 042b 0001 0111 010c 0000020 0100 0100 2003 0f00 0000 1f00 0080 1180

Kind regards
Johannes

hi jasyncfio：
There are plans to optimize java URLs with io_uring？thanks

regard

Do you plan a version based on the FFM API (should be faster): https://youtu.be/8sFt1_7RxGk

Implement asynchronous FileOutputStream for BufferedFile.

According to io_uring_enter man man this can probably be done via IORING_OP_WRITE with offset -1.

Implement asynchronous FileInputStream for DmaFile.

According to io_uring_enter man man this can probably be done via IORING_OP_READ with offset -1.

Hello,

I want to use io_uring for my database prototype [1]. I have interfaces for doing I/O to read and write page fragments, which are only word aligned, but do not have a predefined length, otherwise.

A parent page of the leaf data page fragments in a trie index has at most N (currently set to 4 per default) references to leaf data page fragments. In order to reconstruct a leaf data page in memory at most these N page fragments have to be read. Currently, this is the function to read the page fragments in parallel: https://github.com/sirixdb/sirix/blob/1385d96916015f9703a6ef6e34de0c268e1536a7/bundles/sirix-core/src/main/java/org/sirix/access/trx/page/NodePageReadOnlyTrx.java#L544

I guess it would make sense to switch to Java 19 and to switch to virtual threads for this. In the I/O layer I'd use your library for instance instead of a simple FileChannel based implementation to read page fragments: https://github.com/sirixdb/sirix/blob/master/bundles/sirix-core/src/main/java/org/sirix/io/filechannel/FileChannelReader.java

Do you think it makes sense? However I'm not sure... guess I would have to change the interfaces to return Futures with the pages to do async I/O already in the I/O layer itself: https://github.com/sirixdb/sirix/blob/master/bundles/sirix-core/src/main/java/org/sirix/io/Reader.java

1. Description
2. Simple howto
3. Linux kernel requirements
4. Brief explanation
5. Buffered file examples (simple, registered buffers/files)
6. I/O poll example
7. SQ poll example
8. Dma file examples (simple, registered buffers/files)
9. benchmarks (with comparison to fio)
10. known limitations

Hi Ilya,

Did you make some performance improvements? I wonder if you can provide a new version :-)

kind regards
Johannes

Implement asynchronous FileOutputStream for DmaFile.

According to io_uring_enter man man this can probably be done via IORING_OP_WRITE with offset -1.