qweeze / uring_file Goto Github PK
View Code? Open in Web Editor NEWAsynchronous file I/O with io_uring and asyncio
License: MIT License
Asynchronous file I/O with io_uring and asyncio
License: MIT License
Hi, I tried running this code:
import string
import asyncio
import uring_file
import os
async def main():
path = 'test.txt'
f = uring_file.File(path)
await f.open(os.O_CREAT | os.O_WRONLY)
await f.write(b'hello\nworld')
await f.close()
if __name__ == "__main__":
asyncio.run(main())
I get the following error:
ubuntu@pc:~$ python3 main.py
Exception in callback Uring.eventfd_callback()
handle: <Handle Uring.eventfd_callback()>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/home/ubuntu/uring_file.py", line 48, in eventfd_callback
result = liburing.trap_error(cqe.res)
File "/home/ubuntu/.local/lib/python3.8/site-packages/liburing/wrapper.py", line 27, in trap_error
raise OSError(-no, strerror(-no))
OSError: [Errno 9] Bad file descriptor
These are the system details. It is an AWS machine.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
5.4.0-1038-aws
I installed liburing
from source. Could you tell me if I'm doing something wrong?
First of all, thank you for your example code. Would it be possible that you add a license file for your project (see e.g. the Liburing project)?
Hello,
Thank you for your example code.
I was looking for a library that can operate files using io_uring for my personal interest, and finally I found this repository.
Your example code for integrating io_uring and asyncio was very interesting.
So I forked this repository and I've been writing a new library based on your code.
https://github.com/tree4096/uring_file
Now I achieved excellent R/W performance compared with the other libraries such as aiofiles, aiofile, and anyio.
I tested writing 100 lines of text to 1000 files in 1 second and measured the time to completion.
And I also tested reading 1000 files in 0.1 second.
https://github.com/tree4096/uring_file/blob/master/performance_test.py
Here is the results.
Library | Time [s] | Process Time [s] |
---|---|---|
anyio | 10.69 | 15.62 |
aiofiles | 4.47 | 6.19 |
uring_file ⭐ | 1.92 | 1.88 |
aiofile* | 7.51 | 15.08 |
Library | Time [s] | Process Time [s] |
---|---|---|
anyio | 0.482 | 0.816 |
aiofiles | 0.349 | 0.623 |
uring_file ⭐ | 0.118 | 0.118 |
aiofile* | 0.473 | 0.828 |
*aiofile has Linux AIO mode, but does not work on Ubuntu due to the bug. So it run on C threading mode. See mosquito/caio#17 (comment)
OS: Ubuntu 20.04.4 LTS on WSL2
Kernel ver.: 5.10.102.1-microsoft-standard-WSL2
CPU: AMD Ryzen 7 5800U
SSD: MTFDHBA1T0TDV-1AZ1AABHA
Python ver.: 3.10.5
In these results, uring_file in my repository is much faster than other libs and its process time (the sum of CPU time not include sleeping) is very small because it may run on single thread.
In addition, my library also provides the low level APIs to easily access the io_uring asynchronously.
Here is the sample code.
# Create new Uring
uring = uring_file.Uring(sq_size=8, cq_size=64, session_sq_size=4)
# Get SQE and submit (Open file)
async with uring.session() as session:
sqe = session.get_sqe()
sqe.prep_openat(liburing.AT_FDCWD, b"hello.txt", os.O_RDONLY, 0o644)
# Get result of CQE
fd = sqe.result()
print("FD:", fd)
# Close file
async with uring.session() as session:
# Raw SQE object can be accessed from the _sqe property
liburing.io_uring_prep_close(session.get_sqe()._sqe, fd)
# Same expression as session.get_sqe().prep_close(fd)
# Get default Uring
default_uring = uring_file.get_default_uring()
These APIs enable to access the io_uring like a database session.
The SQEs of multiple sessions are automatically submitted together under high frequency access, and I call it "auto bulk submit".
The auto bulk submit is realized by controlling the task scheduling of asyncio in a little hacky way and it makes greate performance improvement.
And the overflow protection for the CQ is also implemented to prevent the error message, "Device or resource busy".
If your are interested in my library, would you give me some comments or advice?
Awaiting for your answer.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.