protonail / leveldb-jna Goto Github PK
View Code? Open in Web Editor NEWJava JNA (not JNI) adapter for LevelDB
License: MIT License
Java JNA (not JNI) adapter for LevelDB
License: MIT License
I know this is an old project with little activity, but I'm leaving this issue here as a warning to future users. There is a massive memory leak in this wrapper.
I'm using Mojang's LevelDB library, which is based on the latest version of LevelDB 1.18, but supports Windows and adds zlib compression. On both Windows and Ubuntu 14.04 (only two platforms I have to test with), I encounter a very large memory leak where the number of objects allocated outside the Java heap continues to grow until eventually the program uses up all system RAM. I spent 2 days looking into this issue until I finally figured out it was due to my complete ignorance of what this JNA wrapper does behind the scenes.
LevelDB.get
is the main cause of the memory leak, though there may be smaller leaks littered throughout the wrapper. The native LevelDB library returns an malloc
'd byte array to the wrapper, which means it's up to the wrapper to free the byte array. However, this wrapper's LevelDB.get
method returns a copy of the native byte array allocated by the LevelDB library, but doesn't free the original native byte array. This means the original byte array malloc
'd by the native LevelDB library is never freed and persists outside the Java heap, completely invisible to the Java garbage collector.
JNA will not automatically free the byte arrays returned by the native LevelDB library or pointed to by a PointerToReference
. These arrays must be freed manually.
For future users, fork this repo and change LevelDB.get
using one of the following solutions:
Modify the LevelDB.get
function to clear the data being pointed to before returning the copied byte array, like so:
byte[] returnValue = result != null ? result.getPointer().getByteArray(0, (int) resultLength) : null;
if(result != null)
LevelDBNative.leveldb_free(result.getPointer());
return returnValue;
This option also has lots of overhead as Java still has to copy the original byte array to the Java heap.
Modify the LevelDB.get
function to return a ByteBuffer
mapped to the original byte data using:
result.getPointer().getByteBuffer(0, (int) resultLength)
This option will have a little less overhead as Java doesn't have to copy the original byte array; however, the ByteBuffer
returned is a DirectByteBuffer
, which may not be ideal for some applications and use-case scenarios.
Hey Max , (@maxd )
I want to try your library bridge ,
did you have any good testing
over this way of working with leveldb
I am looking for good alternative for
pure java leveldb.
Thanks
Roman.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.