Hey Max , (@maxd )

I want to try your library bridge ,
did you have any good testing
over this way of working with leveldb

I am looking for good alternative for
pure java leveldb.

Thanks
Roman.

I know this is an old project with little activity, but I'm leaving this issue here as a warning to future users. There is a massive memory leak in this wrapper.

Basic Info

I'm using Mojang's LevelDB library, which is based on the latest version of LevelDB 1.18, but supports Windows and adds zlib compression. On both Windows and Ubuntu 14.04 (only two platforms I have to test with), I encounter a very large memory leak where the number of objects allocated outside the Java heap continues to grow until eventually the program uses up all system RAM. I spent 2 days looking into this issue until I finally figured out it was due to my complete ignorance of what this JNA wrapper does behind the scenes.

The Problem

LevelDB.get is the main cause of the memory leak, though there may be smaller leaks littered throughout the wrapper. The native LevelDB library returns an malloc'd byte array to the wrapper, which means it's up to the wrapper to free the byte array. However, this wrapper's LevelDB.get method returns a copy of the native byte array allocated by the LevelDB library, but doesn't free the original native byte array. This means the original byte array malloc'd by the native LevelDB library is never freed and persists outside the Java heap, completely invisible to the Java garbage collector.

JNA will not automatically free the byte arrays returned by the native LevelDB library or pointed to by a PointerToReference. These arrays must be freed manually.

Solutions

For future users, fork this repo and change LevelDB.get using one of the following solutions:

Solution 1

Modify the LevelDB.get function to clear the data being pointed to before returning the copied byte array, like so:

byte[] returnValue = result != null ? result.getPointer().getByteArray(0, (int) resultLength) : null;
if(result != null)
    LevelDBNative.leveldb_free(result.getPointer());
return returnValue;

This option also has lots of overhead as Java still has to copy the original byte array to the Java heap.

Solution 2

Modify the LevelDB.get function to return a ByteBuffer mapped to the original byte data using:

result.getPointer().getByteBuffer(0, (int) resultLength)

This option will have a little less overhead as Java doesn't have to copy the original byte array; however, the ByteBuffer returned is a DirectByteBuffer, which may not be ideal for some applications and use-case scenarios.