meebey / leveldb-sharp Goto Github PK

leveldb-sharp currently copies the values returned from LevelDB using Marshal.PtrToStringAnsi() as the memory was allocated by LevelDB. The issue is that the CLR can not use memory owned / allocated by someone else it can't know when it needs to be released or if it was already released. The simple but not effective solution is to copy the memory to have it CLR owned. To make the situation even worse, the C binding of LevelDB also copies the values returned by LevelDB, see: http://codeofrob.com/entries/the-price-of-abstraction---using-leveldb-in-ravendb.html

After looking into this issue I came up these possible solutions how passing values can work:

• copy unmanaged memory and make the copy managed: Marshal.PtrToStringAnsi()
  • Pros: safe, easy to implement, easy to use
  • Cons: copy overhead
• return the pointer everywhere, as in IntPtr instead of a string or byte[] and let the user access that doing unsafe code blocks
  • Pros: most efficient
  • Cons: unsafe and ugly to use
• implement a Slice type that operates on IntPtr and do unsafe code access on it and release the memory when needed (GCed). This would be basically a safe proxy around the unsafe memory.
  • Pros: safe, easy to use, most efficient

Quoting from Rob's blog:

MemoryStream stream = db.Get(string key)

"Where that stream reads from the start of an unmanaged array to the end of that unmanaged array and then disposes it."

This wouldn't work for us, as LevelDB reads the whole value into memory and returns that, instead of a stream...

I have found a problem when attempting to store data which contains non-ascii characters. "Décor" for example with the accented é.

The issue seems to be rooted in a mis-step with calculating the string length for passing from the managed c# to the un-managed leveldb.

The call to get the expected length uses the UTF8 encoding but then the actual marshaled call passes as the current ANSI code page. In the ANSI page (for a US system) the special é takes only a single byte but in UTF8 is takes 2 bytes.

The end result is that the record is saved with an extra character of garbage, specifically one extra garbage character for each special character in the input.

Thanks for saving me tons of work here. These bindings look great and exactly what I am looking for. I was wondering however how you compiled leveldb on Windows. Some searching revealed few ports but I figured I could ask you since you have probably already experimented with some of them.

Thanks in advance!

I just forked the project to add some features to use in one of my own personal projects.
Everything is fine and smooth until i decided to implement Slices.and I have no clue how my license should look like.I mean I'm not Google and not the original author either.
Could you help me in this?

I am attempting to use your wrapper on a Windows machine (2008 R2) using the standard framework (i.e. not Mono).

When attempting to do a Get, the call to this method fails with the following error:

System.BadImageFormatException - Invalid access to memory location. (Exception from HRESULT: 0x800703E6)

Is this silently failing for you or is the pointer successfully deallocated?

Roman Polunin wrote:

1. in your mapping of leveldb_get you perform Marshal.PtrToStringAnsi on the buffer returned from leveldb; however, you do not call leveldb_free on that buffer. This looks like a memory leak.
   ...

´´´
Die DLL "leveldb": Das angegebene Modul wurde nicht gefunden. (Ausnahme von HRESULT: 0x8007007E) kann nicht geladen werden.

The DLL "leveldb": The selected Module could not be found, (Exception of HRESULT: 0x8007007E) could not be loaded.
´´´
I compiled the whole project over and over and it seems to have no fails in it.
Do I need to download a "leveldb.dll" from somewhere or how do I get that running?

After some initial testing with 1.9.2 I found memory growth issues.
In Native.cs I changes all calls to the free method
From: leveldb_free(ptr);
To: leveldb_free(ref ptr);

And changed the associated import
[DllImport("libleveldb", CallingConvention = CallingConvention.Cdecl)]
public static extern void leveldb_free(ref IntPtr ptr);

All seems well after these changes. Though it does leave me a little uneasy that I'm not finding anyone else needing to make this change.

I'm using the kkowalczyk-leveldb-6f9f0bca2016 windows port of LevelDB itself.

Has anyone used this? I added this project via NuGet to my .NET project and am getting the error above.