jmhodges / levigo Goto Github PK
View Code? Open in Web Editor NEWlevigo is a Go wrapper for LevelDB
License: MIT License
levigo is a Go wrapper for LevelDB
License: MIT License
I need to iterate keys in a finite range, such as start
to end
, in C++
code, i can make it like
for (it->Seek(start); it->Valid(); it->Next()) {
leveldb::Slice key = it->key();
leveldb::Slice value = it->value();
if(options.comparator->compare(key, end) > 0) {
break;
} else {
/**
* process (key, value)
*/
}
}
but in levigo, i cannot get options.comparator
directly.
Could not run it on Win32, I modified it, it would be nice, patch is as follows?
diff --git a/db.go b/db.go
index f958a05..969e857 100644
--- a/db.go
+++ b/db.go
@@ -74,7 +74,7 @@ type Snapshot struct {
func Open(dbname string, o *Options) (*DB, error) {
var errStr *C.char
ldbname := C.CString(dbname)
- defer C.leveldb_free(unsafe.Pointer(ldbname))
+ defer C.free(unsafe.Pointer(ldbname))
leveldb := C.leveldb_open(o.Opt, ldbname, &errStr)
if errStr != nil {
@@ -90,7 +90,7 @@ func Open(dbname string, o *Options) (*DB, error) {
func DestroyDatabase(dbname string, o *Options) error {
var errStr *C.char
ldbname := C.CString(dbname)
- defer C.leveldb_free(unsafe.Pointer(ldbname))
+ defer C.free(unsafe.Pointer(ldbname))
C.leveldb_destroy_db(o.Opt, ldbname, &errStr)
if errStr != nil {
@@ -107,7 +107,7 @@ func DestroyDatabase(dbname string, o *Options) error {
func RepairDatabase(dbname string, o *Options) error {
var errStr *C.char
ldbname := C.CString(dbname)
- defer C.leveldb_free(unsafe.Pointer(ldbname))
+ defer C.free(unsafe.Pointer(ldbname))
C.leveldb_repair_db(o.Opt, ldbname, &errStr)
if errStr != nil {
@@ -260,8 +260,8 @@ func (db *DB) GetApproximateSizes(ranges []Range) []uint64 {
db.Ldb, numranges, startsPtr, startLensPtr,
limitsPtr, limitLensPtr, sizesPtr)
for i := range ranges {
- C.leveldb_free(unsafe.Pointer(starts[i]))
- C.leveldb_free(unsafe.Pointer(limits[i]))
+ C.free(unsafe.Pointer(starts[i]))
+ C.free(unsafe.Pointer(limits[i]))
}
return sizes
}
@@ -273,7 +273,7 @@ func (db *DB) GetApproximateSizes(ranges []Range) []uint64 {
func (db *DB) PropertyValue(propName string) string {
cname := C.CString(propName)
value := C.GoString(C.leveldb_property_value(db.Ldb, cname))
- C.leveldb_free(unsafe.Pointer(cname))
+ C.free(unsafe.Pointer(cname))
return value
}
diff --git a/leveldb_test.go b/leveldb_test.go
index 0d6930b..8180b55 100644
--- a/leveldb_test.go
+++ b/leveldb_test.go
@@ -268,7 +268,7 @@ func TestNilSlicesInDb(t *testing.T) {
if err != nil {
t.Errorf("empty slice key Delete errored: %v", err)
}
-
+ db.Close()
}
func TestIterationValidityLimits(t *testing.T) {
The documentation includes code that go vet
complains about:
for it = it; it.Valid(); it.Next() {
munge(it.Key(), it.Value())
}
Results in go vet complains such as this:
FILENAME.go:124: self-assignment of it to it
Hi!
I'm ran into a compiling issue on Ubuntu 12.10
$ go get github.com/jmhodges/levigo 2
1: error: 'leveldb_free' undeclared (first use in this function)
1: note: each undeclared identifier is reported only once for each function it appears in
Package libleveldb-dev is installed (Version: 0+20120530.gitdd0d562-2).
I've got it compiling with the following patch (Don't know if its wise to map leveldb_free() to free()):
diff --git a/db.go b/db.go
index 619582f..9ef217a 100644
--- a/db.go
+++ b/db.go
@@ -5,6 +5,10 @@ package levigo
#include <stdlib.h>
#include "leveldb/c.h"
+#ifndef leveldb_free
// This function exists only to clean up lack-of-const warnings when
// leveldb_approximate_sizes is called from Go-land.
void levigo_leveldb_approximate_sizes(
diff --git a/iterator.go b/iterator.go
index 1b61a38..cb5e246 100644
--- a/iterator.go
+++ b/iterator.go
@@ -1,8 +1,15 @@
package levigo
-// #cgo LDFLAGS: -lleveldb
-// #include <stdlib.h>
-// #include "leveldb/c.h"
+/*
+#cgo LDFLAGS: -lleveldb
+#include <stdlib.h>
+#include "leveldb/c.h"
+
+#ifndef leveldb_free
+*/
import "C"
import (
I succefully built this project : https://bitbucket.org/robertvazan/leveldb-windows (the only one that i can find to build levelDB under windows)
And i got the leveldb.lib
How i can use it to build levigo?
Thanks in advance if you help me :).
I am trying to make a quick note here. In "db.go",
BTW, as I am learning Go at the moment, here goes the reference.
http://golang.org/doc/articles/c_go_cgo.html
"Conversion between Go and C strings is done with the C.CString, C.GoString, and C.GoStringN functions. These conversions make a copy of the string data."
When I try to run a project with levigo in OS X 10.9 (Mavericks), I get the following error:
# github.com/jmhodges/levigo
../../jmhodges/levigo/batch.go:4:11: fatal error: 'leveldb/c.h' file not found
#include "leveldb/c.h"
^
1 error generated
LevelDB is installed via homebrew, which symlinks leveldb/c.h
into /usr/local/include
.
When I explicitly set cgo flags like this:
CGO_CFLAGS="-I/usr/local/include" CGO_LDFLAGS="-L/usr/local/lib" go run myapp.go
It works, but only on go version 1.2rc2. On 1.1.2 it fails with another message due to this bug: https://code.google.com/p/go/issues/detail?id=4829
This doesn't really seem to be on levigo's end, as cgo/clang is apparently not searching /usr/local/include
for includes for some reason, but I'm filing it here in case anyone else gets stuck on this issue.
SIGSEGV: segmentation violation
PC=0x45fc7d6
signal arrived during cgo execution
github.com/jmhodges/levigo._Cfunc_leveldb_iter_seek(0x0, 0xc2000abed0, 0x8)
github.com/jmhodges/levigo/_obj/_cgo_defun.c:231 +0x2f
github.com/jmhodges/levigo.(*Iterator).Seek(0xc2000008e8, 0xc2000abed0, 0x8, 0xe)
github.com/jmhodges/levigo/_obj/batch.cgo1.go:579 +0x42
...
$ go version
cache := levigo.NewLRUCache(1 << 24) // 16MB
options := levigo.NewOptions()
//options.SetComparator(cmp)
options.SetCache(cache)
options.SetParanoidChecks(true)
options.SetInfoLog(nil)
options.SetWriteBufferSize(1 << 20) // 1MB
options.SetMaxOpenFiles(1023)
options.SetBlockSize(1 << 15) // 32KB
options.SetCompression(1) // snappy compression, 0 == no compression
options.SetCreateIfMissing(true)
levelDb, err := levigo.Open("./store", options)
if err != nil {
panic(err)
}
roptions := levigo.NewReadOptions()
//roptions.SetVerifyChecksums(true)
roptions.SetFillCache(false)
defer roptions.Close()
iter :=levelDb.NewIterator(roptions)
iter.Close()
target := append("t_user", uint8(1))
iter.Seek(target) // <== segmentation violation
hey jeff,
have you ever tried levigo with the basho fork?
-Sean
This issue provides visibility into Renovate updates and their statuses. Learn more
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
What steps will reproduce the problem?
What is the expected output? What do you see instead?
logging should continue after the db is opened, but it doesn't.
There is a workaround, where you can call log.SetOutput() again just after leveldb.Open() and logging resumes.
What version of the product are you using? On what operating system?
go 1.2.1 on amd64 ubuntu 14.04, with leveldb from github on 04/28/14 (sorry, commit # missing as the files were copied to a new repo)
Hi, I'm trying to get a hello world working using go and leveldb.
$ go get github.com/jmhodges/levigo
1: error: 'leveldb_compact_range' undeclared (first use in this function)
1: note: each undeclared identifier is reported only once for each function it appears in
This looks to me like leveldb header/lib files are not found but they are installed in the "usual" place
$ dpkg -L libleveldb-dev
/usr/lib/libleveldb.a
/usr/include/leveldb/db.h
... etc etc etc
$ go version
go version go1
Calling WriteBatch.Delete()
with an empty key, i.e. []byte{}
, panics:
goroutine 38 [running]:
testing.tRunner.func1.1(0x43ddc60, 0xc0005d4060)
/usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:941 +0x3d0
testing.tRunner.func1(0xc0005da000)
/usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:944 +0x3f9
panic(0x43ddc60, 0xc0005d4060)
/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967 +0x15d
github.com/jmhodges/levigo.(*WriteBatch).Delete.func1(0xc0005c22b0, 0x4744c10, 0x0, 0x0)
github.com/jmhodges/[email protected]/batch.go:60 +0x84
github.com/jmhodges/levigo.(*WriteBatch).Delete(0xc0005c22b0, 0x4744c10, 0x0, 0x0)
github.com/jmhodges/[email protected]/batch.go:60 +0x49
A new C API leveldb_free(void)* was eventually added. See http://code.google.com/p/leveldb/issues/detail?id=117
I raised this issue was actually due to non-corresponding calls to malloc/free linked by levigo on Windows as malloc()/free() are not shared system calls crossing modules.
Hello.
Could you please add the binding for this fork of leveldb so it can be built everywhere?
https://github.com/bitcoin/leveldb.git
Thanks in advance
Hello there,
Great package. I was wondering if it's possible to use levigo for something like iterating the DB. I have a DB that I need to iterate over without knowing in advance about one key or the other.
goleveldb has a similar feature:
iter := db.NewIterator(nil, nil)
for iter.Next() {
// Remember that the contents of the returned slice should not be modified, and
// only valid until the next call to Next.
key := iter.Key()
value := iter.Value()
...
}
iter.Release()
err = iter.Error()
...
I was wondering it I am missing it in levigo or if it's just not implemented for now.
Cheers,
Or
Hey,
Would you be able to confirm whether you are still able to compile levigo
executable on darwin,amd64 with a current tip version of Go?
I am able to build on Linux, but not on OSX.
The linker keeps refusing to find symbols. Oddly a couple of weeks ago I had no problem.
I cannot figure out what is going on.
My CGO_LDFLAGS="-L/Users/petar/aux/leveldb/lib -lleveldb -lstdc++"
and I've removed the #cgo -lleveldb directives from the code. I doubt that makes any difference? (It works on Linux with that modification.)
Thank you
Petar
When I run test on Ubuntu 12 (amd64) with leveldb built with -lsnappy.
$ go test github.com/jmhodges/levigo
Here are the errors
leveldb_test.go:128: First size range was 0 leveldb_test.go:131: Second size range was 0
This error is not persist, but just shows up sometimes. Is this really a problem?
Hi,
I noticed that levigo was eating memory after doing many transactions. I'm running it on a VPS with 2GB and it filled all of memory within about 260K transactions of about 20k each.
I added one line on line 153 of db.go (just before the return in Get):
defer C.free(unsafe.Pointer(value))
This solved the memory leak for me and it's been stable for many millions of transactions since.
Have a great day!
Gary
Passing an empty or nil value argument to WriteBatch.Put() causes panic of "index out of range".
This is fairly harmless, but I noticed using go build -extldflags="-v"
that the linker is called with one -lleveldb
per comment in the code base:
0.71 host link: "gcc" "-o" "/tmp/go-build236882147/b001/exe/a.out" "-static" "/tmp/go-link-066068984/go.o" "/tmp/go-link-066068984/000000.o" "/tmp/go-link-066068984/000001.o" "/tmp/go-link-066068984/000002.o" "/tmp/go-link-066068984/000003.o" "/tmp/go-link-066068984/000004.o" "/tmp/go-link-066068984/000005.o" "/tmp/go-link-066068984/000006.o" "/tmp/go-link-066068984/000007.o" "/tmp/go-link-066068984/000008.o" "/tmp/go-link-066068984/000009.o" "/tmp/go-link-066068984/000010.o" "/tmp/go-link-066068984/000011.o" "/tmp/go-link-066068984/000012.o" "/tmp/go-link-066068984/000013.o" "/tmp/go-link-066068984/000014.o" "/tmp/go-link-066068984/000015.o" "/tmp/go-link-066068984/000016.o" "/tmp/go-link-066068984/000017.o" "/tmp/go-link-066068984/000018.o" "/tmp/go-link-066068984/000019.o" "/tmp/go-link-066068984/000020.o" "/tmp/go-link-066068984/000021.o" "/tmp/go-link-066068984/000022.o" "/tmp/go-link-066068984/000023.o" "/tmp/go-link-066068984/000024.o" "/tmp/go-link-066068984/000025.o" "/tmp/go-link-066068984/000026.o" "/tmp/go-link-066068984/000027.o" "/tmp/go-link-066068984/000028.o" "/tmp/go-link-066068984/000029.o" "-lleveldb" "-lleveldb" "-lleveldb" "-lleveldb" "-lleveldb" "-lleveldb" "-lleveldb" "-lleveldb" "-lleveldb" "-static" "-lpthread"
This is related to golang/go#25930 and will hopefully be fixed upstream. Regardless, it might be nice to consolidate all the cgo comments in the code-base into one cgo.go
file, similar to https://github.com/golang/go/blob/master/src/runtime/cgo/cgo.go.
(Not a bug but I thought this would be the most discoverable place for others to find the answer.)
Was there any deficiency in http://code.google.com/p/leveldb-go/ that led you to build this cgo implementation?
OS: MAC OSX
LevelDB: 1.13 (brew install)
Code:
roptions := levigo.NewReadOptions()
iter := db.NewIterator(roptions)
iter.Next()
val := iter.Value()
Error:
SIGSEGV: segmentation violation
PC=0x4195ce2
signal arrived during cgo execution
github.com/jmhodges/levigo._Cfunc_leveldb_iter_next(0x15200120)
github.com/jmhodges/levigo/_obj/_cgo_defun.c:213 +0x2f
github.com/jmhodges/levigo.(*Iterator).Next(0xc200000038)
github.com/jmhodges/levigo/_obj/batch.cgo1.go:547 +0x28
main.main()
Hello
I've done some work to get LevelDB's logs into Go:
Any thoughts on this approach?
It would be nice if the go tool could compile logger.cpp. I think there might be a patch for that floating around on golang-dev.
db, err := LevelDb(dbname)
if err != nil {
fmt.Errorf("can't open db %s", dbname)
panic("fail")
}
defer db.Close()
$go get github.com/jmhodges/levigo
the console give the resutl as flow:
gcc did not produce error at completed:1
on input:
// This function exists only to clean up lack-of-const warnings when
// leveldb_approximate_sizes is called from Go-land.
void levigo_leveldb_approximate_sizes(
leveldb_t* db,
int num_ranges,
char** range_start_key, const size_t* range_start_key_len,
char** range_limit_key, const size_t* range_limit_key_len,
uint64_t* sizes) {
leveldb_approximate_sizes(db,
num_ranges,
(const char* const_)range_start_key,
range_start_key_len,
(const char_ const*)range_limit_key,
range_limit_key_len,
sizes);
}
/* Define intgo when compiling with GCC. */
typedef PTRDIFF_TYPE intgo;
typedef long long intgo;
typedef int intgo;
typedef struct { char *p; intgo n; } GoString;
typedef struct { char *p; intgo n; intgo c; } GoBytes;
GoString GoString(char *p);
GoString GoStringN(char *p, int l);
GoBytes GoBytes(void *p, int n);
char *CString(GoString);
void *_CMalloc(size_t);
void cgo_f_1_1(void) { __typeof(int) *cgo_undefined; }
void *cgo_f_1_2(void) { int __cgo_undefined; }
void cgo_f_1_3(void) { enum { __cgo__undefined = (int)*1 }; }
void cgo_f_2_1(void) { __typeof(leveldb_create_iterator) *cgo_undefined; }
void *cgo_f_2_2(void) { leveldb_create_iterator __cgo_undefined; }
void cgo_f_2_3(void) { enum { __cgo__undefined = (leveldb_create_iterator)*1 }; }
void cgo_f_3_1(void) { __typeof(leveldb_t) *cgo_undefined; }
void *cgo_f_3_2(void) { leveldb_t __cgo_undefined; }
void cgo_f_3_3(void) { enum { __cgo__undefined = (leveldb_t)*1 }; }
void cgo_f_4_1(void) { __typeof(leveldb_put) *cgo_undefined; }
void *cgo_f_4_2(void) { leveldb_put __cgo_undefined; }
void cgo_f_4_3(void) { enum { __cgo__undefined = (leveldb_put)*1 }; }
void cgo_f_5_1(void) { __typeof(leveldb_repair_db) *cgo_undefined; }
void *cgo_f_5_2(void) { leveldb_repair_db __cgo_undefined; }
void cgo_f_5_3(void) { enum { __cgo__undefined = (leveldb_repair_db)*1 }; }
void cgo_f_6_1(void) { __typeof(leveldb_get) *cgo_undefined; }
void *cgo_f_6_2(void) { leveldb_get __cgo_undefined; }
void cgo_f_6_3(void) { enum { __cgo__undefined = (leveldb_get)*1 }; }
void cgo_f_7_1(void) { __typeof(GoBytes) *cgo_undefined; }
void *cgo_f_7_2(void) { GoBytes __cgo_undefined; }
void cgo_f_7_3(void) { enum { __cgo__undefined = (GoBytes)*1 }; }
void cgo_f_8_1(void) { __typeof(leveldb_write) *cgo_undefined; }
void *cgo_f_8_2(void) { leveldb_write __cgo_undefined; }
void cgo_f_8_3(void) { enum { __cgo__undefined = (leveldb_write)*1 }; }
void cgo_f_9_1(void) { __typeof(leveldb_close) *cgo_undefined; }
void *cgo_f_9_2(void) { leveldb_close __cgo_undefined; }
void cgo_f_9_3(void) { enum { __cgo__undefined = (leveldb_close)*1 }; }
void cgo_f_10_1(void) { __typeof(leveldb_snapshot_t) *cgo_undefined; }
void *cgo_f_10_2(void) { leveldb_snapshot_t __cgo_undefined; }
void cgo_f_10_3(void) { enum { __cgo__undefined = (leveldb_snapshot_t)*1 }; }
void cgo_f_11_1(void) { __typeof(leveldb_free) *cgo_undefined; }
void *cgo_f_11_2(void) { leveldb_free __cgo_undefined; }
void cgo_f_11_3(void) { enum { __cgo__undefined = (leveldb_free)*1 }; }
void cgo_f_12_1(void) { __typeof(uint64_t) *cgo_undefined; }
void *cgo_f_12_2(void) { uint64_t __cgo_undefined; }
void cgo_f_12_3(void) { enum { __cgo__undefined = (uint64_t)*1 }; }
void cgo_f_13_1(void) { __typeof(leveldb_property_value) *cgo_undefined; }
void *cgo_f_13_2(void) { leveldb_property_value __cgo_undefined; }
void cgo_f_13_3(void) { enum { __cgo__undefined = (leveldb_property_value)*1 }; }
void cgo_f_14_1(void) { __typeof(leveldb_create_snapshot) *cgo_undefined; }
void *cgo_f_14_2(void) { leveldb_create_snapshot __cgo_undefined; }
void cgo_f_14_3(void) { enum { __cgo__undefined = (leveldb_create_snapshot)*1 }; }
void cgo_f_15_1(void) { __typeof(leveldb_compact_range) *cgo_undefined; }
void *cgo_f_15_2(void) { leveldb_compact_range __cgo_undefined; }
void cgo_f_15_3(void) { enum { __cgo__undefined = (leveldb_compact_range)*1 }; }
void cgo_f_16_1(void) { __typeof(char) *cgo_undefined; }
void *cgo_f_16_2(void) { char __cgo_undefined; }
void cgo_f_16_3(void) { enum { __cgo__undefined = (char)*1 }; }
void cgo_f_17_1(void) { __typeof(size_t) *cgo_undefined; }
void *cgo_f_17_2(void) { size_t __cgo_undefined; }
void cgo_f_17_3(void) { enum { __cgo__undefined = (size_t)*1 }; }
void cgo_f_18_1(void) { __typeof(GoString) *cgo_undefined; }
void *cgo_f_18_2(void) { GoString __cgo_undefined; }
void cgo_f_18_3(void) { enum { __cgo__undefined = (GoString)*1 }; }
void cgo_f_19_1(void) { __typeof(leveldb_destroy_db) *cgo_undefined; }
void *cgo_f_19_2(void) { leveldb_destroy_db __cgo_undefined; }
void cgo_f_19_3(void) { enum { __cgo__undefined = (leveldb_destroy_db)*1 }; }
void cgo_f_20_1(void) { __typeof(leveldb_delete) *cgo_undefined; }
void *cgo_f_20_2(void) { leveldb_delete __cgo_undefined; }
void cgo_f_20_3(void) { enum { __cgo__undefined = (leveldb_delete)*1 }; }
void cgo_f_21_1(void) { __typeof(levigo_leveldb_approximate_sizes) *cgo_undefined; }
void *cgo_f_21_2(void) { levigo_leveldb_approximate_sizes __cgo_undefined; }
void cgo_f_21_3(void) { enum { __cgo__undefined = (levigo_leveldb_approximate_sizes)*1 }; }
void cgo_f_22_1(void) { __typeof(leveldb_release_snapshot) *cgo_undefined; }
void *cgo_f_22_2(void) { leveldb_release_snapshot __cgo_undefined; }
void cgo_f_22_3(void) { enum { __cgo__undefined = (leveldb_release_snapshot)*1 }; }
void cgo_f_23_1(void) { __typeof(CString) *cgo_undefined; }
void *cgo_f_23_2(void) { CString __cgo_undefined; }
void cgo_f_23_3(void) { enum { __cgo__undefined = (CString)*1 }; }
void cgo_f_24_1(void) { __typeof(leveldb_open) *cgo_undefined; }
void *cgo_f_24_2(void) { leveldb_open __cgo_undefined; }
void cgo_f_24_3(void) { enum { __cgo__undefined = (leveldb_open)*1 }; }
int __cgo__1 = __cgo__2;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.