erigontech / mdbx-go Goto Github PK
View Code? Open in Web Editor NEWBindings for https://gitflic.ru/project/erthink/libmdbx/
License: Apache License 2.0
Bindings for https://gitflic.ru/project/erthink/libmdbx/
License: Apache License 2.0
Is there a way to increment 1 to a value in db using txn or do i have to get the value first, add it inside my golang function then only save it into db?
does this mean mdbx is single threaded since it's using c?
Zero-copy reads
Applications with high performance requirements can opt-in to fast, zero-copy reads at the cost of runtime safety. Zero-copy behavior is specified at the transaction level to reduce instrumentation overhead.
maybe we can just call mdbx_cursor_get
directly
err = mdbx_put: MDBX_MAP_FULL: Environment mapsize limit reached
How to resolve this?
looking forward to this repo. great work!
Can we get db snapshot like goleveldb?
https://github.com/syndtr/goleveldb/blob/master/leveldb/db.go#L918
/go/pkg/mod/github.com/torquem-ch/[email protected]/mdbx/dist/mdbx-static.o: No such file or directory
how to resolve this error?
go get github.com/torquem-ch/mdbx-go/mdbx
See: mdbx/env.go file for other bindings example
go mod tidy
go: finding module for package github.com/torquem-ch/mdbx-go/mdbx
go: found github.com/torquem-ch/mdbx-go/mdbx in github.com/torquem-ch/mdbx-go v0.36.2
go: github.com/blockinterface/prospect/database imports
github.com/torquem-ch/mdbx-go/mdbx: github.com/torquem-ch/[email protected]: parsing go.mod:
module declares its path as: github.com/erigontech/mdbx-go
but was required as: github.com/torquem-ch/mdbx-go
xxx@node1:~/projects/psp$ go get github.com/torquem-ch/mdbx-go/mdbx
After make CGO cheaper: #142 - we can re-visit mdbxgo_cursor_get1 helpers - i guess we needed them only to create MDBX_val
inside C. But maybe it's not necessary anymore and we can use mdbx_cursor_get
directly.
Maybe you will need add benchmarks which will trigger getVal0/getVal01/getVal1/getVal2
before moving forward.
System information
which the exact version of Windows you're on ?
I have tried multiple versions of windows and it will reproduce,This crash has been in the version starting in July
Windows 11 21H2 22000.120 1000.22000.120.0
May I ask which version of GO are you on ?
go version go1.16.7 windows/amd64
how much free disk space you have on C: ?
My free disk space have on C: is 1.9T (PM9A1 2T PCIE4.0 NVME m.2)
how much RAM on your system ?
64.0 GB
are you running on a virtualized machine or physical ?
Physical machine
Expected behaviour
run
Actual behaviour
panic
Steps to reproduce the behaviour
just run erigon.exe at windows
Backtrace
Microsoft Windows [版本 10.0.22000.120]
(c) Microsoft Corporation。保留所有权利。
c:\weup\src\github.com\ledgerwatch\mdbx-go\mdbx>go test
Exception 0xc0000005 0x8 0x0 0x0
PC=0x0
runtime: unknown pc 0x356f55
stack: frame={sp:0xb21f1ffc80, fp:0x0} stack=[0x0,0xb21f1ffda0)
000000b21f1ffb80: 0000000000000330 0000000000000000
000000b21f1ffb90: 0000000b00000040 0000000ffa66b000
000000b21f1ffba0: 0000000e1220a000 000000125a66b000
000000b21f1ffbb0: 0000000fcf78a000 00007ffffffe0000
000000b21f1ffbc0: 00007ffed3c6f000 0000000000000000
000000b21f1ffbd0: 0000000002030000 0000000000000000
000000b21f1ffbe0: 0000000000000000 00000007fffffff0
000000b21f1ffbf0: 00007ffd827c8e50 000000c0000ac510
000000b21f1ffc00: 0000026cd04e9510 00007ffd827c8ee9
000000b21f1ffc10: 000000c000089de0 00000000b4bbbdff
000000b21f1ffc20: 0002625a00000000 00ffa66b00001000
000000b21f1ffc30: 0102f2ff00000001 0000000000010000
000000b21f1ffc40: 0000000000ffa66b 0000000000e1220a
000000b21f1ffc50: 00000000ffffffff 0000026cd04e93d0
000000b21f1ffc60: 0000b79ce51fd316 000000c0000ac510
000000b21f1ffc70: 000000c0000ac510 0000000000356f55
000000b21f1ffc80: <00000000001f776f <runtime.semawakeup+47> 0000026cd04e93d0
000000b21f1ffc90: 0000000000000000 0000000000000000
000000b21f1ffca0: 00000000004a01c8 0000000000379b80
000000b21f1ffcb0: 0000100000000009 0000000000010000
000000b21f1ffcc0: 00007ffffffeffff 00000000ffffffff
000000b21f1ffcd0: 000021d800000020 2100001900010000
000000b21f1ffce0: 000000c000089e20 00000000004d65c0
000000b21f1ffcf0: 000000c00008a000 0000000000314aea
000000b21f1ffd00: 000000000022a88a <runtime.callers.func1+138> 00000000002ad51f <testing.callerName+95>
000000b21f1ffd10: 000000c000089f40 0000000000000000
000000b21f1ffd20: 000000c000089e20 00000000004d65c0
000000b21f1ffd30: 000000c000089e20 0000000000230e93 <runtime.asmcgocall+115>
000000b21f1ffd40: 0000000000000000 0000000000000000
000000b21f1ffd50: 0000000000000000 0000000000000001
000000b21f1ffd60: 000000c000089ee0 0000000000000250
000000b21f1ffd70: 000000c000075380 0000000000201440 <runtime.mstart+0>
runtime: unknown pc 0x356f55
stack: frame={sp:0xb21f1ffc80, fp:0x0} stack=[0x0,0xb21f1ffda0)
000000b21f1ffb80: 0000000000000330 0000000000000000
000000b21f1ffb90: 0000000b00000040 0000000ffa66b000
000000b21f1ffba0: 0000000e1220a000 000000125a66b000
000000b21f1ffbb0: 0000000fcf78a000 00007ffffffe0000
000000b21f1ffbc0: 00007ffed3c6f000 0000000000000000
000000b21f1ffbd0: 0000000002030000 0000000000000000
000000b21f1ffbe0: 0000000000000000 00000007fffffff0
000000b21f1ffbf0: 00007ffd827c8e50 000000c0000ac510
000000b21f1ffc00: 0000026cd04e9510 00007ffd827c8ee9
000000b21f1ffc10: 000000c000089de0 00000000b4bbbdff
000000b21f1ffc20: 0002625a00000000 00ffa66b00001000
000000b21f1ffc30: 0102f2ff00000001 0000000000010000
000000b21f1ffc40: 0000000000ffa66b 0000000000e1220a
000000b21f1ffc50: 00000000ffffffff 0000026cd04e93d0
000000b21f1ffc60: 0000b79ce51fd316 000000c0000ac510
000000b21f1ffc70: 000000c0000ac510 0000000000356f55
000000b21f1ffc80: <00000000001f776f <runtime.semawakeup+47> 0000026cd04e93d0
000000b21f1ffc90: 0000000000000000 0000000000000000
000000b21f1ffca0: 00000000004a01c8 0000000000379b80
000000b21f1ffcb0: 0000100000000009 0000000000010000
000000b21f1ffcc0: 00007ffffffeffff 00000000ffffffff
000000b21f1ffcd0: 000021d800000020 2100001900010000
000000b21f1ffce0: 000000c000089e20 00000000004d65c0
000000b21f1ffcf0: 000000c00008a000 0000000000314aea
000000b21f1ffd00: 000000000022a88a <runtime.callers.func1+138> 00000000002ad51f <testing.callerName+95>
000000b21f1ffd10: 000000c000089f40 0000000000000000
000000b21f1ffd20: 000000c000089e20 00000000004d65c0
000000b21f1ffd30: 000000c000089e20 0000000000230e93 <runtime.asmcgocall+115>
000000b21f1ffd40: 0000000000000000 0000000000000000
000000b21f1ffd50: 0000000000000000 0000000000000001
000000b21f1ffd60: 000000c000089ee0 0000000000000250
000000b21f1ffd70: 000000c000075380 0000000000201440 <runtime.mstart+0>
goroutine 6 [syscall]:
github.com/torquem-ch/mdbx-go/mdbx._Cfunc_mdbx_env_create(0xc0000ac510, 0xc000000000)
_cgo_gotypes.go:620 +0x4b
github.com/torquem-ch/mdbx-go/mdbx.NewEnv.func1(0xc0000ac510, 0xc0000ac510)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env.go:137 +0x66
github.com/torquem-ch/mdbx-go/mdbx.NewEnv(0x2ad766, 0x4b1970, 0x41f37d)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env.go:137 +0x4b
github.com/torquem-ch/mdbx-go/mdbx.setupFlags(0x40ba30, 0xc000075200, 0x0, 0x379b80)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env_test.go:496 +0x2d
github.com/torquem-ch/mdbx-go/mdbx.setup(...)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env_test.go:492
github.com/torquem-ch/mdbx-go/mdbx.TestCursor_Txn(0xc000075200)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/cursor_test.go:15 +0x4b
testing.tRunner(0xc000075200, 0x3dec60)
C:/Program Files/Go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
C:/Program Files/Go/src/testing/testing.go:1238 +0x2b3
goroutine 1 [chan receive]:
testing.(*T).Run(0xc000075200, 0x3d4375, 0xe, 0x3dec60, 0x251bed)
C:/Program Files/Go/src/testing/testing.go:1239 +0x2da
testing.runTests.func1(0xc000075080)
C:/Program Files/Go/src/testing/testing.go:1511 +0x7f
testing.tRunner(0xc000075080, 0xc0000bdde0)
C:/Program Files/Go/src/testing/testing.go:1193 +0xef
testing.runTests(0xc000004078, 0x38d460, 0x3d, 0x3d, 0xc03d0ad69a213fa8, 0x8bb3012681, 0x4d6300, 0x3d4109)
C:/Program Files/Go/src/testing/testing.go:1509 +0x310
testing.(*M).Run(0xc0000dc000, 0x0)
C:/Program Files/Go/src/testing/testing.go:1417 +0x1f5
main.main()
_testmain.go:187 +0x145
rax 0x0
rbx 0x26cd04e93d0
rcx 0x26cd04e9558
rdi 0xc0000ac510
rsi 0x7ffd827c8e50
rbp 0xc000089de0
rsp 0xb21f1ffc78
r8 0x2d
r9 0x3e
r10 0x0
r11 0x3ef6e5f10e4be44
r12 0x404ed0
r13 0x0
r14 0x0
r15 0x2030000
rip 0x0
rflags 0x10202
cs 0x33
fs 0x53
gs 0x2b
exit status 2
FAIL github.com/torquem-ch/mdbx-go/mdbx 0.523s
c:\weup\src\github.com\ledgerwatch\mdbx-go\mdbx>
The document link is now 404 in github.
Similar to lmdb: https://github.com/ledgerwatch/lmdb-go/blob/master/lmdb/lmdb.go#L134
But mdbx has config.h file - somehow need to pass all variables without config.h file
we added it in erigon's kv_mdbx.go - probably can push-down it to mdbx-go
we have it in Erigon: kv_mdbx.go
see Makefile and .golangci.yml
take a look at golangcilint linters related to CGO or performance - maybe we can enable them
nice work. possible to add this?
Hi,
Apologies if I'm reading this wrong, it seems to me that the Sequence()
return value is equal to the md_seq
before the increment and not after?
Shouldn't we be reading the value of new
?
func (txn *Txn) Sequence(dbi DBI, increment uint64) (uint64, error) {
var res C.uint64_t
ret := C.mdbx_dbi_sequence(txn._txn, C.MDBX_dbi(dbi), &res, C.uint64_t(increment))
if ret != 0 {
return uint64(res), operrno("mdbx_dbi_sequence", ret)
}
return uint64(res), nil
}
when I want to store the values which abount 100K ~ 1000K, on my ssd disk the benchmark put function is less than 1000 per second.
this is my mdbx env.Open flag :
mdbx.Accede | mdbx.UtterlyNoSync | mdbx.Coalesce | mdbx.NoReadahead
key, val = 20 bytes:
BenchmarkMdbxStorage_Set-8 1152529 1884 ns/op 144 B/op 6 allocs/op
key = 20bytes, val = 512K bytes
BenchmarkMdbxStorage_Set-8 720 1521572 ns/op 141 B/op 6 allocs/op
So, when I store 1 large item, it will take more than 1ms, what is the best way to use mdbx store 100K ~ 1000K item?
package main
import (
"fmt"
"github.com/torquem-ch/mdbx-go/mdbx"
"os"
)
func main() {
env, err := mdbx.NewEnv()
if err != nil {
fmt.Fprintf(os.Stderr, "failed to create mdbx env: %s\n", err)
os.Exit(1)
}
env.SetOption(mdbx.OptMaxDB, 20)
err = env.Open("/tmp/test-mdbx", mdbx.Coalesce|mdbx.LifoReclaim|mdbx.WriteMap, 0664)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to open the env: %s\n", err)
os.Exit(1)
}
idx := 0
updateDbFunc := func(txn *mdbx.Txn) error {
dbi, err := txn.CreateDBI("hello")
if err != nil {
fmt.Fprintf(os.Stderr, "failed to create dbi: %s\n", err)
os.Exit(1)
}
for i := 0; i < 1024*1024*2; i++ {
key := fmt.Sprintf("abcdef%d", idx)
idx += 1
err = txn.Put(dbi, []byte(key), []byte("xyzxyz"), 0)
if err != nil {
// some errors
fmt.Fprintf(os.Stderr, "failed to write to db: %s\n", err)
os.Exit(1)
}
}
env.CloseDBI(dbi)
return nil
}
env.Update(updateDbFunc)
env.Close()
}
In this simple demo, I got errors:
failed to write to db: mdbx_put: MDBX_MAP_FULL: Environment mapsize limit reached
Would someone help illuminate which parat I'm missing?
oops, website is up now. and i got it working. thx
Does this lib support batching multiple inserts and as well as getting multiple keys at once ?
likely need port next PR (or use it as an inspiration): https://github.com/bmatsuo/lmdb-go/pull/99/files#diff-61105a4fb9b9f0a342528e599475968fR165
e.g.
"github.com/torquem-ch/mdbx-go/mdbx"
and then what?
mdbx.Open() <- this is not working. How do you use the DBI? I'm converting boltdb to use this. Can you help?
opening the same path twice results in an MDBX_BUSY
error despite not passing the MDBX_EXCLUSIVE
flag 🤷♂️
panic: mdbx_env_open: MDBX_BUSY: Another write transaction is running,
or environment is already used while opening with MDBX_EXCLUSIVE flag
in the example below there is neither a write transaction running, nor were either opened with the MDBX_EXCLUSIVE
flag:
package main
import (
"fmt"
"io/ioutil"
"github.com/torquem-ch/mdbx-go/mdbx"
)
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
// create a temporary directory
tmpdir, err := ioutil.TempDir("", "mdbx")
check(err)
datpath := fmt.Sprintf("%s/foobar.db", tmpdir)
fmt.Println("datpath:", datpath)
// open environment
env1, err := mdbx.NewEnv()
check(err)
check(env1.Open(datpath, mdbx.NoSubdir, 444))
flag1, err := env1.Flags()
check(err)
fmt.Println("exclusive:", flag1&mdbx.Exclusive)
// open new environment
env2, err := mdbx.NewEnv()
check(err)
check(env2.Open(datpath, mdbx.NoSubdir, 444))
}
cat go.sum
github.com/torquem-ch/mdbx-go v0.23.2 h1:7axXl0leix2v8No+mRzeTV32hJrV1817aKhh+hTEpC8=
github.com/torquem-ch/mdbx-go v0.23.2/go.mod h1:T2fsoJDVppxfAPTLd1svUgH1kpPmeXdPESmroSHcL1E=
Heya,
I'd like to have a play around with db Compaction, which was a feature I was using a bit from boltdb.
Would you consider merging a PR to uncomment the lines below, was there a reason they were commented out?
Erigon need to check this error (by errors.is) in kv_mdbx.go and return nil (erigo's kv interface return nil instead of error when key not found).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.