erigontech / mdbx-go Goto Github PK

Is there a way to increment 1 to a value in db using txn or do i have to get the value first, add it inside my golang function then only save it into db?

does this mean mdbx is single threaded since it's using c?

1. what's the cost of runtime safety?
2. how much faster? 2x? 3x? ( just wondering if worth while to implement over safety issues)

Zero-copy reads
Applications with high performance requirements can opt-in to fast, zero-copy reads at the cost of runtime safety. Zero-copy behavior is specified at the transaction level to reduce instrumentation overhead.

maybe we can just call mdbx_cursor_get directly

err = mdbx_put: MDBX_MAP_FULL: Environment mapsize limit reached

How to resolve this?

looking forward to this repo. great work!

Can we get db snapshot like goleveldb?

https://github.com/syndtr/goleveldb/blob/master/leveldb/db.go#L918

/go/pkg/mod/github.com/torquem-ch/[email protected]/mdbx/dist/mdbx-static.o: No such file or directory

how to resolve this error?

go get github.com/torquem-ch/mdbx-go/mdbx

See: mdbx/env.go file for other bindings example

After make CGO cheaper: #142 - we can re-visit mdbxgo_cursor_get1 helpers - i guess we needed them only to create MDBX_val inside C. But maybe it's not necessary anymore and we can use mdbx_cursor_get directly.

Maybe you will need add benchmarks which will trigger getVal0/getVal01/getVal1/getVal2 before moving forward.

System information

which the exact version of Windows you're on ?
I have tried multiple versions of windows and it will reproduce，This crash has been in the version starting in July
Windows 11 21H2 22000.120 1000.22000.120.0

May I ask which version of GO are you on ?
go version go1.16.7 windows/amd64

how much free disk space you have on C: ?
My free disk space have on C: is 1.9T (PM9A1 2T PCIE4.0 NVME m.2)

how much RAM on your system ?
64.0 GB

are you running on a virtualized machine or physical ?
Physical machine

Expected behaviour
run

Actual behaviour
panic

Steps to reproduce the behaviour
just run erigon.exe at windows

Backtrace

Microsoft Windows [版本 10.0.22000.120]
(c) Microsoft Corporation。保留所有权利。

c:\weup\src\github.com\ledgerwatch\mdbx-go\mdbx>go test
Exception 0xc0000005 0x8 0x0 0x0
PC=0x0

runtime: unknown pc 0x356f55
stack: frame={sp:0xb21f1ffc80, fp:0x0} stack=[0x0,0xb21f1ffda0)
000000b21f1ffb80: 0000000000000330 0000000000000000
000000b21f1ffb90: 0000000b00000040 0000000ffa66b000
000000b21f1ffba0: 0000000e1220a000 000000125a66b000
000000b21f1ffbb0: 0000000fcf78a000 00007ffffffe0000
000000b21f1ffbc0: 00007ffed3c6f000 0000000000000000
000000b21f1ffbd0: 0000000002030000 0000000000000000
000000b21f1ffbe0: 0000000000000000 00000007fffffff0
000000b21f1ffbf0: 00007ffd827c8e50 000000c0000ac510
000000b21f1ffc00: 0000026cd04e9510 00007ffd827c8ee9
000000b21f1ffc10: 000000c000089de0 00000000b4bbbdff
000000b21f1ffc20: 0002625a00000000 00ffa66b00001000
000000b21f1ffc30: 0102f2ff00000001 0000000000010000
000000b21f1ffc40: 0000000000ffa66b 0000000000e1220a
000000b21f1ffc50: 00000000ffffffff 0000026cd04e93d0
000000b21f1ffc60: 0000b79ce51fd316 000000c0000ac510
000000b21f1ffc70: 000000c0000ac510 0000000000356f55
000000b21f1ffc80: <00000000001f776f <runtime.semawakeup+47> 0000026cd04e93d0
000000b21f1ffc90: 0000000000000000 0000000000000000
000000b21f1ffca0: 00000000004a01c8 0000000000379b80
000000b21f1ffcb0: 0000100000000009 0000000000010000
000000b21f1ffcc0: 00007ffffffeffff 00000000ffffffff
000000b21f1ffcd0: 000021d800000020 2100001900010000
000000b21f1ffce0: 000000c000089e20 00000000004d65c0
000000b21f1ffcf0: 000000c00008a000 0000000000314aea
000000b21f1ffd00: 000000000022a88a <runtime.callers.func1+138> 00000000002ad51f <testing.callerName+95>
000000b21f1ffd10: 000000c000089f40 0000000000000000
000000b21f1ffd20: 000000c000089e20 00000000004d65c0
000000b21f1ffd30: 000000c000089e20 0000000000230e93 <runtime.asmcgocall+115>
000000b21f1ffd40: 0000000000000000 0000000000000000
000000b21f1ffd50: 0000000000000000 0000000000000001
000000b21f1ffd60: 000000c000089ee0 0000000000000250
000000b21f1ffd70: 000000c000075380 0000000000201440 <runtime.mstart+0>
runtime: unknown pc 0x356f55
stack: frame={sp:0xb21f1ffc80, fp:0x0} stack=[0x0,0xb21f1ffda0)
000000b21f1ffb80: 0000000000000330 0000000000000000
000000b21f1ffb90: 0000000b00000040 0000000ffa66b000
000000b21f1ffba0: 0000000e1220a000 000000125a66b000
000000b21f1ffbb0: 0000000fcf78a000 00007ffffffe0000
000000b21f1ffbc0: 00007ffed3c6f000 0000000000000000
000000b21f1ffbd0: 0000000002030000 0000000000000000
000000b21f1ffbe0: 0000000000000000 00000007fffffff0
000000b21f1ffbf0: 00007ffd827c8e50 000000c0000ac510
000000b21f1ffc00: 0000026cd04e9510 00007ffd827c8ee9
000000b21f1ffc10: 000000c000089de0 00000000b4bbbdff
000000b21f1ffc20: 0002625a00000000 00ffa66b00001000
000000b21f1ffc30: 0102f2ff00000001 0000000000010000
000000b21f1ffc40: 0000000000ffa66b 0000000000e1220a
000000b21f1ffc50: 00000000ffffffff 0000026cd04e93d0
000000b21f1ffc60: 0000b79ce51fd316 000000c0000ac510
000000b21f1ffc70: 000000c0000ac510 0000000000356f55
000000b21f1ffc80: <00000000001f776f <runtime.semawakeup+47> 0000026cd04e93d0
000000b21f1ffc90: 0000000000000000 0000000000000000
000000b21f1ffca0: 00000000004a01c8 0000000000379b80
000000b21f1ffcb0: 0000100000000009 0000000000010000
000000b21f1ffcc0: 00007ffffffeffff 00000000ffffffff
000000b21f1ffcd0: 000021d800000020 2100001900010000
000000b21f1ffce0: 000000c000089e20 00000000004d65c0
000000b21f1ffcf0: 000000c00008a000 0000000000314aea
000000b21f1ffd00: 000000000022a88a <runtime.callers.func1+138> 00000000002ad51f <testing.callerName+95>
000000b21f1ffd10: 000000c000089f40 0000000000000000
000000b21f1ffd20: 000000c000089e20 00000000004d65c0
000000b21f1ffd30: 000000c000089e20 0000000000230e93 <runtime.asmcgocall+115>
000000b21f1ffd40: 0000000000000000 0000000000000000
000000b21f1ffd50: 0000000000000000 0000000000000001
000000b21f1ffd60: 000000c000089ee0 0000000000000250
000000b21f1ffd70: 000000c000075380 0000000000201440 <runtime.mstart+0>

goroutine 6 [syscall]:
github.com/torquem-ch/mdbx-go/mdbx._Cfunc_mdbx_env_create(0xc0000ac510, 0xc000000000)
_cgo_gotypes.go:620 +0x4b
github.com/torquem-ch/mdbx-go/mdbx.NewEnv.func1(0xc0000ac510, 0xc0000ac510)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env.go:137 +0x66
github.com/torquem-ch/mdbx-go/mdbx.NewEnv(0x2ad766, 0x4b1970, 0x41f37d)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env.go:137 +0x4b
github.com/torquem-ch/mdbx-go/mdbx.setupFlags(0x40ba30, 0xc000075200, 0x0, 0x379b80)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env_test.go:496 +0x2d
github.com/torquem-ch/mdbx-go/mdbx.setup(...)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env_test.go:492
github.com/torquem-ch/mdbx-go/mdbx.TestCursor_Txn(0xc000075200)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/cursor_test.go:15 +0x4b
testing.tRunner(0xc000075200, 0x3dec60)
C:/Program Files/Go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
C:/Program Files/Go/src/testing/testing.go:1238 +0x2b3

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000075200, 0x3d4375, 0xe, 0x3dec60, 0x251bed)
C:/Program Files/Go/src/testing/testing.go:1239 +0x2da
testing.runTests.func1(0xc000075080)
C:/Program Files/Go/src/testing/testing.go:1511 +0x7f
testing.tRunner(0xc000075080, 0xc0000bdde0)
C:/Program Files/Go/src/testing/testing.go:1193 +0xef
testing.runTests(0xc000004078, 0x38d460, 0x3d, 0x3d, 0xc03d0ad69a213fa8, 0x8bb3012681, 0x4d6300, 0x3d4109)
C:/Program Files/Go/src/testing/testing.go:1509 +0x310
testing.(*M).Run(0xc0000dc000, 0x0)
C:/Program Files/Go/src/testing/testing.go:1417 +0x1f5
main.main()
_testmain.go:187 +0x145
rax 0x0
rbx 0x26cd04e93d0
rcx 0x26cd04e9558
rdi 0xc0000ac510
rsi 0x7ffd827c8e50
rbp 0xc000089de0
rsp 0xb21f1ffc78
r8 0x2d
r9 0x3e
r10 0x0
r11 0x3ef6e5f10e4be44
r12 0x404ed0
r13 0x0
r14 0x0
r15 0x2030000
rip 0x0
rflags 0x10202
cs 0x33
fs 0x53
gs 0x2b
exit status 2
FAIL github.com/torquem-ch/mdbx-go/mdbx 0.523s

c:\weup\src\github.com\ledgerwatch\mdbx-go\mdbx>

repository admin has to do it

The document link is now 404 in github.

Similar to lmdb: https://github.com/ledgerwatch/lmdb-go/blob/master/lmdb/lmdb.go#L134
But mdbx has config.h file - somehow need to pass all variables without config.h file

we added it in erigon's kv_mdbx.go - probably can push-down it to mdbx-go
we have it in Erigon: kv_mdbx.go

see Makefile and .golangci.yml

take a look at golangcilint linters related to CGO or performance - maybe we can enable them

nice work. possible to add this?

Hi,

Apologies if I'm reading this wrong, it seems to me that the Sequence() return value is equal to the md_seq before the increment and not after?

Shouldn't we be reading the value of new?

func (txn *Txn) Sequence(dbi DBI, increment uint64) (uint64, error) {
	var res C.uint64_t
	ret := C.mdbx_dbi_sequence(txn._txn, C.MDBX_dbi(dbi), &res, C.uint64_t(increment))
	if ret != 0 {
		return uint64(res), operrno("mdbx_dbi_sequence", ret)
	}
	return uint64(res), nil
}

when I want to store the values which abount 100K ~ 1000K, on my ssd disk the benchmark put function is less than 1000 per second.

this is my mdbx env.Open flag :

mdbx.Accede | mdbx.UtterlyNoSync | mdbx.Coalesce | mdbx.NoReadahead

key, val = 20 bytes:

BenchmarkMdbxStorage_Set-8   	 1152529	      1884 ns/op	     144 B/op	       6 allocs/op

key = 20bytes, val = 512K bytes

BenchmarkMdbxStorage_Set-8   	     720	   1521572 ns/op	     141 B/op	       6 allocs/op

So, when I store 1 large item, it will take more than 1ms, what is the best way to use mdbx store 100K ~ 1000K item?

package main

import (
	"fmt"
	"github.com/torquem-ch/mdbx-go/mdbx"
	"os"
)

func main() {
	env, err := mdbx.NewEnv()
	if err != nil {
		fmt.Fprintf(os.Stderr, "failed to create mdbx env: %s\n", err)
		os.Exit(1)
	}

	env.SetOption(mdbx.OptMaxDB, 20)

	err = env.Open("/tmp/test-mdbx", mdbx.Coalesce|mdbx.LifoReclaim|mdbx.WriteMap, 0664)
	if err != nil {
		fmt.Fprintf(os.Stderr, "failed to open the env: %s\n", err)
		os.Exit(1)
	}

	idx := 0

	updateDbFunc := func(txn *mdbx.Txn) error {
		dbi, err := txn.CreateDBI("hello")
		if err != nil {
			fmt.Fprintf(os.Stderr, "failed to create dbi: %s\n", err)
			os.Exit(1)
		}

		for i := 0; i < 1024*1024*2; i++ {
			key := fmt.Sprintf("abcdef%d", idx)
			idx += 1

			err = txn.Put(dbi, []byte(key), []byte("xyzxyz"), 0)
			if err != nil {
				// some errors
				fmt.Fprintf(os.Stderr, "failed to write to db: %s\n", err)
				os.Exit(1)
			}
		}

		env.CloseDBI(dbi)

		return nil
	}
	env.Update(updateDbFunc)

	env.Close()
}

In this simple demo, I got errors:

failed to write to db: mdbx_put: MDBX_MAP_FULL: Environment mapsize limit reached

Would someone help illuminate which parat I'm missing?

oops, website is up now. and i got it working. thx

Does this lib support batching multiple inserts and as well as getting multiple keys at once ?

likely need port next PR (or use it as an inspiration): https://github.com/bmatsuo/lmdb-go/pull/99/files#diff-61105a4fb9b9f0a342528e599475968fR165

e.g.
"github.com/torquem-ch/mdbx-go/mdbx"

and then what?

mdbx.Open() <- this is not working. How do you use the DBI? I'm converting boltdb to use this. Can you help?

opening the same path twice results in an MDBX_BUSY error despite not passing the MDBX_EXCLUSIVE flag 🤷‍♂️

panic: mdbx_env_open: MDBX_BUSY: Another write transaction is running, 
or environment is already used while opening with MDBX_EXCLUSIVE flag

in the example below there is neither a write transaction running, nor were either opened with the MDBX_EXCLUSIVE flag:

package main

import (
	"fmt"
	"io/ioutil"

	"github.com/torquem-ch/mdbx-go/mdbx"
)

func check(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	// create a temporary directory
	tmpdir, err := ioutil.TempDir("", "mdbx")
	check(err)
	datpath := fmt.Sprintf("%s/foobar.db", tmpdir)
	fmt.Println("datpath:", datpath)

	// open environment
	env1, err := mdbx.NewEnv()
	check(err)
	check(env1.Open(datpath, mdbx.NoSubdir, 444))
	flag1, err := env1.Flags()
	check(err)
	fmt.Println("exclusive:", flag1&mdbx.Exclusive)

	// open new environment
	env2, err := mdbx.NewEnv()
	check(err)
	check(env2.Open(datpath, mdbx.NoSubdir, 444))
}

cat go.sum
github.com/torquem-ch/mdbx-go v0.23.2 h1:7axXl0leix2v8No+mRzeTV32hJrV1817aKhh+hTEpC8=
github.com/torquem-ch/mdbx-go v0.23.2/go.mod h1:T2fsoJDVppxfAPTLd1svUgH1kpPmeXdPESmroSHcL1E=

Heya,

I'd like to have a play around with db Compaction, which was a feature I was using a bit from boltdb.
Would you consider merging a PR to uncomment the lines below, was there a reason they were commented out?

https://github.com/torquem-ch/mdbx-go/blob/f3edade9116463fb327849863a971544f9e3abf2/mdbx/env.go#L242-L276

Erigon need to check this error (by errors.is) in kv_mdbx.go and return nil (erigo's kv interface return nil instead of error when key not found).