nutsdb / nutsdb Goto Github PK

key不变, value变化
重复N次, 数据库可能存了N个,不会覆盖/清理

尝试用Merge方法, 也不会清理空间

I was wondering why nutsdb is so fast, so I did a quick review on your code, and there is no flush/sync call in the code, you should call it to ensure durability and update the benchmark. 💪

Also, the README should mention the byte endianness issue as well, and the isolation level is not clear.

I wanted to try this db. I wanted to set 10M keys, read them sequentially, then write 10k random values and read 10k random values. I did that with other databases(bolt, badger, bitcask..) so I wanted to compare nuts as well. The thing is that no matter what, the transaction just keeps on going. It creates these 8MB files and writes at under 3MB/s speed but no matter how long I wait, it just keeps on going without finishing.

I tried a single transaction and I tried multiple transactions and setting 100 values at a time, but the same result.

So I am quite baffled as to why this is happening?

• Support persistence
• Discard mmap package (this package maybe has bugs about Flush function)
• Discard EntryIdxMode options: HintAndRAMIdxMode and HintAndMemoryMapIdxMode
• Add new EntryIdxMode options: HintKeyValAndRAMIdxMode and HintKeyAndRAMIdxMode
• Fix when fn is nil
• Update README && CHANGELOG

看到文档关于索引模式的介绍, 我得出以下结论.
请帮忙看理解是否正确? (有3个问题,Q1/Q2/Q3)

HintKeyValAndRAMIdxMode

• (默认) 最快: 内存使用最大, key和value都放在内存,
• **Q1**数据量受限于内存? 我10G的数据量,就需要10GB内存? (如果不是, 有没有一个数据给参考)
• 支持其他结构 list\set\sorted set

HintKeyAndRAMIdxMode

• 中等速度, 内存中等,
• 适合 value远大于key的场景效果
• **Q2**以后有可能支持 sorted set吗?
• **Q3**目前的GoLevelDB 最像这个模式?

HintBPTSparseIdxMode

• 最低速度, 内存最小, 极限省内存模式.

参考:

issue #33 里面说到

• 总元数据量15G左右,结果吃掉我80G的内存,而且我用的是HintKeyAndRAMIdxMode的方式
• 消耗内存是数据量的 3.46倍左右
  所以我对于 索引模式的消耗内存情况不是很清晰了...

文档:

如果使用HintKeyAndRAMIdxMode的选项，读速度比默认配置低很多。道理很简单，默认配置是全内存索引，但是HintKeyAndRAMIdxMode的模式，是内存索引+磁盘混合的方式，但是这个选项模式可以保存远大于内存的数据。特别是value远大于key的场景效果更明显。
当前版本使用HintKeyValAndRAMIdxMode、 HintKeyAndRAMIdxMode和HintBPTSparseIdxMode 这三种作为db启动的时候索引模式。 默认使用HintKeyValAndRAMIdxMode。在基本的功能的string数据类型（put、get、delete、rangeScan、PrefixScan）这三种模式都支持。HintKeyValAndRAMIdxMode，作为数据库默认选项，他是全内存索引，读写性能都很高。他的瓶颈在于内存。如果你内存够的话，这种默认是适合的。另一种模式HintKeyAndRAMIdxMode，他会把value存磁盘，通过索引去找offset，这种模式特别适合value远大于key的场景，他的读性能要比起默认模式要降低不少。HintBPTSparseIdxMode这个模式（v0.4.0之后支持）这个模式非常省内存，使用多级索引，测试10亿数据，只占用80几MB的内存，但是读性能比较差，需要自己加缓存加速。具体看自己的要求选择模式。

天下没有银弹，NutsDB也有他的局限，比如随着数据量的增大，索引变大，启动会慢, 还有为了保持保->高 性能的写如->入，目前版本没有做实时的flush/sync操作，依赖内核的回写操作。只想说NutsDB还有很多优化和提高的空间，由于本人精力以及能力有限。所以把这个项目开源出来。更重要的是我认为一个项目需要有人去使用，有人提意见才会成长。

在忘记某个bucket名字时，知道所有的bucket信息可能有帮助，谢谢！

• [New feature] Support mmap loading file
• [Bug Fix] Fix tx bug when a tx commits
• [Change] Add rwmanager interface
• [Change] Add new options: RWMode, SyncEnable and StartFileLoadingMode
• [Change] Clean up some codes
• [Change] Update README && CHANGELOG

Describe the solution you'd like

Hello, NutsDB is great KV store. I want to use NutsDB in my big project and I look for callback function feature on key eviction by TTL. For example, "bigcache" project have a custom callback function (OnRemove). It is a be great if callback function will be in NutsDB.

In my case, I have a real files on the file system and TTL in memory KV store for these files. When TTL time for key is gone, then with callback I could do remove file from file system.

I think this is useful feature and may be easy to implement in NutsDB.

Thank You.

以前一直用 http://ssdb.io/.

其中有一个选项

leveldb.compression 压缩硬盘上的数据
最好设置为 yes! 如果是 yes, 一般你能存储 10 倍硬盘空间的数据, 而且性能会更>好.

如果不压缩, 我是不是要在存入前, 自行用zlib等进行压缩

Describe the bug
tx.GetAll undefined (type *nutsdb.Tx has no field or method GetAll)

To Reproduce

	err := NDB.View(
		func(tx *nutsdb.Tx) error {
			entries, err := tx.GetAll(bucket)
			if err != nil {
				return err
			}

			for _, entry := range entries {
				fmt.Println(string(entry.Key), string(entry.Value))
			}

			return nil
		})

Expected behavior
tx.GetAll should be available, accoring to:
https://xujiajun.cn/nutsdb/#get-all

What actually happens
An error :-(

please complete the following information :

• OS: Mac OS 10.14.6
• NutsDB Version : 0.4.0

Is your feature request related to a problem? Please describe.
The GetAll function should return an iterator to the key-value instead of reading the entire db into the memory since that'd make us dependable on the amount of system memory.

Describe the solution you'd like
Use callbacks to allow user to handle the key-value items returned from the db.

missing " at https://github.com/xujiajun/nutsdb/blob/f5d505c82c271735c25e2201baed0ed390a188be/README.md#L367

bucket：= []byte("user_list")

https://godoc.org/github.com/syndtr/goleveldb/leveldb#DB.OpenTransaction
as the doc says that golevel does support transaction.

`

GoLevelDB GoLevelDB 是 google 开源的 leveldb 的 go 语言版本的实现。他的性能很高，特别是写性能，据官方 c++版本说可以到 40w+次写 /秒，他基于 LSM tree 实现。他不支持事务。

`
为什么说 不支持呢 ？

• [New feature] Support EntryIdxMode: HintBPTSparseIdxMode
• [New feature] Support GetAll() function for all models
• [Bug Fix] Fix error too many open files in system
• [Bug Fix] Fix constant 2147483648 overflows int
• [Bug Fix] Fix when the number of files waiting to be merged not at least 2
• [Bug Fix] Fix data pollution when executing the merge method
• [Change] Modify Records type && Entries type
• [Change] Refactor for tx Commit function
• [Change] Update Iterating over keys about README
• [Change] Fix some grammatical mistakes about README
• [Change] Rename variable for func ReadBPTreeRootIdxAt
• [Change] Add issue templates
• [Change] Update README && CHANGELOG

是不是没有清空或者直接删除bucket的操作

目前已经用在业务上，希望能增加分布式

Describe the bug
Iterating the key-value store doesn't work with HintBPTSparseIdxMode indexing option.

To Reproduce
Steps to reproduce the behavior(Be specific!): Run this

err := c.db.View(
		func(tx *nutsdb.Tx) error {
			entries, err := tx.GetAll(bucketName)
			if err != nil {
				return err
			}

			for _, entry := range entries {
				callback(entry.Key, entry.Value)
			}
			return nil
		},
	)

Expected behavior
Keys and values should be iterated.

What actually happens
It says bucket not found, while debugging, i see that the routine doesn't has a separate function for ActiveBPTTree, and instead it works with normal indexing.

Screenshots

This seems like the issue -

if index, ok := tx.db.BPTreeIdx[bucket]; ok {
		records, err := index.All()
		if err != nil {
			return nil, ErrBucketEmpty
		}

		entries, err = tx.getHintIdxDataItemsWrapper(records, ScanNoLimit, entries, RangeScan)
		if err != nil {
			return nil, ErrBucketEmpty
		}
	}

please complete the following information :

• OS: Ubuntu 16.04
• NutsDB Version Latest

比如下面这样写，

db.Update(nil)
db.View(nil)

会出现死锁错误：fatal error: all goroutines are asleep - deadlock!

thank u.

已用于生产环境，目前来说还不错，就是用得比较多的方法如exists，incr，llen等都得自己实现一遍，建议作者可以加上去。

与其计划做成分布式，作者不如先考虑让这个项目的各种方法以及内置类型尽量向redis靠齐，成为嵌入版本的redis，这样受众反而会多一些，因为可以降低学习成本。

Can you add a similar ForEach() function like in boltdb?
if i known only bucket name and want get all keys, values.
Thanks

GOARCH=386
GOOS=windows
github.com\xujiajun\nutsdb\ds\zset\sortedset.go:291:13: constant 2147483648 overflows int
查看代码发现有一个初始值：
limit := 1 << 31

limit值int型

请问nutsdb有没有在生产环境使用的案例呢？

支持iterator
Iterator应该是范围扫描常用的方式，我们使用GO+CGO+RocksDB构建应用，就大量使用到Iterator。

希望支持iterator
nutsdb是个原生GO语言kv数据库，希望能增加正向iterator和反向iterator的功能，我们就可以拿来试用替换rocksdb，再也不用CGO了:)

Issue

I'm not able to execute tx.LSet over db object. It throws the error err bucket. Although it is working fine for tx.Get as well as tx.Set.

Version

v0.4.0

Sample code

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		if err := tx.LSet(bucket, key, 0, values[0]); err != nil {
			return err
		}

		return nil
	}); err != nil {
        fmt.Println(err.Error())
}

问题描述

如题

版本

v0.5.0

逻辑

使用Update写入数据，然后使用View查询，报bucket is empty

代码

package main

import (
    "fmt"
    "github.com/xujiajun/nutsdb"
    "log"
)

func main() {
    opt := nutsdb.DefaultOptions
    opt.Dir = "/tmp/nutsdb" //这边数据库会自动创建这个目录文件
    db, err := nutsdb.Open(opt)
    if err != nil {
        log.Fatal(err)
    }   
    defer db.Close()

    for i := 0; i < 100; i++ {
        err := db.Update(func(tx *nutsdb.Tx) error {
            bucket := "test"
            key := []byte("myList")
            val := []byte(fmt.Sprintf("myList: %d\n", i)) 
            return tx.RPush(bucket, key, val)
        })
        if err != nil {
            log.Println(err)
        }
    }   

    if err := db.View(
        func(tx *nutsdb.Tx) error {
            bucket := "test"
            entries, err := tx.GetAll(bucket)
            if err != nil {
                return err 
            }

            for _, entry := range entries {
                fmt.Println(string(entry.Key), string(entry.Value))
            }

            return nil 
        }); err != nil {
        log.Println(err)
    }   
}

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/xujiajun/nutsdb.(*DB).getMaxFileIDAndFileIDs(0xc000044aa0, 0x0, 0xc000069d28, 0x81f8ec, 0x91d5bc)
D:/gopath/src/github.com/xujiajun/nutsdb/db.go:347 +0x33d
github.com/xujiajun/nutsdb.(*DB).buildIndexes(0xc000044aa0, 0x3, 0x30000076f25401)
D:/gopath/src/github.com/xujiajun/nutsdb/db.go:616 +0x36
github.com/xujiajun/nutsdb.Open(0x91d5bc, 0x3, 0x0, 0x0, 0x800000, 0x1, 0x1, 0x1, 0xa509ff, 0xc000069eb8, ...)
D:/gopath/src/github.com/xujiajun/nutsdb/db.go:174 +0x164
leibe.cn/roomMessageOutput/db/ndb.GetNDB(0xd79500)
D:/gopath/src/leibe.cn/roomMessageOutput/db/ndb/ndb.go:12 +0xc5
main.main()
D:/gopath/src/leibe.cn/roomMessageOutput/easy.go:16 +0x3b

代码：
package ndb

import (
"github.com/xujiajun/nutsdb"
"log"
)
var NDB *nutsdb.DB

func GetNDB() *nutsdb.DB {
opt := nutsdb.DefaultOptions
opt.Dir = ./
NDB, err := nutsdb.Open(opt)
if err != nil {
log.Fatal(err,NDB)
}
return NDB
}

• already exist a key with TTL

tx.Put(bucket, key, val, 60)  // now TTL is 60s

// 30s after
tx.Put(bucket, key, val, 60) // what's TTL now?

• already exist a persistent key

tx.Put(bucket, key, val, 0) // put a persistent key

// some times later
tx.Put(bucket, key, val, 60) // the key was persistent or would expired after 60s?

Describe the bug
目前观察到当goarch=386时似乎无法正常存入数据并且无报错.

相关代码
数据库初始化
数据插入
数据读取

相关issue
Mrs4s/go-cqhttp#93

Hello! I'm trying to write a console viewer for your database. I wander is there a way to list all buckets?
It looks like I should just iterate keys of db.BPTreeIdx is it the right way?

Hi @xujiajun,

Do we have a checklist, which is being tracked to make this production-ready? What are the major issues we are facing to have it production-ready?

package main

import (
"fmt"
"github.com/xujiajun/nutsdb"
"log"
"os"
"time"
)

var (
DBBucket = "test"
DBDir = "/tmp/nutsdb"
)

func main() {
_ = os.RemoveAll(DBDir)
opt := nutsdb.DefaultOptions
opt.Dir = DBDir
db, err := nutsdb.Open(opt)
if err != nil {
log.Fatal(err)
}
defer db.Close()
_ = db.Update(func(tx *nutsdb.Tx) error {
key := []byte("testKey")
tx.RPush(DBBucket, key, []byte(time.Now().String()))
time.Sleep(time.Second)
tx.RPush(DBBucket, key, []byte(time.Now().String()))
time.Sleep(time.Second)
tx.RPush(DBBucket, key, []byte(time.Now().String()))
time.Sleep(time.Second)
tx.RPush(DBBucket, key, []byte(time.Now().String()))
time.Sleep(time.Second)

	return nil
})

_ = db.View(func(tx *nutsdb.Tx) error {
	key := []byte("testKey")

	b, _ := tx.LPop(DBBucket, key)
	fmt.Println(string(b))
	return nil
})

_ = db.View(func(tx *nutsdb.Tx) error {
	key := []byte("testKey")

	b, _ := tx.LPop(DBBucket, key)
	fmt.Println(string(b))
	return nil
})

_ = db.View(func(tx *nutsdb.Tx) error {
	key := []byte("testKey")

	b, _ := tx.LRange(DBBucket, key, 0, -1)
	for _, b1 := range b {
		fmt.Println(string(b1))
	}
	return nil
})

}

这是选择的模式有问题吗？

https://github.com/xujiajun/gokvstore-bench

对 commit() 函数实现有点疑惑：如果 commit 时写数据到磁盘成功，但是更新 txid 为已提交（插入B+树）失败，报错给 client 后宕机，nutsdb 重启时会根据数据文件重新生成索引，这时候会更新 txid 为已提交。

请问，我理解的有问题吗？这样会导致提交失败的变更实际是可见的～

如果我没有保存过一个key，我尝试去get它，返回错误 ErrKeyNotFound
if i never put a key, i get it , it will return ErrKeyNotFound.

如果我曾经保存过一个key，然后删除了，接着又尝试去get它。返回错误ErrNotFoundKey。
if i had put a key, then delet it , now i get it , it will return ErrNotFoundKey.

请教一下，这两个错误的意思的区别是什么呢？难道我使用过程中还得区分它们吗？
could your tell me different between ErrKeyNotFound and ErrNotFoundKey?

thanks ..

LRem描述为：
从指定bucket里面的指定的key的列表里移除前 count 次出现的值为 value 的元素。 这个 count 参数通过下面几种方式影响这个操作：

但并没有参数指定value，查看源码可发现该实现错误地类似于了LTrim。

opt := nutsdb.DefaultOptions
	opt.Dir = "./nutsdb"
	opt.SegmentSize = 8388608
	opt.EntryIdxMode = nutsdb.HintKeyAndRAMIdxMode
	opt.SyncEnable = true
	opt.RWMode = nutsdb.FileIO
	opt.NodeNum = 1

Can`t start connect to db if set HintKeyAndRAMIdxMode.
First start is normal , but second start showing error.

从代码上看，无论是HintKeyAndRAMIdxMode还是HintKeyValAndRAMIdxMode， list等类型的数据都会在磁盘写一份，func (tx *Tx) Commit() error {}逻辑里面并不区分具体的数据结构，都一股脑往磁盘里面写进去，同时tx.buildIdxes(writesLen)又会把数据全部写在内存的list链表里面一次，是这样的吗？如果我理解的没错，觉得这是个大问题

Hi @xujiajun,

By looking at the code is not clear that we're getting values of our own pending writes when reading within a transaction. I might be probably missing something.

Thanks.

Example:
https://github.com/xujiajun/nutsdb/blob/5705bf9fa79506cbc5b32bf9cf5a0a4940e84716/tx_bptree.go#L25

RT

支持哈希表

hset/hget还是比较常用，作者打算何时支持一下，如何可以，我可以参与进来

希望支持分布式 动态迁移

nutsdb是个高性能的kv存储，但是现在这种单机模式在后期数据量非常大时就会出现瓶颈，作者是否考虑支持分布式，支持动态扩容和迁移，类似像codis的slot的设计将数据分散到不同node上。

Cards can be added to your board to track the progress of issues and pull requests. You can also add note cards, like this one!

Is your feature request related to a problem? Please describe.
Yes. I currently need support for iterating through all existing buckets and then deleting each one including all their objects. I have not been able to find a satisfactory workaround.

Describe the solution you'd like
Adding tx.ListBuckets() and tx.DeleteBucket(bucket string) functions.

Describe alternatives you've considered
I have considered maintaining a bucket list and object count myself as a workaround but it's not ideal since for every tx.Put() I need to first check if the entity already exists or not in order to increase the bucket counter. This makes the approach very inefficient.

除HintKeyValAndRAMIdxMode 外, List open报错 db.buildIndexes error: err EntryIdxMode option set

func init() {
	var err error
	opt := nutsdb.Options{
		EntryIdxMode:         nutsdb.HintKeyValAndRAMIdxMode, // key在内存
		SegmentSize:          8 * 1024 * 1024,
		NodeNum:              1,
		RWMode:               nutsdb.FileIO,
		SyncEnable:           true,
		StartFileLoadingMode: nutsdb.MMap,
		Dir:                  "C:\\db",
	}
	db, err = nutsdb.Open(opt)
	if err != nil {
		log.Fatal(err)
	}
}

if err := db.Update(
		func(tx *nutsdb.Tx) error {
			bucket := "bucketForList"
			key := []byte("myList")
			val := []byte("val1")
			return tx.RPush(bucket, key, val)
		}); err != nil {
		log.Fatal(err)
	}