bytestorage / flydb Goto Github PK
View Code? Open in Web Editor NEWThe high-performance kv storage engine based on bitcask paper made in golang
License: Apache License 2.0
The high-performance kv storage engine based on bitcask paper made in golang
License: Apache License 2.0
What can I need to contribute if I join your developer team?
Questions about the Raft module:
What are the benefits of dividing nodes into Master and Slave? For Raft, both read and write operations should go through the Raft Leader. Originally, it is not possible to perform read-write separation operations. To increase concurrency, it can only be achieved through Regions. Each Region contains multiple replicas, each on a Raft node. The Leader of each Region can be different Raft nodes, allowing concurrent operations on each Region and improving concurrency. If the state machine is manipulated without going through the Leader, linear consistency cannot be guaranteed. For a key-value storage, it’s not just metadata that maintains linear consistency.Or is the goal not strong consistency?
Does the Master store the operation logs sent by the clients, such as Get, Put, Del, etc.?
What is the workflow for client operations? Is it that the client sends operation commands to the Leader of the Master cluster, and after the Leader replicates the operation logs, it commits and applies the logs to update the state machine? However, the state machine is not on the Master node but on the Slave nodes. How should the Apply operation be performed on the Slave cluster? Is it directly invoking RPC methods on each Slave to update the state machine? How is successful Apply ensured?
Do the nodes in the Slave store the same data? If they are the same, how can linear consistency be maintained in the Slave cluster when it is not a Raft cluster?
Some personal suggestions:
Combine the Raft nodes and state machine nodes (DB), with the state machine on top of the Raft module. The client operation flow can be designed as follows: the client sends operation requests to the Raft Leader, the Leader replicates and commits the logs, and applies the logs to the upper-level state machine, then directly returns the result to the client (for read operations, optimization can be done using Lease Index, skipping the disk write for reading logs and the Leader determining if it is still the Leader's network IO, which performs well in terms of performance). This simplifies the complexity of the system architecture and better ensures linear consistency for read and write operations,and it can also reduce additional network IO(Client between Slave or Master between Slave,just Client between Raft Node).
In this case, the Master does not need to maintain additional metadata such as heartbeat and directory tree for the Slave.
Partition the keys based on hash or range, and implement operations such as automatic splitting when the partition becomes too large or automatic merging when it becomes too small.
I think TiKV's distributed architecture is really worth learning.
Hello,
Please share the address of the bitcask thesis!
Thanks!
We need a new in-memory index, SkipList, to allow users to choose the index that is more efficient for the business during development.
In order to avoid serious problems such as data directory failure or data file corruption resulting in data loss, we need to perfect a data backup function for FlyDB.
we should use github action to check code and unit-test
You can change the data file format to 32kb block based on the wal format of RocksDB to speed up data read and write.
The wal log format already exists in lib and can be optimized based on it.
this method is listen master leader change
In the design process of FlyDB's storage engine, it draws on the design idea of bitcask, and maintains all the keys in memory. If MVCC is implemented on this basis, it will also maintain the corresponding version information in memory, which may lead to a sharp expansion of memory capacity.
If you take on issue#94, you will have to consider this issue and hopefully come up with a good optimization solution.
this method is register self to master, it should use grpc method
You need to define a method NewFlyDbCluster,param can be master address list and slave address list. Interacting between nodes through GRPC and building master and slave clusters
Implementing leader election for internal master clusters through the raft library requires defining several interfaces, such as obtaining leaders, joining the cluster, etc
Need to create several tests to test
Similarly, a lower-level bug occurs when testing the merge file function. The hintIndex file is created and an error occurs when opening a database instance :no such file or directory.
When I looked at the merge file's method for loading the hint index, the loadIndexFromHintFile function, it returned err instead of nil when the file didn't exist and needed to be created.
This is a very basic question that strikes me as amazing.
Hopefully, in subsequent iterations, you'll notice the tiny details.
If a data operation, such as data insertion or deletion, fails, you need to perform the operation again according to the operation log.
FlyDB needs to add a http service.
请教个楼主个问题
合并完成后,内存中的索引哈希表必须要更新,因为这些数据所在的文件已经改变了。请问这个具体的更新过程是怎么样的呢?
flydb 目前的功能并不是十分的完善,因此我希望有人可以参与进来,不管你是 DB 的爱好者,或者是专家,甚至刚入门的小白,其实都能够对 flydb 进行相关的贡献。
目前项目比较缺人,如果有意愿的话,欢迎加我 vx:qishen_on 或 qq:1050026498 进行交流!
We need a new in-memory index, HashMap, to allow users to choose the index that is more efficient for the business during development.
this method is send heartbeat to master, for range time.tick, it should use grpc method
In the current FlyDB startup process, all the data is loaded and the index is built, which can take a long time if the database contains a lot of data.
So we need a new strategy to speed up the start-up.
this method is update slave self message
When I was functional testing flydb's merge code, I found a file read/write bug. When I was building and writing data to the database instance and finally performing the merge operation, I was prompted with an error message indicating insufficient permissions to create.
After some debugging, it is found that the problem is in the getMergePath function of the merge file. The lack of the basic directory of the database results in an error. Hope to fix it as soon as possible.
FlyDB currently only implements a very simple ACID transaction model SSI, which only utilizes a global lock to ensure serialization of the transaction.
In order to be able to implement more complex multi-version concurrency control to ensure that there are no incalculable losses caused by the simple model of concurrency control SSI in the production environment.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.