Giter Club home page Giter Club logo

claudb's Introduction

ClauDB

ClauDB is a REDIS implementation in Java. At the moment is in development and only implements a small subset of commands and features. The objetive is implement a full functional one-to-one replacement for REDIS (2.8 branch).

You will probably wonder why I do this, the answer is I do it Just For Fun.

Why ClauDB?

Initially I called this project TinyDB, but there's another project with the same name, so, I've decided to chage to ClaudDB.

Clau is 🔑 in Valencià, a language spoken in eastern Spain, and ClauDB is a key-value database.

Implemented commands

Server
  • FLUSHDB
  • INFO
  • TIME
  • SYNC
  • SLAVEOF
  • ROLE
Connection
  • ECHO
  • PING
  • QUIT
  • SELECT
Key
  • DEL
  • EXISTS
  • KEYS
  • RENAME
  • TYPE
  • EXPIRE
  • PERSIST
  • TTL
  • PTTL
String
  • APPEND
  • DECRBY
  • DECR
  • GET
  • GETSET
  • INCRBY
  • INCR
  • MGET
  • MSET
  • MSETNX
  • SET (with NX, PX, NX and XX options)
  • SETEX
  • SETNX
  • STRLEN
Hash
  • HDEL
  • HEXISTS
  • HGETALL
  • HGET
  • HKEYS
  • HLEN
  • HMGET
  • HMSET
  • HSET
  • HVALS
List
  • LPOP
  • LPUSH
  • LINDEX
  • LLEN
  • LRANGE
  • LSET
  • RPOP
  • RPUSH
Set
  • SADD
  • SCARD
  • SDIFF
  • SINTER
  • SISMEMBER
  • SMEMBERS
  • SPOP
  • SRANDMEMBER
  • SREM
  • SUNION
Sorted Set
  • ZADD
  • ZCARD
  • ZRANGEBYSCORE
  • ZRANGE
  • ZREM
  • ZREVRANGE
  • ZINCRBY
Pub/Sub
  • SUBSCRIBE
  • UNSUBSCRIBE
  • PSUBSCRIBE
  • PUNSUBSCRIBE
  • PUBLISH
Transactions
  • MULTI
  • EXEC
  • DISCARD
Scripting
  • EVAL
  • EVALSHA
  • SCRIPT LOAD
  • SCRIPT EXISTS
  • SCRIPT FLUSH

Design

ClauDB is implemented using Java8. Is single thread, like REDIS. It uses asynchronous IO (netty) and reactive programing paradigm (rxjava).

Requests come from IO threads and enqueues to rxjava single thread scheduler. Then IO thread is free to process another request. When request is done, the response is sended to client asyncronously. Then, every request is managed one by one, in a single thread, so there's no concurrency issues to care about.

Features

Now only implements a subset of REDIS commands, but is usable.

ClauDB also supports persistence compatible with REDIS, RDB dumps and AOF journal. It can create compatible RDB files you can load in a REDIS server. Removed since 2.0 version

Now ClauDB support master/slave replication, a master can have multiple slaves, but at the moment slaves can't have slaves.

Also implements partially the Pub/Sub subsystem.

Changes in 2.0 version

The RDB file format support has been dropped and replaced with h2 MVStore.

This allows ClauDB to have file persistence and off heap memory support.

Another important change, now ClauDB has been splited in several subprojects:

  • claudb-lib: implementation of the server and commands.
  • claudb-app: command line application to run standalone server and client.
  • claudb-junit4: implements a Junit4 compatible @Rule to use in junit4 based tests. Example
  • claudb-junit5: implements a Junit5 compatible extension to use in junit5 based tests. Example

Performance

Performance is quite good, not as good as REDIS, but it's good enough for Java.

This is ClauDB

$ redis-benchmark -t set,get -h localhost -p 7081 -n 100000 -q
SET: 82576.38 requests per second, p50=0.303 msec
GET: 93896.71 requests per second, p50=0.287 msec

And this is REDIS

$ redis-benchmark -t set,get -h localhost -p 6379 -n 100000 -q
SET: 148148.14 requests per second, p50=0.167 msec
GET: 147710.48 requests per second, p50=0.175 msec

In my laptop (Intel Core i7-1065G7, with 32G of RAM, running linux)

BUILD

You need to clon the repo:

$ git clone https://github.com/tonivade/claudb.git

ClauDB uses Gradle as building tool, but you don't need Gradle installed, just type:

$ ./gradlew build

This scripts automatically download Gradle and then runs the tasks.

Or if you have Gradle installed, just type

$ gradle build

Create all-in-one jar

$ ./gradlew fatJar

DOCKER

You can create your own docker images for ClauDB using the provided Dockerfile

$ docker build -t claudb .

And then run the image

$ docker run -p 7081:7081 claudb

USAGE

You can start a new server listening in default port 7081.

$ wget https://repo1.maven.org/maven2/com/github/tonivade/claudb-app/2.0.1/claudb-app-2.0.1-all.jar
$ java -jar claudb-2.0.1-all.jar

or using jgo utility

$ jgo com.github.tonivade:claudb-app:2.0.1:com.github.tonivade.claudb.Server

Parameters:

Option                  Description
------                  -----------
-N                      enable keyspace notifications (experimental)
-O                      enable off heap memory (experimental)
-P [String: file name]  enable persistence (experimental) (default: ./claudb.data)
-V                      verbose mode
-h <String: host>       define listen host (default: localhost)
--help                  print help
-p <Integer: port>      define listen port (default: 7081)

Also you can use inside your project using Maven

<dependency>
    <groupId>com.github.tonivade</groupId>
    <artifactId>claudb</artifactId>
    <version>2.0.1</version>
</dependency>

Or gradle

compile 'com.github.tonivade:claudb:2.0.1'

Or embed in your source code

    RespServer server = ClauDB.builder().host("localhost").port(7081).build();
    server.start(); 

Native Image

Now is possible to generate a native image thanks to graalvm. You can generate one with this command:

$ ./gradlew clean nativeImage

Some features are not available like lua runtime and offheap memory.

TODO

  • Ziplist and Maplist encoding not implemented yet.
  • Master/Slave replication improvements. Slave with Slaves
  • Partitioning?
  • Clustering?
  • Geo Commands

Continuous Integration

Java CI with Gradle Codacy Badge Codacy Badge Maven Central Join the chat at https://gitter.im/tonivade/claudb

Stargazers over time

Stargazers over time

LICENSE

ClauDB is released under MIT License

claudb's People

Contributors

dependabot-preview[bot] avatar dependabot[bot] avatar tonivade avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

claudb's Issues

latest release 1.8.0 not working with java 8, only with java 11

It seems there are some incompatibilities in some java.nio.ByteBuffer methods in java 8 and java 11.

When you use cross compilation with java 11 to java 8, like in this project, the byte code generated is incorrect and not compatible with java 8.

See jetty/jetty.project#3244

java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
	at com.github.tonivade.resp.protocol.SafeString.append(SafeString.java:106)
	at com.github.tonivade.claudb.command.string.AppendCommand.lambda$execute$0(AppendCommand.java:32)
	at com.github.tonivade.claudb.data.Database.merge(Database.java:78)
	at com.github.tonivade.claudb.command.string.AppendCommand.execute(AppendCommand.java:30)
	at com.github.tonivade.claudb.command.DBCommandWrapper.executeDBCommand(DBCommandWrapper.java:94)
	at com.github.tonivade.claudb.command.DBCommandWrapper.execute(DBCommandWrapper.java:82)
	at com.github.tonivade.claudb.command.CommandRule.execute(CommandRule.java:133)
	at com.github.tonivade.claudb.command.string.AppendCommandTest.testExecute(AppendCommandTest.java:24)

Remove logback, replace with slf4j-api

When using claudb for tests I get an SLF4J error complaining about multiple bindings on the classpath:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/me/.m2/repository/org/slf4j/slf4j-nop/1.7.30/slf4j-nop-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/Users/me/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

mvn dependency:tree tells me that claudb depends on resp-server which depends on logback.

[INFO] \- com.github.tonivade:claudb:jar:1.7.1:compile
[INFO]    +- com.github.tonivade:resp-server:jar:0.16.0:compile
[INFO]    |  +- io.netty:netty-all:jar:4.1.47.Final:compile
[INFO]    |  +- io.reactivex.rxjava3:rxjava:jar:3.0.0:compile
[INFO]    |  +- com.github.tonivade:purefun-core:jar:1.8:compile
[INFO]    |  |  \- com.github.tonivade:purefun-annotation:jar:1.8:compile
[INFO]    |  \- ch.qos.logback:logback-classic:jar:1.2.3:runtime
[INFO]    |     \- ch.qos.logback:logback-core:jar:1.2.3:runtime

Can logback be replaced with slf4j-api?

SADD multiple members function missing

I am trying to add multiple members to a set but from your code, apparently, you do not handle it.

SADD mykey member1 member2 member3

or with Jedis:

jedis.sadd("mykey", "member1", "member2", "member3")

both return cardinality 1 (expected 3)

Issue with list commands in 2.1

Hi

After attempting to bump our claudb dependency from 2.0.1 to 2.1 one of our unit test started failing on what looks like a bug somewhere in lpush/rpush/lpop/rpop commands.

Simple reproducer (can be copied directly into ClauDBServerTest):

@Test
public void testPushPull() {
  execute(jedis -> {
    long push = jedis.rpush("key", "val1", "val2");
    String pop1 = jedis.rpop("key");
    String pop2 = jedis.rpop("key");
    assertThat(push, is(2l));
    assertThat(pop1, equalTo("val2"));
    assertThat(pop2, equalTo("val1"));
  });
}

It fails on the last assert where pop2 is null instead of expected val1.

I checked the changes and the lpush and rpush commands were both changed, so I would expect a bug was introduced here.

importRDB Error

 by: java.io.IOException: not supported: 172
	at com.github.tonivade.claudb.persistence.RDBInputStream.parse(RDBInputStream.java:116)
	at com.github.tonivade.claudb.DBServerState.importRDB(DBServerState.java:107)
	at com.github.tonivade.claudb.ClauDB.lambda$importRDB$2(ClauDB.java:136)
	at io.reactivex.internal.operators.observable.ObservableCreate.subscribeActual(ObservableCreate.java:40)
	at io.reactivex.Observable.subscribe(Observable.java:12284)
	at io.reactivex.internal.operators.observable.ObservableObserveOn.subscribeActual(ObservableObserveOn.java:45)
	at io.reactivex.Observable.subscribe(Observable.java:12284)
	at io.reactivex.internal.operators.observable.ObservableBlockingSubscribe.subscribe(ObservableBlockingSubscribe.java:81)
	... 20 common frames omitted

Bigset implementation

Implements the data structure bitset and it's related commands setbit, getbit, etc...

Migration to vavr.io collections

Collection data types are saved inside TinyDB using this backends:

  • List: java.util.LinkedList
  • Set: java.util.LInkedHashSet
  • Zset: custom SortedSet implementation
  • Hash: java.util.LinkedHashMap

The objective is replace each collection with it's counterparts in vavr library

  • List: io.vavr.collection.LinkedList
  • Set: io.vavr.collection.LinkedHashSet
  • Map: io.vavr.collection.LinkedHashMap

The exception is SortedSet that is a custom implementation of java.util.NavigableSet

Unexpected end of stream in rpop with jedis client

Jedis version: 2.9.0
Resp-server 0.10.0
Netty: 4.1.24
claudb: 1.2.0

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
	at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199)
	at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
	at redis.clients.jedis.Protocol.process(Protocol.java:151)
	at redis.clients.jedis.Protocol.read(Protocol.java:215)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
	at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:259)
	at redis.clients.jedis.BinaryJedis.rpop(BinaryJedis.java:1260)
	at redis.clients.jedis.BinaryShardedJedis.rpop(BinaryShardedJedis.java:303)

Are you using claudb? Please tell me how!!

If you are using claudb, It would be awesome if you tell me how you are using it.

  • Environment: testing or production
  • Purpose: cache, redis integration tests, realtime analysis...
  • Which features are you using: lua scripting, persistence, off heap cache, keyspace events...
  • Which features do you miss: geospatial searches, some command or data type ...

If you want to help me, please write a comment to this issue with this information.

Thanks a lot!

Support sentinel commands

Have the ability to support sentinel commands and implements (maybe partially) its functionality.

Scripting support

Add support to lua scripting, compatible with redis, or add new scripting capabilities like javascript, jruby and jithon.

Support for HSCAN command

https://redis.io/commands/hscan/

When using Redisson with ClauDB:
org.redisson.client.RedisException: ERR unknown command 'HSCAN'. channel: [id: 0x1b56c5bb, L:/127.0.0.1:54510 - R:127.0.0.1/127.0.0.1:6379] command: (HSCAN), promise: java.util.concurrent.CompletableFuture@3a653de6[Not completed, 1 dependents], params: [JUnit_SESSION_MAP, 0, COUNT, 10]

Implement set command options

Now, the implementation of set command is too simple and doesn't support additional options added in REDIS

    EX seconds -- Set the specified expire time, in seconds.
    PX milliseconds -- Set the specified expire time, in milliseconds.
    NX -- Only set the key if it does not already exist.
    XX -- Only set the key if it already exist.

see documentation

build error

  • I'm not familiar with gradle, and there is the build error message: image
  • I'm using ./gradlew buld in the project root path
  • using the Intellij IDEA gradle build tools also have the same error.

Issues with events

keyspace events bad info

1) "psubscribe"
2) "__key*__:*"
3) (integer) 1
1) "pmessage"
2) "__key*__:*"
3) "__keyevent__@0__:a"
4) "set"
1) "pmessage"
2) "__key*__:*"
3) "__keyspace__@0__:set"
4) "set"

expected

1) "psubscribe"
2) "__key*__:*"
3) (integer) 1
1) "pmessage"
2) "__key*__:*"
3) "__keyspace@0__:a"
4) "set"
1) "pmessage"
2) "__key*__:*"
3) "__keyevent@0__:set"
4) "a"

Remove keys when expire

When a key has expired, it's still there

127.0.0.1:7081> keys *
1) "b"
2) "a"

But keys a and b has been expired

127.0.0.1:7081> get a
(nil)
127.0.0.1:7081> get b
(nil)
127.0.0.1:7081> keys *

TinyDB needs a process to remove expired keys before accessing keys

Deploy fatJar in maven

If you want to run quickly in console the server, you need to checkout the source code, generate the fat jar manually.

It would be nice to have this artifact deployed in maven

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.