stephenmcd / curiodb Goto Github PK

View Code? Open in Web Editor NEW

510.0 510.0 47.0 1.19 MB

Distributed NoSQL Database

Home Page: http://curiodb.jupo.org

License: BSD 2-Clause "Simplified" License

Scala 98.35% Python 1.65%

curiodb's People

Contributors

Stargazers

Watchers

curiodb's Issues

Scala 2.11

Upgrading to Scala 2.11 gives 10-20% better performance. Here are my results:

Scala 2.10, second run:

PING_INLINE: 74626.87 requests per second
PING_BULK: 52356.02 requests per second
SET: 51546.39 requests per second
GET: 64102.56 requests per second
INCR: 71428.57 requests per second
LPUSH: 64516.13 requests per second
LPOP: 66666.66 requests per second
SADD: 73529.41 requests per second
SPOP: 57471.27 requests per second
LPUSH (needed to benchmark LRANGE): 73529.41 requests per second
LRANGE_100 (first 100 elements): 36630.04 requests per second
LRANGE_300 (first 300 elements): 15948.96 requests per second
LRANGE_500 (first 450 elements): 10395.01 requests per second
LRANGE_600 (first 600 elements): 8733.62 requests per second
MSET (10 keys): 30959.75 requests per second

Scala 2.11.7, second run:

PING_INLINE: 59523.81 requests per second
PING_BULK: 76923.08 requests per second
SET: 81300.81 requests per second
GET: 84745.77 requests per second
INCR: 72992.70 requests per second
LPUSH: 47393.37 requests per second
LPOP: 68493.15 requests per second
SADD: 84745.77 requests per second
SPOP: 84033.61 requests per second
LPUSH (needed to benchmark LRANGE): 77519.38 requests per second
LRANGE_100 (first 100 elements): 39215.69 requests per second
LRANGE_300 (first 300 elements): 16611.29 requests per second
LRANGE_500 (first 450 elements): 11947.43 requests per second
LRANGE_600 (first 600 elements): 9182.74 requests per second
MSET (10 keys): 35842.29 requests per second

127.0.0.1

Fresh Install of the latest master apears to always try to connect to 127.0.0.1 which will fail if you try and build a cluster between multiple machines (i.e over a private network).

Error Messages:

Jan 21 06:53:41 db01 bash[24020]: [WARN] [01/21/2017 06:53:41.374] [curiodb-akka.remote.default-remote-dispatcher-43] [akka.tcp://[email protected]:9001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fcuriodb%40127.0.0.1%3A9001-74] Association with remote system [akka.tcp://[email protected]:9001] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:9001]] Caused by: [Connection refused: /127.0.0.1:9001]
Jan 21 06:53:49 db01 bash[24020]: [WARN] [01/21/2017 06:53:49.370] [curiodb-akka.remote.default-remote-dispatcher-41] [akka.tcp://[email protected]:9001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fcuriodb%4010.128.21.3%3A9001-75] Association with remote system [akka.tcp://[email protected]:9001] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:9001]] Caused by: [No route to host]
Jan 21 06:53:49 db01 bash[24020]: [WARN] [01/21/2017 06:53:49.378] [curiodb-akka.remote.default-remote-dispatcher-44] [akka.tcp://[email protected]:9001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fcuriodb%4010.128.21.2%3A9001-76] Association with remote system [akka.tcp://[email protected]:9001] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:9001]] Caused by: [No route to host]
Jan 21 06:53:51 db01 bash[24020]: [WARN] [01/21/2017 06:53:51.401] [curiodb-akka.remote.default-remote-dispatcher-44] [akka.tcp://[email protected]:9001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fcuriodb%40127.0.0.1%3A9001-77] Association with remote system [akka.tcp://[email protected]:9001] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:9001]] Caused by: [Connection refused: /127.0.0.1:9001]

Configuration:

curiodb {

  // Addresses listening for clients.
  listen = [
    "tcp://10.128.21.1:6379"    // TCP server using Redis protocol.
  ]

  // Duration settings (either time value, or "off").
  persist-after = 1 second    // Like "save" in Redis.
  sleep-after   = 10 seconds  // Virtual memory threshold.
  expire-after  = off         // Automatic key expiry.

  transactions {
    timeout   = 3 seconds     // Max time a transaction may take to run.
    isolation = repeatable    // "repeatable", "committed", or "uncommitted".
    on-error  = rollback      // "commit" or "rollback".
  }

  commands {
    timeout  = 1 second       // Max time a command may take to run.
    disabled = [SHUTDOWN]     // List of disabled commands.
    debug    = off            // Print debug info for every command run.
  }

  // Cluster nodes.
  nodes = {
    db01: "tcp://10.128.21.1:9001"
    db02: "tcp://10.128.21.2:9001"
    db03: "tcp://10.128.21.3:9001"
  }

  // Current cluster node (from the "nodes" keys above).
  node = db01

}

It's also not even listening on port 6379 which is configured. This is using OpenJDK, I'll test out Suns next.

I will be frank, I totally forgot we already shared curiodb in RedisWeekly. I did not even take the the time to try it with Redsmin! I don't know if you did but I will be glad to give you feedbacks on how well it integrates with Redis GUIs :)

var + mutable state inside actors

Curious about the choice of using mutable variables for some collections inside actors, are there specific reason to not stick with val + mutable or better var + immutable beside maybe the need of some methods of the mutable version of the collection? Great work by the way @stephenmcd !

MSETNX behaves wrong

127.0.0.1:6379> get a
(nil)
127.0.0.1:6379> get b
(nil)
127.0.0.1:6379> get c
(nil)
127.0.0.1:6379> set b b
OK
127.0.0.1:6379> msetnx a a b b c c
(integer) 1
127.0.0.1:6379> get a
"a"
127.0.0.1:6379> get b
"b"
127.0.0.1:6379> get c
"c"

The reason is AggregateMSetNX extends AggregateBroadcast which overrides begin method, it just check Command's first key. So if I set the second key already, it doesn't validate existence right. So is it really necessary to broadcast to all KeyNodes for behaviors like MSETNX?
Also is there a plan to write some test cases?

Why SETNX always return true?

It just foward to SET.

lost update in transactions

#/usr/bin/env python
#coding: utf-8

import redis

from multiprocessing import Process

def get_value(host, port, var):
    r = redis.Redis(host=host, port=port)
    return r.get(var)

def lost_update(host, port, var):
    r = redis.Redis(host=host, port=port)
    pipe = r.pipeline()
    pipe.incr(var, 1)
    pipe.execute()
    #for i in range(0, 10):
    #    pipe.incr(var, 1)
    #    pipe.execute();


if __name__ == "__main__":
    host = '127.0.0.1'
    port = 6370
    var = "test1"
    print get_value(host, port, var)

    p1 = Process(target=lost_update, args=(host, port, var))
    p2 = Process(target=lost_update, args=(host, port, var))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    print get_value(host, port, var)

run the above code, output:

43
44

if you uncomment the code, run it, sometimes it will response timeout.

problem 1:
when one keyNode send _MULTI to itself(node() ! command), if may has receive another _MULTI from another clientNode, so both clientNodes won't wait(inTransaction will fail), they will update a data node concurrently, and update at their own data version.

problem 2:
two clientNode can send _MULTI to every keyNode at the same time, so some keyNodes may receive _MULTI from clientNode1, some keyNodes receive _MULTI from clientNode2, and no clientNodes will finish the transaction.

and i have a question, why keynodes should have multi-version?

publish on maven central

It would be useful to have an artifact on maven central. One of the advantages is to start the application from coursier.

commands in transaction executed not sequently

test 1: mget number1 number3 should return 4 15.
redis 127.0.0.1:6370> get number3
"15"
redis 127.0.0.1:6370> multi
OK
redis 127.0.0.1:6370> mget number1 number3
QUEUED
redis 127.0.0.1:6370> set number3 20
QUEUED
redis 127.0.0.1:6370> exec

1. "4"
2. "20"
OK

test 2: dbsize in transaction should return 11, not 12.
redis 127.0.0.1:6370> dbsize
(integer) 11
redis 127.0.0.1:6370> multi
OK
redis 127.0.0.1:6370> dbsize
QUEUED
redis 127.0.0.1:6370> set number2 1
QUEUED
redis 127.0.0.1:6370> exec

(integer) 12
OK

the reason is the commands are sent sequently, but not executed sequently.

What is a .hgignore file. It doesn't work under my fork.

NOT recover its value from disk after "sleep-after = 10000"

I find some unexpected behavior:

127.0.0.1:6379> get abc
"2"
127.0.0.1:6379> get abc
"2"
127.0.0.1:6379> set abc 3
OK
127.0.0.1:6379> get abc
""
127.0.0.1:6379>


[info] [DEBUG] [07/17/2015 16:15:11.281] [curiodb-akka.actor.default-dispatcher-22] [akka://curiodb/user/keys/c7/0-string-abc] received handled message Sleep
[info] [DEBUG] [07/17/2015 16:15:11.281] [curiodb-akka.actor.default-dispatcher-22] [akka://curiodb/user/keys/c7/0-string-abc] stopped



[info] [DEBUG] [07/17/2015 16:15:16.818] [curiodb-akka.actor.default-dispatcher-16] [akka://curiodb/user/server/$a] received handled message Received(ByteString(42, 50, 13, 10, 36, 51, 13, 10, 103, 101, 116, 13, 10, 36, 51, 13, 10, 97, 98, 99, 13, 10))
[info] [DEBUG] [07/17/2015 16:15:16.819] [curiodb-akka.actor.default-dispatcher-16] [akka://curiodb/user/keys/c7] received handled message Routable(Command(Vector(get, abc),0,Some(Actor[akka://curiodb/user/server/$a#-1963201235])))
[info] [DEBUG] [07/17/2015 16:15:16.820] [curiodb-akka.actor.default-dispatcher-16] [akka://curiodb/user/keys/c7] now supervising Actor[akka://curiodb/user/keys/c7/0-string-abc#2063326557]
[info] [DEBUG] [07/17/2015 16:15:16.820] [curiodb-akka.actor.default-dispatcher-18] [akka://curiodb/user/keys/c7/0-string-abc] started (curiodb.StringNode@510b0c72)
[info] [DEBUG] [07/17/2015 16:15:16.822] [curiodb-akka.actor.default-dispatcher-18] [akka://curiodb/user/keys/c7/0-string-abc] received handled message Command(Vector(get, abc),0,Some(Actor[akka://curiodb/user/server/$a#-1963201235]))
[info] [DEBUG] [07/17/2015 16:15:16.822] [curiodb-akka.actor.default-dispatcher-16] [akka://curiodb/user/server/$a] received handled message Response(abc,)
[info] [DEBUG] [07/17/2015 16:15:17.831] [curiodb-akka.actor.default-dispatcher-3] [akka://curiodb/user/keys/c7] received handled message Persist
[info] [DEBUG] [07/17/2015 16:15:17.833] [curiodb-akka.actor.default-dispatcher-19] [akka://curiodb/user/keys/c7] received handled message SaveSnapshotSuccess(SnapshotMetadata(c7,0,1437120917831))
[info] [DEBUG] [07/17/2015 16:15:17.835] [curiodb-akka.actor.default-dispatcher-3] [akka://curiodb/user/keys/c7] received handled message DeleteSnapshotsSuccess(SnapshotSelectionCriteria(0,1437120917830))
[info] [INFO] [07/17/2015 16:15:17.835] [curiodb-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:9001/user/keys/c7] ============Actor[akka://curiodb/system/akka.persistence.snapshot-store.local#-1687772525]============

stephenmcd / curiodb Goto Github PK

curiodb's People

Contributors

Stargazers

Watchers

Forkers

curiodb's Issues

Scala 2.11

127.0.0.1

Awesome

var + mutable state inside actors

MSETNX behaves wrong

Why SETNX always return true?

lost update in transactions

publish on maven central

commands in transaction executed not sequently

What is a .hgignore file. It doesn't work under my fork.

NOT recover its value from disk after "sleep-after = 10000"

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent