taoensso / carmine Goto Github PK
View Code? Open in Web Editor NEWRedis client + message queue for Clojure
Home Page: https://www.taoensso.com/carmine
License: Eclipse Public License 1.0
Redis client + message queue for Clojure
Home Page: https://www.taoensso.com/carmine
License: Eclipse Public License 1.0
Right now, you must do:
(car/close-listener listener)
It would be nice to do:
(with-open [listener my-listener] ... )
The necessary change is:
(defrecord Listener [connection handler state]
java.io.Closeable
(close [this]
(close-listener this)))
The test suite errors out on my machine so no pull request included.
While profiling a performance-critical Clojure app using Carmine (using the VisualVM sampler), it looks like the call to (satisfies? IConnectionPool opts)
in connections/pooled-coon
is extremely expensive (taking as much CPU time as all the rest of the work done in Carmine+redis). This is because the call to satisfies?
ultimately results in a Class.getInterfaces
reflection call.
Furthermore, from looking at the code I can't see why this check is even necessary; a comment mentions "; Pass through pre-made pools", but I can't see any case or the code or documentation where a caller would be passing an IConnectionPool object rather than a simple map of options.
If the IConnectionPool case is no longer necessary then presumably we can just remove the satisfies?
check all together; if it still is then maybe we can avoid reflection by passing in the information about its type in an argument rather than using reflection?
Thanks!
Tundra is not compatible with clojure 1.4.0.
I had started to work on cleaning up the project file and ran into some hitches. I was trying to remove expectations
as a production dependency, but it seems as though all of your libraries have it as well.
Consider using :pedantic? :abort
to detect version range problems and clean up accordingly. I stopped because I didn't know how you would like to handle 1.4 compatibility.
It's been released for a while now -- would clean up user projects that need to exclude 0.1.1 from carmine.
Hey,
Little bug in my upgrade from 2.3.1. When I try to run the following code with an empty spec map (ie, local redis), all's well:
(car/with-new-pubsub-listener (:spec conf)
{"results" (fn [[message-type topic-name msg]]
(broadcast! msg))}
(car/subscribe "results"))
However, when conf is {:uri "redistogo-url"}
, this exception shows up:
java.lang.NullPointerException
at clojure.core$deref_future.invoke(core.clj:2108)
at clojure.core$deref.invoke(core.clj:2129)
at taoensso.carmine.protocol$get_parsed_replies.invoke(protocol.clj:203)
at taoensso.carmine.connections$make_new_connection$fn__25792.invoke(connections.clj:55)
at taoensso.carmine.connections$make_new_connection.invoke(connections.clj:56)
at paddleguru.websocket.PubSub.start_BANG_(form-init1409143132977261857.clj:5)
at paddleguru.lifecycle$start_system$fn__10186.invoke(lifecycle.clj:15)
at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:63)
any ideas?
Program can not start.
OS:
CentOS release 5.4 (Final)
Linux web1 2.6.18-243.el5 #1 SMP Mon Feb 7 18:47:27 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:306)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:213)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at taoensso.nippy$loading__4784__auto__.invoke(nippy.clj:1)
at taoensso.nippy__init.load(Unknown Source)
at taoensso.nippy__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at taoensso.carmine.protocol$loading__4784__auto__.invoke(protocol.clj:1)
at taoensso.carmine.protocol__init.load(Unknown Source)
at taoensso.carmine.protocol__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5302)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at taoensso.carmine$loading__4784__auto__.invoke(carmine.clj:1)
at taoensso.carmine__init.load(Unknown Source)
at taoensso.carmine__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at test_test.core$loading__4784__auto__.invoke(core.clj:1)
at test_test.core__init.load(Unknown Source)
at test_test.core__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:415)
at test_test.core.<clinit>(Unknown Source)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.0.53-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /tmp/snappy-1.0.53-libsnappyjava.so)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1807)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1703)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at org.xerial.snappy.SnappyNativeLoader.load(SnappyNativeLoader.java:39)
... 82 more
Exception in thread "main" org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:223)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at taoensso.nippy$loading__4784__auto__.invoke(nippy.clj:1)
at taoensso.nippy__init.load(Unknown Source)
at taoensso.nippy__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at taoensso.carmine.protocol$loading__4784__auto__.invoke(protocol.clj:1)
at taoensso.carmine.protocol__init.load(Unknown Source)
at taoensso.carmine.protocol__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5302)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at taoensso.carmine$loading__4784__auto__.invoke(carmine.clj:1)
at taoensso.carmine__init.load(Unknown Source)
at taoensso.carmine__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at test_test.core$loading__4784__auto__.invoke(core.clj:1)
at test_test.core__init.load(Unknown Source)
at test_test.core__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at clojure.lang.RT.loadClassForName(RT.java:2056)
at clojure.lang.RT.load(RT.java:419)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:415)
at test_test.core.<clinit>(Unknown Source)
Could not find the main class: test_test.core. Program will exit.
When using get
on a key that was set using the setbit
function, get
returns a java.lang.String representation of the string returned by redis, instead of returning a byte array.
From the carmine README:
(wcar* (car/set "bin-key" (byte-array 50))
(car/get "bin-key"))
=> ["OK" #<byte[] [B@7c3ab3b4>]
This example shows that if set
was used with a byte-array then we get back a byte-array when using get
. If we have a key which was created with the setbit
redis function, we get back a java.lang.String
with wrong representation of the bytes (I guess something to do with signed vs unsigned bytes) and is impossible to get the underlying bits in order to see which bits are on or off.
There should be a way to coerce car/get
to byte-array
Example of the problem:
in redis-cli: setbit somekey 8 1
The facts:
car/get
will return a string with 2 16-bit chars or 4 bytesTo work around this problem, I'm using lua script instead of car/get
to get the right string and convert it to some kind of byte array.
Is there a way to work around this problem with carmine?
Test case:
=> (def server1-conn {:pool {}, :spec {:uri "redis://test"}})
#'/server1-conn
=> (def foo (car/with-new-pubsub-listener server1-conn {}))
#'/foo
=> foo
#taoensso.carmine.Listener{:connection #taoensso.carmine.connections.Connection{:socket #<Socket Socket[addr=/127.0.0.1,port=6379,localport=54537]>, :spec {:listener? true, :pubsub-listener? true, :spec {:uri "redis://test"}, :pool {}, :host "127.0.0.1", :port 6379}, :in-stream #<DataInputStream java.io.DataInputStream@61ba71dc>, :out-stream #<BufferedOutputStream java.io.BufferedOutputStream@3913e70e>}, :handler #<Atom@4b4bb49e: #<core$fn__1366$fn__1368 websocket_handler.core$fn__1366$fn__1368@6f2c508f>>, :state #<Atom@465d154e: {}>}
First thank you for carmin, I really like the library and enjoy using it
The following is a bit confusing:
(wcar (car/hset "atom:q" :foo 2))
(wcar (car/hgetall* "atom:q" )); -> {:foo "2"}
(wcar (car/hset "atom:q" :foo {:bar 5}))
(wcar (car/hgetall* "atom:q" )); -> {:foo {:bar 5}}
Now iv read the code and I understand that in the first case no nippy serialization takes place and redis converts the int into a string, in the second case nippy is serializing the entire value so we back a number as we expect,
I don't see an easy fix here but this is an inconsistency which isn't clear to newcomers.
Best regards
Ronen
I'm using carmine.message-queue and I seem to get this error when the handler function returns a non-nil value:
2013-Aug-08 20:17:55 +0545 thinkpad FATAL [taoensso.carmine.message-queue] - Worker error!! java.lang.IllegalArgumentException: No value supplied for key: 1
java.lang.IllegalArgumentException: No value supplied for key: 1
PersistentHashMap.java:77 clojure.lang.PersistentHashMap.create
message_queue.clj:217 taoensso.carmine.message-queue.Worker/fn[fn]
message_queue.clj:208 taoensso.carmine.message-queue.Worker/fn
core.clj:1836 clojure.core/binding-conveyor-fn[fn]
AFn.java:18 clojure.lang.AFn.call
FutureTask.java:334 java.util.concurrent.FutureTask$Sync.innerRun
FutureTask.java:166 java.util.concurrent.FutureTask.run
ThreadPoolExecutor.java:1145 java.util.concurrent.ThreadPoolExecutor.runWorker
ThreadPoolExecutor.java:615 java.util.concurrent.ThreadPoolExecutor$Worker.run
Thread.java:722 java.lang.Thread.run
But seems to work fine when the handler is made to return nil
. Is this expected behaviour?
Last update was over 3 months ago (as of 22 June 2013): https://github.com/antirez/redis-doc/blob/master/commands.json
See taoensso.carmine.tundra.faraday
as an example implementation. A disk-based DataStore could function similarly, but use unique files for each key.
Might be best to hash the key names and use files under a number of hash prefix subdirectories as Git does.
Hi!
I use https://github.com/clojure/data.json 2.0 and it has incompatible changes with older versions.
cut:
Release 0.2.0 on 2012-Oct-12
Breaking API changes: renamed core functions
New :key-fn and :value-fn permit flexible transformation of values when reading & writing JSON
Support for reading large integers as BigInt
Optional support for reading decimals as BigDecimal
Performance improvements
Do you have plans to update json lib?
I consistently get this error when performing blocking operations unless I set the connection timeout to zero.
For example if I use:
(car/make-conn-spec :host "localhost" :timeout 5 :port 6379 :db 8)
And proceed to use:
(wcar (car/brpoplpush "testing" "processing" 2))
I immediately get a timeout:
SocketTimeoutException Read timed out java.net.SocketInputStream.socketRead0 (SocketInputStream.java:-2)
However, if I set the initial connection timeout explicitly to zero
(car/make-conn-spec :host "localhost" :timeout 0 :port 6379 :db 8)
And use:
(wcar (car/brpoplpush "testing" "processing" 2))
This is fine.
According to the Redis spec, BRPOPLPUSH takes a timeout argument including zero which is not supposed to time out. Is this correct that the connection timeout is affecting the timeout argument that BRPOPLPUSH takes?
I've tried to use following code without success:
(wcar* (car/evalsha checksum [] ["Java,Clojure"]))
Lua's table ARGV is empty in my script.
Could you please add example of usage evalsha? Unit test would be also great.
Redis version: 2.8.4
UPD:
Actually ARGV table is not empty. It contains one element with empty value.
Test code in Lua script:
for key,value in pairs(ARGV) do print(key, "=", value) end
Result output in Redis logs:
1 =
Iv saw your todo mentioning success/fail reply from the handle-fn,
Currently if a handler fails it will cause a retry later on, this isn't always desirable (and indeed the user can catch exceptions in order to prevent this but the semantics are wrong)
Iv created a branch with explicit return status for retry/success (an error is basically another form of retry)
https://github.com/narkisr/carmine/commit/b478be489349396ed4898748d110d8d652609bcb
Let me know what you think,
Ronen
There's some mind-bogglingly cool avenues to explore here - particularly once Redis gets reliable Pub/Sub (expected "soonish" last time I checked).
Possibly major applications for large, distributed Clojure software (think distributed high performance message channels, etc.).
Want to wait first on a finalised Cluster/Sentinel design since the design there may have implications here.
Keyspace notifications (2.8+) also give some very interesting possibilities in this area, esp. when coupled with Tundra.
Using [carmine "0.8.2-SNAPSHOT"]
(ns cartest.core
(:require
[carmine (core :as r)]
))
(def pool (r/make-conn-pool :test-while-idle? true))
(def spec-server1 (r/make-conn-spec :host "127.0.0.1"
:db 0
:port 6379))
(defmacro redis
"Basically like (partial with-conn pool spec-server1)."
[& body] `(r/with-conn pool spec-server1 ~@body))
(defn -main
[& args]
(println
(redis
(r/hgetall "hit:1"))
))
;; (url /admin/ added 20120606 user honza datetime 2012-06-06 00:42:30)
;;
;; Expected:
;; {:url /admin/ :added 20120606 :user honza :datetime <date time inst>}
;;
;; redis-cli:
;; hgetall hit:1
;; 1) "url"
;; 2) "/admin/"
;; 3) "added"
;; 4) "20120606"
;; 5) etc...
Disclaimer: I'm a Clojure n00b and there might be something obviously wrong with my code. I apologize in advance.
Redis doesn't support setting a TTL on a hash field. Simulating this behavior is possible with a combination hash key and key->expire-at-udt sorted set.
The ZSET maintenance and garbage collection could be automated with a simple Lua script.
At a minimum, would need 2 commands:
hpsetex
which sets the hash field, sets an expireAT
type udt, and runs a gc.hpgetex
which operates like hget
but discards expired keys, and potentially runs a gc - maybe with a random probability? Note that the decision to gc or not-gc would need to be made client-side for purposes of replication.The following will cause an error because :foo
will be replaced by the value for :f
in car/lua key-vars.
(car/lua "if redis.call('hget', _:f, _:bar) then redis.call('hget', _:foo, _:b) end" {:f "f" :foo "foo"} {:b "b" :bar "bar"})
This becomes a problem if you wanted to use names for your vars that contain the same words in them i.e key, name-key. Instead of a straight string replace on line 176 of carmine.clj you could wrap the match
variable in an exact regex (re-pattern (str "^" match "$"))
.
Workaround is to name your key-vars distinctly so that they don't contain the same subset of letters i.e instead of :f
and :foo
name them :foo
and :bar
.
Redis 2.8 is getting keyspace notificactions: http://redis.io/topics/notifications. Would def. be worth investigating integration with Tundra for (semi?)-auto key marking.
Hey Petter, Im struggling here with the composition of wcar,
While it works great on sequences of redis actions it does not work on clojure functions:
(defn some-redis-action [..]
(car/add "..." ))
(wcar (car/add "...") (some-redis-action)); the function will complain on missing wcar
Am I missing somthing or there is a way to compose Clojure functions within a pipeline
Thanks
Ronen
I'm away most of this weekend, but will take a look first thing Monday...
Hello Peter,
I've got a strange error during playing with sandbox project. Maybe you could help me..
I've got an exception:
java.lang.Exception: ERR Error running script (call to f_6507f875a7bdb3f843f59e2369875e092d1e9f84): @user_script:11: user_script:11: JSON parser does not support UTF-16 or UTF-32
protocol.clj:129 taoensso.carmine.protocol/get-basic-reply
protocol.clj:188 taoensso.carmine.protocol/get-parsed-reply
protocol.clj:216 taoensso.carmine.protocol/get-parsed-replies
db.clj:47 ziggurat.db/run-lua-script[fn]
db.clj:47 ziggurat.db/run-lua-script[fn]
db.clj:47 ziggurat.db/run-lua-script
RestFn.java:423 clojure.lang.RestFn.invoke
db.clj:42 ziggurat.db/create-post
RestFn.java:408 clojure.lang.RestFn.invoke
db.clj:54 ziggurat.db/find-posts
routes.clj:22 ziggurat.routes/fn
core.clj:94 compojure.core/make-route[fn]
core.clj:40 compojure.core/if-route[fn]
core.clj:25 compojure.core/if-method[fn]
core.clj:107 compojure.core/routing[fn]
core.clj:2443 clojure.core/some
core.clj:107 compojure.core/routing
RestFn.java:139 clojure.lang.RestFn.applyTo
core.clj:619 clojure.core/apply
core.clj:112 compojure.core/routes[fn]
core.clj:107 compojure.core/routing[fn]
core.clj:2443 clojure.core/some
core.clj:107 compojure.core/routing
RestFn.java:139 clojure.lang.RestFn.applyTo
core.clj:619 clojure.core/apply
core.clj:112 compojure.core/routes[fn]
json.clj:21 ring.middleware.json/wrap-json-body[fn]
middleware.clj:44 noir.util.middleware/wrap-request-map[fn]
keyword_params.clj:27 ring.middleware.keyword-params/wrap-keyword-params[fn]
nested_params.clj:65 ring.middleware.nested-params/wrap-nested-params[fn]
params.clj:55 ring.middleware.params/wrap-params[fn]
middleware.clj:12 hiccup.middleware/wrap-base-url[fn]
format_params.clj:98 ring.middleware.format-params/wrap-format-params[fn]
format_response.clj:113 ring.middleware.format-response/wrap-format-response[fn]
multipart_params.clj:103 ring.middleware.multipart-params/wrap-multipart-params[fn]
validation.clj:140 noir.validation/wrap-noir-validation[fn]
cookies.clj:72 noir.cookies/noir-cookies[fn]
cookies.clj:160 ring.middleware.cookies/wrap-cookies[fn]
session.clj:142 noir.session/noir-flash[fn]
flash.clj:14 ring.middleware.flash/wrap-flash[fn]
session.clj:97 noir.session/noir-session[fn]
session.clj:43 ring.middleware.session/wrap-session[fn]
cookies.clj:160 ring.middleware.cookies/wrap-cookies[fn]
Var.java:415 clojure.lang.Var.invoke
reload.clj:18 ring.middleware.reload/wrap-reload[fn]
stacktrace.clj:17 ring.middleware.stacktrace/wrap-stacktrace-log[fn]
stacktrace.clj:80 ring.middleware.stacktrace/wrap-stacktrace-web[fn]
jetty.clj:18 ring.adapter.jetty/proxy-handler[fn]
(Unknown Source) ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle
HandlerWrapper.java:116 org.eclipse.jetty.server.handler.HandlerWrapper.handle
Server.java:363 org.eclipse.jetty.server.Server.handle
AbstractHttpConnection.java:483 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest
AbstractHttpConnection.java:920 org.eclipse.jetty.server.AbstractHttpConnection.headerComplete
AbstractHttpConnection.java:982 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete
HttpParser.java:635 org.eclipse.jetty.http.HttpParser.parseNext
HttpParser.java:235 org.eclipse.jetty.http.HttpParser.parseAvailable
AsyncHttpConnection.java:82 org.eclipse.jetty.server.AsyncHttpConnection.handle
SelectChannelEndPoint.java:628 org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle
SelectChannelEndPoint.java:52 org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run
QueuedThreadPool.java:608 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
QueuedThreadPool.java:543 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run
Thread.java:662 java.lang.Thread.run
Lua script (just a test):
-- Find all existed tags
local rawTags = redis.call('smembers', 'tags')
local existedTags = {}
for i = 1, #rawTags do
local rawTag = rawTags[i]
table.insert(existedTags, cjson.decode(rawTag))
end
-- Find or create post tags
local postInfo = cjson.decode(ARGV[1])
local tagNames = postInfo.tags
local tags = {}
for i = 1, #tagNames do
local tagName = tagNames[i]
table.insert(tags, tagName)
end
return '{"ok": "' .. #tags .. '"}'
All my sources and Lua scripts have encoding "UTF-8", but script fails on line:
local postInfo = cjson.decode(ARGV[1])
I transfer there (ARGV[1]) the following JSON object: "{"tags": ["Java", "Clojure"]}"
I found that this error message is sent by Redis (http://download.redis.io/redis-stable/deps/lua/src/lua_cjson.c), but still don't understand where is the problem..
Thanks in advance for any help!
conns/make-conn-pool
calls a memoized function that returns a connection-pool depending on if you pass in either :none
or options.
If I want to create a pool and tear it down, I have no way to get a fresh pool -> make-conn-pool
will return the closed pool.
A common use for this would be testing.
Hi Peter,
sorry that I'm obviously too stupid to make a simple Redis connection with Carmine.
I want to connect to a Redis server running in a virtual environment (IP address 172.17.0.3). Redis client (redis-cli) from shell works fine. No connection problems. (Pinging to the virtual IP address from within my Leiningen REPL session works also nicely.)
When using Carmine with the following settings:
(def server1-conn
{:pool :none, :spec {:hostname "172.17.0.3" :port 6379}})
(defmacro wcar* [& body] `(car/wcar server1-conn ~@body))
and invoking
(wcar* (car/get "foo"))
in my Leiningen REPL session I get a Java socket connection error. When I start a local Redis server on IP address 127.0.0.1 the command runs (no changes to server1-conn) but it connects to precisely the new Redis instance at 127.0.0.1 not to the virtual IP address. So it seems my server1-conn settings are disregarded. Syntax error? I checked source code in Obi-Wan Kenobi style (esp. @ http://tinyurl.com/o7zf8tv) but didn't help me.
Best regards,
Hans-J.
I went to upgrade to the latest version 2.2.2 (I was using 2.2.0) and noticed that all my usages of carmine have suddenly broken.
The stacktrace is:
java.lang.Exception: Unknown pool option: :pool
at taoensso.carmine.connections$set_pool_option.invoke(connections.clj:89)
at clojure.core.protocols$fn__6022.invoke(protocols.clj:79)
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13)
at clojure.core$reduce.invoke(core.clj:6177)
at taoensso.carmine.connections$conn_pool$fn__1756.invoke(connections.clj:112)
at clojure.lang.Delay.deref(Delay.java:33)
at clojure.core$deref.invoke(core.clj:2128)
at taoensso.carmine.connections$conn_pool.doInvoke(connections.clj:103)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at taoensso.carmine.connections$pooled_conn.invoke(connections.clj:141)
Note I am still using the macro with-conn as defined in carmine.clj. with-conn passes the :pool and :spec options to wcar. I am more than happy to start using the wcar macro, however, I need to control which pool I pass to wcar.
My carmine usage looks like this:
(defrecord RedisConnection [pool settings])
(defmacro with-redis-conn [redis-connection & body]
`(redis-client/with-conn (:pool ~redis-connection) (:settings ~redis-connection) ~@body))
Waiting on Redis change: http://goo.gl/LPhIO
I found some error logs in our server:
2014-ไธๆ-20 16:57:07 +0800 xxx.com FATAL [taoensso.carmine.message-queue] - Worker error! java.lang.Exception: Carmine connection error
java.lang.Exception: Carmine connection error
connections.clj:154 taoensso.carmine.connections/pooled-conn
message_queue.clj:282 taoensso.carmine.message-queue/handle1[fn]
RestFn.java:442 clojure.lang.RestFn.invoke
message_queue.clj:313 taoensso.carmine.message-queue/handle1
message_queue.clj:353 taoensso.carmine.message-queue.Worker/start-polling-loop!
message_queue.clj:356 taoensso.carmine.message-queue.Worker/fn[fn]
core.clj:1819 clojure.core/binding-conveyor-fn[fn]
AFn.java:18 clojure.lang.AFn.call
FutureTask.java:262 java.util.concurrent.FutureTask.run
ThreadPoolExecutor.java:1145 java.util.concurrent.ThreadPoolExecutor.runWorker
ThreadPoolExecutor.java:615 java.util.concurrent.ThreadPoolExecutor$Worker.run
Thread.java:744 java.lang.Thread.run
It seems that connect to redis failed,but the worker didn't process messages any more.Is it the expected behaviour? How to prevent worker exited unexpected?
Currently a connection pool is cached in pool-cache
atom against pool-opts we provide to create it.
We use wcar as follows
(wcar {:pool {<pool_opts>}
:spec {<server_opts>}}
<redis-operations>)
Assume that we always want to use max-active connection 16. Hence pool-opts
will be {:max-active 16}
. But we need to connect two different instances for different operations server1
and server2
(for both server we need max-active as 16). So our call becomes
For Server1
(wcar {:pool {:max-active 16}
:spec {:host "server1"}}
<redis1-operations>)
For Server2
(wcar {:pool {:max-active 16}
:spec {:host "server2"}}
<redis2-operations>)
I was assuming that a seperate pool will be created for server1
and server2
. But after reading pooled-conn
and conn-pool
I found out that pools are maintained against pool-opts and have nothing to do with server-spec. Hence if I provide same pool-opts for both servers only one pool will be created. So If there are n-servers they will share same pool of connections. May be I am interpreting wcar
API wrongly, but I think each server-spec should have its pool.
/cc @vedang
The documentation is getting a little long for a single-page README. Think it'd be a better idea to switch to a GitHub wiki where some structure can be added and there's more freedom to expand on examples, etc.
To start with, the whole README could just be dumped as a single page - then structure slowly introduced.
I see in the comments the handler-fn should either throw an exception or return :retry. How many times will a message be retried? Is there a growing pause between retries?
For example, if I want to retry 10 times with an exponentially growing pause between retries, is this possible?
[PATCH] closeable-pool support -- includes a type-hint on the return
of make-conn-pool, extending Closeable to ConnectionPool, and a call to close
in the main test.
---
src/taoensso/carmine.clj | 3 ++-
src/taoensso/carmine/connections.clj | 9 ++++++---
test/taoensso/carmine/tests/main.clj | 7 ++++---
3 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/src/taoensso/carmine.clj b/src/taoensso/carmine.clj
index e6a77fc..003b85c 100644
--- a/src/taoensso/carmine.clj
+++ b/src/taoensso/carmine.clj
@@ -18,6 +18,7 @@
(defn make-conn-pool
"For option documentation see http://goo.gl/EiTbn"
+ ^taoensso.carmine.connections.ConnectionPool
[& options]
(let [;; Defaults adapted from Jedis
defaults {:test-while-idle? true
@@ -418,4 +419,4 @@
(with-conn nil nil (ping)) ; Degenerate pool, default spec
(with-conn (make-conn-pool) (make-conn-spec) (ping))
(with-conn pool spec (ping))
- (with-conn pool spec "invalid" (ping) (ping)))
\ No newline at end of file
+ (with-conn pool spec "invalid" (ping) (ping)))
diff --git a/src/taoensso/carmine/connections.clj b/src/taoensso/carmine/connections.clj
index a1267a0..94f9316 100644
--- a/src/taoensso/carmine/connections.clj
+++ b/src/taoensso/carmine/connections.clj
@@ -4,7 +4,8 @@
{:author "Peter Taoussanis"}
(:require [taoensso.carmine (utils :as utils) (protocol :as protocol)])
(:import java.net.Socket
- [java.io BufferedInputStream DataInputStream BufferedOutputStream]
+ [java.io BufferedInputStream DataInputStream BufferedOutputStream
+ Closeable]
[org.apache.commons.pool KeyedPoolableObjectFactory]
[org.apache.commons.pool.impl GenericKeyedObjectPool]))
@@ -49,7 +50,9 @@
(get-conn [this spec] (.borrowObject pool spec))
(release-conn [this conn] (.returnObject pool (get-spec conn) conn))
(release-conn [this conn exception] (.invalidateObject pool (get-spec conn)
- conn)))
+ conn))
+ Closeable
+ (close [this] (.close pool)))
(defn make-new-connection
"Actually creates and returns a new socket connection."
@@ -99,4 +102,4 @@
:time-between-eviction-runs-ms (.setTimeBetweenEvictionRunsMillis pool v)
:min-evictable-idle-time-ms (.setMinEvictableIdleTimeMillis pool v)
(throw (Exception. (str "Unknown pool option: " opt))))
- pool)
\ No newline at end of file
+ pool)
diff --git a/test/taoensso/carmine/tests/main.clj b/test/taoensso/carmine/tests/main.clj
index ac32ddb..715d058 100644
--- a/test/taoensso/carmine/tests/main.clj
+++ b/test/taoensso/carmine/tests/main.clj
@@ -3,14 +3,14 @@
(:require [clojure.string :as str]
[taoensso.carmine :as car]
[taoensso.carmine.utils :as utils]
- [taoensso.nippy :as nippy]))
+ [taoensso.nippy :as nippy])
+ (:import [taoensso.carmine.connections ConnectionPool]))
;;;; TODO
;; * Refactor this whole ns: fixtures, consistent (= x const) ordering, etc.
;; * Bring over tests from redis-clojure, etc.
;;;; Setup
-
(def p (car/make-conn-pool))
(def s (car/make-conn-spec))
(defmacro wcar [& body] `(car/with-conn p s ~@body))
@@ -356,4 +356,5 @@
(car/ping)))))
(is (= k-val (wcar (car/get k))))))
-(clean-up!) ; Leave with a fresh db
\ No newline at end of file
+(clean-up!) ; Leave with a fresh db
+(.close ^ConnectionPool p) ; Close the connection pool
--
1.8.1.4
I was using a custom Closeable
record to pass to wcar
.
#me.Redis{:pool #taoensso.carmine.connections.ConnectionPool{:pool #<GenericKeyedObjectPool org.apache.commons.pool.impl.GenericKeyedObjectPool@1515d8a6>}, :spec {:port 6379, :host "localhost", :name "redis"}}
On 2.2.1
, this is valid. On 2.2.2
it is not.
(wcar my-redis-obj (llen 'foo'))
;; Exception Unknown pool option: :pool taoensso.carmine.connections/set-pool-option (connections.clj:89)
Please apply a backwards compatible patch.
Newer options like EX for the SET command are not supported in carmine as a result of this.
This is a feature request. The lua-script helper is great, but recently I had a need to pass an array of values to a script. This can't easily be done in the current setup, as both the KEYS and ARGV parameter groups are maps to be used for interpolation. I had to use eval*, which has completely different syntax.
I think a variant of the lua-script helper that interpolates KEYS but lets me pass ARGV as a simple array of values that I can access using ARGV[1] etc would be quite useful in many cases.
http://redis.io/topics/sentinel-clients
https://npmjs.org/package/node-sentinel
Waiting for sentinel client guidelines to be finalized.
If there is a connection error in a Redis listener, Carmine does not try to reconnect, and the subscriber function will receive no further messages.
Example:
(def listener
(car/with-new-pubsub-listener my-conn
{"p" #(log/debug %)}
(car/subscribe "p")))
After restarting Redis I assumed that the client would reconnect, and I would see a "subscribe" message in the logs. But this is not the case: neither the subscribe message nor any further messages on the pub/sub channel is delivered to the listener.
What's even worse, I cannot even detect whether the connection is alive:
user=> (.conn-alive? (:connection listener))
true
Handlers registered with the "with-new-listener" or "new-pubsub-listener" are not called again after they raise an exception.
The reason is that no exception handling in the event handler "while" loop:
https://github.com/ptaoussanis/carmine/blob/master/src/taoensso/carmine.clj#L350
This is the rare case when Throwable should be catched, I guess. Until the fix the obvious workaround is handling every exception in our handler.
I wrote a small caching wrapper around carmine that adds some logic to caching values - for instance, it fetches from an in-process cache first before falling back to carmine and ultimately calling a function if the cached value isn't there.
This behaviour can be tested in isolation at the unit level so I don't necessarily need nor want to have redis running to run these tests.
However, stubbing macros - such as wcar
and with-conn
- is problematic and makes writing such tests a bit difficult.
I dug into carmine's source and ended up implementing a couple of fake records that make writing such tests a lot easier. One can write something like this:
;; make-fake-conn-pool is the stubbing function
(fact "tests my function that wraps carmine"
(with-redefs [cache/pool (make-fake-conn-pool)
protocol/get-replies! (constantly "leo")]
(cache/external-fetch "name") => "leo"
(provided (redis/get "name") => irrelevant)))
I then thought about creating a small lib called carmine-mock - akin to ring-mock - that would provide these testing facilities but I then wondered if it's worth this into carmine instead?
Also not sure this is the best interface yet - it's a first draft really so regardless, feedback is welcome.
While running the message queue example (https://github.com/ptaoussanis/carmine#message-queue) on Mac OS X 10.9 (Mavericks) with Redis 2.6.16 or 2.7.105 (2.8.0-rc5) it takes hundreds of milliseconds or even seconds for the message to arrive at the other end. I pretty much copy pasted the example code verbatim with the worker in one terminal (via a lein repl) and the sending party in another.
I have not tried this on another operating system yet, but the redis-benchmark shows nothing out of the ordinary. I did run this on both a Macbook Air (late 2011) and a hackintosh with 32GB RAM.
Potentially also migrate to Expectations from clojure.test
?
This may be a case of just needing to add instructions, but I have found that while it works in my preferred development environment of Ubuntu with Java 7, it does not work in the equivalent Mac OS environment. I have so far not traced the source of the problem, other than snappy-java (presumably the native lib that the snappy-java JNI interface uses) not being on the class path. Is there something that needs to be installed?
Here is the stack trace:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at carmine.serialization$eval1106$loading__4784__auto____1107.invoke(serialization.clj:1)
at carmine.serialization$eval1106.invoke(serialization.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5302)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at carmine.protocol$eval1087$loading__4784__auto____1088.invoke(protocol.clj:1)
at carmine.protocol$eval1087.invoke(protocol.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5302)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at carmine.core$eval1081$loading__4784__auto____1082.invoke(core.clj:1)
at carmine.core$eval1081.invoke(core.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at foo.redis.impl$eval1075$loading__4784__auto____1076.invoke(impl.clj:1)
at foo.redis.impl$eval1075.invoke(impl.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at testapp.redis$eval1069$loading__4784__auto____1070.invoke(redis.clj:1)
at testapp.redis$eval1069.invoke(redis.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:930)
at foo.core$eval112$loading__4784__auto____113.invoke(core.clj:1)
at foo.core$eval112.invoke(core.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:512)
at foo.demo$eval89$loading__4784__auto____90.invoke(demo.clj:1)
at foo.demo$eval89.invoke(demo.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval49$fn__70.doInvoke(NO_SOURCE_FILE:1)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:601)
at user$eval49.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.eval(Compiler.java:6477)
at clojure.core$eval.invoke(core.clj:2797)
at clojure.main$eval_opt.invoke(main.clj:297)
at clojure.main$initialize.invoke(main.clj:316)
at clojure.main$null_opt.invoke(main.clj:349)
at clojure.main$main.doInvoke(main.clj:427)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:419)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.main.main(main.java:37)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
at java.lang.Runtime.loadLibrary0(Runtime.java:845)
at java.lang.System.loadLibrary(System.java:1084)
at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
... 173 more
Exception in thread "main" org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at carmine.serialization$eval1106$loading__4784__auto____1107.invoke(serialization.clj:1)
at carmine.serialization$eval1106.invoke(serialization.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5302)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at carmine.protocol$eval1087$loading__4784__auto____1088.invoke(protocol.clj:1)
at carmine.protocol$eval1087.invoke(protocol.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5302)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at carmine.core$eval1081$loading__4784__auto____1082.invoke(core.clj:1)
at carmine.core$eval1081.invoke(core.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at foo.redis.impl$eval1075$loading__4784__auto____1076.invoke(impl.clj:1)
at foo.redis.impl$eval1075.invoke(impl.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at foo.redis$eval1069$loading__4784__auto____1070.invoke(redis.clj:1)
at foo.redis$eval1069.invoke(redis.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:930)
at foo.core$eval112$loading__4784__auto____113.invoke(core.clj:1)
at foo.core$eval112.invoke(core.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:512)
at foo.demo$eval89$loading__4784__auto____90.invoke(demo.clj:1)
at foo.demo$eval89.invoke(demo.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval49$fn__70.doInvoke(NO_SOURCE_FILE:1)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:601)
at user$eval49.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.eval(Compiler.java:6477)
at clojure.core$eval.invoke(core.clj:2797)
at clojure.main$eval_opt.invoke(main.clj:297)
at clojure.main$initialize.invoke(main.clj:316)
at clojure.main$null_opt.invoke(main.clj:349)
at clojure.main$main.doInvoke(main.clj:427)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:419)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.main.main(main.java:37)
I'm having problems with receiving multiple values from sorted sets and hashes. Lists appear to be working great, but I've just started using the library and my primary focus has been on sorted sets, so far.
The steps to reproduce it:
[redis-2.4.15]$ ./src/redis-cli
redis 127.0.0.1:6379> del ztest
(integer) 1
redis 127.0.0.1:6379> zincrby ztest 1 "foo"
"1"
redis 127.0.0.1:6379> zincrby ztest 2 "bar"
"2"
redis 127.0.0.1:6379> zrange ztest 0 -1
1) "foo"
2) "bar"
[carmine:master]$ lein repl
user=> (load "taoensso/carmine")
nil
user=> (require '[taoensso.carmine :as r])
nil
user=> (def spec (r/make-conn-spec))
#'user/spec
user=> (def pool (r/make-conn-pool :max-active 8))
#'user/pool
user=> (r/with-conn pool spec (r/ping))
"PONG"
user=> (r/with-conn pool spec (r/zrange "ztest" 0 -1))
["foo"] ;; expected ["foo", "bar"]
user=> (r/with-conn pool spec (r/zrevrange "ztest" 0 -1))
["bar"] ;; expected ["bar", "foo"]
user=> (r/with-conn pool spec (r/zrevrange "ztest" 0 2))
["bar"] ;; expected ["bar", "foo"]; using 0 2 to show it's not a problem of handling negatives.
Hello there,
I'm trying to create a Redis adapter for Hyperion and I would like to use Carmine for the redis adapter. Unfortunately I'm having some issues using your API.
I'll be getting a hash of options from the user, but there's no easy way passing that hash to r/make-conn-spec
. Going past that because the connection spec will be created dynamically its hard to use the carmine macro.
What I would like to do is create a method that takes the options passed by the user, and passes back a function that behaves like the carmine macro. This is what I have so far.
(defmacro with-conn [pool spec body]
`(r/with-conn ~pool ~spec ~@body))
(defn open-db [options]
(let [defaults {:host "127.0.0.1" :port 6379 :password nil :timeout 0 :db 0}
pool (r/make-conn-pool :max-active 8)
connection-spec (merge defaults options)]
(fn [& body]
(with-conn pool connection-spec body))))
But this doesn't work. Do you know of a way that would fix these issues?
Thank you for the great project,
Eric
I'm trying to use Carmine to create my own pub/sub server. The relevant portion of my handler looks like the following:
(with-channel request connection
(on-receive connection
(fn [data]
(car/with-conn redis-pool redis-server
(println "publishing" data "to" pub-channel)
(car/publish pub-channel data))))
(car/with-new-pubsub-listener
redis-server {sub-channel (fn [msg]
(println "sending" msg "to" connection)
(send! connection msg))}
(car/psubscribe sub-channel)))
Now, the "publishing" message shows up every time I send new data to the socket from my browser. However, the "sending" message only prints when I renew my connection by refreshing my browser. I tried looking into the listener source code and couldn't figure out where I am going wrong. I've tried out my sub-channel regex multiple times inside of redis-cli
, and it looks like it should work.
What am I missing? Am I not using listeners properly?
(wcar (car/spop "foo" "bar"))
should be
(wcar (car/spop "foo"))
in the 'Basic commands' section.
This isn't really a bug report yet, I'm rather looking for hints and pointers. I have a system where some values transported to/from Redis seem to get corrupted. Sometimes.
Case #1: I have a lua script that does this:
1384505478.269537 [0 5.9.23.176:50292] "EVALSHA" "c9ff551721953469eb600c6a9057b5d6ace939ed" "2" "customer" "186kvcq0a-LOrFzrozoRBmVizb"
1384505478.269571 [0 lua] "get" "t2r:customer:186kvcq0a-LOrFzrozoRBmVizb"
[...]
the script in question essentially does this:
local t2r_key = 't2r:' .. _:customer .. ':' .. _:tid
local ruid = redis.call('get', t2r_key)
local created = false
[...]
return {ruid, created}
the elided code doesn't matter, it's an "if" that is not taken in this case. The lua script basically just performs a single GET here and returns the value along with a boolean (false) flag.
Problem is, sometimes the values returned sometimes make no sense. When the problem occurs, Clojure logging code prints the two received values as something similar to "[ 2". Investigating this closer shows that the "ruid" value is "\x00>\x03\b\n\x00[", while it should be:
redis 127.0.0.1:6379> get t2r:customer:186kvcq0a-LOrFzrozoRBmVizb
"165854"
To give you an idea of "sometimes", in this case it happens in about 1.5% of calls.
Case #2: There is also another problem, which looks related. It is in a completely different section of the code, does not involve lua scripting at all, and does not involve reading anything from redis. This is stats code that only stores values (strings, in this case), which arrive to redis in this form:
1384505479.011301 [0 5.9.23.176:50420] "ZINCRBY" "stats:customer:s:2013-11-15" "1" "\x00>\x01\x00\x03"
The "\x00>\x01\x00\x03" should normally be a human-readable string. My Clojure logs show no such corruption there, it's only visible in redis monitor.
Things of note:
Software involved: Redis 2.6.16, Clojure 1.5.1, Carmine 1.12.0.
Any ideas?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.