bwlewis / rredis Goto Github PK
View Code? Open in Web Editor NEWR client for Redis
Home Page: http://illposed.net/
R client for Redis
Home Page: http://illposed.net/
function is not adding correnct value to list, it can be a character encoding problem
cann`t use pipeline, which will be very slow if redis server stored huge keys
In the manual it says that calling redisIncr
after redisSet
could be easily done. The example from the manual does not seems to work though:
> redisSet('foo', 1)
[1] "OK"
> redisIncr('foo')
Error in .getResponse() : ERR value is not an integer or out of range
If I set the integer value from command line (with redis-cli
) the redisIncr
works like a charm so it seems that the redisSet
does not save the value as integer but as R object? From command line the saved foo
(set to: 1) looks like:
"X\\x00\x00\x00\x02\x00\x02\\x00\x00\x02\x03\x00\x00\x00\x00\x0e\x00\x00\x00\x01?\xf0\x00\x00\x00\x00\x00\x00"
If I set the value with charToRaw
, it seems to work:
> redisSet('foo', charToRaw('1'))
[1] "OK"
> redisIncr('foo')
[1] 2
The documentation might be updated if my findings look right.
rredis seems to be ok for small sized, data, but when I try to store a large object the connection seems to die:
> redisSet("test", rnorm(1e8))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR Protocol error: invalid bulk length
In addition: Warning messages:
1: In writeBin(v, con) : problem writing to connection
2: In writeBin(.raw("\r\n"), con) : problem writing to connection
Any idea what is going on?
I think I am running into some sort of race condition in the hash get functions that receive bulk replies. When I make a request for a lot of data using either HMGET or HGETALL, I can get the following error:
Error in if (nchar(l) < 2) { : argument is of length zero
Calls: redisCmd ... replicate -> sapply -> lapply -> FUN -> .getResponse
I am running rredis 1.6.6 against Redis server version 2.4.10. The code I am running to test this is:
redisConnect('redis host');
a = redisCmd('HGETALL', 'test_redis_bug', raw=TRUE)
redisClose();
test_redis_bug
is a hash containing 400 vectors of 51,698 doubles and a single vector of 51,698 character strings, all of which have been manually run through serialize()
before storage (this is also why they are being retrieved in raw mode).
For small enough requests, I will never see this problem, while for large enough requests it happens every time. There is a middle ground where it happens inconsistently. “Large” seems to vary depending on the speed of the connection to the Redis server, but over our metro Ethernet (which runs at tens of megabits per second), I see consistent problems on this 400 x 50,000 example.
I did a little bit of poking around and when I see this error, l
is character(0)
. If I read the next line, it becomes an empty string, and if I read the line after that I get "$<some length>"
, as expected.
I suspect a race condition because putting a Sys.sleep(1)
at the top of .getResponse()
fixes everything.
I was able to improve the situation by looping the l <- readLines(con=con, n=1)
until it got something real, but I then less consistently see an error where readLines()
ran out of material to read before reading an end-of-line, which will be followed by a garbled message on my next pull (which I theorize is data from the first request that didn’t get there in time).
It's been frustrating to try to figure out more because trying to look at anything that happens before the error seems to introduce enough delay that the race isn't triggered. Let me know if I can do anything to help in tracking this down.
Hello!
Wanted to use this, but i need to be able to specify which database to connect to.
Seems like this is currently not possible.
P
Hi Bryan,
With the help of your package is there any way to perform RESP serialization of an R dataframe?
If not do you think it's good idea to include such functionality.
Thank you,
Sergey.
instead of serialization, redisHMSet
applied charToRaw
on characters and lost information. for example
> letters
[1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s" "t" "u" "v" "w" "x" "y" "z"
> redisHMSet("test", list(letters = letters))
[1] "OK"
Warning message:
In charToRaw(values[[j]]) :
argument should be a character vector of length 1
all but the first element will be ignored
> redisHGet("test", "letters")
[1] "a"
>
the problem is below line
> redisHMSet
function (key, values)
{
a <- c(alist(), list(.raw("HMSET")), list(.raw(key)))
fieldnames <- lapply(names(values), charToRaw)
for (j in 1:length(values)) {
a <- c(a, fieldnames[j])
<<<< if (is.character(values[[j]])) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
a <- c(a, list(charToRaw(values[[j]])))
else (a <- c(a, list(.cerealize(values[[j]]))))
}
do.call(".redisCmd", a)
}
Is this work as design or an issue?
Hi Bryan,
Thank you for your work on rredis! I have a suggestion, can you update the CRAN package? Since that rredis has new features like the authenticated connection, it would be great that it serves the community. Thanks!
1.6.7 introduces a timeout parameter for redisConnect() that is giving me trouble with R 2.15.1 under OS X 10.8 (Mountain Lion). The issue appears to be that timeouts that large just don't work on Mountain Lion, as the call consistently fails with timeout > 100,000,000, while my colleague's 10.7 machine and our Ubuntu 10.04 servers are fine. I can clearly just set the timeout in the call, but it seems like setting the default to 100,000,000 (~3 years) might be nice. I'm not sure if socketConnection might also allow 0 to mean "no timeout".
[R.app GUI 1.52 (6188) x86_64-apple-darwin9.8.0]
> library(rredis)
> redisConnect("redis-equinix")
Error in socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) :
cannot open the connection
In addition: Warning message:
In socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) :
redis-equinix:6379 cannot be opened
> redisConnect("redis-equinix", timeout=100000000L)
> redisConnect("redis-equinix", timeout=100000001L)
Error in socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) :
cannot open the connection
In addition: Warning message:
In socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) :
redis-equinix:6379 cannot be opened
>
It seems like inserting multiple list elements is not possible with the current redisLPush/redisRPush implementation (despite this being supported in redis).
Would it make sense to implement this (e.g. by breaking a vector of values into individual list items and inserting them)? A flag could be used to enable this option (to provide backwards compatibility).
Blocking operations like redisBRPop
, redisBLPop
can hang when communication with the Redis server is interrupted. This is a bug in the internal burn
function.
See the internal .raw
function: https://github.com/bwlewis/rredis/blob/master/R/redis-internal.R#L110
The approach defined there is used almost everywhere now already with the notable exception of redis "String" operations like redisSet
, redisMSet
, etc.
The original intent of this package is to make R objects always the default, letting users convert them to raw values if they desire to inter-operate with other programs.
However, I don't really see the harm in relaxing that slightly for scalar character values in R. The only real chance for problems I see might be in various character encoding data being messed up.
Ideas? Comments?
consider the following, set a key from 3 different clients:
SET from: "abcd"
Is this 7x bloat necessary? And is there a workaround in the form of an option/setting?
Hi~
Could you make hyperloglog redis command like pfcount command ?
Using redis to exchange data between multiple clients I found problems setting a value in R (using rredis) and reading it back from the redis client or from node: the value I get back is not the one I put in, but a string of "strange characters".
It works perfectly only if I get the value back in R or if I read a value stored by the redis client.
> library(rredis)
> redisConnect(timeout=300)
> redisSet('x',10)
[1] "OK"
> redisGet('x')
[1] 10
redis 127.0.0.1:6379> get x
"X\n\x00\x00\x00\x02\x00\x03\x00\x00\x00\x02\x03\x00\x00\x00\x00\x0e\x00\x00\x00\x01@$\x00\x00\x00\x00\x00\x00"
redis 127.0.0.1:6379> set x 5
OK
> redisGet('x')
[1] "5"
I get the same results using rredis 1.6.7 either with R 3.0.0-64bit (on OSX 10.8) and R 2.15.2-64bit (on Windows 7). Redis version is 2.6.12 (on OSX 10.8), but I tried also with a free redis on-line service.
Hi
Just a quick question. In the vignette, the size limitation of a R object is indicated below.
When I checked its data types, however, the maximum number of elements of lists, sets and hashes is 2^32 - 1 (http://redis.io/topics/data-types) while the maximum size of its key and value is 512 MB.
So I guess, for example, if a R vector is created, as long as each value (and the key) doesn't exceed 512 MB, it'd be possible to add.
Please inform if the guess is incorrect.
Regards
Jaehyeon
redisConnect
refuses to use integer ports (e.g. redisConnect("localhost", 6379L)
will throw an error). The culprit is
stopifnot(class(port) == "numeric")
which should read
stopifnot(is.numeric(port))
(integers are numeric, but they have the class "integer"
).
Hey,
I'm having a performance issue problem with running Redis (2.6) on Ubuntu 12.04.
After filling a database by executing the following command in a redis-cli
eval "for i=1,100 do redis.call('sadd', i, 1); end; return 1" 0
Which fills a 100 sets with one element. Then in R running the following commands:
library(rredis)
redisConnect()system.time (lapply(1:100, function(i) { redisSMembers(as.character(i)) }))
user system elapsed
0.104 0.044 4.000system.time (lapply(1:100, function(i) { runif(1) }))
user system elapsed
0.000 0.000 0.001
The operation takes about 4 second elapsed time, which seems a bit long especially with regard to the user time ;-). Happens consistently as I just iterated over 400k sets with two Redis operations which lasted about 7,5 hours (26060.933 seconds to be precise).
redis-benchmark results in acceptable performance, so I'm kinda at a loss what's going here! It works fine on my Mac, happens with both 32 and 64 bit R versions (latest major release).
redisTTL()
function is returning undocumented value of -2 for a non-existent key. It should probably return -1 or the documentation should contain the description of another return value of -2.
The next planned CRAN package update is September 15th, 2013.
If there is something you want in the package let me know before then!
Problems of redisZRange
in combination with redisKeys
e.g. redisZRange(redisKeys("A*Sensor"))
caused by f <- match.call()
in .redisCmd
which takes the call from redisKeys
.
I'm not sure, where to fix it. The most central way would be in .redisCmd
, but then match.call
has to be replaced.
Using rredis 1.6.3, redisHMSet doesn't seem to work:
redisHMSet('A', list(x=1,y=2,z=3))
[1] TRUE
redisHKeys('A')
NULL
redisHSet('A', 'x', 1)
[1] TRUE
redisHKeys('A')
[[1]]
[1] "x"
Starting from an empty Redis:
redisFlushAll()
[1] "OK"
redisKeys()
NULL
redisHMSet('A', list(x=1,y=2,z=3))
[1] TRUE
redisKeys()
[1] 120 1
It also gives errors when the list of fields and values is not exactly 3 elements that make me suspect something is going wrong internally:
redisHMSet('A', list(x=1))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR wrong number of arguments for 'hmset' command
redisHMSet('A', list(x=1,y=2))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR wrong number of arguments for 'hmset' command
redisHMSet('A', list(x=1,y=2,z=3))
[1] TRUE
redisHMSet('A', list(x=1,y=2,z=3,w=4))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR wrong number of arguments for HMSET
I encountered this error caused by this line in redis-internal.R
writeBin(.raw(hdr), con)
which is supposedly thrown if con
is not binary, and not NULL either. I don't see any how it can happen, looking through the code of rredis.
And in fact, when I call rredis::redisGet
locally there was no issue. Yet the error popped out for all the calls via OpenCPU.
It took me quite a while and still have no clue how or where con
can become a non-binary connection. I don't think it is the problem of either rredis or OpenCPU. I must have missed some very basic thing...
Hi,
using rredis 1.6.9 on R 3.1.2 I get the following with the example of redisIncrBy
> redisSet('x','5')
[1] "OK"
> redisIncrBy('x','3')
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR value is not an integer or out of range
The problem might be an encoding issue (?). Because that is what I get in redis:
redis 127.0.0.1:6379> get x
"X\n\x00\x00\x00\x02\x00\x03\x01\x02\x00\x02\x03\x00\x00\x00\x00\x10\x00\x00\x00
\x01\x00\x04\x00\t\x00\x00\x00\x015"
redis 127.0.0.1:6379> set x "5"
OK
redis 127.0.0.1:6379> get x
"5"
redis 127.0.0.1:6379> incrby x 1
(integer) 6
Ciao,
Stefan
Some get commands have a 'raw' argument:
redisGet(key, raw = FALSE)
redisMGet(keys, raw = FALSE)
redisHGet(key, field, raw = FALSE)
Some do not:
redisHGetAll(key)
redisHMGet(key, fields)
We've taken to using redisCmd() to call a base Redis command when we need a raw version that isn't provided, but it would be nice if this wasn't necessary. Is there some rationale behind which commands have raw versions and which do not?
On startup, I am consistently getting a fail on the first attempt to connect, but it works on the subsequent attempt(s):
library(rredis)
ref <- redisConnect(returnRef=TRUE)
# Error in .openConnection(host = host, port = port, nodelay = nodelay, :
# Error setting TCP_NODELAY
ref <- redisConnect(returnRef=TRUE)
I get the error with or without returnRef
, with non-default host/port, and with timeout specified or not. I've started R with --vanilla --no-site-file --no-init-file
to ensure I'm not poisoning it somehow.
It does not fail with nodelay=FALSE
.
> sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rredis_1.7.0
loaded via a namespace (and not attached):
[1] compiler_3.2.3 tools_3.2.3
turns out I had an issue with one of our systems that seems to have been resolved by updating to master, perhaps you should tag it as 1.7.1.
I got this response from redisKeys
$ redisKeys("*")
Error in readBin(con, raw(), 1000000L) :
negative length vectors are not allowed
According to bwlewis/doRedis#4, this error message should be issued when exceeding the data size limit. But this is definitely not the case of redisKeys
. There was probably some error around the connection, because after I connected again, I got the correct response with no error:
$ redisConnect('myserver')
$ redisKeys("*")
[1] "jobs:1.alive.1" "x" "jobs:counter" "jobs:1.start.3" "jobs:1.alive.3"
[6] "jobs" "jobs:1" "jobs:1.start.1" "jobs:workers" "jobs:1.env"
So there should probably be a different error message issued in this case.
Hi,
redisHIncrByFloat
function (key, field, value, ...)
{
.redisCmd(.raw("HINCRBY"), .raw(key), .raw(field),
.raw(as.character(value)), ...)
}
It should be calling HINCRBYFLOAT.
Thanks for your work on Rredis - we use it extensively.
David
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.