Hello, do you have any examples of pub/sub for keyspace notifications with async contexts?
Has this been tried before? Do you have any observations to share?

Thank you.

hey, bjosv
It does not work with command time opt, here's my code
handle_ = redisClusterContextInit();
redisClusterSetOptionAddNodes(handle_, addrs_.c_str());
struct timeval timeout = {0, 5000};
redisClusterSetOptionTimeout(handle_, timeout);
redisClusterConnect2(handle_);
redisReply *reply = (redisReply *)redisClusterCommand(handle_, "GET %s", key.c_str());

Some response still cost more than 5 ms !

Hi Team,

Is there any api similar to "redisClusterAsyncCommandArgv" (uses argc and argv), but that should be for a Node(uses ToNode).

Thanks!

It seems that cmake is not installing the file dict.h because it's missing from CMakeLists.h?

install(FILES hircluster.h adlist.h hiarray.h dict.h
  DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis_cluster)

Hello,

I'm trying to use the Async API. Everything works fine, the load is balanced on my masters. My test software shows an output of 160k req/s (only GET commands)
If I set my connection to READONLY, only the 3 master instances seems to be used (doing a simple "top" shows only 3 instances working)

Here is a code extract of what i do (removed error checks):

redisClusterAsyncContext * cc = redisClusterAsyncConnect("127.0.0.1:30001", HIRCLUSTER_FLAG_NULL);
uv_loop_t * loop = (uv_loop_t *)malloc(sizeof(uv_loop_t));
uv_loop_init(loop);
redisClusterLibuvAttach(cc, loop);
redisClusterAsyncContext(cc, "READONLY");
redisClusterAsyncCommand(cc, &__cb_getKey, &priv, "GET key");

Is this the correct way to handle this ? Also what is the purpose of the second parameter of redisClusterAsyncConnect ?

Thanks

Hi,

cluster_update_route_by_addr() function today invokes a 'cluster nodes' command and waits synchronously till the response is available. Since its a blocking call, and hiredis context is single threaded, all other operations go for a halt till the response is received. This impacts the performance of the system, since all other calls application uses are async implemented, and any code flow that reaches here has to wait till the call gets over.

This ticket is a request for an async implementation for supporting the cluster nodes/slots command.

Thanks.

Hi,

Came across one more crash scenario.
The top frames look similar to ticket raised before: #113

However, the bt leading to it is different.

[Current thread is 1 (Thread 0x7f22b4e9ba40 (LWP 38))]
(gdb) bt
#0  actx_get_by_node (acc=acc@entry=0x7b1a20, node=0x1c351e0)
    at hircluster.c:3631
#1  0x00007f22b84ef01c in redisClusterAsyncFormattedCommand (acc=acc@entry=0x7b1a20,
    fn=fn@entry=0x7f22b815d040 <optimized out>, privdata=privdata@entry=0x0,
    cmd=0xac9c90 <optimized out>, len=86)
    at hircluster.c:4124
#2  0x00007f22b84ef31b in redisClustervAsyncCommand (acc=0x7b1a20,
    fn=0x7f22b815d040 <optimized out>, privdata=0x0, format=<optimized out>,
    ap=ap@entry=0x7ffdb96af2b0) at hircluster.c:4251
#3  0x00007f22b84ef42e in redisClusterAsyncCommand (acc=<optimized out>, fn=<optimized out>, privdata=<optimized out>, format=<optimized out>)
    at hircluster.c:4265

Frames 4 and 5 are application frames, where it processes callback from hiredis and again issues a new command for execution.

#6  0x00007f22b84ef6e5 in redisClusterAsyncRetryCallback (ac=<optimized out>, r=0x0, privdata=0x7b75b0)
    at hircluster.c:4016
#7  0x00007f22b84d0e09 in __redisRunCallback (reply=0x0, cb=0x7ffdb96af690, ac=0x1e091a0)
    at hiredis/async.c:288
#8  __redisAsyncFree (ac=0x1e091a0) at hiredis/async.c:310
#9  0x00007f22b84d1618 in redisAsyncFree (ac=0x1e091a0) at hiredis/async.c:375
#10 0x00007f22b84e5ee5 in cluster_node_deinit (node=node@entry=0x8865c0)
    at hircluster.c:281
#11 0x00007f22b84e6015 in cluster_node_deinit (node=0x8865c0)
    at hircluster.c:264
#12 dictClusterNodeDestructor (privdata=<optimized out>, val=0x8865c0)
    at hircluster.c:133
#13 0x00007f22b84e5616 in _dictClear (ht=0x88ef80) at dict.c:179
#14 dictRelease (ht=0x88ef80) at dict.c:194
#15 0x00007f22b84eac44 in cluster_update_route_by_addr (port=<optimized out>, ip=<optimized out>, cc=<optimized out>)
    at hircluster.c:1390
#16 cluster_update_route (cc=cc@entry=0x7b1920) at hircluster.c:1463
#17 0x00007f22b84efa65 in actx_get_after_update_route_by_slot (slot_num=4682, acc=0x7b1a20)
    at hircluster.c:3737
#18 redisClusterAsyncRetryCallback (ac=<optimized out>, r=0x919570, privdata=0x169d230)
    at hircluster.c:3972
#19 0x00007f22b84d190c in __redisRunCallback (reply=<optimized out>, cb=0x7ffdb96af920, ac=0x7b9650)
    at hiredis/async.c:288
#20 redisProcessCallbacks (ac=0x7b9650) at hiredis/async.c:580

Below frames, eventloop detects a socket event and calls into hiredis.

Looks like there is multiple back and forth jumping between hiredis and application, and eventually leading to a crash. The node, table, slots are all having junk data at frame 0. However, nodes dictionary is intact similar to how ticket #113 was having.

Atleast the below piece of code explains why nodes dictionary is intact.

    /* Replace cc->nodes before releasing the old dict since
     * the release procedure might access cc->nodes. */
    oldnodes = cc->nodes;
    cc->nodes = nodes;
    if (oldnodes != NULL) {
        dictRelease(oldnodes);
    }

But we don't update slots and table as part of this, not sure if that is intentional. Can you please check this stack and see what can lead to this/suggest a solution?

cc: @bjosv

Thanks.

Hello,

We are using hiredis-cluster Release 0.8.1, and this issue is with regards to a crash we see in our performance testing.

We use async hiredis APIs, and invoke them both from the application and also from hiredis callbacks.
Our application is multi-threaded, and hiredis context object & hiredis API calls are mutex protected.
Our adapter is designed as per the reference hiredis adapter, and we don't do anything special besides calling into hiredis library.

We see the following crash a few hours into our test following an adapter timeout:

Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x1c
Process uptime: not available

Thread 0 (crashed)
0 libhiredis_cluster.so.0.8!redisClusterAsyncRetryCallback + 0x263
1 libhiredis.so.1.0.0!__redisAsyncFree + 0x69
2 libhiredis.so.1.0.0!redisProcessCallbacks + 0x1a0
3 redisLibmyeventHandler(int, unsigned int, base::SystemInfo) [hiredis_libmyevent.h : 89 + 0x5]

Please help debug the issue. Let me know if you need any further information.

Thank you.

Hey hiredis-cluster team,

We encountered an issue where hiredis-cluster did not update the cluster routing after it was scaled down (one redis node removed from cluster).
The issue results in a failure to connect to (now) the non-existing node, specifically for our use case (AWS managed Redis) - DNS resolution error: Redis error: Name or service not known. For other services it might be an IP address not responding (timeout error).

It seems that the issue is due to the list of nodes already being populated, but the trigger for a remapping is only MOVED errors. In our case the driver would return to redisReconnect() for the configured node and would try the same host again and again.

This situation is quite standard when using Redis clusters in autoscale mode, please consider supporting it.

Thanks!

P.S we are using a Lua wrapper for the hiredis-cluster - which uses hiredis-cluster 0.6.0.

Hello,

We are using hiredis-cluster Release 0.8.1, and this issue is with regards to a crash we see in our performance testing.

We use async hiredis APIs, and invoke them both from the application and also from hiredis callbacks.
Our application is multi-threaded, and hiredis context object & hiredis API calls are mutex protected.

We noticed the following crash a few hours into our test:

Crash reason: SIGSEGV /0x00000080
Crash address: 0x0
Process uptime: not available

Thread 13 (crashed)
0 libhiredis_cluster.so.0.8!actx_get_by_node [hircluster.c : 3630 + 0x0]
1 libhiredis_cluster.so.0.8!redisClusterAsyncFormattedCommand [hircluster.c : 4117 + 0x8]
2 libhiredis_cluster.so.0.8!redisClustervAsyncCommand [hircluster.c : 4244 + 0x16]
3 libhiredis_cluster.so.0.8!redisClusterAsyncCommand [hircluster.c : 4258 + 0x5]

Kindly help resolve this crash.

Thank you.

Although hiredis-cluster is mainly for Redis Cluster, it is useful to also support Redis standalone instances. The benefit is that the same client and application code can be used regardless of the Redis configuration.

• When connecting, we send a CLUSTER SLOTS command to figure out the slot mapping. If Redis replies with an error indicating that it's not in cluster mode, we can handle this implicitly and return a success code to the caller.
• When updating the slot mapping, if we know that we're in standalone mode, there is no point sending the CLUSTER SLOTS command to Redis. It also does no harm doing so, but it's an extra call to Redis, so this bullet point is not of the highest priority.
• We could allow a standalone option when connecting to the cluster, which is just an indication meaning that we guess that we're not in cluster mode. CLUSTER SLOTS is not sent when connecting, but if we later receive a MOVED redirect, the slot mapping is fetched and we switch to cluster mode. The default should be cluster mode though, since this is primarily a cluster client.

int redisClusterSetOptionAddNodes(redisClusterContext *cc, const char *addrs) {
int ret;
sds *address = NULL;
int address_count = 0;
int i;

if (cc == NULL) {
    return REDIS_ERR;
}

address = sdssplitlen(addrs, strlen(addrs), CLUSTER_ADDRESS_SEPARATOR,
                      strlen(CLUSTER_ADDRESS_SEPARATOR), &address_count);
if (address == NULL) {
    __redisClusterSetError(cc, REDIS_ERR_OOM, "Out of memory");
    return REDIS_ERR;
}

if (address_count <= 0) {
    __redisClusterSetError(cc, REDIS_ERR_OTHER,
                           "invalid server addresses (example format: "
                           "127.0.0.1:1234,127.0.0.2:5678)");
    sdsfreesplitres(address, address_count);
    return REDIS_ERR;
}

for (i = 0; i < address_count; i++) {
sdstrim(address[i],"\t ");  // support format "127.0.0.1:1234, 127.0.0.2:5678",  trim blank, tab
    ret = redisClusterSetOptionAddNode(cc, address[i]);
    if (ret != REDIS_OK) {
        sdsfreesplitres(address, address_count);
        return REDIS_ERR;
    }
}

sdsfreesplitres(address, address_count);

return REDIS_OK;

}

Excuse me, my application scenario is that multiple threads will call the redis api to write message. What can I do with the connection handle "redisClusterContext *cc"? Create one per thread

Hi,

We are trying to issue 'cluster info' as follows:

int status = redisClusterAsyncCommand(cc, clusterInfoCallback, (char *)"THE_ID", "CLUSTER INFO");

This returns REDIS_ERR, with cc->err set to "Parse command error. Cmd type: 0, state: 5, break position: 15."

What could be the issue here?

Thank you.

My project contains libev instead of libevent, so what can I do to modify the code in purpose of making it adapt to libev？

Hi @bjosv,

We came across a case wherein hiredis-cluster seems to be trying to connect to a stale redis-server ip while issuing commands to redis server, this connection flapping kept on happening (approx 2hrs) until one more redeploy was done for the application image (which has redis-cluster integrated).
From the server-side logs, it looks like this ip was alive earlier for some period of time in the past, but not for the previous couple of hours atleast when the issue was seen. There was an image upgrade done on the redis server nodes which could have resulted in the change of ips.

As part of the callbacks registered by application with hiredis, the below prints were seen as part of the redis command send calls.

[2023-01-12T07:47:30.263102][disconnectCallback]Error: Server closed the connection

...connectCallback print for 192.168.228.26

[2023-01-12T07:48:26.189608][disconnectCallback]Error: Timeout

...connectCallback print for 192.168.228.26

[2023-01-12T07:48:28.191117][disconnectCallbackk]Error: Timeout

...connectCallback print for 192.168.228.26

[2023-01-12T07:48:34.186333][disconnectCallback]Error: Connection reset by peer

...connectCallback print for 192.168.228.26

[2023-01-12T09:12:53.184634][disconnectCallback][Disconnected from 192.168.228.26:6379

kubectl get pods output was collected when the issue was seen, and at that point the list of ips are as captured below. 3 primary nodes and 2 replica per node (adds up to total 9 nodes) as shown below. Here, the ip which redis-cluster is trying to connect - 192.168.228.26 is not appearing.

  1 NAME                                READY   STATUS    RESTARTS   AGE    IP                NODE                                    NOMINATED NODE   READINESS GATES
  2 rediscluster-node-for-redis-9cj4g   2/2     Running   0          169m   192.168.229.229   cl-60-node-5-dcc808-4b2c09f7b029c460    <none>           <none>
  3 rediscluster-node-for-redis-9vvzf   2/2     Running   0          165m   192.168.228.4     cl-60-node-9-dcc808-4b2c09f7b029c460    <none>           <none>
  4 rediscluster-node-for-redis-h5lw6   2/2     Running   0          168m   192.168.100.63    cl-60-node-4-dcc808-4b2c09f7b029c460    <none>           <none>
  5 rediscluster-node-for-redis-mx8gs   2/2     Running   0          168m   192.168.149.14    cl-60-node-1-dcc808-4b2c09f7b029c460    <none>           <none>
  6 rediscluster-node-for-redis-qwrqg   2/2     Running   0          168m   192.168.33.99     cl-60-node-6-dcc808-4b2c09f7b029c460    <none>           <none>
  7 rediscluster-node-for-redis-slbj7   2/2     Running   0          166m   192.168.164.92    cl-60-node-3-dcc808-4b2c09f7b029c460    <none>           <none>
  8 rediscluster-node-for-redis-t92lx   2/2     Running   0          169m   192.168.100.3     cl-60-node-4-dcc808-4b2c09f7b029c460    <none>           <none>
  9 rediscluster-node-for-redis-wgbdq   2/2     Running   0          166m   192.168.41.246    cl-60-node-10-dcc808-4b2c09f7b029c460   <none>           <none>
 10 rediscluster-node-for-redis-z5dr7   2/2     Running   0          169m   192.168.124.200   cl-60-node-2-dcc808-4b2c09f7b029c460    <none>           <none>

Do you have some clue on what could be resulting in this behaviour?

I want to get all keys in redis cluster，and i use "(redisReply *)redisClusterCommand(g_redis_cluster_conn, "KEYS *")", but , this returns NULL

please help me, much THX

Background
Redis 6 introduced Access Control List that can allow certain connections to be limited in terms of the commands that can be executed and the keys that can be accessed.

The Redis AUTH command was also extended in Redis 6, to enable authentication as a specific user:

AUTH <username> <password>

Requested functionality
Currently the API only supports the legacy authentication method using password via:

int redisClusterSetOptionPassword(redisClusterContext *cc,
                                  const char *password);

API support for specifying a username is needed.

Misc
Redis 6 will internally use the username default if no username is given in AUTH

Links
https://redis.io/topics/acl
https://redis.io/commands/auth

hello,
i have a redis cluster with three master and three slave. when one master or slave node is down, function command() will return error "server closed connection". i try to set retry option but useless, is there any ways to solve it? thanks (my request way is Synchronous blocking )

Recently my Redis cluster was down (getting a CLUSTERDOWN reply), but hiredis-cluster showed the following error: too many cluster redirect.

Looking at the code, ie.

Retries can happen for other reasons as well (like CLUSTERDOWN), so I suggest to change max_redirect_count to max_retry_count and update the error messages (I assume initially there was only a retry on redirects?)

Hello,

I am having trouble understanding how to use pipelining for bulk uploading keys. As I have recently discovered, the redis-cli does not support pipelining for cluster mode, unless I can ensure that keys are pushed to the same slots. But I cannot do this as I am pushing several million keys in a bulk upload using --pipe.

I have an awk script that with cluster mode disabled works really well with the sh awk_script.sh | redis-cli --pipe command structure, because it can upload my ~2million keys (hashes - HSET) in under a second.

It seems as though the hiredis-cluster library supports pipelining in cluster mode, so my plan was to call the script from within my C/C++ code and pipeline/stream it to redisClusterAppendCommand or redisClusterAppendCommandArgv. But how would freeing the reply object sequentially work in this case, as I see in the README that after every appendCommand I will need to free the reply object? The reason I use awk is for the speed and a one-liner that can parse my .txt file with ease. Will it be possible to pipe the output of my awk script to one of the pipelining commands for redis?

Hello,

We have a test that repeatedly kills all the Master Nodes & brings the cluster back up every 5 seconds, while the application is continuously making Async API calls using redisClusterAsyncCommand()

During this we noticed the following assert:

File: /home/osboxes/hiredis-cluster/hircluster.c Line: 3694: 0 [0] /usr/local/lib/libhiredis_cluster.so.0.7(actx_get_by_node+0x69) [0x7ffff7f92cd4] [1] /usr/local/lib/libhiredis_cluster.so.0.7(redisClusterAsyncFormattedCommand+0x25e) [0x7ffff7f93963] [2] /usr/local/lib/libhiredis_cluster.so.0.7(redisClustervAsyncCommand+0xc2) [0x7ffff7f93d68] [3] /usr/local/lib/libhiredis_cluster.so.0.7(redisClusterAsyncCommand+0xca) [0x7ffff7f93e5a]

The code at the above line points to NOT_REACHED() call inside actx_get_by_node()

How do we avoid the assert here?
We understand redisClusterAsyncCommand() will fail in this case which is fine, but we would like to continue waiting till the Nodes are back up rather that exiting at this point.

Thank you.

I see redisClusterSetOptionPassword(redisClusterContext *cc, ) in hircluster.h but how this can be used with redisClusterAsyncContext ?

In async example the context is created together with connect

redisClusterAsyncContext *cc =
redisClusterAsyncConnect(nodes.c_str(), HIRCLUSTER_FLAG_NULL);
if (cc && cc->err) {
printf("Error: %s\n", cc->errstr);
return 1;
}

Is it possible to pass redisClusterContext with password to redisClusterAsyncContext before connect ?

I am looking for correct tools to compile it on Windows 10. Any help ? Tried by through cmake and following

pkg-config_0.26-1_win32.zip, gettext-runtime_0.18.1.1-2_win32.zip, glib_2.28.8-1_win32.zip for pkg-config with cmake but it still fails with following error:

D:\_redis_lib>cmake.exe -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDOWNLOAD_HIREDIS=ON ..
Detected version: 0.7.0
Downloading dependency 'hiredis'..
Detected version: 1.0.0
-- Checking for module 'glib-2.0'
--   No package 'glib-2.0' found
-- Configuring done
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
hiredis_cluster;hiredis;LIBEVENT_LIBRARY
    linked by target "ct_connection" in directory D:/_redis_lib/tests
    linked by target "ct_out_of_memory_handling" in directory D:/_redis_lib/tests
    linked by target "ct_connection_ipv6" in directory D:/_redis_lib/tests
    linked by target "ct_pipeline" in directory D:/_redis_lib/tests
    linked by target "ct_async" in directory D:/_redis_lib/tests
    linked by target "ct_specific_nodes" in directory D:/_redis_lib/tests
    linked by target "clusterclient_async" in directory D:/_redis_lib/tests
    linked by target "clusterclient_reconnect_async" in directory D:/_redis_lib/tests

CMake Error at CMakeLists.txt:132 (TARGET_LINK_LIBRARIES):
  Target "hiredis_cluster" links to:

    hiredis::hiredis

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

Will it block when the cluster node connection fails?

Hello,

Is it possible to connect to Slaves in order to perform some GET operations?
If yes, how can this be achieved? Also, will we get the connect/disconnect callbacks from the Slaves too?
Kindly advise.

Thank you.

I am trying to build "hiredis-cluster" with USE_SSL=1 to connect to an Elasticache Redis cluster and as a minimal test, I am trying to run example_tls.c with my specific cluster node/host and port number.

I am assuming it is a linking issue on my part, except that none of the other "hiredis-cluster" functions are triggering an out-of-scope. But, the crux of the issue is while compiling I get:

#27 68.99 app/src/test_tls_cluster.cpp: In function 'int main(int, char**)':
#27 68.99 app/src/test_tls_cluster.cpp:30:5: error: 'redisClusterSetOptionEnableSSL' was not declared in this scope
#27 68.99      redisClusterSetOptionEnableSSL(cc, ssl);
#27 68.99      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#27 69.00 app/src/test_tls_cluster.cpp:30:5: note: suggested alternative: 'redisClusterSetOptionMaxRetry'
#27 69.00      redisClusterSetOptionEnableSSL(cc, ssl);
#27 69.00      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#27 69.00      redisClusterSetOptionMaxRetry
#27 69.04 make: *** [makefile:39: app/bin/test_tls_cluster] Error 1
------
executor failed running [/bin/sh -c chmod u+x ./ILOG_COS_20.10_LINUX_X86_64.bin && 	./ILOG_COS_20.10_LINUX_X86_64.bin -f install.properties && 	make clean && make USE_SSL=1 && 	rm -rf ./CPLEX && 	rm -f ./ILOG_COS_20.10_LINUX_X86_64.bin && 	rm -f ./install.properties && 	rm -f ./makefile && 	rm -rf ./app/src && 	rm -rf ./hiredis && 	rm -rf /usr/include/hiredis;]: exit code: 2

Some details:
I am compiling in a docker container. The following is how I am building hiredis_cluster in the container:

... Some stuff above that works ...
RUN git clone https://github.com/redis/hiredis.git --depth 1 && \
	cd hiredis && \
	make USE_SSL=1 && make install && \
	mkdir /usr/include/hiredis && \
	cp libhiredis.so /usr/lib/ && \
	cp libhiredis_ssl.so /usr/lib && \
	cp libhiredis.a /usr/lib/ && \
	cp libhiredis_ssl.a /usr/lib && \
	cp hiredis.h /usr/include/hiredis/ && \
	cp hiredis_ssl.h /usr/include/hiredis/ && \
	cp read.h /usr/include/hiredis/ && \
	cp sds.h /usr/include/hiredis/ && \
	cp alloc.h /usr/include/hiredis/ && \
	cp async.h /usr/include/hiredis/ && \
	cp -r adapters /usr/include/hiredis/adapters/ && \
	ldconfig

WORKDIR /app

# Install hiredis-cluster dependencies:
RUN git clone https://github.com/Nordix/hiredis-cluster.git --depth 1 && \
	mkdir /usr/include/hiredis_cluster && \
	cd hiredis-cluster && \
	mkdir install && \
	make USE_SSL=1 CFLAGS="-I/usr/include -D_XOPEN_SOURCE=600" LDFLAGS="-L/usr/local/lib" clean install &&\
	cp -r /usr/local/include/hiredis_cluster/ /usr/include/hiredis_cluster/ && \
	cp /usr/local/lib/libhiredis_cluster.so /usr/lib/ && \
	cp /usr/local/lib/libhiredis_cluster.a /usr/lib/ && \
	ldconfig
... Some stuff below that installs other packages that another program requires ...

I then call my specific make file to build test_tls_cluster.cpp which is literally just the example_tls.c script from the repo with my specific Elasticache Server and a path pointing to a ca-certificate.crt file. The make file builds the binary from the source files via the g++ compiler as shown above in the error message:
g++ -std=c++17 app/src/test_tls_cluster.cpp -I./app/include -I/usr/include -L/usr/lib/libhiredis_cluster.a -L/usr/local/lib/libhiredis.a -L/usr/local/lib/libhiredis_ssl.a -lm -lpthread -o app/bin/test_tls_cluster

Can anyone tell me what I am doing wrong?

In some special occasions, we hope that redis can be partitioned.
So, does hiredis cluster support select command after updating the slot mapping or connecting to the cluster?

when run example_ipv4
seems can't connect redis server
redis-sever vesion 7.0

my code:
redisClusterContext *cc = redisClusterContextInit();
redisClusterSetOptionAddNodes(cc, "127.0.0.1:30001");
redisClusterSetOptionConnectTimeout(cc, timeout);
redisClusterSetOptionRouteUseSlots(cc);
redisClusterConnect2(cc);
if (cc && cc->err) {
printf("Error: %s\n", cc->errstr);
// handle error
exit(-1);
}

Hi,

We have a test where multiple clients are connected to the cluster and continuously performing CRUD operations on a set of keys.

When we restart a Redis node, we notice that some clients re-connect to the node within seconds after the node has come back up.

Some other clients on the other hand take up to 5 minutes to re-connect.
On these nodes, we see that we keep receiving connect callbacks with status being REDIS_ERR, and errstr of redisAsyncContext says "Connection timed out".
The connection eventually gets re-established.

Kindly help debug the issue.

Thank you.

By running a static analyzer tool during CI we could find dead code, faulty initiations and more.
Options might be Coverity or Clang static analyzer, or other.

We are running Redis server ver 6.2.7 with hiredis ver 1.1.0 & hiredis-cluster ver 0.9

1] In our existing load test we create a single async context (redisClusterAsyncContext *asyncCtx) using which we perform SET/GET operations on a 3 Node cluster. This test works well. We have performed multiple long duration runs with this test on > 1 million keys without any issues.

2] For testing pub/sub, we have now created another connection with a new context: redisClusterAsyncContext *asyncCtxPS:

asyncCtxPS = redisClusterAsyncContextInit();
redisClusterSetOptionAddNodes(asyncCtxPS->cc, cluster_ip);
redisClusterConnect2(asyncCtxPS->cc);

As connections with Redis Nodes get established as a result of the key operations performed in step 1 above, we trigger subscription in the connect callback:

void connectCallback(const redisAsyncContext *ac, int status) {
nodeIterator ni;
cluster_node *clusterNode;
....
....
initNodeIterator(&ni, asyncCtx->cc);
while ((clusterNode = nodeNext(&ni)) != NULL) {
if (ac == clusterNode->acon) {
redisClusterAsyncCommandToNode(asyncCtx, clusterNode, configCb, NULL, "CONFIG SET notify-keyspace-events Eh");
redisClusterAsyncCommandToNode(asyncCtxPS, clusterNode, eventCb, NULL, "PSUBSCRIBE __key*__:*");
....
....

At present, the above two pub/sub commands are invoked only on the first connected Node, not on all 3 Nodes.

void eventCb(redisClusterAsyncContext *cc, void *r, void *privdata) {
redisReply *reply = (redisReply *)r;

if (r == NULL) {
return;
}

if (reply->type == REDIS_REPLY_ARRAY) {
for (int j = 0; j < reply->elements; j++) {
printf("%u) %s\n", j, reply->element[j]->str);
}
}
}

In response, we see the following in the callback:

0) psubscribe
1) __key*__:*
2) (null)

(null) here doesn't seem correct. Subscription is successful if we run the above PSUBSCRIBE directly on redis-cli. What could be the issue here?

A couple of seconds later, we notice a crash:

#0 0x00007ffff7fa70d3 in redisClusterAsyncCallback (ac=0x5d94d0, r=0x0, privdata=0x5da940)
at /....../hircluster.c:3794
#1 0x00007ffff7fb45f2 in __redisRunCallback (ac=0x5d94d0, cb=0x5da9a0, reply=0x0)
at /....../async.c:311
#2 0x00007ffff7fb4979 in __redisAsyncFree (ac=0x5d94d0)
at /....../async.c:386
#3 0x00007ffff7fb4bbf in __redisAsyncDisconnect (ac=0x5d94d0)
at /....../async.c:450
#4 0x00007ffff7fb52b2 in redisProcessCallbacks (ac=0x5d94d0)
at /....../async.c:625
#5 0x00007ffff7fb560a in redisAsyncRead (ac=0x5d94d0)
at /....../async.c:717
#6 0x00007ffff7fb569b in redisAsyncHandleRead (ac=0x5d94d0)
at /....../async.c:738
#7 0x00000000004eb13d in redisLibmyeventHandlerRead (fd=<optimized out>, arg=0x5d7de0)
at /....../hiredis_libmyevent.h:27

Line 625 of async.c is:

605 if (__redisShiftCallback(&ac->replies,&cb) != REDIS_OK) {
606 /*
607 * A spontaneous reply in a not-subscribed context can be the error
608 * reply that is sent when a new connection exceeds the maximum
609 * number of allowed connections on the server side.
610 *
611 * This is seen as an error instead of a regular reply because the
612 * server closes the connection after sending it.
613 *
614 * To prevent the error from being overwritten by an EOF error the
615 * connection is closed here. See issue #43.
616 *
617 * Another possibility is that the server is loading its dataset.
618 * In this case we also want to close the connection, and have the
619 * user wait until the server is ready to take our request.
620 */
621 if (((redisReply*)reply)->type == REDIS_REPLY_ERROR) {
622 c->err = REDIS_ERR_OTHER;
623 snprintf(c->errstr,sizeof(c->errstr),"%s",((redisReply*)reply)->str);
624 c->reader->fn->freeObject(reply);
625 __redisAsyncDisconnect(ac);
626 return;
627 }

If we disable pub/sub in step 2 above, the test runs well again.
Please help debug the issue. Let us know if any addition information is required.

Thank you.

Thanks.

Hello,

We have a 3-Node cluster on which we perform CRUD operations from our application using async hiredis-cluster API calls.

If we add a 4th Master node to this cluster while this test is still running, we start seeing redisClusterAsyncCommand failures for some of the new commands, and callbacks with NULL replies for some of commands already invoked for which we are waiting for replies.

It could be that hiredis-cluster is not in sync with the redistributed hash slots after addition of the new Master.

Kindly let us know if this scenario is supported, and works fine for you.

Thank you for the support.

Hi there,

I finished installing the hiredis-cluster with the instructions. Now I am running the given example code(https://github.com/Nordix/hiredis-cluster/tree/master/examples/src)/example.c) to connect my redis cluster, but it always failed during gcc with errors. Does anyone have meet this issue? Thank you!

notls.c:(.text+0x20): undefined reference to redisClusterContextInit'
notls.c:(.text+0x37): undefined reference to redisClusterSetOptionAddNodes' notls.c:(.text+0x4e): undefined reference to redisClusterSetOptionConnectTimeout'
notls.c:(.text+0x5a): undefined reference to redisClusterSetOptionRouteUseSlots' notls.c:(.text+0x66): undefined reference to redisClusterConnect2'
notls.c:(.text+0xc3): undefined reference to redisClusterCommand' notls.c:(.text+0xef): undefined reference to freeReplyObject'
notls.c:(.text+0x10e): undefined reference to redisClusterCommand' notls.c:(.text+0x13a): undefined reference to freeReplyObject'
notls.c:(.text+0x146): undefined reference to redisClusterFree'

Regards

cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_SSL=OFF ..

-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Detected version: 0.8.1
Downloading dependency 'hiredis'..
Scanning dependencies of target hiredis-populate
[ 11%] Creating directories for 'hiredis-populate'
[ 22%] Performing download step (git clone) for 'hiredis-populate'
Cloning into 'hiredis'...
Note: switching to 'v1.1.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at c14775b Prepare for v1.1.0 GA
[ 33%] No patch step for 'hiredis-populate'
[ 44%] Performing update step for 'hiredis-populate'
fatal: invalid upstream 'origin/v1.1.0'
fatal: No rebase in progress?
CMake Error at /root/hiredis-cluster/build/_deps/hiredis-subbuild/hiredis-populate-prefix/tmp/hiredis-populate-gitupdate.cmake:102 (message):

Failed to rebase in: '/root/hiredis-cluster/build/_deps/hiredis/'.

You will have to resolve the conflicts manually

make[2]: *** [CMakeFiles/hiredis-populate.dir/build.make:97: hiredis-populate-prefix/src/hiredis-populate-stamp/hiredis-populate-update] Error 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/hiredis-populate.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

CMake Error at /usr/share/cmake-3.16/Modules/FetchContent.cmake:915 (message):
Build step for hiredis failed: 2
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FetchContent.cmake:1006 (__FetchContent_directPopulate)
CMakeLists.txt:91 (FetchContent_Populate)

-- Configuring incomplete, errors occurred!
See also "/root/hiredis-cluster/build/CMakeFiles/CMakeOutput.log".

Does anybody know, why I'm having this dependancy problem while trying to compile library from a source?

Does hiredis-cluster fix this bug？

hiredis-vip : master
date : 2019-11-25
hiredis version in hircluster.h : 1.0.0

In my code, I got a redisClusterContext,
I called
struct timeval tv = { _timeout_ms / 1000, (_timeout_ms % 1000) * 1000};
_context = redisClusterConnectWithTimeout(_host.c_str(), tv, HIRCLUSTER_FLAG_NULL);
if (redisClusterSetOptionTimeout(_context,tv) != REDIS_OK) { /* log on error */ }

Why it won't work, if I set _timeout_ms=10 , my redis command timer still get a number bigger than 10, say 29 maybe...

I was trying to use redisClusterCommandArgv to get data from redis cluster.
Both "set, get, del" work well, but failed for "keys" like "keys xxxxxxxx:*".
Is "keys" command unsupported?

Kindly help with the following queries:

1. Is there a way to figure out the number of master nodes at any given point?
2. Is there a way to figure out the number of active nodes (master + replica) nodes at any given point?

Thank you.

Hi!

We're looking into using hiredis-cluster in our application, but we're unclear on the copyright on a few files without license header (command.c, hiarray.c, hircluster.c, hiutil.c, win32.h and their headers).

Based on the COPYING file in the root folder we believe it's BSD 3-Clause. Could we have this clarified by adding a license header to all files? Some files already have a license header (adlist.c, crc16.c etc.).

Thank you!

Hey, guys.
I want to use this in nginx for redis cluster support. But right now, nginx's event is not supported in the adpater list.

Just a minor issue I noticed while building the project. It's supposed to be version 0.6.0, but it's building as version 0.5.0. It looks like CMakeLists.txt is getting the version from the preprocessor directives in hicluster.h. They should be updated to:

#define HIREDIS_CLUSTER_MAJOR 0
#define HIREDIS_CLUSTER_MINOR 6
#define HIREDIS_CLUSTER_PATCH 0
#define HIREDIS_CLUSTER_SONAME 0.6

I want to chose database 1 instead of databse 0. It seems i need to call "select" command for every slots

We have noticed that redisClusterAsyncConnect() blocks if the server is unavailable/unreachable. We understand that the timeout option is available for commands only after context creation, but is there a way to make redisClusterAsyncConnect() return error if the connection creation has not succeeded after a certain amount of time?

Thank you.

Hi

would it be possible to add support for the commands I've listed in the title ?

Thanks

There's no current way to support the use case where we want to have support for -ASK or -MOVED with the BITFIELD command. The sole option is to send the BITFIELD command to a single node having the knowledge where this specific key resides (as explained in #50 ). This forces the client to have a sub-optimal flow when using the command.

As the BITFIELD command will always apply to a single key (single slot), there is also no need to parse the sub-commands. Also, redis executes all bitfield sub-commands in the same execution (see https://github.com/redis/redis/blob/unstable/src/bitops.c#L1136)

Optionally, we can support the BITFIELD_RO as a bonus because the only difference is that it supports the GET command.

We are using hiredis-cluster/hiredis C client library to access a Redis server (version 5.0.7)
It has been observed when the library makes an async connection to the server, the MULTI command is failing (the library returns a NULL reply pointer)
However, when the library makes a synchronous connection to the server, the MULTI command works as expected.
We need to understand if it is a known paradigm that MULTI will not work on asynchronous connections with Redis server or if this might be a limitation of the hiredis-cluster/hiredis library itself.

We used the following --

redisClusterAsyncCommand(context, callback, client_data, "MULTI");

In the callback
void callback(redisClusterAsyncContext *cc, void *r, void *client_data)

redisReply *reply = (redisReply *)r; <--- this comes as NULL

Thanks for the help.

i use hiredis cluster like this

ctx = redisClusterContextInit();
redisClusterSetOptionAddNodes(ctx, redis_url.c_str());
if (!password.empty()) {
redisClusterSetOptionPassword(ctx, password.c_str());
}
redisClusterSetOptionRouteUseSlots(ctx);
redisClusterConnect2(ctx);`

my question is redisClusterAsyncContext have redisClusterAsyncSetDisconnectCallback process dissconnect event, but redisClusterContext not have correspond method, does it means when i use redisClusterContext, it automatic processing dissconnect?