hmartiro / redox Goto Github PK

Can get segfaults if the server goes down while running commands. Get back to the previous behavior, where the disconnect callback is invoked.

Issue:

[ 41%] Built target redox
[ 50%] Linking CXX executable pub_sub
/usr/bin/ld: libredox.so.0.3.0: undefined reference to `hi_malloc(unsigned long)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/pub_sub.dir/build.make:87: pub_sub] Error 1
make[1]: *** [CMakeFiles/Makefile2:80: CMakeFiles/pub_sub.dir/all] Error 2
make: *** [Makefile:152: all] Error 2
Build complete (4 errors, 0 warnings): /home/laxman/eclipse-workspace/redox/build/default

Hi, I am trying to integrate redox pubsub into my Qt GUI application, for publishing financial tick/depth data. I simply connect to both publisher and subscriber.To test redox, I subscribe to a topic with a callback function to handle message received, which simply append the message received to a QPlainTextEdit. And then I publish a message. What happens is that the QPlainTextEdit is not refreshing. Only when I move my mouse or resize the GUI window or click other application window, the message then is displayed in the QPlainTextEdit widget. Looks like by calling publish(), I am blocking the Qt eventloop. Please kindly help. Do I need to use redox pubsub in a seperate Thread? and how? thx

Hello there. Please add this library to conan center. To easy install in projects.

https://conan.io/center/

Thanks.

Create with CPack

Output of the Loop test:

[ RUN ] RedoxTest.Loop
[ASYNC] GET redox_test:a: 18
/home/bramv/src/redox/test/test.cpp:75: Failure
Value of: c.reply()
Actual: "18"
Expected: value
Which is: "20"
[ FAILED ] RedoxTest.Loop (116 ms)

On a lot of HGET request, I expect my response to be nil. Unfortunately, when that happens, I get a log message such as:

(2016-02.05 13:38:55) [Warning] HGET 37108 8: Nil reply.

The homepage of redox says that the logs can be managed and redirected at will, but there is no documentation showing how to do so.

Maybe I'm not supposed to do an HGET if I know the response might be nil, but in an case I'm developing a C++ web server, and I need to be able to redirect redox's (or hiredis ?) logs to whatever stream the server is using.

Maybe there should be an example on how to do that on the project's homepage.

I am trying to benchmark Redis using Redox by executing hundreds of thousands of consecutive GETs; however, after an arbitrary amount (usually in the 10k-100k range) of GETs, redox blocks indefinitely on one. That is, some arbitrary get never returns to the calling function, even though the ones before it did successfully. And how many before this block occurs varies every time I run the program.

I'm convinced that the problem lies with Redox and not with my Redis setup since I was able to replace code calling the Redox API with corresponding calls to the Hiredis API and it then worked fine (didn't block at any point), but I might be wrong in what the cause is.

I also encountered the same problem trying to mass-insert KV pairs into Redis using Redox's SET; however, Redis docs specifically mention that the recommended way is using this instead, so it's not as big of an issue.

My setup details: I'm using Redis with an LRU caching policy, no persistence/snapshotting, 28G memory allocated to Redis, and a data set of size ~8G, with each key of size 1000B. All data is loaded into memory prior to the process execution, so it never goes to disk.

I'm trying to perform authentication via the AUTH command. This works just fine through Redox but I can't do the same for Subscriber since it doesn't allow command execution.

Are there any plans to add this? For now I just forked the code and added a getter for the internal Redox inside the subscriber but it would be really nice if the upstream codebase supported this. Especially since I don't think it's such an odd request to be able to authenticate before subscribing to channels.

I can create a PR for this, just let me know how you would like to achieve this (exposing the Redox might not be so nice)

Hi,

I was able to compile my application cpp but when I tried to link, I have the below error:

/tmp/ccIM35WN.o: In function main': MyApp.cpp:(.text+0x47f): undefined reference toredox::Redox::connect(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, std::function<void (int)>)'
collect2: error: ld returned 1 exit status

Thanks,
Mike

I've opened the issue at redis/hiredis#602 but it might be related to redox.

So that we can have debug/trace log messages that are included at compile time only when compiled in Debug mode. Right now, we can set the log level but having arguments of debug and trace messages evaluated slows down the system.

Hi

I'd like to know how to handle reconnection with your library?

I'm currenlty setting the connection handler and I'd like to try to reconnect to the redis server in case of unexpected disconnection.

For that, I've tried to first call redox::disconnect() and then redox::connect, but the program abort in the redox::connect because the event loop thread has not been joined (when the event_loop thread is reset in the redox::connect, the previous thread is destroyed without having being joined before).

But even if I join the thread in the disconnect method, my program crashes with a segfault.
Here is a backtrace:

[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace:66): segmentation fault
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 1 ecctv_server 0x000000010829e670 sigcrash_handler(int) + 16
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 2 libsystem_platform.dylib 0x00007fff87f34f1a _sigtramp + 26
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 3 ??? 0x0a007fd46af0cdc8 0x0 + 720716490683239880
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 4 libc++.1.dylib 0x00007fff83bfbb6f _ZNSt3__15mutex4lockEv + 9
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 5 libredox.0.dylib 0x000000010871cf4f redox::Command<redisReply*>::free() + 31
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 6 libredox.0.dylib 0x000000010872b069 redox::Subscriber::stop() + 777
[ERR ] [2015-07-10 - 11:19:17] (ecctv_server::log_backtrace_line:56): 7 libredox.0.dylib 0x000000010872ad4e redox::Subscriber::disconnect() + 14

What shall we do if we want to reconnect? Build a new Redox object? Or calling specific methods that does the work?

Sorry for opening an issue for this, I did not find other means of contacting you.

I would like to invite your community to write a review for an alternative Redis client library that is being proposed for inclusion in boost. As the author of a redis client yourself your opinion would be highly valuable. For more information see https://lists.boost.org/Archives/boost/2023/01/253871.php.

Thanks

Can this lib be built on and for Windows? Either way, you should specify it explicitly in the description to avoid this question in the future :)

Hello sir, what is the maximum number of key, value pairs that can be stored in redox database? Is this similar to redis? And how do i check if given value exists for given key? right now when value is not present it shows errror: [FATAL] Error getting key 5000000: Status code 1

Need autogenerated documentation.

Hi,

I am using Redis with authentication so doing this:
rdx.command({"auth", "mypassword"}, [](Command& c) {
if(!c.ok()) return;
cout << "Auth is done." << endl;
});

And seeing this error message:
(2015-12.10 22:26:02) [Error] auth mypassword: Received reply of type 5, expected type 3.

in spite of this error, the next commands are processed fine.

Could you explain what is going on?

Regards,

Ian

When we get an error reply, hiredis gives us a string. Provide this string as a parameter c.err() or c.error(). Also stick in our own messages on WRONG_TYPE, NIL_REPLY, etc.

Unclear why, build with debug symbols and inspect.

Hi,

I used this trick for the reconnection bug.

I leaked on 'ev_realloc()' when I trying a connection/reconnection.

Thanks,
Myushu

Basic methods for sets, hashes, etc

I have just build the redox package . Now how do i link the headers and libraries in cmakelist.txt file?

cmake_minimum_required(VERSION 2.6 FATAL_ERROR)

project(pcl_visualizer_viewports)

find_package(PCL 1.2 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS} /home/rob/redox/include/)
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

add_executable (pcl_visualizer_demo pcl_visualizer_demo.cpp)
target_link_libraries (pcl_visualizer_demo ${PCL_LIBRARIES} /home/rob/redox/build/)

the error comming is:

/usr/bin/ld: CMakeFiles/pcl_visualizer_demo.dir/pcl_visualizer_demo.cpp.o: undefined reference to symbol 'redisAsyncHandleWrite'
//usr/lib/x86_64-linux-gnu/libhiredis.so.0.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
CMakeFiles/pcl_visualizer_demo.dir/build.make:367: recipe for target 'pcl_visualizer_demo' failed
make[2]: *** [pcl_visualizer_demo] Error 1

Either require specific version, or just submodule it. It's a small library, might just be worth submoduling. Would it be smart to do the same with libev?

I have a single program that SET hundreds of keys each second.

Is calling rdx.set(key, value) for each key OK or should I use the MULTI Redis command?

In the background, is redox bulking the commands?

Hi,

I am seeing memory problems with repeated invocations of the SMEMBERS command.

I have a program that registers to a topic and then periodically runs commandSync, conceptually similarly to this:

...
mySubscriber.subscribe("__keyevent@"+redis_database+"__:expired", parse_expire, subscribed, unsubscribed);
....
auto parse_expire = [](const string& topic, const string& msg) {
Command<std::vector<std::string>>& c = rdx.commandSync<std::vector<std::string>>({"SMEMBERS", "key"});
c.free();
}

If I run the code without the SMEMBERS call, after several thousand messages it has a resident set size of about 4MB.

However, with SMEMBERS it blows to over 100 MBs. I did not find a way to force redox to deallocate this unused memory.

If I replace commandSynch with command and remove c.free() the situation just gets worse.

Is there any workaround?

Right now, the event loop calls ev_run(loop, EVRUN_NOWAIT) in a loop to maximize throughput. However, this hogs CPU on the event loop thread. If we just call ev_run(loop), such that the loop is internal to libev, throughput slows down up to 5x. Maybe use a CV to run the loop slower until we have a lot of commands to process.

I am getting the following errors when building on Mac OSX El Capitan 10.11 from master:

.../redox/include/redox/client.hpp:390:56: warning: dependent nested name specifier 'Command<ReplyT>::' for friend class declaration is not supported; turning off access control for 'Redox' [-Wunsupported-friend]
  template <class ReplyT> friend void Command<ReplyT>::free();
                                      ~~~~~~~~~~~~~~~~~^
.../redox/include/redox/client.hpp:393:56: warning: dependent nested name specifier 'Command<ReplyT>::' for friend class declaration is not supported; turning off access control for 'Redox' [-Wunsupported-friend]
  template <class ReplyT> friend void Command<ReplyT>::processReply(redisReply *r);
                                      ~~~~~~~~~~~~~~~~~^
In file included from .../redox/src/subscriber.cpp:22:
In file included from .../redox/include/redox/subscriber.hpp:23:
.../redox/include/redox/client.hpp:390:56: warning: dependent nested name specifier 'Command<ReplyT>::' for friend class declaration is not supported; turning off access control for 'Redox' [-Wunsupported-friend]
  template <class ReplyT> friend void Command<ReplyT>::free();
                                      ~~~~~~~~~~~~~~~~~^
.../redox/include/redox/client.hpp:393:56: warning: dependent nested name specifier 'Command<ReplyT>::' for friend class declaration is not supported; turning off access control for 'Redox' [-Wunsupported-friend]
  template <class ReplyT> friend void Command<ReplyT>::processReply(redisReply *r);

Is there a config file where we can edit the logging level? Or alternatively is there a way to do so programmatically? This seems like the relevant class.

Right now, there is an explicit pause in the disconnect() method of Subscriber. It seems hiredis requires us to unsubscribe from everything and have no more pending commands, or it gives this type of error:

speed_test_pubsub: async.c:289: __redisAsyncDisconnect: Assertion__redisShiftCallback(&ac->replies,((void *)0)) == -1' failed.`

Contact hiredis guys to see a nicer way around this? Maybe have a cv that waits until there are no more pending hiredis requests, or cancel them somehow.

I'm assuming this variation shouldn't be a problem, but if I were to execute:

rdx.command<string>({"multi"});
rdx.command<string>({"SET", "number", "8"});
rdx.command<string>({"publish", "info", "number_updates"});
rdx.command<string>({"exec"});

• would it execute as an atomic transaction with proper ordering?
• If my string message is large (5 - 10 MB) would it cause any problems, I mean regarding performance and correctness? (by performance I'm only concerned if it has extra overhead compared to hiredis)

I've tried to get the SCAN command to work, but ran into a problem that the redis reply is not processed correctly.

According to the Redis documentation:

SCAN, SSCAN, HSCAN and ZSCAN return a two elements multi-bulk reply, where the first element is a string representing an unsigned 64 bit number (the cursor), and the second element is a multi-bulk with an array of elements.

When using redox, the only result I get back is the first element (cursor). While debugging, I noticed that the reader in the redis context does have the entire result (including the 2nd part), so this doesn't seem to be a hiredis issue.

I need some help, i get this error after I compile the "hello world" example with the given command: g++ hello.cpp -o hello -std=c++11 -lredox -lev -lhiredis . So the compilation works good, without errors, but the execution gives me this error: "error while loading shared libraries: libredox.so.0: cannot open shared object file: No such file or directory"

NOTE: Please try not to be mean when you explain to me the problem, I'm kind of a noob with those things. I am thinking that I didn't do the installation correctly.

Right now, there is support for binary data as a value when explicitly quoted as the last parameter of a string. There should be a method of supplying arbitrary binary arguments. Maybe an overload of command() that takes a vector of strings?