azuqua / fred.rs Goto Github PK

A Redis Client for Rust based on Futures and Tokio

License: MIT License

Rust 99.76% Shell 0.24%

rust tokio futures-rs redis azuqua-io-role-library async cluster

fred.rs's Issues

Implement `Error` trait for `RedisError`

Hi! Thanks for creating this library, it has a really nice interface. I was wondering whether it would be possible to implement the std::error::Error trait for RedisError? This would mean that the ? operator could be used in functions returning Result<T, Box<dyn Error>>.

std::string::String not the right choice for RedisKey and RedisValue?

From https://doc.rust-lang.org/std/string/struct.String.html
A UTF-8 encoded, growable string.

From https://redis.io/topics/data-types-intro
Redis keys are binary safe, this means that you can use any binary sequence as a key, from a string like "foo" to the content of a JPEG file.

I'd really like to use this crate, but since RedisKey is represented as String and RedisValue as either i64 or String, one can't really safely set arbitrary binary data, or am I mistaken here? Sure, I could always do some String::from_utf8_unchecked magic, but it's a) unsafe and b) not nice having to do this.

Imo, the String representation of RedisKey and RedisValue should always be [u8] or rather Vec<u8> internally (or std::ffi::CString if that's more convenient), as Redis makes no statement about UTF-8 encoding, which is inherent in Rusts String type (see above). Therefore, using a type with such implications (UTF-8-encoded) and overhead (when trying to convert [u8] to String, if even possible) doesn't really make sense and is imo a limiting factor for a Redis client. Or maybe there is a plausible reason that I just don't see. :)

Nonetheless, not being able to use arbitrary binary data makes this crate unusable for me. Curiously enough, redis-protocol crate (same author, used by this crate) does use [u8] and Vec<u8> for bulk strings.

9.0.1 In sentinel mode with resp3 : ERR: Protocol Error: Cannot start a stream while already inside a stream.#

I use kvrocks 2.8.0 & redis7.2-sentinel with freds 9.0.1 use resp3

When I develop on my local machine with a single instance, everything is fine. When I deploy online with sentinel mode, the access will report the following error (not every time, but often)

I think the authentication is wrong, but the question is how do I set the redis password when I use ServerConfig::Sentinel? (I have seted the password for Sentinel)

❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#
❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#
❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#
❯ systemctl restart api
❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#

the api log (this first line is api restarted)


09:01:45 api.sh[18289]:   INFO t3::srv: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/t3-0.1.9/src/srv.rs:10: http://[::]:8850


09:02:20 api.sh[18289]:   WARN fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]:   WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.24
2.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set
09:02:20 api.sh[18289]:   WARN fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]:   WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.24
2.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set
09:02:20 api.sh[18289]:   WARN fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]:  ERROR re::err: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/re-0.1.5/src/err.rs:28: Protocol Error: Cannot start a stream while already
 inside a stream.
09:02:20 api.sh[18289]:   WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.242.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set

there are two error

fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]: WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.242.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set

The relevant code is as follows:

https://github.com/i18n-api/mod.pub/blob/main/ping/src/kvrocks.rs

use r::fred::interfaces::KeysInterface;
use t3::IntoResponse;

pub async fn get() -> re::Result<impl IntoResponse> {
  let r: Option<String> = r::R.get("").await?;
  Ok(r.unwrap_or("Kvrocks".into()))
}

I wrap freds in xkv https://docs.rs/crate/xkv/0.1.26/source/Cargo.toml

 pub fn sentinel(
    service_name: impl Into<String>,
    hosts: Vec<fred::types::Server>,
    username: Option<String>,
    password: Option<String>,
  ) -> ServerConfig {
    ServerConfig::Sentinel {
      service_name: service_name.into(),
      hosts,
      username: Some(username.unwrap_or_else(|| "default".into())),
      password,
    }
  }

Support for PSUBSCRIBE/PUNSUBSCRIBE

Would you consider adding support for PSUBSCRIBE/PUNSUBSCRIBE commands?

Details at: https://redis.io/topics/pubsub

Migrate from tokio-core to tokio.

Hello!
As i understand tokio-core are deprecated in favor of tokio.
At this moment i can't integrate tokio-core libraries into tokio based frameworks, like actix.
Did you have a plans for migration to tokio from tokio-core?

GEO related commands

Hello, I can see that there is not GEO related commands implemented, such as GEOADD, GEODIST and so on.

I need those to my use-case, can you help? It is hard to implement?

Support for evaluate scripts

Are there any plans for supporting commands EVAL/EVALSHA?

How to close the redis tcp connection manually

my code has so many connection, i want to close tcp connection manually

[SECURITY] Bumping pretty_env_logger dependency

pretty_env_logger 0.4.0 depends on env_logger 0.7.1, which depends on atty.

atty has a long-standing security problem with unaligned reads and has not been fixed, it is likely abandoned. This is a long-standing issue.

Bumping pretty_env_logger to 0.5.0 will resolve the security problem.

Disable CLIENT SETNAME command on connection init

Is there a way to disable CLIENT SETNAME being sent upon connection?

This isn't supported in twemproxy.

Support for pipelining

Does it support pipeline more when connected to multi-shard cluster?

`close` commands on `sync::borrowed::RedisClientRemote` and `sync::owned::RedisClientRemote`

To kill the underlying channels.

Redis array response is wrongly translated

I have implemented a Redis module that made a custom command available in Redis. This command returns an array with a single value of type integer.

When I run the command on redis-cli, It is correctly displayed as an array with a single element.

127.0.0.1:6379> udsf.delrec recordID1 RealmID1 StorageID
1) (integer) 0
127.0.0.1:6379>

However, when executing this command through fred, RedisValue received is of type RedisValue::Integer (and not RedisValue::Array). It works when an array contains more than one element.

It seems like fred is translating an array with only one element to a basic type. Since this is a common response format that is being shared across multiple commands (with some commands filling in more than one element), this is breaking the application parsing of response.

Broken Parsing of Nested Arrays from pipeline or Lua script

I was hoping to optimize a certain number of serial ZINTER calls to the Redis server by calling creating them in a pipeline and then subsequently parsing into a Vec<Vec<String>>. However the final result seems to always make each individual String into it's own Vec as opposed to having each ZINTER result as a Vec. Below is an example

use fred::prelude::*

// given the following SETs/ZSETs in Redis
// - Scores and SET types are irrelevant as this works for both unordered and ordered sets
// - Also shows up even when returning a nested table from a lua script and trying to parse into a Vec<Vec<_>>
// - Not certain if this parsing failure also occurs with other methods like SMEMBER/SINTER but I suspect it does
// a - { "hello", "world", "I", "am", "broken" }
// b - {"world", "I", "not"}
// c - {"hello", "am", "other"}

let compare_key = "a";
let other_keys = vec!["b", "c"];
let pool = RedisPool::new(...)?;
let pipeline = pool.pipeline();
for key in other_keys() {
    pipeline
        .zinter<Vec<String>, _, _>(vec![compare_key, key], None, None, false)
        .await?;
}
let result: Vec<Vec<String>> = pipeline.all().await?;
println!("{:?}", &result);
// Result ends up being something like -> [  [ "world" ],   [ "I" ],   [ "hello" ],   [ "am" ]  ]
//  as opposed to -> [  [  "world", "I"  ], [  "hello", "am" ]  ]

What is the best way to increase the `default_broadcast_channel_capacity`

I am noticing when subscribing to pubsub channels with a "bursty" volume of data I often see Lagged errors. What is the best way to increase the default capacity of 32?

Can't resume automatically after redis cluster reboot

I encountered a problem when using fred. I use the fred cluster mode. If the cluster redis restarts, fred cannot recover automatically. How to solve this problem.

[WARN] [2023.04.04 01:13:08.914] [127.0.0.1] [fred::multiplexer::utils] [/root/.cargo/registry/src/github.com-1ecc6299db9ec823/fred-5.2.0/src/multiplexer/utils.rs(1229)] - fred-mbn8GkCsjF: Error creating or using backchannel for cluster nodes: Redis Error - kind: IO, details: Os { code: 110, kind: TimedOut, message: "Connection timed out" }

let config = RedisConfig {
server: ServerConfig::Clustered {
hosts: nodes,
},

    ..Default::default()

};

0.2.4 fails to build with "--features enable-tls"

Recent release of 0.2.4 fails to build when using enable-tls feature.

I've tried with only this feature specified, as well as all features, but no dice.

cargo build --features enable-tls results in the following errors plus numerous warnings...

Additionally, version 0.2.3 is no longer available via crates.io...

error[E0308]: match arms have incompatible types
   --> src/multiplexer/mod.rs:188:58
    |
188 |           let (multiplexer, multiplexer_ft, commands_ft) = match transports {
    |  __________________________________________________________^
189 | |           Either::A(transports) => {
190 | |             let multiplexer = Multiplexer::new(
191 | |               mult_config.clone(),
...   |
242 | |           }
243 | |         };
    | |_________^ expected struct `tokio_tls::TlsStream`, found struct `tokio_core::net::TcpStream`
    |
    = note: expected type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
               found type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
note: match arm with an incompatible type
   --> src/multiplexer/mod.rs:216:36
    |
216 |             Either::B(transports) => {
    |  ____________________________________^
217 | |             let multiplexer = Multiplexer::new(
218 | |               mult_config.clone(),
219 | |               mult_message_tx.clone(),
...   |
241 | |             (multiplexer, multiplexer_ft, commands_ft)
242 | |           }
    | |___________^

error[E0308]: match arms have incompatible types
   --> src/multiplexer/mod.rs:357:54
    |
357 |       let (multiplexer, multiplexer_ft, commands_ft) = match transport {
    |  ______________________________________________________^
358 | |       Either::A((redis_sink, redis_stream)) => {
359 | |         let multiplexer = Multiplexer::new(
360 | |           config.clone(),
...   |
401 | |       }
402 | |     };
    | |_____^ expected struct `tokio_tls::TlsStream`, found struct `tokio_core::net::TcpStream`
    |
    = note: expected type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
               found type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
note: match arm with an incompatible type
   --> src/multiplexer/mod.rs:380:48
    |
380 |         Either::B((redis_sink, redis_stream)) => {
    |  ________________________________________________^
381 | |         let multiplexer = Multiplexer::new(
382 | |           config.clone(),
383 | |           message_tx.clone(),
...   |
400 | |         (multiplexer, multiplexer_ft, commands_ft)
401 | |       }
    | |_______^

Support async/await in redis cluster

Instead of returning future , its good to achieve the things via async and await.

azuqua / fred.rs Goto Github PK

fred.rs's Issues

Recommend Projects

Recommend Topics

Recommend Org