Giter Club home page Giter Club logo

fred.rs's Introduction

Notice

This repository has moved to a new location. The crates.io location will not change, so nothing needs to change in your toml if you're using that.

The newest version is a complete rewrite on async/await, new tokio, etc. It also includes a number of new features and breaking changes to the API. See the new repository for more information.

Changes to the 2.x major version will be made here, if necessary, but all other development will happen in the new repository. When the 2.x major version is fully deprecated this repository will be archived.

Fred

LICENSE Build Status Crates.io API docs

A Redis client for Rust based on Futures and Tokio that supports PubSub commands, clustered Redis deployments, and more.

Install

With cargo edit.

cargo add fred

Features

  • Supports clustered Redis deployments.
  • Optional built-in reconnection logic with multiple backoff policies.
  • Publish-Subscribe interface.
  • Supports ElastiCache, including TLS support.
  • Gracefully handle live cluster rebalancing operations.
  • Flexible interfaces for different use cases.
  • Supports various scanning functions.
  • Automatically retry requests under bad network conditions.
  • Built-in tracking for network latency and payload size metrics.
  • Built-in mocking layer for running tests without a Redis server.
  • A client pooling interface to round-robin requests among a pool of connections.

Example

extern crate fred;
extern crate tokio_core;
extern crate futures;

use fred::RedisClient;
use fred::owned::RedisClientOwned;
use fred::types::*;

use tokio_core::reactor::Core;
use futures::{
  Future,
  Stream
};

fn main() {
  let config = RedisConfig::default();

  let mut core = Core::new().unwrap();
  let handle = core.handle();

  println!("Connecting to {:?}...", config);
  
  let client = RedisClient::new(config, None);
  let connection = client.connect(&handle);
  
  let commands = client.on_connect().and_then(|client| {
    println!("Client connected.");
    
    client.select(0)
  })
  .and_then(|client| {
    println!("Selected database.");
    
    client.info(None)
  })
  .and_then(|(client, info)| {
    println!("Redis server info: {}", info);
    
    client.get("foo")
  })
  .and_then(|(client, result)| {
    println!("Got foo: {:?}", result);
    
    client.set("foo", "bar", Some(Expiration::PX(1000)), Some(SetOptions::NX))
  })
  .and_then(|(client, result)| {
    println!("Set 'bar' at 'foo'? {}.", result);
    
    client.quit()
  });

  let (reason, client) = match core.run(connection.join(commands)) {
    Ok((r, c)) => (r, c),
    Err(e) => panic!("Connection closed abruptly: {}", e) 
  };

  println!("Connection closed gracefully with error: {:?}", reason);
}

See examples for more.

Redis Cluster

Clustered Redis deployments are supported by this module by specifying a RedisConfig::Clustered variant when using connect or connect_with_policy. When creating a clustered configuration only one valid host from the cluster is needed, regardless of how many nodes exist in the cluster. When the client first connects to a node it will use the CLUSTER NODES command to inspect the state of the cluster.

In order to simplify error handling and usage patterns this module caches the state of the cluster in memory and maintains connections to each node in the cluster. In the event that a node returns a MOVED or ASK error the client will pause to rebuild the in-memory cluster state. When the local cluster state and new connections have been fully rebuilt the client will begin processing commands again. Any requests sent while the in-memory cache is being rebuilt will be queued up and replayed when the connection is available again.

Additionally, this module will not acknowledge requests as having finished until a response arrives, so in the event that a connection dies while a request is in flight it will be retried multiple times (configurable via features below) when the connection comes back up.

Logging

This module uses pretty_env_logger for logging. To enable logs use the environment variable RUST_LOG with a value of trace, debug, warn, error, or info. See the documentation for env_logger for more information.

When a client is initialized it will generate a unique client name with a prefix of fred-. This name will appear in nearly all logging statements on the client in order to associate client and server operations if logging is enabled on both.

Elasticache and occasional NOAUTH errors

When using Amazon Elasticache and Redis auth, NOAUTH errors can occasionally occur even though the client has previously authenticated during the session. This prevents further commands from being processed successfully until the connection is rebuilt. While the exact cause of the NOAUTH response from Elasticache is not fully known, it can be worked around by using the reconnect-on-auth-error compiler feature. When enabled, NOAUTH errors are treated similarly to other general connection errors and if a reconnection policy is configured then the client will automatically rebuild the connection, re-auth and continue on with the previous command. By default this behaviour is not enabled.

Features

Name Default Description
enable-tls Enable TLS support. This requires OpenSSL (or equivalent) dependencies.
ignore-auth-error x Ignore auth errors that occur when a password is supplied but not required.
reconnect-on-auth-error A NOAUTH error is treated the same as a general connection failure and the client will reconnect based on the reconnection policy.
mocks Enable the mocking layer, which will use local memory instead of an actual redis server.
super-duper-bad-networking Increase the number of times a request will be automatically retried from 3 to 20. A request is retried when the connection closes while waiting on a response.

Environment Variables

Name Default Description
FRED_DISABLE_CERT_VERIFICATION false Disable certificate verification when using TLS features.

Tests

To run the unit and integration tests:

cargo test -- --test-threads=1

Note a local Redis server must be running on port 6379 and a clustered deployment must be running on ports 30001 - 30006 for the integration tests to pass.

Beware: the tests will periodically run flushall.

TODO

  • Expand the mocking layer to support all commands.
  • More commands.
  • Blocking commands.
  • Distribute reads among slaves.
  • Transactions.
  • Lua.

Contributing

See the contributing documentation for info on adding new commands. For anything more complicated feel free to file an issue and we'd be happy to point you in the right direction.

fred.rs's People

Contributors

aembke avatar alecembke-okta avatar bibhasharyal-okta avatar danielkeller-okta avatar giselleserate-okta avatar harveyrook-okta avatar j-crowe avatar josephalfredo-okta avatar kevinwilson541 avatar mjc-gh avatar richardfarr-okta avatar ronnychan-okta avatar samuelxing-okta avatar virginiachiu-okta avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fred.rs's Issues

GEO related commands

Hello, I can see that there is not GEO related commands implemented, such as GEOADD, GEODIST and so on.

I need those to my use-case, can you help? It is hard to implement?

Implement `Error` trait for `RedisError`

Hi! Thanks for creating this library, it has a really nice interface. I was wondering whether it would be possible to implement the std::error::Error trait for RedisError? This would mean that the ? operator could be used in functions returning Result<T, Box<dyn Error>>.

Migrate from tokio-core to tokio.

Hello!
As i understand tokio-core are deprecated in favor of tokio.
At this moment i can't integrate tokio-core libraries into tokio based frameworks, like actix.
Did you have a plans for migration to tokio from tokio-core?

Broken Parsing of Nested Arrays from pipeline or Lua script

I was hoping to optimize a certain number of serial ZINTER calls to the Redis server by calling creating them in a pipeline and then subsequently parsing into a Vec<Vec<String>>. However the final result seems to always make each individual String into it's own Vec as opposed to having each ZINTER result as a Vec. Below is an example

use fred::prelude::*

// given the following SETs/ZSETs in Redis
// - Scores and SET types are irrelevant as this works for both unordered and ordered sets
// - Also shows up even when returning a nested table from a lua script and trying to parse into a Vec<Vec<_>>
// - Not certain if this parsing failure also occurs with other methods like SMEMBER/SINTER but I suspect it does
// a - { "hello", "world", "I", "am", "broken" }
// b - {"world", "I", "not"}
// c - {"hello", "am", "other"}

let compare_key = "a";
let other_keys = vec!["b", "c"];
let pool = RedisPool::new(...)?;
let pipeline = pool.pipeline();
for key in other_keys() {
    pipeline
        .zinter<Vec<String>, _, _>(vec![compare_key, key], None, None, false)
        .await?;
}
let result: Vec<Vec<String>> = pipeline.all().await?;
println!("{:?}", &result);
// Result ends up being something like -> [  [ "world" ],   [ "I" ],   [ "hello" ],   [ "am" ]  ]
//  as opposed to -> [  [  "world", "I"  ], [  "hello", "am" ]  ]

Can't resume automatically after redis cluster reboot

I encountered a problem when using fred. I use the fred cluster mode. If the cluster redis restarts, fred cannot recover automatically. How to solve this problem.

[WARN] [2023.04.04 01:13:08.914] [127.0.0.1] [fred::multiplexer::utils] [/root/.cargo/registry/src/github.com-1ecc6299db9ec823/fred-5.2.0/src/multiplexer/utils.rs(1229)] - fred-mbn8GkCsjF: Error creating or using backchannel for cluster nodes: Redis Error - kind: IO, details: Os { code: 110, kind: TimedOut, message: "Connection timed out" }

let config = RedisConfig {
server: ServerConfig::Clustered {
hosts: nodes,
},

    ..Default::default()

};

std::string::String not the right choice for RedisKey and RedisValue?

From https://doc.rust-lang.org/std/string/struct.String.html
A UTF-8 encoded, growable string.

From https://redis.io/topics/data-types-intro
Redis keys are binary safe, this means that you can use any binary sequence as a key, from a string like "foo" to the content of a JPEG file.

I'd really like to use this crate, but since RedisKey is represented as String and RedisValue as either i64 or String, one can't really safely set arbitrary binary data, or am I mistaken here? Sure, I could always do some String::from_utf8_unchecked magic, but it's a) unsafe and b) not nice having to do this.

Imo, the String representation of RedisKey and RedisValue should always be [u8] or rather Vec<u8> internally (or std::ffi::CString if that's more convenient), as Redis makes no statement about UTF-8 encoding, which is inherent in Rusts String type (see above). Therefore, using a type with such implications (UTF-8-encoded) and overhead (when trying to convert [u8] to String, if even possible) doesn't really make sense and is imo a limiting factor for a Redis client. Or maybe there is a plausible reason that I just don't see. :)

Nonetheless, not being able to use arbitrary binary data makes this crate unusable for me. Curiously enough, redis-protocol crate (same author, used by this crate) does use [u8] and Vec<u8> for bulk strings.

9.0.1 In sentinel mode with resp3 : ERR: Protocol Error: Cannot start a stream while already inside a stream.#

I use kvrocks 2.8.0 & redis7.2-sentinel with freds 9.0.1 use resp3

When I develop on my local machine with a single instance, everything is fine. When I deploy online with sentinel mode, the access will report the following error (not every time, but often)

image

I think the authentication is wrong, but the question is how do I set the redis password when I use ServerConfig::Sentinel? (I have seted the password for Sentinel)

❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#
❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#
❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#
❯ systemctl restart api
❯ curl http://127.0.0.1:8850/pingKvrocks
ERR: Protocol Error: Cannot start a stream while already inside a stream.#

the api log (this first line is api restarted)


09:01:45 api.sh[18289]:   INFO t3::srv: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/t3-0.1.9/src/srv.rs:10: http://[::]:8850


09:02:20 api.sh[18289]:   WARN fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]:   WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.24
2.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set
09:02:20 api.sh[18289]:   WARN fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]:   WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.24
2.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set
09:02:20 api.sh[18289]:   WARN fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
09:02:20 api.sh[18289]:  ERROR re::err: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/re-0.1.5/src/err.rs:28: Protocol Error: Cannot start a stream while already
 inside a stream.
09:02:20 api.sh[18289]:   WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.242.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set

there are two error

  1. fred::router::responses: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/router/responses.rs:302: fred-7B0fMbYhpK: En
    ding reader task from 38.242.220.222:2000 due to Some(Redis Error - kind: Protocol, details: Cannot start a stream while already inside a stream.)
  2. 09:02:20 api.sh[18289]: WARN fred::protocol::utils: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fred-9.0.1/src/protocol/utils.rs:275: fred-7B0fMbYhpK: [38.242.220.222:2000] Dropping unused auth warning: ERR Client sent AUTH, but no password is set

The relevant code is as follows:

https://github.com/i18n-api/mod.pub/blob/main/ping/src/kvrocks.rs

use r::fred::interfaces::KeysInterface;
use t3::IntoResponse;

pub async fn get() -> re::Result<impl IntoResponse> {
  let r: Option<String> = r::R.get("").await?;
  Ok(r.unwrap_or("Kvrocks".into()))
}

I wrap freds in xkv https://docs.rs/crate/xkv/0.1.26/source/Cargo.toml

 pub fn sentinel(
    service_name: impl Into<String>,
    hosts: Vec<fred::types::Server>,
    username: Option<String>,
    password: Option<String>,
  ) -> ServerConfig {
    ServerConfig::Sentinel {
      service_name: service_name.into(),
      hosts,
      username: Some(username.unwrap_or_else(|| "default".into())),
      password,
    }
  }
image

Redis array response is wrongly translated

I have implemented a Redis module that made a custom command available in Redis. This command returns an array with a single value of type integer.

When I run the command on redis-cli, It is correctly displayed as an array with a single element.

127.0.0.1:6379> udsf.delrec recordID1 RealmID1 StorageID
1) (integer) 0
127.0.0.1:6379> 

However, when executing this command through fred, RedisValue received is of type RedisValue::Integer (and not RedisValue::Array). It works when an array contains more than one element.

It seems like fred is translating an array with only one element to a basic type. Since this is a common response format that is being shared across multiple commands (with some commands filling in more than one element), this is breaking the application parsing of response.

0.2.4 fails to build with "--features enable-tls"

Recent release of 0.2.4 fails to build when using enable-tls feature.

I've tried with only this feature specified, as well as all features, but no dice.

cargo build --features enable-tls results in the following errors plus numerous warnings...

Additionally, version 0.2.3 is no longer available via crates.io...

error[E0308]: match arms have incompatible types
   --> src/multiplexer/mod.rs:188:58
    |
188 |           let (multiplexer, multiplexer_ft, commands_ft) = match transports {
    |  __________________________________________________________^
189 | |           Either::A(transports) => {
190 | |             let multiplexer = Multiplexer::new(
191 | |               mult_config.clone(),
...   |
242 | |           }
243 | |         };
    | |_________^ expected struct `tokio_tls::TlsStream`, found struct `tokio_core::net::TcpStream`
    |
    = note: expected type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
               found type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
note: match arm with an incompatible type
   --> src/multiplexer/mod.rs:216:36
    |
216 |             Either::B(transports) => {
    |  ____________________________________^
217 | |             let multiplexer = Multiplexer::new(
218 | |               mult_config.clone(),
219 | |               mult_message_tx.clone(),
...   |
241 | |             (multiplexer, multiplexer_ft, commands_ft)
242 | |           }
    | |___________^

error[E0308]: match arms have incompatible types
   --> src/multiplexer/mod.rs:357:54
    |
357 |       let (multiplexer, multiplexer_ft, commands_ft) = match transport {
    |  ______________________________________________________^
358 | |       Either::A((redis_sink, redis_stream)) => {
359 | |         let multiplexer = Multiplexer::new(
360 | |           config.clone(),
...   |
401 | |       }
402 | |     };
    | |_____^ expected struct `tokio_tls::TlsStream`, found struct `tokio_core::net::TcpStream`
    |
    = note: expected type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_tls::TlsStream<tokio_core::net::TcpStream>, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
               found type `(std::rc::Rc<multiplexer::multiplexer::Multiplexer<tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>, tokio_io::codec::Framed<tokio_core::net::TcpStream, protocol::types::RedisCodec>>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>, std::boxed::Box<dyn futures::Future<Error=error::RedisError, Item=()>>)`
note: match arm with an incompatible type
   --> src/multiplexer/mod.rs:380:48
    |
380 |         Either::B((redis_sink, redis_stream)) => {
    |  ________________________________________________^
381 | |         let multiplexer = Multiplexer::new(
382 | |           config.clone(),
383 | |           message_tx.clone(),
...   |
400 | |         (multiplexer, multiplexer_ft, commands_ft)
401 | |       }
    | |_______^

[SECURITY] Bumping pretty_env_logger dependency

pretty_env_logger 0.4.0 depends on env_logger 0.7.1, which depends on atty.

atty has a long-standing security problem with unaligned reads and has not been fixed, it is likely abandoned. This is a long-standing issue.

Bumping pretty_env_logger to 0.5.0 will resolve the security problem.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.