We should test our clients on a wide range of language and redis server versions, instead of on a single one.
This should be done once we have more minutes for our Github actions - ATM we have precious few.

We've seen that when running 1000 concurrent tasks with 100 bytes, rustls harms performance significantly. we should investigate the reason for this.

We can consider to use https://docs.rs/trust-dns-resolver/latest/trust_dns_resolver/ which can get the host system config. I will add an item to our new shiny backlog.
/etc/resolv.conf
https://docs.rs/trust-dns-resolver/latest/trust_dns_resolver/#using-the-host-system-config

docs.rsdocs.rs
trust_dns_resolver - Rust
The Resolver is responsible for performing recursive queries to lookup domain names.

docs.rsdocs.rs
trust_dns_resolver - Rust
The Resolver is responsible for performing recursive queries to lookup domain names.

[X] 1. Provide supporting materials for Design Review
[X] 2. Provide Project Resources
[X] 3. Provide supporting material for Threat Model Review
[] 4. Review supporting material with your Security Guardian
[] 4a. Have your Guardian post their notes to the ticket correspondence
[] 5. Complete the AppSec Review Process Training
[] 6. Complete the Pentest Questionnaire
[] 7. Change ticket Status to Assigned once Items 1-6 are completed

• Connection timeout standalone
• Connection timeout cluster
• Command timeout
• standalone Heartbeat
• cluster Heartbeat

The client is composed of multiple layers, each operating asynchronously from the others:

• Wrapper (in JS/Python/ etc.) →
• core API →
• core client →
• core connection →
• MPSC channel (in order to allow multiplexing) →
• parser ->
• TCP stream →
• server.

Responses are communicated back, in the inverse order.
We need to define what’s the meaning of connection & response timeouts - what should the user expect these values to mean. We assume that the user expects that the timeouts represent the amount of time they are willing to wait for success on the external-most layers - the wrapper. So a timeout represents the amount of time to wait for a request / connection attempt to pass from the wrapper to the server, and the response to pass through all layers back to the wrapper.
We will also set an internal timeout, so that messages that are sent to a hanging server will cause the core client to timeout, just so that the core’s state will match the wrapper’s state.
We won’t retry operations, unless we receive EAGAIN errors from the TCP connection, or MOVED, ASK, CLUSTERDOWN, LOADING or TRYAGAIN errors from a cluster node.

defaults: connection timeout 250ms, response timeout 250 ms. TCP keepalive - TBD.

reconnection

There are two types of reconnection: during client creation and in steady state. As a client is being created, we will try to establish a connection with a timeout that will be exposed to the user. That timeout will encompass the whole connection process, including multiple retries with exponential backoff between them. During the initialize phase, we shall raise an exception only if a fatal error occurs, for example: standalone endpoint given to cluster client, all seed nodes are unreachable. However, if the client is partially functional, e.g.: we can get the cluster topology from at least one of the seed nodes or we can connect only to the primary in standalone, we shall allow the user to use this client and we shall work on establishing all connections / refreshing the cluster topology asynchronously on a background job on the core side. When the client is in steady-state, reconnecting will always be handled by the background job and be transparent to the user. The backoff configurations for the steady-state reconnection will be set internally and won’t be configurable by the user.

The background job is supposed to act as a connection watchdog and do the following:

1. Monitor the connections' health
2. If a connection is broken, keep trying to reconnect until the connection is established using exponential backoff and jitter (e.g. - without limiting the number of retries, as long as the client is alive).

If a user tries to execute a command and the client doesn’t have a ready connection at that time, a connection error will be raised to the user.

Timeouts will not trigger reconnection attempts, since they might represent a busy server that is still available, and reconnection will only increase the load on the server. We will try to reconnect if the connection is broken on sent messages, but we will need to understand how and to which value to set this timeout.

defaults: exponential backoff - unlimited retries, time between retries: uniform distribution up to 100ms * 2^current retry. Initial connection timeout - TBD.

Node:

• bitmap
• cluster management
• connection management
• generic
• geo
• hash
• hyperloglog
• list
• scripting
• server management
• set
• sorted set
• stream
• string
• transactions
• pub / sub

Python:

• bitmap
• cluster management
• connection management
• generic
• geo
• hash
• hyperloglog
• list
• scripting
• server management
• set
• sorted set
• stream
• string
• transactions
• pub / sub

The last time it was tested, cargo-udeps failed to handle dev-dependencies correctly. Once this is fixed, we should integrate it into our CI.

Support password rotation using a credential provider, such as IAM roles

this can be reproduced both in node & C# - we have 2 test suites, and only the logs in the first suite that runs appear on console.

MVP commands support (150) - 90 commands left, not including STREAM, PUBSUB, LUA.

• Python wrapper
• Consume git repo

retry command when receiving loafing error from replica

The current implementation doesn't support partial covered slots.
We should allow this mode to support partial functionality during failover in clusters without replicas

Redis Streams support

https://github.com/redis-rs/redis-rs/pull/842/files#diff-3081b035ea7b3a6c6fbf36bdcc857b7d1405e36fbe98df550dc7ba187275a638

If we execute a command to all nodes/primaries in this function, we're not handling CME errors (e.g. MOVED, ASK..) nor refreshing topology.
I think we should have a wrapper function that decide which nodes to send the command to, and an inner function with the logic of executing the request.
e.g.

fn request (...) {
  nodes = get_nodes_to_be_executed()
  for node in nodes:
     node.execute_request()
}

fn execute_request(...) {
        loop {
            // Get target address and response.
            let (addr, rv) = {
   ...
                (addr, func(conn))
            };
            match rv {
                Ok(rv) => return Ok(rv),
                Err(err) => {
       ...
                    }
}
}

• #233
• #234
• #235
• #236

The last time I tested this, it gave erroneous results on the python library. It should be re-examined after a couple of versions.

• C#
• Python
• TS

The socket allows any process on the machine to read or write from Redis, without any additional protections.

The same might be true of init_file.

sample tests -

#[cfg(test)]
mod tests {
    use crate::{init_console, init_file, log_trace, FILE_DIRECTORY};
    use rand::{distributions::Alphanumeric, Rng};
    use std::{
        fs::{read_dir, read_to_string},
        io,
    };

    fn generate_random_string(length: usize) -> String {
        rand::thread_rng()
            .sample_iter(&Alphanumeric)
            .take(length)
            .map(char::from)
            .collect()
    }

    fn get_file_contents(file_name: &str) -> String {
        let files = read_dir(FILE_DIRECTORY).unwrap();
        let file = files
            .into_iter()
            .find(|path| {
                path.as_ref()
                    .unwrap()
                    .path()
                    .file_name()
                    .unwrap()
                    .to_str()
                    .unwrap()
                    .starts_with(file_name)
            })
            .unwrap();
        read_to_string(file.unwrap().path()).unwrap()
    }

    #[test]
    fn log_to_file_works_after_multiple_inits() {
        let identifier = generate_random_string(10);
        init_file(crate::Level::Trace, identifier.as_str());
        init_file(crate::Level::Trace, identifier.as_str());
        log_trace(identifier.clone(), "foo");

        let contents = get_file_contents(identifier.as_str());

        assert!(
            contents.contains(identifier.as_str()),
            "Contens: {}",
            contents
        );
        assert!(contents.contains("foo"), "Contens: {}", contents);
    }

    #[test]
    fn log_to_console_works_after_multiple_inits() {
        let identifier = generate_random_string(10);

        init_console(crate::Level::Trace);
        init_console(crate::Level::Trace);
        log_trace(identifier.clone(), "foo");

        // let mut stdout = io::stdout().lock();
        // let lines = stdout.lines();

        // let contents = get_file_contents(identifier.as_str());

        // assert!(
        //     contents.contains(identifier.as_str()),
        //     "Contens: {}",
        //     contents
        // );
        // assert!(contents.contains("foo"), "Contens: {}", contents);
    }

    #[test]
    fn log_to_console_works_after_file_init() {}

    #[test]
    fn log_to_file_works_after_console_init() {
        let identifier = generate_random_string(10);
        init_console(crate::Level::Trace);
        init_file(crate::Level::Trace, identifier.as_str());
        log_trace(identifier.clone(), "foo");

        let contents = get_file_contents(identifier.as_str());

        assert!(
            contents.contains(identifier.as_str()),
            "Contens: {}",
            contents
        );
        assert!(contents.contains("foo"), "Contens: {}", contents);
    }
}

If the application threads are calling exit() it can shutdown the socket listener thread without cleaning the socket file.
If we'll go with the socket approach, we need to consider some kind of shutdown hooks/ use at_exit to ensure we delete the socket file when the process is closed.

At the moment pipeline commands must be in the same slot:
https://github.com/nihohit/redis-rs/blob/2437344c804ec109ac1cceeec0c2204268ccb59a/redis/src/cluster_async/mod.rs#L146
If the pipeline doesn't execute requests with multi/exec, we should be able to handle commands from different slots by routing them to the relevant nodes

• Python pip
• JavaScript - npm

• Round robin algo (replicas only, replicas + primary) cluster

PubSub support.
We will support only RESP3. For the push notification.
RESP3 and pubsub in redis-rs currently in progress.

How to handle disconnections? both in standalone and CME.

• Core rust, support pub sub in cluster.
• maintain subscription state after disconnect
• Add to the socket manager option to push out of band messages
• Python
• node

• values - random strings instead of recurring 0's. The strings should be recomputed before each usage instead of reused.
• Write JSON results to file. Receive file name as a command-line argument.
• Write a program that receives names of all JSON result files in a run, and collects them into a CSV file
• Clear the DB before each benchmark, in order to ensure that past benchmarks don't affect our results.
• Take address + port as an optional command-line argument, to allow benchmarking against remote server.
• (consider) Take optional parameter to disable non-babushka benchmarks, if we only want to measure a change's improved perf.

Read from replica
Sub tasks:

• #228
• #229
• #230
• #231
• #232

Task was destroyed but it is pending!
task: <Task pending name='Task-229' coro=<RedisAsyncSocketClient._reader_loop() done, defined at /home/ubuntu/babushka/python/python/pybushka/async_socket_client.py:160> wait_for=>

Since some clients create threads, we should test performance not only for many actions on a single client, but also for many concurrent clients.

currently the tests require that a redis server will be running on the machine, which leaves room for errors.