driverx / aioredis-cluster Goto Github PK

View Code? Open in Web Editor NEW

24.0 4.0 7.0 545 KB

Redis Cluster support extension for aioredis

License: MIT License

Makefile 0.27% Python 99.48% Cython 0.25%

redis redis-client cluster python redis-cluster redis-cluster-client python36 python37 python38 python39

aioredis-cluster's People

Contributors

Stargazers

Watchers

Forkers

vadimpushtaev yprocfly jacekll erastov roman901 anukkrit149 aamalev

aioredis-cluster's Issues

Support decode_responses in initialization / create_redis_cluster

Passing in a decode_responses=True into the create_redis_cluster initializer would be convenient. This would call a .decode('utf-8') or similar on all results before returning them

It's currently implemented in both redis-py and aioredis

Migration to aioredis 2.0.0

Hi, are there any plans to migrate to aioredis 2.0.0?

I am running into occasional ConnectionResetError and this issue suggests 2.0.0 might fix it (maybe...)

PubSub support for multiple channels and context managers

Hi @DriverX , I was testing the sharded ssubscribe and subscribe function and noticed that PubSubs in your library are limited to a single channel, getting error:

ValueError: Only one channel supported in cluster mode.

Are there plans to support multiple channels? Clients should be able to subscribe to multiple channels so long as they belong to the same slot. https://redis.io/commands/ssubscribe/.

i.e.result = await redis_cluster.ssubscribe('{key}:123', '{key}:456'

Also, is it possible to open a Pub/Sub context manager to handle subscription messages?
In aioredis, this is possible like this example: https://aioredis.readthedocs.io/en/latest/examples/#pubsub.

i.e.

channels = ['{key}:123', '{key}:456']
async with self.redis_cluster.pubsub() as pubsub:
    await pubsub.ssubscribe(*channels)

    async for result in pubsub.listen():
        yield result

Many thanks!

PSubscribe subscribe to single (master) node in cluster

Expected Behavior:
As per Redis documentation, Every node of a Redis cluster generates events about its own subset of the keyspace, However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.
https://redis.io/topics/notifications
Current Behavior:
aioredis-cluster lib has provided "psubscribe(self, pattern, *patterns)" to subscribe to patterns but this method subscribe to only one of the master node in cluster.
Question: Any recommended way to subscribe to all nodes of cluster to receive all keyspace events with the APIs supported by aioredis-cluster.
Or
Is there any way of subscribing dynamically to slave node turn into master in case of fail over (master brought down)?

Expected Behavior:

is there any way to discover cluster nodes endpoint using aioredis-cluster??

Read-only mode

Hi thank for your work !

I have a cluster with replicas, is there a way to favor read from replicas (follower node) since I could have a read-only cluster client for my app.

Passing custom ssl.SSLContext object to aioredis-cluster fails

aioredis-cluster version in use 2.3.0
python 3.10.2

Expected behavior - Passing ssl.SSLContext object to aioredis_cluster.create_redis_cluster is successful.

Current behavior - App startup fails when connecting to Redis due to following error:

  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/_aioredis/stream.py", line 21, in open_connection
    transport, _ = await get_event_loop().create_connection(lambda: protocol, host, port, **kwds)
  File "uvloop/loop.pyx", line 2069, in create_connection
  File "uvloop/loop.pyx", line 2064, in uvloop.loop.Loop.create_connection
  File "uvloop/sslproto.pyx", line 517, in uvloop.loop.SSLProtocol._on_handshake_complete
  File "uvloop/sslproto.pyx", line 477, in uvloop.loop.SSLProtocol._start_handshake
AttributeError: 'function' object has no attribute 'wrap_bio'

The code to build the ssl.SSLContext object is as follows:

        ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
        ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2
        ssl_context.set_ciphers(CNSA_CIPHERS)

        ssl_context.check_hostname = False
        ssl_context.verify_mode = ssl.CERT_NONE

        return ssl_context

It looks like an ssl error but I am creating SSLContext in this way for other infrastructure services, like postgres/elasticsearch/kafka and passing the object to the corresponding python library and it is working fine.
Additionally when passing just a bool to aioredis_cluster i.e. enable_ssl=True it is succesful, but I have no way to easily prove what the SSLContext is.

This call is successful, where ENABLE_REDIS_SSL is a bool True. The docs say ssl can be a Union[bool, ssl.SSLContext, None]:

redis_connector = await aioredis_cluster.create_redis_cluster(
            [REDIS_URL], password=REDIS_PASSWORD, ssl=ENABLE_REDIS_SSL
        )

My understanding is that when you pass a bool "True" to aioredis_cluster.create_redis_cluster it creates a default ssl.SSLContext object, in a similar fashion to how I am creating it in the code.

Here is the full stack trace:

File "/home/appuser/.local/lib/python3.10/site-packages/starlette/routing.py", line 621, in lifespan
    async with self.lifespan_context(app):
  File "/home/appuser/.local/lib/python3.10/site-packages/starlette/routing.py", line 518, in __aenter__
    await self._router.startup()
  File "/home/appuser/.local/lib/python3.10/site-packages/starlette/routing.py", line 598, in startup
    await handler()
  File "/app/./[REDACTED]/main.py", line 67, in app_startup
    await create_redis_cluster_connection(
  File "/home/appuser/.local/lib/python3.10/site-packages/[REDACTED]/application_services/redis/redis_wrapper.py", line 50, in create_redis_cluster_connection
    redis_cluster_connection = await aioredis_cluster.create_redis_cluster(
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/factory.py", line 116, in create_redis_cluster
    cluster = await create_cluster(
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/factory.py", line 82, in create_cluster
    await cluster._init()
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/cluster.py", line 534, in _init
    await self._manager._init()
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/manager.py", line 396, in _init
    await self.reload_state()
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/manager.py", line 164, in reload_state
    return await self._load_state(self._reload_count)
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/manager.py", line 360, in _load_state
    state = await self._fetch_state(init_addrs, reload_id, self._state)
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/manager.py", line 274, in _fetch_state
    raise last_err
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/manager.py", line 206, in _fetch_state
    pool = await self._pooler.ensure_pool(addr)
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/pooler.py", line 70, in ensure_pool
    pool = await self._create_pool((addr.host, addr.port))
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/cluster.py", line 758, in _create_default_pool
    return await self._create_pool(addr)
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/cluster.py", line 778, in _create_pool
    pool = await create_pool(addr, **{**default_opts, **opts})
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/aioredis/pool.py", line 76, in create_pool
    await pool._fill_free(override_min=False)
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/pool.py", line 421, in _fill_free
    conn = await self._create_new_connection(self._address)
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/pool.py", line 444, in _create_new_connection
    conn: AbcConnection = await create_connection(
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/aioredis/connection.py", line 94, in create_connection
    reader, writer = await asyncio.wait_for(
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/appuser/.local/lib/python3.10/site-packages/aioredis_cluster/_aioredis/stream.py", line 21, in open_connection
    transport, _ = await get_event_loop().create_connection(lambda: protocol, host, port, **kwds)
  File "uvloop/loop.pyx", line 2069, in create_connection
  File "uvloop/loop.pyx", line 2064, in uvloop.loop.Loop.create_connection
  File "uvloop/sslproto.pyx", line 517, in uvloop.loop.SSLProtocol._on_handshake_complete
  File "uvloop/sslproto.pyx", line 477, in uvloop.loop.SSLProtocol._start_handshake
AttributeError: 'function' object has no attribute 'wrap_bio'

Any ideas or pointers on why passing a default created ssl.SSLContext object fails but passing a bool True is succesful?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.