cabol / nebulex_redis_adapter Goto Github PK

We setup Redis Cluster without Sentinel and instead of getting data consistently we received
*MOVED Redirection
instead of it following the redirect when we try to get data. Please help, maybe it's our setup.

config :client_compass, AppCache,
pools: [
primary: [
url: "redis://172.16.43.5:6379,172.16.43.6:6380,172.16.43.7:6381,172.16.43.5:6381,172.16.43.6:6379,172.16.43.7:6380"
]
]

Any help would be appreciated.

The current config for Redis Cluster uses the :master_nodes and :conn_opts options, but it is quite confusing how they work. The goal is to simplify it by using only :conn_opts.

The :conn_opts option

For Redis Cluster mode,:conn_opts will define the Redis client options but for the configuration endpoint. This is where the client should connect to send the "CLUSTER SHARDS" (Redis >= 7) or "CLUSTER SLOTS" (Redis < 7) command to get the cluster information and set it up on the client side.

The :override_master_host option

Defines whether the given master host should be overridden with the configuration endpoint or not. Defaults to false. By default, the adapter uses the host returned by the "CLUSTER SHARDS" (Redis >= 7) or "CLUSTER SLOTS" (Redis < 7) command. One may consider set it to true for tests when using Docker for example, or when Redis nodes are behind a load balancer that Redis doesn't know the endpoint of. See Redis docs for more information.

Update to Nebulex v2.1.0 and instrument the adapter with the recommended Telemetry events, such as the built-in adapters.

Hi, we're trying to use nebulex_redis_adapter master branch with v2.0.0-rc.0, but compilation errors occur:

** (CompileError) lib/nebulex_cluster.ex:51: Nebulex.Object.__struct__/0 is undefined, cannot expand struct
Nebulex.Object. Make sure the struct name is correct. If the struct name exists and is correct but it
still cannot be found, you likely have cyclic module usage in your code
    (stdlib 3.13.2) lists.erl:1358: :lists.mapfoldl/3
    (elixir 1.11.0) src/elixir_fn.erl:15: anonymous fn/4 in :elixir_fn.expand/3
    (stdlib 3.13.2) lists.erl:1358: :lists.mapfoldl/3
could not compile dependency :nebulex_cluster, "mix compile" failed. You can recompile this dependency with
"mix deps.compile nebulex_cluster", update it with "mix deps.update nebulex_cluster" or clean it with
"mix deps.clean nebulex_cluster"

Is there a timeline where this lib will become compatible? If not, do you have any pointers to help us get the most simplistic case working (i.e. without worrying about distributed caching)?

We're seeing quite a few exceptions due to our Redis connection timing out when using this adaptor. Is there a way of having a configuration to send regular PING commands to Redis to keep connections alive while using this adaptor.

Implement default cluster support based on sharding distribution model (using consistent-hashing maybe?) – similar to Nebulex.Adapters.Dist.

Refactor/rename (related issue #45):

• Option :codec to :serializer.
• Module NebulexRedisAdapter.Codec to NebulexRedisAdapter.Serializer.
• Module NebulexRedisAdapter.Codec.StringCodec to NebulexRedisAdapter.Serializer.Serializable.

It has a lot of improvements and it's the current version supported by the newest new_relic_agent. Since it doesn't seem to be much work to be done is it ok if submit a pull request doing the change?

Redix changelog:
https://github.com/whatyouhide/redix/blob/main/CHANGELOG.md

in my config/config.exs，i change the host address to remote address， but it does not take effect，what else do I need to modify?

config :my_app, MyApp.RedisCache,
conn_opts: [
# Redix options
host: "127.0.0.1", // change host is invalid
port: 6379
]

Hello,

Recently, the in-memory cache for my application has become too large and it is causing my Kubernetes pods to be OOM killed. I suspect that the memory usage is also impacting my app performance. I am trying to use the Redis adapter as a caching solution with a Redis cluster I have set up on my k8s cluster. It's a StatefulSet with 6 nodes. I have verified that the cluster is set up properly and I am currently using it to cache some temporary location data for the apps.

The issue I am having is that when I try to write to the cache, it returns the error

** (Redix.Error) MOVED 866 10.8.17.53:6379
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter/command.ex:103: NebulexRedisAdapter.Command.handle_command_response/1
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:480: NebulexRedisAdapter.put/6
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:475: anonymous fn/7 in NebulexRedisAdapter.put/6
    (telemetry 1.0.0) /build/services/api/deps/telemetry/src/telemetry.erl:293: :telemetry.span/3
    (nebulex 2.3.1) lib/nebulex/cache/entry.ex:42: Nebulex.Cache.Entry.put/4

When I read from the cache, I get no such error.

Here is my configuration using all 6 Redis hostnames as possible master nodes.

config :my_app, MyApp.Cache,
  mode: :redis_cluster,
  master_nodes: [
    [
      host: "redis-cluster-0.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-1.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-2.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-3.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-4.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-5.redis-cluster-headless"
    ]
  ],
  conn_opts: []

Any help would be greatly appreciated. Thank you

Implement a Registry-based routing pool strategy instead of using atoms and create them in runtime.

Based on Andrea's Leopardi article Process pools with Elixir's Registry

Currently, bay default, all values except the counters are stored as Nebulex.Object.t() (before to put the data on Redis, the object is encoded as binary). The idea is to allow storing raw strings, not within the object, and improve the memory consumption for string data types.

Hi
Thanks for all the hard work on this library. We are busy implementing it but have run into the issue that our redis cluster is setup with sentinel support for primary discovery with secondary replicas.

Is sentinel support on the roadmap? If not are you open to PRs for its implementation?

thanks

This is explained well in https://paraxial.io/blog/elixir-rce

Basically, the code below will print I was executed!!!

f = fn _, _ -> IO.puts "I was executed!!!"; {:halt, []} end
serialized = :erlang.term_to_binary(f)
serialized |> :erlang.binary_to_term() |> Enum.to_list()

In Nebulex context this means that any iteration over structures fetched from Nebulex creates an RCE vulnerability through Redis.
Yes, normally Redis is not writable from the outside.... But still it's a "security weakness".

One potential fix would be using https://hexdocs.pm/plug_crypto/Plug.Crypto.html#non_executable_binary_to_term/2

The downsides are:

• dependency on another library plug_crypto (a small one and with no other runtime deps)
• performance implications (because non_executable_binary_to_term would iterate over each result fetched from the cache)
• backward in-compatibility (what if someone indeed wanted to store anonymous functions in the cache 🤷)

@cabol I'm happy to make a PR if you suggest a resolution for the above concerns 🙏

When executing a Redis pipeline, it may return {:ok, results} but the results may contain one or more Redis errors (Redix.Error structs). Currently, this scenario is not handled by the adapter causing unexpected behavior. The proposed solution is to check the pipeline results and raise an error if any Redis error is found within them.

Is there a way to have an existing redix connection started and give it to NebulexRedisAdapter/Nebulex at runtime?

Background:

Because we can't really call code from within config.exs et al, this fails to build a release:

config :my_app, MyApp.Cache,
  # config for NebulexRedisAdapter from config/prod.exs
  conn_opts: [
    url: System.get_env("REDIS_ENDPOINT"),
    password: {System, :get_env, ["REDIS_TOKEN"]},
    sync_connect: true,
    socket_opts: [
      customize_hostname_check: [
        match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
      ]
    ]
  ]

Specifically the match_fun call to :public_key Erlang module leads to mix release returning an error:

** (Protocol.UndefinedError) protocol Enumerable not implemented for %Config.Provider{config_path: {:system, "RELEASE_SYS_CONFIG", ".config"}, extra_config: [kernel: [start_distribution: true]], providers: [{Config.Reader, {{:system, "RELEASE_ROOT", "/releases/0.1.0/runtime.exs"}, [env: :prod, target: :host, imports: :disabled]}}], prune_runtime_sys_config_after_boot: false, reboot_system_after_config: true, validate_compile_env: [{:retentions, [:plug_telemetry], {:ok, [level: :info]}}, {:mime, [:types], :error}]} of type Config.Provider (a struct). This protocol is implemented for the following type(s): Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.Stream, DBConnection.PrepareStream, Timex.Interval, Jason.OrderedObject, HashSet, Range, Map, Function, List, Stream, Date.Range, HashDict, GenEvent.Stream, MapSet, File.Stream, IO.Stream
…

While this isn't a very good error, my only change was adding conn_opts for redis, and I was able to narrow it down.

The use case: I want to delete all key that matches "prefix*". Nebulex gives us Nebulex.Cache.delete_all, but that doesn't work as redis has no native methods to delete all matching keys, so right now this method only works if we send nil to delete the whole thing.

So, the best way to do this in redis to get all matching keys then delete them in a single DEL call sending a list of keys. Getting all matching keys can be done using Nebulex.Cache.all but there's no way to delete multiple keys at once, forcing us to do multiple calls

tl,dr: Please implement Nebulex.Cache.delete in a way we can send a list of keys to be deleted

The idea is to use KEYS command to query data, in this case, only keys can be queried, due to command limitations in Redis.

Given:

defmodule MyCache do
  use Nebulex.Cache,
    otp_app: :nebulex,
    adapter: NebulexRedisAdapter
end

then

iex> MyCache.stats()
** (UndefinedFunctionError) function MyCache.stats/0 is undefined or private, but the behaviour Nebulex.Cache expects it to be present
    (my_app 0.1.0) MyCache.stats()

This leaves us unable to obtain Telemetry metrics for this cache.

Hi! I think there's some kind of difference in the Redis response for the CLUSTER SLOTS command between v6 and v7. In v7 I was getting this:

iex(5)> {:ok, cluster_slots} = Redix.command(conn, ["CLUSTER", "SLOTS"])
{:ok,
 [
   [
     0,
     16383,
     ["172.20.0.2", 6379, "71053754ec7ec7bba4396ec6355634afc1a9eb18", []]
   ]
 ]}

which made my project break:

** (Mix) Could not start application …
    ** (EXIT) shutdown: failed to start child: …
        ** (EXIT) shutdown: failed to start child: {Redix, {0, 16383}}
            ** (EXIT) an exception was raised:
                ** (MatchError) no match of right hand side value: [["172.20.0.2", 6379, "71053754ec7ec7bba4396ec6355634afc1a9eb18", []]]
                    (nebulex_redis_adapter 2.0.0) lib/nebulex_redis_adapter/redis_cluster/supervisor.ex:21: NebulexRedisAdapter.RedisCluster.Supervisor.init/1
                    (stdlib 3.15.2) supervisor.erl:330: :supervisor.init/1
                    (stdlib 3.15.2) gen_server.erl:423: :gen_server.init_it/2
                    (stdlib 3.15.2) gen_server.erl:390: :gen_server.init_it/6
                    (stdlib 3.15.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

After some investigation I noticed that we were using redis:latest docker image so I compared versions with my teammates and they had v6. I switched to that version too and the project stopped breaking.

After rebuilding my docker container, v6 response for CLUSTER SLOTS was:

iex(3)> {:ok, cluster_slots} = Redix.command(conn, ["CLUSTER", "SLOTS"])
{:ok,
 [[0, 16383, ["172.20.0.15", 6379, "e191e13a8cef4f17b4412e8f7660aee00f655e0f"]]]}

Hope it helps.

We have a much larger application to which we are trying to add a Redis cluster. I kept getting issues with MOVED errors and disconnections. So, I wanted to try using the cluster in much simpler application to see if I could get it to work. I'm not sure if my configuration is off or if clustering is not working.

The local cluster is 7.2.3 installed following: Understand, Setup and Test a Redis Cluster in less than 10 minutes. I had the same issues with Bitnami's docker compose cluster.

defp deps do
  [
    {:nebulex, "2.5.2"},
    {:nebulex_redis_adapter, "2.3.1"}
  ]
end

config :testcache, Testcache.Cache,
  mode: :redis_cluster,
  redis_cluster: [
    configuration_endpoints: [
      conn_opts: [
        host: "127.0.0.1",
        port: 30001
      ]
    ]
  ]

defmodule Testcache.Cache do
  use Nebulex.Cache, otp_app: :testcache, adapter: NebulexRedisAdapter
end

iex(1)> Testcache.Cache.start_link([])
{:ok, #PID<0.650.0>}
iex(2)> Testcache.Cache.put("foo", "bar")
:ok
iex(3)> Testcache.Cache.put("hello", "world")
** (Redix.Error) MOVED 866 127.0.0.1:30001...

Hello,

Thanks for the great lib! I can see that quite some updates have happened since last release on hex.pm.

Would you be able to release?

thanks,

Hey Cabol,

First of all fantastic library, it's super easy to use and works extremely nicely. We've had a small issue building a repeatable docker image for our elixir umbrella app. This is the error that is thrown when including the nebulex_redis_adapter

web_1    | 17:54:26.884 [info] Application jchash exited: exited in: :jchash.start(:normal, [])
web_1    |     ** (EXIT) an exception was raised:
web_1    |         ** (UndefinedFunctionError) function :jchash.start/2 is undefined or private
web_1    |             (jchash) :jchash.start(:normal, [])
web_1    |             (kernel) application_master.erl:277: :application_master.start_it_old/4
web_1    | {"Kernel pid terminated",application_controller,"{application_start_failure,jchash,{bad_return,{{jchash,start,[normal,[]]},{'EXIT',{undef,[{jchash,start,[normal,[]],[]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,277}]}]}}}}}"}
web_1    | Kernel pid terminated (application_controller) ({application_start_failure,jchash,{bad_return,{{jchash,start,[normal,[]]},{'EXIT',{undef,[{jchash,start,[normal,[]],[]},{application_master,start_it_old
web_1    |
web_1    | Crash dump is being written to: erl_cra

Tested with: nebulex_redis_adapter ~> "1.0.0" & nebulex_redis_adapter ~> "1.1.0"

Currently the nebulex redis adapter is not grabbing and passing the ssl config option to Redix.

Currently, the adapter encodes and decodes Redis keys and values automatically underneath, it encodes any Elixir term to a binary and decodes binaries into Elixir terms. The idea is to allow the user to optionally provide a custom codec with the option :codec. The value must be a module implementing the NebulexRedisAdapter.Codec behaviour.

This is how the behaviour may look like:

defmodule NebulexRedisAdapter.Codec do
  @moduledoc """
  Codec interface.
  """

  @doc """
  Encodes `key` with the given `opts`.
  """
  @callback encode_key(key :: term, opts :: [term]) :: iodata

  @doc """
  Encodes `value` with the given `opts`.
  """
  @callback encode_value(value :: term, opts :: [term]) :: iodata

  @doc """
  Decodes `key` with the given `opts`.
  """
  @callback decode_key(key :: binary, opts :: [term]) :: term

  @doc """
  Decodes `value` with the given `opts`.
  """
  @callback decode_value(value :: binary, opts :: [term]) :: term
end

Currently, the Redis adapter provides the extended functions command!/3 and pipeline!/3, but they are bang functions, which means they raise an exception when an error occurs. The idea is to add support for the counterpart functions command/3 and pipeline/3, returning Ok/Error tuples.

New Redis Cluster management strategy

Currently, the Redis Cluster setup is very naive, when the cache is started, it tries to get the cluster configuration from Redis and then configure the supervision tree based on it. The problem is it is static once the cache starts, and it doesn't handle cluster changes or "MOVED" errors. The idea is to provide a smarter cluster management strategy able to handle cluster changes as well as "MOVED" errors dynamically.

Strategy highlights

• A dynamic supervision tree for handling the connection pools to the different master nodes.
• A configuration manager process in charge of configuring the hash slot map whenever there is a change in the cluster to a "MOVED" error.
• When the configuration manager runs, it should block all commands execution until the hash slot map is ready to use.
• Whenever there is a "MOVED" error, the configuration manager is called to re-configure the cluster and once it is done, the failed command is executed again.
• Remove the :master_nodes option which is quite confusing and use only :conn_opts instead. The idea here is to configure only the configuration endpoint via the :conn_opts; this is where the client should connect to send the "CLUSTER SHARDS" (Redis >= 7) or "CLUSTER SLOTS" (Redis < 7) command to get the cluster information and set it up on the client side.

Allow passing Redis client option to the extended cache functions like command and pipeline.

Currently, these functions look like this:

command(key \\ nil, name \\ __MODULE__, command)

command!(key \\ nil, name \\ __MODULE__, command)

pipeline(key \\ nil, name \\ __MODULE__, commands)

pipeline!(key \\ nil, name \\ __MODULE__, commands)

They don't admit extra options. Therefore, the idea is to make them like so:

command(command, opts \\ [])

command!(command, opts \\ [])

pipeline(commands, opts \\ [])

pipeline!(commands, opts \\ [])

Add add :key and :name to the supported options.