cabol / nebulex_redis_adapter Goto Github PK
View Code? Open in Web Editor NEWNebulex adapter for Redis
Home Page: https://hexdocs.pm/nebulex_redis_adapter/NebulexRedisAdapter.html
License: MIT License
Nebulex adapter for Redis
Home Page: https://hexdocs.pm/nebulex_redis_adapter/NebulexRedisAdapter.html
License: MIT License
We setup Redis Cluster without Sentinel and instead of getting data consistently we received
*MOVED Redirection
instead of it following the redirect when we try to get data. Please help, maybe it's our setup.
config :client_compass, AppCache,
pools: [
primary: [
url: "redis://172.16.43.5:6379,172.16.43.6:6380,172.16.43.7:6381,172.16.43.5:6381,172.16.43.6:6379,172.16.43.7:6380"
]
]
Any help would be appreciated.
The current config for Redis Cluster uses the :master_nodes
and :conn_opts
options, but it is quite confusing how they work. The goal is to simplify it by using only :conn_opts
.
:conn_opts
optionFor Redis Cluster mode,:conn_opts
will define the Redis client options but for the configuration endpoint. This is where the client should connect to send the "CLUSTER SHARDS" (Redis >= 7) or "CLUSTER SLOTS" (Redis < 7) command to get the cluster information and set it up on the client side.
:override_master_host
optionDefines whether the given master host should be overridden with the configuration endpoint or not. Defaults to false
. By default, the adapter uses the host returned by the "CLUSTER SHARDS" (Redis >= 7) or "CLUSTER SLOTS" (Redis < 7) command. One may consider set it to true
for tests when using Docker for example, or when Redis nodes are behind a load balancer that Redis doesn't know the endpoint of. See Redis docs for more information.
Update to Nebulex v2.1.0
and instrument the adapter with the recommended Telemetry events, such as the built-in adapters.
Hi, we're trying to use nebulex_redis_adapter
master branch with v2.0.0-rc.0
, but compilation errors occur:
** (CompileError) lib/nebulex_cluster.ex:51: Nebulex.Object.__struct__/0 is undefined, cannot expand struct
Nebulex.Object. Make sure the struct name is correct. If the struct name exists and is correct but it
still cannot be found, you likely have cyclic module usage in your code
(stdlib 3.13.2) lists.erl:1358: :lists.mapfoldl/3
(elixir 1.11.0) src/elixir_fn.erl:15: anonymous fn/4 in :elixir_fn.expand/3
(stdlib 3.13.2) lists.erl:1358: :lists.mapfoldl/3
could not compile dependency :nebulex_cluster, "mix compile" failed. You can recompile this dependency with
"mix deps.compile nebulex_cluster", update it with "mix deps.update nebulex_cluster" or clean it with
"mix deps.clean nebulex_cluster"
Is there a timeline where this lib will become compatible? If not, do you have any pointers to help us get the most simplistic case working (i.e. without worrying about distributed caching)?
We're seeing quite a few exceptions due to our Redis connection timing out when using this adaptor. Is there a way of having a configuration to send regular PING
commands to Redis to keep connections alive while using this adaptor.
Implement default cluster support based on sharding distribution model (using consistent-hashing maybe?) – similar to Nebulex.Adapters.Dist
.
Refactor/rename (related issue #45):
:codec
to :serializer
.NebulexRedisAdapter.Codec
to NebulexRedisAdapter.Serializer
.NebulexRedisAdapter.Codec.StringCodec
to NebulexRedisAdapter.Serializer.Serializable
.It has a lot of improvements and it's the current version supported by the newest new_relic_agent. Since it doesn't seem to be much work to be done is it ok if submit a pull request doing the change?
Redix changelog:
https://github.com/whatyouhide/redix/blob/main/CHANGELOG.md
in my config/config.exs,i change the host address to remote address, but it does not take effect,what else do I need to modify?
config :my_app, MyApp.RedisCache,
conn_opts: [
# Redix options
host: "127.0.0.1", // change host is invalid
port: 6379
]
Hello,
Recently, the in-memory cache for my application has become too large and it is causing my Kubernetes pods to be OOM killed. I suspect that the memory usage is also impacting my app performance. I am trying to use the Redis adapter as a caching solution with a Redis cluster I have set up on my k8s cluster. It's a StatefulSet with 6 nodes. I have verified that the cluster is set up properly and I am currently using it to cache some temporary location data for the apps.
The issue I am having is that when I try to write to the cache, it returns the error
** (Redix.Error) MOVED 866 10.8.17.53:6379
(nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter/command.ex:103: NebulexRedisAdapter.Command.handle_command_response/1
(nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:480: NebulexRedisAdapter.put/6
(nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:475: anonymous fn/7 in NebulexRedisAdapter.put/6
(telemetry 1.0.0) /build/services/api/deps/telemetry/src/telemetry.erl:293: :telemetry.span/3
(nebulex 2.3.1) lib/nebulex/cache/entry.ex:42: Nebulex.Cache.Entry.put/4
When I read from the cache, I get no such error.
Here is my configuration using all 6 Redis hostnames as possible master nodes.
config :my_app, MyApp.Cache,
mode: :redis_cluster,
master_nodes: [
[
host: "redis-cluster-0.redis-cluster-headless"
],
[
host: "redis-cluster-1.redis-cluster-headless"
],
[
host: "redis-cluster-2.redis-cluster-headless"
],
[
host: "redis-cluster-3.redis-cluster-headless"
],
[
host: "redis-cluster-4.redis-cluster-headless"
],
[
host: "redis-cluster-5.redis-cluster-headless"
]
],
conn_opts: []
Any help would be greatly appreciated. Thank you
Implement a Registry-based routing pool strategy instead of using atoms and create them in runtime.
Based on Andrea's Leopardi article Process pools with Elixir's Registry
Currently, bay default, all values except the counters are stored as Nebulex.Object.t()
(before to put the data on Redis, the object is encoded as binary). The idea is to allow storing raw strings, not within the object, and improve the memory consumption for string data types.
Hi
Thanks for all the hard work on this library. We are busy implementing it but have run into the issue that our redis cluster is setup with sentinel support for primary discovery with secondary replicas.
Is sentinel support on the roadmap? If not are you open to PRs for its implementation?
thanks
This is explained well in https://paraxial.io/blog/elixir-rce
Basically, the code below will print I was executed!!!
f = fn _, _ -> IO.puts "I was executed!!!"; {:halt, []} end
serialized = :erlang.term_to_binary(f)
serialized |> :erlang.binary_to_term() |> Enum.to_list()
In Nebulex context this means that any iteration over structures fetched from Nebulex creates an RCE vulnerability through Redis.
Yes, normally Redis is not writable from the outside.... But still it's a "security weakness".
One potential fix would be using https://hexdocs.pm/plug_crypto/Plug.Crypto.html#non_executable_binary_to_term/2
The downsides are:
plug_crypto
(a small one and with no other runtime deps)non_executable_binary_to_term
would iterate over each result fetched from the cache)@cabol I'm happy to make a PR if you suggest a resolution for the above concerns 🙏
When executing a Redis pipeline, it may return {:ok, results}
but the results
may contain one or more Redis errors (Redix.Error
structs). Currently, this scenario is not handled by the adapter causing unexpected behavior. The proposed solution is to check the pipeline results and raise an error if any Redis error is found within them.
Is there a way to have an existing redix connection started and give it to NebulexRedisAdapter/Nebulex at runtime?
Background:
Because we can't really call code from within config.exs et al, this fails to build a release:
config :my_app, MyApp.Cache,
# config for NebulexRedisAdapter from config/prod.exs
conn_opts: [
url: System.get_env("REDIS_ENDPOINT"),
password: {System, :get_env, ["REDIS_TOKEN"]},
sync_connect: true,
socket_opts: [
customize_hostname_check: [
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
]
]
]
Specifically the match_fun
call to :public_key
Erlang module leads to mix release
returning an error:
** (Protocol.UndefinedError) protocol Enumerable not implemented for %Config.Provider{config_path: {:system, "RELEASE_SYS_CONFIG", ".config"}, extra_config: [kernel: [start_distribution: true]], providers: [{Config.Reader, {{:system, "RELEASE_ROOT", "/releases/0.1.0/runtime.exs"}, [env: :prod, target: :host, imports: :disabled]}}], prune_runtime_sys_config_after_boot: false, reboot_system_after_config: true, validate_compile_env: [{:retentions, [:plug_telemetry], {:ok, [level: :info]}}, {:mime, [:types], :error}]} of type Config.Provider (a struct). This protocol is implemented for the following type(s): Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.Stream, DBConnection.PrepareStream, Timex.Interval, Jason.OrderedObject, HashSet, Range, Map, Function, List, Stream, Date.Range, HashDict, GenEvent.Stream, MapSet, File.Stream, IO.Stream
…
While this isn't a very good error, my only change was adding conn_opts for redis, and I was able to narrow it down.
The use case: I want to delete all key that matches "prefix*". Nebulex gives us Nebulex.Cache.delete_all
, but that doesn't work as redis has no native methods to delete all matching keys, so right now this method only works if we send nil
to delete the whole thing.
So, the best way to do this in redis to get all matching keys then delete them in a single DEL call sending a list of keys. Getting all matching keys can be done using Nebulex.Cache.all
but there's no way to delete multiple keys at once, forcing us to do multiple calls
tl,dr: Please implement Nebulex.Cache.delete
in a way we can send a list of keys to be deleted
The idea is to use KEYS command to query data, in this case, only keys can be queried, due to command limitations in Redis.
Given:
defmodule MyCache do
use Nebulex.Cache,
otp_app: :nebulex,
adapter: NebulexRedisAdapter
end
then
iex> MyCache.stats()
** (UndefinedFunctionError) function MyCache.stats/0 is undefined or private, but the behaviour Nebulex.Cache expects it to be present
(my_app 0.1.0) MyCache.stats()
This leaves us unable to obtain Telemetry metrics for this cache.
Hi! I think there's some kind of difference in the Redis response for the CLUSTER SLOTS command between v6 and v7. In v7 I was getting this:
iex(5)> {:ok, cluster_slots} = Redix.command(conn, ["CLUSTER", "SLOTS"])
{:ok,
[
[
0,
16383,
["172.20.0.2", 6379, "71053754ec7ec7bba4396ec6355634afc1a9eb18", []]
]
]}
which made my project break:
** (Mix) Could not start application …
** (EXIT) shutdown: failed to start child: …
** (EXIT) shutdown: failed to start child: {Redix, {0, 16383}}
** (EXIT) an exception was raised:
** (MatchError) no match of right hand side value: [["172.20.0.2", 6379, "71053754ec7ec7bba4396ec6355634afc1a9eb18", []]]
(nebulex_redis_adapter 2.0.0) lib/nebulex_redis_adapter/redis_cluster/supervisor.ex:21: NebulexRedisAdapter.RedisCluster.Supervisor.init/1
(stdlib 3.15.2) supervisor.erl:330: :supervisor.init/1
(stdlib 3.15.2) gen_server.erl:423: :gen_server.init_it/2
(stdlib 3.15.2) gen_server.erl:390: :gen_server.init_it/6
(stdlib 3.15.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
After some investigation I noticed that we were using redis:latest
docker image so I compared versions with my teammates and they had v6. I switched to that version too and the project stopped breaking.
After rebuilding my docker container, v6 response for CLUSTER SLOTS was:
iex(3)> {:ok, cluster_slots} = Redix.command(conn, ["CLUSTER", "SLOTS"])
{:ok,
[[0, 16383, ["172.20.0.15", 6379, "e191e13a8cef4f17b4412e8f7660aee00f655e0f"]]]}
Hope it helps.
We have a much larger application to which we are trying to add a Redis cluster. I kept getting issues with MOVED errors and disconnections. So, I wanted to try using the cluster in much simpler application to see if I could get it to work. I'm not sure if my configuration is off or if clustering is not working.
The local cluster is 7.2.3 installed following: Understand, Setup and Test a Redis Cluster in less than 10 minutes. I had the same issues with Bitnami's docker compose cluster.
defp deps do
[
{:nebulex, "2.5.2"},
{:nebulex_redis_adapter, "2.3.1"}
]
end
config :testcache, Testcache.Cache,
mode: :redis_cluster,
redis_cluster: [
configuration_endpoints: [
conn_opts: [
host: "127.0.0.1",
port: 30001
]
]
]
defmodule Testcache.Cache do
use Nebulex.Cache, otp_app: :testcache, adapter: NebulexRedisAdapter
end
iex(1)> Testcache.Cache.start_link([])
{:ok, #PID<0.650.0>}
iex(2)> Testcache.Cache.put("foo", "bar")
:ok
iex(3)> Testcache.Cache.put("hello", "world")
** (Redix.Error) MOVED 866 127.0.0.1:30001...
Hello,
Thanks for the great lib! I can see that quite some updates have happened since last release on hex.pm.
Would you be able to release?
thanks,
Hey Cabol,
First of all fantastic library, it's super easy to use and works extremely nicely. We've had a small issue building a repeatable docker image for our elixir umbrella app. This is the error that is thrown when including the nebulex_redis_adapter
web_1 | 17:54:26.884 [info] Application jchash exited: exited in: :jchash.start(:normal, [])
web_1 | ** (EXIT) an exception was raised:
web_1 | ** (UndefinedFunctionError) function :jchash.start/2 is undefined or private
web_1 | (jchash) :jchash.start(:normal, [])
web_1 | (kernel) application_master.erl:277: :application_master.start_it_old/4
web_1 | {"Kernel pid terminated",application_controller,"{application_start_failure,jchash,{bad_return,{{jchash,start,[normal,[]]},{'EXIT',{undef,[{jchash,start,[normal,[]],[]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,277}]}]}}}}}"}
web_1 | Kernel pid terminated (application_controller) ({application_start_failure,jchash,{bad_return,{{jchash,start,[normal,[]]},{'EXIT',{undef,[{jchash,start,[normal,[]],[]},{application_master,start_it_old
web_1 |
web_1 | Crash dump is being written to: erl_cra
Tested with: nebulex_redis_adapter ~> "1.0.0"
& nebulex_redis_adapter ~> "1.1.0"
Currently the nebulex redis adapter is not grabbing and passing the ssl config option to Redix.
Currently, the adapter encodes and decodes Redis keys and values automatically underneath, it encodes any Elixir term to a binary and decodes binaries into Elixir terms. The idea is to allow the user to optionally provide a custom codec with the option :codec
. The value must be a module implementing the NebulexRedisAdapter.Codec
behaviour.
This is how the behaviour may look like:
defmodule NebulexRedisAdapter.Codec do
@moduledoc """
Codec interface.
"""
@doc """
Encodes `key` with the given `opts`.
"""
@callback encode_key(key :: term, opts :: [term]) :: iodata
@doc """
Encodes `value` with the given `opts`.
"""
@callback encode_value(value :: term, opts :: [term]) :: iodata
@doc """
Decodes `key` with the given `opts`.
"""
@callback decode_key(key :: binary, opts :: [term]) :: term
@doc """
Decodes `value` with the given `opts`.
"""
@callback decode_value(value :: binary, opts :: [term]) :: term
end
Currently, the Redis adapter provides the extended functions command!/3
and pipeline!/3
, but they are bang functions, which means they raise an exception when an error occurs. The idea is to add support for the counterpart functions command/3
and pipeline/3
, returning Ok/Error tuples.
Currently, the Redis Cluster setup is very naive, when the cache is started, it tries to get the cluster configuration from Redis and then configure the supervision tree based on it. The problem is it is static once the cache starts, and it doesn't handle cluster changes or "MOVED"
errors. The idea is to provide a smarter cluster management strategy able to handle cluster changes as well as "MOVED"
errors dynamically.
"MOVED"
error."MOVED"
error, the configuration manager is called to re-configure the cluster and once it is done, the failed command is executed again.:master_nodes
option which is quite confusing and use only :conn_opts
instead. The idea here is to configure only the configuration endpoint via the :conn_opts
; this is where the client should connect to send the "CLUSTER SHARDS" (Redis >= 7) or "CLUSTER SLOTS" (Redis < 7) command to get the cluster information and set it up on the client side.Allow passing Redis client option to the extended cache functions like command
and pipeline
.
Currently, these functions look like this:
command(key \\ nil, name \\ __MODULE__, command)
command!(key \\ nil, name \\ __MODULE__, command)
pipeline(key \\ nil, name \\ __MODULE__, commands)
pipeline!(key \\ nil, name \\ __MODULE__, commands)
They don't admit extra options. Therefore, the idea is to make them like so:
command(command, opts \\ [])
command!(command, opts \\ [])
pipeline(commands, opts \\ [])
pipeline!(commands, opts \\ [])
Add add :key
and :name
to the supported options.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.