Presence stops working after ~1 week about phoenix_pubsub HOT 5 OPEN

Can you provide more information about your setup? Approximate scale as far as number of simultaneous users/presence entries, etc? As you're hitting a system limit error, something is out of bounds, possibly the number of entries we build in our matchspec, but we've never seen this before so more information would be helpful. Thanks!

from phoenix_pubsub.

The error message says:

** (SystemLimitError) a system limit has been reached due to errors at the given arguments:

How many nodes are you running? My guess is that the nodes are never marked as permanently down, so the list only grows, until the match specification grows too long. There is a permdown_period, have you changed it in any way?

from phoenix_pubsub.

This is a fly.io cluster in 8 regions, with ~2k concurrent users. There is typically ~10 machines in the cluster, and they stop automatically when there's nobody connected to them for 15 minutes, and start again when there's demand.

I did not change permdown_period.

from phoenix_pubsub.

It might be possible to avoid building massive match specs for very large sets of values leveraging is_map_key with a match spec like:

map = :maps.from_keys(values_to_fiter_out, [])
[{{{key, :_, :"$1"}, :"$2", {:"$3", :_}}, [{:not, {:is_map_key, :"$3", map}}], [{:"$1", :"$2"}]}]

This should hopefully avoid this crash (separate issue is why the list grows so big in the first place). There's also a need to asses performance of this solution vs what's used right now.

from phoenix_pubsub.

Note: on fly I think node names are chosen randomly during startup, so I think Jose's guess is correct

from phoenix_pubsub.