Deadlock when calling lasp:query/1 about lasp HOT 2 OPEN

Looking at the traces now I see that the lasp:query call in step 1 blocks because the lasp_distribution_backend process is already handling a different request - it's probably coming from a lasp:declare_dynamiccall.

from lasp.

I looked more at the traces and found the following call flows leading to this deadlock:

1. A partisan_peer_service_client receives a TCP(?) message on a socket in a handle_info function. It decodes the message into {state, Tag, LocalState}, calls handle_message which will send a {connected, ...} message to the partisan_pluggable_peer_service_manager process.
2. The partisan_pluggable_peer_service_manager receives the {connected, ...} message in the handle_info function, calls partisan_peer_service_events:update which will call gen_event:sync_notify and blocks here until the partisan_peer_service_events process handles the notification.
3. The partisan_peer_service_events process receives the update event in the handle_event function and will call lasp_membership:update_membership (that's the callback in the state), which in turn eventually calls lasp_distribution_backend:declare_dynamic and blocks here until the lasp_distribution_backend process handles the declaration.
4. The lasp_distribution_backend process receives the {declare_dynamic, ...} call in the handle_call function, eventually calls a lasp_storage_backend:get and blocks here until the lasp_storage_backend process handles the {get, ...} call. The problem is that the lasp_storage_backend process is blocked (see step 2 in the call flow below).

The other call flow:

1. The lasp_state_based_synchronization_backend receives a {state_sync, ...} message (I think it is received from a timer), calls the init_state_sync function which calls lasp_storage_backend:fold and it blocks here until the call returns.
2. The lasp_storage_backend process receives a {fold, Function, ...} request where Function is the function created in lasp_state_based_synchronization_backend:init_state_sync. Then eventually calls lasp_ets_storage_backend:fold with the same function and blocks here until the call returns.
3. The lasp_ets_storage_backend process receives the {fold, Function, ...} request and calls ets:fold(Function, ...) which will call the inline function from lasp_state_based_synchronization_backend:init_state_sync. This anonymous function calls lasp_synchronization_backend:send which will call partisan_pluggable_peer_service_manager:cast_message. Despite it's name this function ends up doing a gen_server:call on the partisan_pluggable_peer_service_manager process, but that's blocked (see step 2 in the above call flow).

So it looks like the race is between two flows: an external TCP packet triggers a partisan_pluggable_peer_service_manager -> lasp_storage_backend chain (see the first flow above), while a scheduled {state_sync, ...} message triggers a lasp_storage_backend -> partisan_pluggable_peer_service_manager chain (see the second flow above). If these happen about the same time, we have the deadlock.

I don't know what these flows supposed to do, my guess based on variable and function names is that both want to update some kind of internal state, who are in the cluster. I looked into the '[email protected]' ets table in a different run and saw this:

[{{<<"_membership">>,lwwregister},
  {dv,{state_lwwregister,{1611854259,
                          ['[email protected]',
                           '[email protected]',
                           '[email protected]']}},
      [],[],[],lwwregister,
      [{clock,[{<<112,81,31,35,249,226,203,99,138,234,42,146,124,20,143,158,12,
                  185,230,4>>,
                9}]},
       {dynamic,true}],
      0,[],[]}}]

so it looks line host2 is missing (maybe that's being added).

from lasp.