Unexpected behavior: bad_return application start failure using Mnesia outside of Honeydew about honeydew HOT 5 CLOSED

Huh, that's interesting. I'm not sure what caused that, my only guess without being able to inspect the environment is that mnesia was waiting on one of the other nodes in the cluster to become ready.

from honeydew.

I'm still not sure what was causing this, but if anyone else runs into it, deleting my mnesia file and redeploying appeared to work.

from honeydew.

@koudelka - well, it ended up coming back, but resulting in another error:

{"Kernel pid terminated",application_controller,"{application_start_failure,growglad,{bad_return,{{'Elixir.Growglad.Application',start,[normal,[]]},{'EXIT',{{badmatch,{error,{{{badmatch,{timeout,['honeydew_:queue','honeydew_:queue_in_progress']}},[{'Elixir.Honeydew.Queue.Mnesia',init,2,[{file,\"lib/honeydew/queue/mnesia.ex\"},{line,88}]},{'Elixir.Honeydew.Queue',init,1,[{file,\"lib/honeydew/queue.ex\"},{line,97}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]},{child,undefined,queue,{'Elixir.Honeydew.Queue',start_link,[[queue,'Elixir.Honeydew.Queue.Mnesia',[{ram_copies,['[email protected]']}],{'Elixir.Honeydew.Dispatcher.LRU',[]},{'Elixir.Honeydew.FailureMode.Abandon',[]},nil,false]]},transient,5000,worker,['Elixir.Honeydew.Queue']}}}},[{'Elixir.Growglad.Application',start,2,[{file,\"lib/growglad/application.ex\"},{line,34}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,277}]}]}}}}}"}

I scaled down to one instance, to see if there was some dissonance between the different nodes. Deleted the Mnesia file and rebuilt it, and can't seem to get rid of this error.

The environment is:

• Erlang/OTP 22 [erts-10.7.2.5] [source] [64-bit] [smp:8:4] [ds:8:4:10] [async-threads:1] [hipe]
• Elixir 1.9.4 (compiled with Erlang/OTP 22)

The only thing that's changed is how I'm starting Honeydew:

:ok = Honeydew.start_queue(:queue)
:ok = Honeydew.start_workers(:queue, Growglad.Workers)

from honeydew.

Oh sorry, by "inspect the environment" I meant "poke around the machines." :)

The line in question is here, it looks like mnesia itself is waiting for the tables to become available, it's out of Honeydew's hands at that point.

You might try commenting out your start_queue line and then bringing the application up. Then looking at the mnesia status on all the nodes in your cluster, one of them might be having trouble creating the tables.

Check your inter-node connectivity, one of the nodes your specified as an mnesia secondary may not be contactable.

Does the mnesia schema exist on all of your nodes, too? Without the schema on all nodes, it might not be able to create the tables.

from honeydew.

I appreciate the feedback! I ended up toying around with it, and it appears there's some conflict with Pow when run with its Mnesia cache as well. It appears as though they're both creating a schema on each deploy and either:

• They're obliterating the tables on net split
• Pow is starting Mnesia and then Honeydew is throwing an error because it's not able to start Mnesia and see's that its queue is already in progress

I could be totally wrong, but this my assumption after verifying inter-node connectivity and running both Pow and Honeydew across multiple nodes. I went back to using Redis for Pow's cache for now, but when I get more time I'll look into why this is happening and try to fix it.

I'm thinking I'll need to manually create the schema for both Pow and Honeydew and do some kind of custom handler for starting the Queue. 🤷‍♂️ I'll keep you posted though! Have a happy 2021!

from honeydew.