Handle delete_and_terminate throws in recovery

[the-mikedavis]

ra_server:recover/1 folded through the log but did not catch the throw for delete_and_terminate. This leads to a crash loop on boot if the server appends a deletion command, does not apply it, and then reboots. We can try/catch and transition to terminating_follower instead.

I saw this in a 3.13 cluster recently and saw that it's still possible on the latest main, at least theoretically. I'll post a redacted snippet of the raft log I captured where this occurred.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[the-mikedavis]

Raft log dump (redacted)...

Persisted last applied was index 21, right before the deletion command.

[{0,0,undefined},
 {1,1,{noop,#{ts => 123456789},3}},
 {2,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {update_config,
          #{name => my_vhostname_my_queuename,
            max_length => undefined,max_bytes => undefined,
            created => 123456789,
            queue_resource => {resource,<<"my_vhostname">>,queue, <<"my_queuename">>},
            dead_letter_handler =>
                {at_most_once,
                    {rabbit_quorum_queue,dead_letter_publish,
                        [{resource,<<"my_vhostname">>,exchange,<<"my_exchangename">>},
                         <<"yada_yada">>,
                         {resource,<<"my_vhostname">>,queue,<<"my_queuename">>}]}},
            become_leader_handler =>
                {rabbit_quorum_queue,become_leader,
                    [{resource,<<"my_vhostname">>,queue,<<"my_queuename">>}]},
            overflow_strategy => drop_head,delivery_limit => undefined,
            expires => 190000,msg_ttl => 180000,
            single_active_consumer_on => false,
            max_in_memory_length => undefined,
            max_in_memory_bytes => undefined}},
      await_consensus}},
 {3,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {4,1,
  {'$usr',
      #{ts => 123456789},
      {enqueue,Pid,1, #mc{...}},
      {notify,0,Pid}}},
 {5,1,
  {'$usr',#{ts => 123456789},{down,Pid,normal},noreply}},
 {6,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {7,1,
  {'$usr',
      #{ts => 123456789},
      {enqueue,Pid,1,#mc{...}},
      {notify,0,Pid}}},
 {8,1,
  {'$usr',#{ts => 123456789},{down,Pid,normal},noreply}},
 {9,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {10,1,
  {'$usr',
      #{ts => 123456789},
      {enqueue,Pid,1,#mc{...}},
      {notify,0,Pid}}},
 {11,1,
  {'$usr',#{ts => 123456789},{down,Pid,normal},noreply}},
 {12,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {13,1,
  {'$usr',
      #{ts => 123456789},
      {enqueue,Pid,1,#mc{...}},
      {notify,0,Pid}}},
 {14,1,
  {'$usr',#{ts => 123456789},{down,Pid,normal},noreply}},
 {15,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {16,1,
  {'$usr',
      #{ts => 123456789},
      {enqueue,Pid,1,#mc{...}},
      {notify,0,Pid}}},
 {17,1,
  {'$usr',#{ts => 123456789},{down,Pid,normal},noreply}},
 {18,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {19,1,
  {'$usr',
      #{ts => 123456789},
      {enqueue,Pid,1,#mc{...}},
      {notify,0,Pid}}},
 {20,1,
  {'$usr',#{ts => 123456789},{down,Pid,normal},noreply}},
 {21,1,{'$usr',#{ts => 123456789},{timeout,expire_msgs},noreply}},
 {22,1,
  {'$ra_cluster',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      delete,await_consensus}},
 {23,1,
  {'$usr',
      #{ts => 123456789,
        from => {Pid, [alias|Reference]}},
      {register_enqueuer,Pid},
      await_consensus}},
 {24,2,{noop,#{ts => 123456789},3}},
 {25,2,{'$usr',#{ts => 123456789},{timeout,expire_msgs},noreply}},
 {26,2,
  {'$usr',
      #{ts => 123456789,
        from =>
            {Pid,
             [alias|Reference]}},
      {update_config,
          #{name => my_vhostname_my_queuename,
            max_length => undefined,max_bytes => undefined,
            created => 1785155367902,
            queue_resource => {resource,<<"my_vhostname">>,queue,<<"my_queuename">>},
            dead_letter_handler =>
                {at_most_once,
                    {rabbit_quorum_queue,dead_letter_publish,
                        [{resource,<<"my_vhostname">>,exchange,<<"my_exchangename">>},
                         <<"yada_yada">>,
                         {resource,<<"my_vhostname">>,queue,<<"my_queuename">>}]}},
            become_leader_handler =>
                {rabbit_quorum_queue,become_leader,
                    [{resource,<<"my_vhostname">>,queue,<<"my_queuename">>}]},
            overflow_strategy => drop_head,delivery_limit => undefined,
            expires => 190000,msg_ttl => 180000,
            single_active_consumer_on => false,
            max_in_memory_length => undefined,
            max_in_memory_bytes => undefined}},
      await_consensus}},
 {27,2,{'$usr',#{ts => 123456789},{timeout,expire_msgs},noreply}}]

Log crash snippet...

[error] <0.7905.0> ** State machine my_vhostname_my_queuename terminating
[error] <0.7905.0> ** Last event = {internal,go}
[error] <0.7905.0> ** When server state  = [{id,
[error] <0.7905.0>                           {my_vhostname_my_queuename,
...
[error] <0.7905.0>                             effective_handle_aux_fun => {handle_aux,6}}}]
[error] <0.7905.0> ** Reason for termination = error:{bad_return_from_state_function,
[error] <0.7905.0>                                    {delete_and_terminate,
[error] <0.7905.0>                                     #{log =>
[error] <0.7905.0>                                        {ra_log,
...
[error] <0.7905.0>                                       {wrap_reply,ok}}]}}
[error] <0.7905.0> ** Callback modules = [ra_server_proc]
[error] <0.7905.0> ** Callback mode = [state_functions,state_enter]
[error] <0.7905.0> ** Stacktrace =
[error] <0.7905.0> **  [{gen_statem,loop_state_callback_result,11,
[error] <0.7905.0>                  [{file,"gen_statem.erl"},{line,1549}]},
[error] <0.7905.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]
[error] <0.7905.0>
[error] <0.7905.0>   crasher:
[error] <0.7905.0>     initial call: ra_server_proc:init/1
[error] <0.7905.0>     pid: <0.7905.0>
[error] <0.7905.0>     registered_name: my_vhostname_my_queuename
[error] <0.7905.0>     exception error: {bad_return_from_state_function,
[error] <0.7905.0>                       {delete_and_terminate,
[error] <0.7905.0>                        #{log =>
[error] <0.7905.0>                           {ra_log,
...