How to recover RabbitMQ after running out of disk space

fix-rabbitmq-after-disk-space-full.markdown

If RabbitMQ wont start and show the following in the startup logs:

# cat  /var/log/rabbitmq/startup_log


BOOT FAILED
===========

Error description:
   {error,corrupt_cluster_status_files,[]}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

Stack trace:
   [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1},
    {rabbit_node_monitor,prepare_cluster_status_files,0},
    {rabbit,'-boot/0-fun-1-',0},
    {rabbit,start_it,1},
    {init,start_it,1},
    {init,start_em,1}]

{"init terminating in do_boot",{rabbit,failure_during_boot,{error,corrupt_cluster_status_files,[]}}}

# cat /var/log/rabbitmq/[email protected]

=INFO REPORT==== 25-Mar-2014::04:43:36 ===
Error description:
   {error,corrupt_cluster_status_files,[]}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

Stack trace:
   [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1},
    {rabbit_node_monitor,prepare_cluster_status_files,0},
    {rabbit,'-boot/0-fun-1-',0},
    {rabbit,start_it,1},
    {init,start_it,1},
    {init,start_em,1}]


=INFO REPORT==== 25-Mar-2014::04:44:24 ===
Error description:
   {error,corrupt_cluster_status_files,[]}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

Stack trace:
   [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1},
    {rabbit_node_monitor,prepare_cluster_status_files,0},
    {rabbit,'-boot/0-fun-1-',0},
    {rabbit,start_it,1},
    {init,start_it,1},
    {init,start_em,1}]


=INFO REPORT==== 25-Mar-2014::04:44:39 ===
Error description:
   {error,corrupt_cluster_status_files,[]}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

Stack trace:
   [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1},
    {rabbit_node_monitor,prepare_cluster_status_files,0},
    {rabbit,'-boot/0-fun-1-',0},
    {rabbit,start_it,1},
    {init,start_it,1},
    {init,start_em,1}]


=INFO REPORT==== 25-Mar-2014::04:46:27 ===
Error description:
   {error,corrupt_cluster_status_files,[]}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

Stack trace:
   [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1},
    {rabbit_node_monitor,prepare_cluster_status_files,0},
    {rabbit,'-boot/0-fun-1-',0},
    {rabbit,start_it,1},
    {init,start_it,1},
    {init,start_em,1}]


=INFO REPORT==== 25-Mar-2014::04:48:06 ===
Error description:
   {error,corrupt_cluster_status_files,[]}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

Stack trace:
   [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1},
    {rabbit_node_monitor,prepare_cluster_status_files,0},
    {rabbit,'-boot/0-fun-1-',0},
    {rabbit,start_it,1},
    {init,start_it,1},
    {init,start_em,1}]

Manually purge the Queues by removing them from /var/lib/rabbitmq/mnesia/ and then starting the service.

# rm -fr /var/lib/rabbitmq/mnesia/*
# service rabbitmq-server start

We don't actually have to purge the entire mnesia dir. All that we actually need is to just remove the rabbitmq cluster_nodes.config file and then restart rabbit.

rm /var/lib/rabbitmq/mnesia/rabbit@<hostname>/cluster_nodes.config
service rabbitmq-server start

Thanks! This worked for me ;)

Though this won't so much recover - as delete all your data and start over again

rm -fr /var/lib/rabbitmq/mnesia/*

Does this command delete all rabbit users? I lost them after this command

Hi,
Can anyone let me know how to catch exception at publisher end when rabbitmq disk is full.
this.XXChannel.sendNotification().send(notificationSendEventMessage);
I am expecting some exception while executing above line incase disk is full.

I am using spring cloud stream for my configuration.

rm -fr /var/lib/rabbitmq/mnesia/*

Does this command delete all rabbit users? I lost them after this command

Yes. Need to add back the user using rabbitmqctl command