Free Disk Space Alarms

RabbitMQ

Features
Get Started
Support
Community
Docs
Blog

Overview

When free disk space drops below a configured limit (50 MB by default), an alarm will be triggered and all producers will be blocked.

The goal is to avoid filling up the entire disk which will lead all write operations on the node to fail and can lead to RabbitMQ termination.
How it Works

To reduce the risk of filling up the disk, all incoming messages are blocked. Transient messages, which aren’t normally persisted, are still paged out to disk when under memory pressure, and will use up the already limited disk space.

If the disk alarm is set too low and messages are paged out rapidly, it is possible to run out of disk space and crash RabbitMQ in between disk space checks (at least 10 seconds apart). A more conservative approach would be to set the limit to the same as the amount of memory installed on the system (see the configuration below).

An alarm will be triggered if the amount of free disk space drops below a configured limit.

The free space of the drive or partition that the broker database uses will be monitored at least every 10 seconds to determine whether the disk alarm should be raised or cleared.

Monitoring will begin on node start. It will leave a log entry like this:

2019-04-01 12:02:11.564 [info] <0.329.0> Enabling free disk space monitoring
2019-04-01 12:02:11.564 [info] <0.329.0> Disk free limit set to 950MB

Free disk space monitoring will be disabled on unrecognised platforms, causing an entry such as the one below:

2019-04-01 11:04:54.002 [info] <0.329.0> Disabling disk free space monitoring

When running RabbitMQ in a cluster, the disk alarm is cluster-wide; if one node goes under the limit then all nodes will block incoming messages.

RabbitMQ periodically checks the amount of free disk space. The frequency with which disk space is checked is related to the amount of space at the last check. This is in order to ensure that the disk alarm goes off in a timely manner when space is exhausted. Normally disk space is checked every 10 seconds, but as the limit is approached the frequency increases. When very near the limit RabbitMQ will check as frequently as 10 times per second. This may have some effect on system load.

When free disk space drops below the configured limit, RabbitMQ will block producers and prevent memory-based messages from being paged to disk. This will reduce the likelihood of a crash due to disk space being exhausted, but will not eliminate it entirely. In particular, if messages are being paged out rapidly it is possible to run out of disk space and crash in the time between two runs of the disk space monitor. A more conservative approach would be to set the limit to the same as the amount of memory installed on the system (see the configuration section below).
Configuring Disk Free Space Limit

The disk free space limit is configured with the disk_free_limit setting. By default 50MB is required to be free on the database partition (see the description of file locations for the default database location). This configuration file sets the disk free space limit to 1GB:

disk_free_limit.absolute = 1000000000

Or you can use memory units (KB, MB GB etc.) like this:

disk_free_limit.absolute = 1GB

Or, using the classic config format:

[{rabbit, [{disk_free_limit, 1000000000}]}].

It is also possible to set a free space limit relative to the RAM in the machine. This configuration file sets the disk free space limit to the same as the amount of RAM on the machine:

disk_free_limit.relative = 1.0

Or, using the classic config format:

[{rabbit, [{disk_free_limit, {mem_relative, 1.0}}]}].

The limit can be changed while the broker is running using the rabbitmqctl set_disk_free_limit disk_limit command or rabbitmqctl set_disk_free_limit mem_relative fraction command. This command will take effect until next node restart.

The corresponding configuration setting should also be changed when the effects should survive a node restart.
Getting Help and Providing Feedback

If you have questions about the contents of this guide or any other topic related to RabbitMQ, don’t hesitate to ask them on the RabbitMQ mailing list.
Help Us Improve the Docs ❤️

If you’d like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!
In This Section

Server Documentation
    Configuration
    Management UI
    Monitoring
    Production Checklist
    TLS Support
    Feature Flags
    Distributed RabbitMQ
    Clustering
    Reliable Delivery
    Backup and restore
    Alarms
        Memory Alarms
        Disk Alarms
        Memory Use
        Flow Control
    Memory Use
    Networking
    Virtual Hosts
    High Availability (pacemaker)
    Access Control (Authorisation)
    Authentication Mechanisms
    LDAP
    Lazy Queues
    Internal Event Exchange
    Firehose (Message Tracing)
    Manual Pages
    Windows Quirks
Client Documentation
Plugins
News
Protocol
Our Extensions
Building
Previous Releases
License

RabbitMQ

Features
Get Started
Support
Community
Docs
Blog

Copyright © 2007-2020 VMware, Inc. or its affiliates. All rights reserved. Terms of Use, Privacy and Trademark Guidelines