FlowControl

Thereare two flow control mechanisms in RabbitMQ. Both work by exertingTCP backpressure on connections that are publishing too fast. Theyare:

  • Aper-connection mechanism that prevents messages being publishedfaster than they can be routed to queues.

  • Aglobal mechanism that prevents any messages from being publishedwhen the memoryusage exceeds a configured threshold or free disk space dropsbelow a configured threshold.

Bothmechanisms will temporarily block connections - the serverwill pause reading from the sockets of connected clients which sendcontent-bearing methods (such as basic.publish) which have beenblocked. Connection heartbeat monitoring will be disabled too.

Theintent here is to introduce a flow control mechanism that throttlesproducers but lets consumers continue unaffected. However, sinceAMQP permits producers and consumers to operate on the same channel,and on different channels of a single connection, this logic isnecessarily imperfect. In practice that does not pose any problemsfor most applications since the throttling is observable merely as adelay. Nevertheless, other design considerations permitting, it isadvisable to only use individual AMQP connections for eitherproducing or consuming.

Per-ConnectionFlow Control

RabbitMQwill block connections which are publishing too quickly for queuesto keep up. No configuration is required.

Ablocked connection will show a state of blocked in rabbitmqctl andthe management plugin HTTP API. Typically such fast-publishingconnections will be blocked and unblocked several times per second,so inspecting a connection's state may not show per-connection flowcontrol happening clearly. The last_blocked_by and last_blocked_agefields in rabbitmqctl and the management API can be inspected todetermine whether flow control is taking place.

Themanagement plugin web UI unifies this information into a singlefield, showing flow for connections which have recently been in flowcontrol.

Memory-BasedFlow Control

TheRabbitMQ server detects the total amount of RAM installed in thecomputer on startup and when rabbitmqctlset_vm_memory_high_watermark fraction is executed. Bydefault, when the RabbitMQserver uses above 40% of the installed RAM, it raises a memory alarmand blocks all connections. Once the memory alarm hascleared (e.g. due to the server paging messages to disk ordelivering them to clients) normal service resumes.

Whenrunning RabbitMQ in a cluster,the memory alarm is cluster-wide; if one node goes over the limitthen all nodes will block connections.

Thedefault memory threshold is set to 40% of installed RAM. Note thatthis does not prevent the RabbitMQ server from using more than 40%,it is merely the point at which publishers are throttled. Erlang'sgarbage collector can, in the worst case, cause double the amount ofmemory to be used (by default, 80% of RAM). It is stronglyrecommended that OS swap or page files are enabled.

32-bitarchitectures tend to impose a per process memory limit of 2GB.Common implementations of 64-bit architectures (i.e. AMD64 and IntelEM64T) permit only a paltry 256TB per process. 64-bit Windowsfurther limits this to 8TB. However, note that even under 64-bitOSes, a 32-bit process frequently only has a maximum address spaceof 2GB.

Configuringthe Memory Threshold

Thememory threshold at which the flow control is triggered can beadjusted by editing the configurationfile. The example below sets the threshold to the default valueof 0.4: 

[{rabbit, [{vm_memory_high_watermark, 0.4}]}].

Thedefault value of 0.4 stands for 40% of installed RAM or 40% ofavailable virtual address space, whichever is smaller. E.g. on a32-bit platform, if you have 4GB of RAM installed, 40% of 4GB is1.6GB, but 32-bit Windows normally limits processes to 2GB, so thethreshold is actually to 40% of 2GB (which is 820MB).

Avalue of 0 makes the memory alarm go off immediately and thusdisables all publishing (this may be useful if you wish to disablepublishing globally; use rabbitmqctl set_vm_memory_high_watermark0). To prevent the memory alarm from going off at all, set some highmultiplier such as 100.

Thememory limit is appended to the RABBITMQ_NODENAME.log filewhen the RabbitMQ server starts: 

=INFO REPORT==== 29-Oct-2009::15:43:27 ===Memory limit set to 2048MB.

Thememory limit may also be queried using the rabbitmqctl statuscommand.

Thethreshhold can be changed while the broker is running using therabbitmqctl set_vm_memory_high_watermark fraction command.This command will take effect until the broker shuts down. Thecorresponding configuration setting should also be changed when theeffects should survive a broker restart. The memory limit may changeon systems with hot-swappable RAM when this command is executedwithout altering the threshhold, due to the fact that the totalamount of system RAM is queried.

LimitedAddress Space

Whenrunning RabbitMQ inside a 32 bit Erlang VM in a 64 bit OS, theaddressable memory is limited. The server will detect this and log amessage like: 

=WARNING REPORT==== 19-Dec-2013::11:27:13 ===Only 2048MB of 12037MB memory usable due to limited address space.

Thememory alarm system is not perfect. While stopping publishing willusually prevent any further memory from being used, it is quitepossible for other things to continue to increase memory use.Normally when this happens and the physical memory is exhausted theOS will start to swap. But when running with a limited addressspace, running over the limit will cause the VM to crash.

Itis therefore strongly recommended that when running on a 64 bit OSyou use a 64 bit Erlang VM.

Configuringthe Paging Threshold

Beforethe broker hits the high watermark and blocks publishers, it willattempt to free up memory by instructing queues to page theircontents out to disc. Both persistent and transient messages will bepaged out (the persistent messages will already be on disc but willbe evicted from memory).

Bydefault this starts to happen when the broker is 50% of the way tothe high watermark (i.e. with a defaulthigh watermark of 0.4, this is when 20% of memory is used).To change this value, modify thevm_memory_high_watermark_paging_ratioconfiguration from its default value of 0.5. Forexample: 

[{rabbit, [{vm_memory_high_watermark_paging_ratio, 0.75},{vm_memory_high_watermark, 0.4}]}].

Theabove configuration starts paging at 30% of memory used, and blockspublishers at 40%.

Itis possible to set vm_memory_high_watermark_paging_ratio to agreater value than 1.0. In this case queues will not page theircontents to disc. If this causes the memory alarm to go off, thenproducers will be blocked as explained above.

Unrecognisedplatforms

Ifthe RabbitMQ server is unable to recognise your system, it willappend a warning to the RABBITMQ_NODENAME.log file. It thenassumes than 1GB of RAM is installed: 

=WARNING REPORT==== 29-Oct-2009::17:23:44 ===Unknown total memory size for your OS {unix,magic_homebrew_os}. Assuming memory size is 1024MB.

Inthis case, the vm_memory_high_watermark configuration value is usedto scale the assumed 1GB RAM. With the default value ofvm_memory_high_watermark set to 0.4, RabbitMQ's memory threshold isset to 410MB, thus it will throttle producers whenever RabbitMQ isusing more than 410MB memory. Thus when RabbitMQ can't recognizeyour platform, if you actually have 8GB RAM installed and you wantRabbitMQ to throttle producers when the server is using above 3GB,set vm_memory_high_watermark to 3.

Itis advised you do not set the threshold above 50% of your installedRAM.

Disk-BasedFlow Control

RabbitMQcan also block producers when free disk space drops below a certainlimit. This is a good idea since even transientmessages can be paged to disk at any time, and running out of diskspace can cause the server to crash. By default RabbitMQ will blockproducers, and prevent memory-based messages from being paged todisk, when free disk space drops below 50MB. This will reduce butnot eliminate the likelihood of a crash due to disk space beingexhausted. In particular, if messages are being paged out rapidly itis possible to run out of disk space and crash in the time betweentwo runs of the disk space monitor. A more conservative approachwould therefore be to set the limit to the same as the amount ofmemory installed on the system (see below).

Globalflow control will be triggered if the amount of free disk spacedrops below a configured limit. The free space of the drive orpartition that the broker database uses will be monitored everyminute to determine whether the alarm should be raised or cleared.Monitoring will start as soon as the broker starts up, causing anentry in the broker logfile: 

=INFO REPORT==== 23-Jun-2012::14:52:41 ===Disk free limit set to 953MB

Monitoringwill be disabled on unrecognised platforms, causing an entry such asthe one below: 

=WARNING REPORT==== 23-Jun-2012::15:45:29 ===Disabling disk free space monitoring

Whenrunning RabbitMQ in a cluster, the disk alarm is cluster-wide; ifone node goes under the limit then all nodes will block connections.

RabbitMQwill periodically check the amount of free disk space. The frequencywith which disk space is checked is related to the amount of spaceat the last check (in order to ensure that the disk alarm goes offin a timely manner when space is exhausted). Normally disk space ischecked every 10 seconds, but as the limit is approached thefrequency increases. When very near the limit RabbitMQ will check asfrequently as 10 times per second. This may have some effect onsystem load.

Configuringthe Disk Free Space Limit

Thedisk free space limit is configured with the disk_free_limitsetting. By default 50MB is required to be free on thedatabase partition. (See the description of filelocations for the default location). This configuration filesets the disk free space limit to 1GB: 

[{rabbit, [{disk_free_limit, 1000000000}]}].

Itis also possible to set a free space limit relative to the RAM inthe machine. This configuration file sets the disk free space limitto the same as the amount of RAM on the machine: 

[{rabbit, [{disk_free_limit, {mem_relative, 1.0}}]}].

ClientNotifications

RabbitMQprovides an extension to AMQP 0-9-1 that allows clients to benotifiedwhen a connection is blocked.