MongoDB性能优化之连接优化

本文是BoxedIce公司遇到的一个真实案例，如果你正在使用MongoDB的话，此文又是一个实用教材。

BoxedIce在plugin directory项目中添加了新的WEB机器结点后，发现其MongoDB集群的性能开始受到影响，表现在请求的高峰时期，MongoDB的响应时间变得非常长。

通过监控发现，在其高峰时间MongoDB的连接数达到了1100～1500左右，由于每个连接需要使用10M（stack size默认为10240）的内存，这导致相当大的内存开销。

BoxedIce的处理方法是，首先通过优化连接池，将连接数控制在了800个左右，然后通过修改内核的stack size值，从默认的10M修改到1M，使连接占用的内存大大减少。

After adding new web nodes for the launch of our plugin directory, we started seeing performance problems with our MongoDB database cluster. The symptom was increased response times in the main web app and using mongostat we were able to see this was being caused by queue spikes in MongoDB as queries were backing up waiting to get executed.

We spent several days optimising queries and even the servers mongod itself was running on (increasing RAM, removing non-essential services, disabling cron jobs) but continued to see the spikes even with significantly reduced load and more than sufficient memory for the indexes + data size.

Through discussions with 10gen, the company behind MongoDB, we narrowed down the problem to the number of connections to the main cluster. This had gone from ~1100 per mongod node to ~1500 (as a result of increased web nodes and traffic). It turns out that every connection has a fairly large overhead – 10MB on Linux – and this requires sufficient RAM to accommodate all connections. Even with the increased RAM and reduced load, connection overhead + data size + index size well exceeded the total available RAM.

So we tweaked our connection pooling and optimised how the web nodes use themongos routers to reduce the number of connections to around 800. This helped significantly but we were still concerned with the per connection overhead of 10MB.

This value is based on the Linux stack size, which defaults to 10240:

david@rs1a ~: ulimit -a
...
stack size (kbytes, -s) 10240

10gen suggested that they had done some testing on changing this value to 1024to reduce the overhead to just 1MB. However, this hasn’t been as extensively tested as with the defaults so we decided to implement on one of our shards to test. The improvement was immediately noticable and after a period of testing, we deployed this change to all our servers.

On CentOS / Red Hat, this can be changed in the /etc/security/limits.conf file by adding the following 2 lines:

david hard stack 1024
david soft stack 1024

You should replace david with the name of the user MongoDB runs as. Log out and log back in and run ulimit -s to confirm the change has taken effect, then restartmongod. At low loads, you may not see any effect but as usage increases this significantly reduces the amount of RAM you need.