postfix性能调整(一)

Postfix Performance Tuning

---

Purpose of Postfix performance tuning

The hints and tips in this document help you improve theperformance of Postfix systems that already work. If your Postfixsystem is unable to receive or deliver mail, then you need to solvethose problems first, using the DEBUG_README document as guidance.

For tuning external content filter performance, first read therespective information in the FILTER_README and SMTPD_PROXY_READMEdocuments. Then make sure to avoid latency in the content filtercode. As much as possible avoid performing queries against externaldata sources with a high or highly variable delay. Your contentfilter will run with a small concurrency to avoid CPU/memorystarvation, and if any latency creeps in, content filter throughputwill suffer. High volume environments should avoid RBL lookups,complex database queries and so on.

Topics on mail receiving performance:

• General mail receiving performance tips
• Doing more work with your SMTP server processes
• Slowing down SMTP clients that make many errors
• Measures against clients that make too many connections

Topics on mail delivery performance:

• General mail delivery performance tips
• Tuning the frequency of deferred mail delivery attempts
• Tuning the number of simultaneous deliveries
• Tuning the number of recipients per delivery

Other Postfix performance tuning topics:

• Tuning the number of Postfix processes
• Tuning the number of processes on the system
• Tuning the number of open files orsockets

The following tools can be used to measure mail system performanceunder artificial loads. They are normally not installed with Postfix.

• smtp-source, SMTP/LMTP messagegenerator
• smtp-sink, SMTP/LMTP message dump
• qmqp-source, QMQP message generator
• qmqp-sink, QMQP message dump

General mail receiving performancetips

• Read and understand the maildrop queue, incoming queue,and active queue discussions in the QSHAPE_README document.

• Run a local name server to reduce slow-down due to DNSlookups. If you run multiple Postfix systems, point each local nameserver to a shared forwarding server to reduce the number of lookupsacross the upstream network link.

• Eliminate unnecessary LDAP lookups, by specifying a domainfilter. This eliminates lookups for addresses in remote domains,and eliminates lookups of partial addresses. See ldap_table(5) fordetails.

When Postfix responds slowly to SMTP clients:

• Look for obvious signsof trouble as described in the DEBUG_README document, andeliminate those problems first.

• Turn off your header_checks and body_checks patterns andsee if the problem goes away.

• Turn off chrootoperation as described in the DEBUG_README document and seeif the problem goes away.

• If Postfix logs the SMTP client as "unknown" then you havea name service problem: the name server is bad, or the resolv.conffile contains bad information, or some packet filter is blockingthe DNS requests or replies.

• If the number of smtpd(8) processes has reached the processlimit as specified in master.cf, new SMTP clients must wait untila process becomes available. Increase the number of processes ifmemory permits. See the instructions given under "Tuning the number of Postfix processes".

Doing more work with your SMTP serverprocesses

With Postfix versions 2.0 and earlier, the smtpd(8) serverpauses before reporting an error to an SMTP client. The idea iscalled tar pitting. However, these delays also slow down Postfix.When the smtpd(8) server replies slowly, sessions take more time,so that more smtpd(8) server processes are needed to handle theload. When your Postfix smtpd(8) server process limit is reached,new clients must wait until a server process becomes available.This means that all clients experience poor performance.

You can speed up the handling of smtpd(8) server error repliesby turning off the delay:

/etc/postfix/ main.cf: # Not needed with Postfix 2.1 smtpd_error_sleep_time = 0

With the above setting, Postfix 2.0 and earlier can serve moreSMTP clients with the same number SMTP server processes. The nextsection describes how Postfix deals with clients that make a largenumber of errors.

Slowing down SMTP clients that make many errors

The Postfix smtpd(8) server maintains a per-session error count.The error count is reset when a message is transferred successfully,and is incremented when a client request is unrecognized orunimplemented, when a client request violates access restrictions, or whensome other error happens.

As the per-session error count increases, the smtpd(8) serverchanges behavior and begins to insert delays into the responses.The idea is to slow down a run-away client in order to limit resourceusage. The behavior is Postfix version dependent.

IMPORTANT: These delays slow down Postfix, too. When too muchdelay is configured, the number of simultaneous SMTP sessions willincrease until it reaches the smtpd(8) server process limit, and newSMTP clients must wait until an smtpd(8) server process becomes available.

Postfix version 2.1 and later:

• When the error count reaches $smtpd_soft_error_limit(default: 10), the Postfix smtpd(8) server delays all non-error anderror responses by $smtpd_error_sleep_time seconds (default: 1second).

• When the error count reaches $smtpd_hard_error_limit(default: 20) the Postfix smtpd(8) server breaks the connection.

Postfix version 2.0 and earlier:

• When the error count is less than $smtpd_soft_error_limit(default: 10) the Postfix smtpd(8) server delays all error replies by$smtpd_error_sleep_time (1 second with Postfix 2.0, 5 seconds withPostfix 1.1 and earlier).

• When the error count reaches $smtpd_soft_error_limit,the Postfix smtpd(8) server delays all responses by "error count"seconds or $smtpd_error_sleep_time, whichever is more.

• When the error count reaches $smtpd_hard_error_limit(default: 20) the Postfix smtpd(8) server breaks the connection.

Measures against clients that make too many connections

Note: these features use the Postfix anvil(8) service, introducedwith Postfix version 2.2.

The Postfix smtpd(8) server can limit the number of simultaneousconnections from the same SMTP client, as well as the connectionrate and the rate of certain SMTP commands from the same client.These statistics are maintained by the anvil(8) server (translation:if anvil(8) breaks, then connection limits stop working).

IMPORTANT: These limits must not be used to regulate legitimatetraffic: mail will suffer grotesque delays if you do so. The limitsare designed to protect the smtpd(8) server against abuse byout-of-control clients.

smtpd_client_connection_count_limit (default: 50)

The maximum number of connections that an SMTP client may makesimultaneously.

smtpd_client_connection_rate_limit (default: no limit)

The maximum number of connections that an SMTP client may makein the time interval specified with anvil_rate_time_unit (default:60s).

smtpd_client_message_rate_limit (default: no limit)

The maximum number of message delivery requests that an SMTP clientmay make in the time interval specified with anvil_rate_time_unit(default: 60s).

smtpd_client_recipient_rate_limit (default: no limit)

The maximum number of recipient addresses that an SMTP clientmay specify in the time interval specified with anvil_rate_time_unit(default: 60s).

smtpd_client_new_tls_session_rate_limit (default: no limit)

The maximum number of new TLS sessions (without usingthe TLS session cache) that an SMTP client may negotiate in thetime interval specified with anvil_rate_time_unit (default: 60s).

smtpd_client_event_limit_exceptions (default: $ mynetworks)

SMTP clients that are excluded from connection and ratelimits specified above.

General mail delivery performance tips

• Read and understand the maildrop queue, incoming queue,active queue and deferred queue discussions in the QSHAPE_READMEdocument.

• In case of slow delivery, run the qshape tool as describedin the QSHAPE_README document.

• Submit multiple recipients per message instead of submittingmessages with only a few recipients.

• Submit mail via SMTP instead of /usr/sbin/sendmail. Youmay have to adjust the smtpd_recipient_limit parameter setting.

• Don't overwhelm the disk with mail submissions. Optimizethe mail submission rate by tuning the number of parallel submissionsand/or by tuning the Postfix in_flow_delay parameter setting.

• Run a local name server to reduce slow-down due to DNSlookups. If you run multiple Postfix systems, point each local nameserver to a shared forwarding server to reduce the number of lookupsacross the upstream network link.

• Reduce the smtp_connect_timeout and smtp_helo_timeoutvalues so that Postfix does not waste lots of time connectingto non-responding remote SMTP servers.

• Use a dedicated mail delivery transport for problematicdestinations, with reduced timeouts and with adjusted concurrency.See "Tuning the number of simultaneous deliveries"below.

• Use a fallback_relay host for mail that cannot be deliveredupon the first attempt. This "graveyard" machine can use shorterretry times for difficult to reach destinations. See "Tuning the frequency of deferred mail deliveryattempts" below.

• Speed up disk updates with a large (64MB) persistent writecache. This allows disk updates to be sorted for optimal accessspeed without compromising file system integrity when the systemcrashes.

• Use a solid-state disk (a persistent RAM disk). Thisis an expensive solution that should be used in combinationwith short SMTP timeouts and a fallback_relay "graveyard"machine that delivers mail for problem destinations.

Tuning the number of simultaneous deliveries

Although Postfix can be configured to run 1000 SMTP clientprocesses at the same time, it is rarely desirable that it makes1000 simultaneous connections to the same remote system. For thisreason, Postfix has safety mechanisms in place to avoid thisso-called "thundering herd" problem.

The Postfix queue manager implements the analog of the TCP slowstart flow control strategy: when delivering to a site, send asmall number of messages first, then increase the concurrency aslong as all goes well; reduce concurrency in the face of congestion.

• The initial_destination_concurrency parameter (default: 5)controls how many messages are initially sent to the same destinationbefore adapting delivery concurrency. Of course, this setting iseffective only as long as it does not exceed the process limit andthe destination concurrency limit for the specific mail transportchannel.

• The default_destination_concurrency_limit parameter (default:20) controls how many messages may be sent to the same destinationsimultaneously. You can override this setting for specific messagedelivery transports by taking the name of the master.cf entryand appending "_destination_concurrency_limit".

Examples of transport specific concurrency limits are:

• The local_destination_concurrency_limit parameter (default:2) controls how many messages are delivered simultaneously to thesame local recipient. The recommended limit is low because deliveryto the same mailbox must happen sequentially, so massive parallelismis not useful. Another good reason to limit delivery concurrencyto the same recipient: if the recipient has an expensive shellcommand in her .forward file, or if the recipient is a mailing listmanager, you don't want to run too many instances of those processesat the same time.

• The default smtp_destination_concurrency_limit of 20 seemsenough to noticeably load a system without bringing it to its knees.Be careful when changing this to a much larger number.

The above default values of the concurrency limits work wellin a broad range of situations. Knee-jerk changes to these parametersin the face of congestion can actually make problems worse.Specifically, large destination concurrencies should never be thedefault. They should be used only for transports that deliver mailto a small number of high volume domains.

A common situation where high concurrency is called for is ongateways relaying a high volume of mail from between the Internetand an intranet mail environment. Approximately half the mail(assuming equal volumes inbound and outbound) will be destinedfor the internal mail hubs. Since the internal mail hubs will bereceiving all external mail exclusively from the gateway, it isreasonable to configure the gateway to make greater demands on thecapacity of the internal SMTP servers.

The tuning of the inbound concurrency limits need not be trialand error. A high volume capable mailhub should be able to easilyhandle 50 or 100 (rather than the default 20) simultaneous connections,especially if the gateway forwards to multiple MX hosts. When allMX hosts are up and accepting connections in a timely fashion,throughput will be high. If any MX host is down and completelyunresponsive, the average connection latency rises to at least 1/N* $smtp_connection_timeout, if there are N MX hosts. This limitsthroughput to at most the destination concurrency * N /$smtp_connection_timeout.

For example, with a destination concurrency of 100 and 2 MXhosts, each host will handle up to 50 simultaneous connections. Ifone MX host is down and the default SMTP connection timeout is 30s,the throughput limit is 100 * 2 / 30 ~= 6 messages per second. Thissuggests that high volume destinations with good connectivity andmultiple MX hosts need a lower connection timeout, values as lowas 5s or even 1s can be used to prevent congestion when one ormore, but not all MX hosts are down.

If necessary, set a higher transport_destination_concurrency_limit(in main.cf since this is a queue manager parameter) and a lowersmtp_connection_timeout (with a "-o" override in master.cf sincethis parameter has no per-transport name) for the relay transportand any transports dedicated for specific high volume destinations.

Tuning the number of recipients per delivery

The default_destination_recipient_limit parameter (default:50) controls how many recipients a Postfix delivery agent will sendwith each copy of an email message. You can override this settingfor specific Postfix delivery agents. For example,"uucp_destination_recipient_limit = 100" would limit the number ofrecipients per UUCP delivery to 100.

If an email message exceeds the recipient limit for somedestination, the Postfix queue manager breaks up the list ofrecipients into smaller lists. Postfix will attempt to send multiplecopies of the message in parallel.

IMPORTANT: Be careful when increasing the recipient limit permessage delivery; some smtpd(8) servers abort the connection when theyrun out of memory or when a hard recipient limit is reached, sothat the message will never be delivered.

The smtpd_recipient_limit parameter (default: 1000) controlshow many recipients the Postfix smtpd(8) server will take perdelivery. The default limit is more than any reasonable SMTP clientwould send. The limit exists to protect the local mail systemagainst a run-away client.

Tuning the frequency of deferred mail delivery attempts

When a Postfix delivery agent (smtp(8), local(8), etc.) isunable to deliver a message it may blame the message itself, or itmay blame the receiving party.

• When the delivery agent blames the message, the queuemanager gives the queue file a time stamp into the future, so itwon't be looked at for a while. By default, the amount of time tocool down is the amount of time that has passed since the messagearrived. This results in so-called exponential backoff behavior.

• When the delivery agent blames the receiving party (forexample a local recipient user, or a remote host), the queue managernot only advances the queue file time stamp, but also puts thereceiving party on a "dead" list so that it will be skipped forsome amount of time.

This process is governed by a bunch of little parameters.

queue_run_delay (default: 300 seconds; before Postfix 2.4:1000s)

How oftenthe queue manager scans the queue for deferred mail.

minimal_backoff_time (default: 300 seconds; before Postfix2.4: 1000s)

Theminimal amount of time a message won't be looked at, and the minimalamount of time to stay away from a "dead" destination.

maximal_backoff_time (default: 4000 seconds)

Themaximal amount of time a message won't be looked at after a deliveryfailure.

maximal_queue_lifetime (default: 5 days)

How longa message stays in the queue before it is sent back as undeliverable.Specify 0 for mail that should be returned immediately after thefirst unsuccessful delivery attempt.

bounce_queue_lifetime (default: 5 days, available with Postfixversion 2.1 and later)

How long a MAILER-DAEMON messagestays in the queue before it is considered undeliverable. Specify0 for mail that should be tried only once.

qmgr_message_recipient_limit (default: 20000)

Thesize of many in-memory queue manager data structures. Among others,this parameter limits the size of the short-term, in-memory listof "dead" destinations. Destinations that don't fit the list arenot added.

transport_destination_concurrency_failed_cohort_limit

Controls when a destination is considered "dead". Thisparameter is critical with a non-zero transport_destination_rate_delay, with a reduced transport_destination_concurrency_limit, or witha reduced initial_destination_concurrency.

IMPORTANT: If you increase the frequency of deferred maildelivery attempts, or if you flush the deferred mail queue frequently,then you may find that Postfix mail delivery performance actuallybecomes worse. The symptoms are as follows:

• The active queue becomes saturated with mail that hasdelivery problems. New mail enters the active queue only whenan old message is deferred. This is a slow process that usuallyrequires timing out one or more SMTP connections.

• All available Postfix delivery agents become occupiedtrying to connect to unreachable sites etc. New mail has to waituntil a delivery agent becomes available. This is a slow processthat usually requires timing out one or more SMTP connections.

When mail is being deferred frequently, fixing the problem isalways better than increasing the frequency of delivery attempts.However, if you can control only the delivery attempt frequency,consider using a dedicated fallback_relay "graveyard" machine forbad destinations, so that these destinations do not ruin theperformance of normalmail deliveries.

Tuning the number of Postfix processes

The default_process_limit configuration parameter gives directcontrol over how many daemon processes Postfix will run. As ofPostfix 2.0 the default limit is 100 smtp client processes, 100smtp server processes, and so on. This may overwhelm systems withlittle memory, as well as networks with low bandwidth.

You can change the global process limit by specifying anon-default default_process_limit in the main.cf file. For example,to run up to 10 smtp client processes, 10 smtp server processes,and so on:

/etc/postfix/ main.cf: default_process_limit = 10

You need to execute "postfix reload" to make the change effective.The limits are enforced by the Postfix master(8) daemon which doesnot automatically read main.cf when it changes.

You can override the process limit for specific Postfix daemonsby editing the master.cf file. For example, if you do not wish toreceive 100 SMTP messages at the same time, but do not want tochange the process limits for local mail deliveries, you couldspecify:

/etc/postfix/ master.cf: # ==================================================================== # service type private unpriv chroot wakeup maxproc command + args # (yes) (yes) (yes) (never) (100) # ==================================================================== . . . smtp inet n - - - 10 smtpd . . . Tuning the number of processes on the system

• MacOS X will run out of process slots when you increasePostfix process limits. The following works with OSX 10.4 and OSX10.5.

  MacOS X kernel parameters can be specified in /etc/sysctl.conf.

  /etc/sysctl.conf: kern.maxproc=2048 kern.maxprocperuid=2048

  Unfortunately these can't simply be set on the fly with "sysctl-w". You also have to set the following in /etc/launchd.conf sothat the root user after boot will have the right process limit(2048). Otherwise you have to always run ulimit -u 2048 as root,then start a user shell, and then start processes for things totake effect.

  /etc/launchd.conf: limit maxproc 2048

  Once these are in place, reboot the system. After that, the limits will stay in place.

Tuning the number of open files or sockets

When Postfix opens too many files or sockets, processes willabort with fatal errors, and the system may log "file table full"errors.

• Depending on your Postfix and operating system versionsyou may need to recompile Postfix if you need more than 1024 filedescriptors per process:

  • No recompilation is needed for Postfix version 2.4and later, when it was compiled for systems that support BSD kqueue(2)(FreeBSD 4.1, NetBSD 2.0, OpenBSD 2.9), Solaris 8 /dev/poll, orLinux 2.6 epoll(4).

  • Otherwise, Postfix needs to be recompiled to override thedefault FD_SETSIZE value.

• Reduce the number of processes as described under "Tuning the number of Postfix processes" above. Fewer processes need fewer open files and sockets.

• Configure the kernel for more open files and sockets.The details are extremely system dependent and change with theoperating system version. Be sure to verify the following informationwith your system tuning guide:

  • Some FreeBSD kernel parameters can be specified in/boot/loader.conf, and some can be specified in /etc/sysctl.confor changed with sysctl commands.Which is which depends on the version.

    kern.ipc.maxsockets="5000"kern.ipc.nmbclusters="65536"kern.maxproc="2048"kern.maxfiles="16384"kern.maxfilesperproc="16384"

  • Linux kernel parameters can be specified in /etc/sysctl.confor changed with sysctl commands:

    fs.file-max=16384kernel.threads-max=2048

  • Solaris kernel parameters can be specified in /etc/system,as described in the SolarisFAQ entry titled "How can I increase the number of filedescriptors per process?"

    * set hard limit on file descriptorsset rlim_fd_max = 4096* set soft limit on file descriptorsset rlim_fd_cur = 1024