mod_jk的优化配置

Optimal mod_jk configuration

 

There are many potential problems associated with the default configuration of mod_jk.  Let's say it is perfectly adequate for a very low traffic website, but when pushing any moderate to high load to mod_jk, there will be connection problems.  This is not due to any bug in mod_jk whatsoever, however, it is because the default configuration makes no assumption about your existing hardware or potential load, so, therefore, it is not tuned accordingly.

 

Note that the configuration recommendations here are optimal as a base configuration to avoid many of the common problems users experience with mod_jk. There exist many other useful optimizations, but these depend on the environment and web application in use. See http://tomcat.apache.org/connectors-doc/reference/workers.html for details on all available mod_jk properties.

 

Let's take a look at a typical default configuration for Apache/Tomcat/mod_jk:

workers.properties

worker.list=loadbalancer,status

worker.node1.port=8009
worker.node1.host=node1.mydomain.com
worker.node1.type=ajp13
worker.node1.lbfactor=1

worker.node2.port=8009
worker.node2.host= node2.mydomain.com
worker.node2.type=ajp13
worker.node2.lbfactor=1

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2

worker.status.type=status

 

JBoss Web's (Tomcat) server.xml AJP snippet:

<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"
         emptySessionPath="true" enableLookups="false" redirectPort="8443" ></Connector>

 

Apache's httpd.conf:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

 

 

The above configuration, under load, may cause mod_jk to be very slow and unresponsive, cause http errors, and cause half-closed connections.   These problems can arise because there are no connection timeouts specified to take care of orphaned connections, no error handling properties defined in workers.properties, and no connection limits set in Apache and Tomcat.

 

 

First off, lets take care of Tomcat:

 

Configuring server.xml:

 

The main concern with server.xml is setting the connectionTimeout which

sets the SO_TIMEOUT of the underlying socket.  So when a connection in

Tomcat hasn't had a request in the amount of time specified by

connectionTimeout, then the connection dies off.  This is necessary because if the connection hasn't been used for a certain period of

time then there is the chance that it is half-close on the mod_jk end.

If the connection isn't closed there will be an inflation of threads

which can over time hit the maxThreads count in Tomcat then Tomcat will

not be able to accept any new connections.  A connectionTimeout of 600000 (10 minutes) is a good number to start out with.  There may be a situation where the connections are not being recycled fast enough, in this instance the connectionTimeout could be lowered to 60000 or 1 minute.

 

When setting connectionTimeout in Tomcat, mod_jk should also have

connect_timeout/prepost_timeout set, which allows detection that the

Tomcat connection has been closed and preventing a retry request.

 

The recommended value of maxThreads is 200 per CPU, so here we assume the server is a single core machine.  If it has been quad core, we could push that value to 800, and more depending on RAM and other machine specs.

 

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="200"
           connectionTimeout="600000"></Connector>

 

 

 

 

Configuring workers.properties:

 

See comments inline.

 

worker.list=loadbalancer,status

worker.template.port=8009
worker.template.type=ajp13
worker.template.lbfactor=1
#ping_timeout was introduced in 1.2.27
worker.template.ping_timeout=1000
#ping_mode was introduced in 1.2.27, if not using 1.2.27 please specify connect_timeout=10000 and prepost_timeout=10000 as an alternative
worker.template.ping_mode=A
worker.template.socket_timeout=10
#It is not necessary to specify connection_pool_timeout if you are running the worker mpm 
worker.template.connection_pool_timeout=600

#Referencing the template worker properties makes the workers.properties shorter and more concise
worker.node1.reference=worker.template
worker.node1.host=192.168.1.2

worker.node2.reference=worker.template
worker.node2.host=192.168.1.3

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=True

worker.status.type=status

 

The key points in the above workers.properties is we've added limits for the connections mod_jk makes.  With the base configuration, socket timeouts default to infinite. The other important properties are ping_mode and ping_timeout which handle probing a connection for errors and connection_pool_timeout which must be set to equal server.xml's connectionTimeout when using the prefork mpm.  When these two values are the same, after a connection has been inactive for x amount of time, the connection in mod_jk and Tomcat will be closed at the same time, preventing a half-closed connection.

 

 

 

Configuring Apache

 

Make note that maxThreads for the AJP connection should coincide with

the MaxClients set in Apache's httpd.conf.  MaxClients needs to be set

in the correct module in Apache.

 

 

This can be determined by running httpd -V:

 

# httpd -V

Server version: Apache/2.2.3
Server built:   Sep 11 2006 09:43:05
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.8
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

 

Which tells me the Server MPM is Prefork.  This is not always 100% accurate so you should also view the output of /etc/sysconfig/httpd to see if the following line is there: HTTPD=/usr/sbin/httpd.worker.  If it is commented out you are running prefork, otherwise if uncommented worker.

 

httpd.conf:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
MaxClients       200
MaxRequestsPerChild  0
</IfModule>

 

Or if Apache is using worker, it is
<IfModule worker.c>
StartServers         2
MaxClients         200
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

 

MaxRequestsPerChild is 0, this is the recommended value when using

mod_jk as mod_jk keeps open persistent connections.  The key values in

the above configuration are MaxClients and MaxRequestsPerChild, the rest

of the values are left as default.  Note that MaxRequestsPerChild is

recommended to be 0 however the value may need to be greater than 0

depending on if Apache is used for other modules also, especially in the

case of resource leakage.

 

 

Advanced worker-mpm Configuration

 

To get the most out of your mod_jk setup you should be using Apache's worker mpm which provides a definite performance improvement over the prefork mpm.  The following section will detail how to configure Apache/mod_jk/Tomcat with the worker mpm and the math behind the configuration.

 

Let's start out with the worker mpm configuration

<IfModule mpm_worker_module>

ThreadLimit 100
StartServers 5
MaxClients 1000
MinSpareThreads 100
MaxSpareThreads 1000
ThreadsPerChild 100
MaxRequestsPerChild 0
</IfModule>

 

The optimal configuration completely depends on the hardware being used and the load requirements.  But a general rule of thumb, keep processes low and thread count high.  To determine the number of processes Apache will use simply divide MaxClients by ThreadPerChild.  So in this case MaxClients (1000) / ThreadsPerChild (100) = Processes (10), so Apache will allocate a maximum of 100 threads per each 10 child processes resulting in a total of 1000 possible clients.

 

Now to translate this to mod_jk, mod_jk maintains a connection pool for each worker defined in workers.properties.  By default with Apache mod_jk sets connection_pool_size to ThreadsPerChild, so in the above case that would translate to 100, giving 1000 possible connections to JBoss.  This may or may not be desired.

 

Let's take a common example, there will be 3 JBoss servers that combined needed to be able to handle 900 concurrent connections

 

worker.list=loadbalancer,status
 

worker.template.type=ajp13
worker.template.port=8009
worker.template.ping_mode=A
worker.template.connection_pool_size=30
worker.template.socket_timeout=10
worker.template.retries=20
 

worker.node1.reference=worker.template
worker.node1.host=192.168.0.101

worker.node2.reference=worker.template
worker.node2.host=192.168.0.102

worker.node3.reference=worker.template
worker.node3.host=192.168.0.103
 

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2, node3
worker.loadbalancer.sticky_session=True

worker.status.type=status

 

The above configuration tells mod_jk to multiplex 30 connections to the available Apache processes, which is 10 processes.  So that means 30 connections multiplexed over 10 processes gives 300 possible connections to each backend worker.  Furthermore, the total connections able to be used in this configuration from Apache is 900 which means  100 connections will be left over for static content or whatnot.

 

 

Next configure maxThreads in each ajp connector to match the above.

 

Node 1 ajp connector:

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="300"
           connectionTimeout="600000"></Connector>

 

Node 2 ajp connector:

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="300"
           connectionTimeout="600000"></Connector>

 

Node 3 ajp connector:

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="300"
           connectionTimeout="600000"></Connector>

 

Remember when using connectionTimeout which is always recommended, prepost_timeout and connect_timeout also need to be set, which is done.  I'm not showing sticky session configuration, but that covered in the main mod_jk article in using mod_jk with JBoss.