jboss性能优化（一）

JBoss tuning tip 1: Tunethe garbage collector

One strength of the J2SE platform isthat it shields the developer from the complexity of memory allocation.However, once garbage collection is the principal bottleneck, it is worthunderstanding some aspects of this hidden implementation

An object is considered garbage when itcan no longer be reached from any pointer in the running program. The moststraightforward garbage collection algorithms simply iterate over everyreachable object. Any objects left over are then considered garbage. The timethis approach takes is proportional to the number of live objects,

The complete address space reserved for object memory can be divided intothe young and tenured generations.

The young generationconsists of eden and two survivor spaces. Most objects are initially allocatedin eden. One survivor space is empty at any time, and serves as the destinationof any live objects in eden and the other survivor space during the nextcopying collection. Objects are copied between survivor spaces in this wayuntil they are old enough to be tenured (copiedto the tenured generation).

A third generation closely related tothe tenured generation is the permanent generation which holds data needed bythe virtual machine to describe objects that do not have an equivalence at theJava language level. For example objects describing classes and methods arestored in the permanent generation

Use the the command line option -verbose:gc causesinformation about the heap and garbage collection to be printed at eachcollection. For example, here is output from a large server application:

 It's demonstrated that an application that spends 10% of its timein garbage collection can lose 75% of its throughput when scaled out to 32processors
(http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html)

JBoss tuning tip 2: Set-Xms and -Xmx to the same value

 

By default, the virtual machine grows orshrinks the heap at each collection to try to keep the proportion of free spaceto live objects at each collection within a specific range.

Setting -Xms and -Xmx to the same value. Thisincrease predictability by removing the most important sizing decision from thevirtual machine.

JBoss tuning tip 3: Useserver VM

The server JVM is better suited to longer running applications. To enable itsimply set the -server option on the command line.

JBoss tuning tip 4: Turn off distributed gc

The RMI system provides a reference counting distributed garbage collectionalgorithm. This system works by having the server keep track of which clientshave requested access to remote objects running on the server. When a referenceis made, the server marks the object as "dirty" and when a clientdrops the reference, it is marked as being "clean.". However thissystem is quite expensive and by defaultruns every minute.

Set it to run every 30 minute at least
-Dsun.rmi.dgc.client.gcInterval=1800000
-Dsun.rmi.dgc.server.gcInterval=1800000

JBoss tuning tip 6: Turn on parallel gc

If you have multiple proessors you can do your garbage collection with multiplethreads. By default the parallel collector runs a collection thread perprocessor, that is if you have an 8 processor box then you'll garbage collectyour data with 8 threads. In order to turn on the parallel collector use theflag -XX:+UseParallelGC. You can also specify how manythreads you want to dedicate to garbage collection using the flag-XX:ParallelGCThreads=8.

JBoss tuning tip7: Don't use Huge heaps, use a cluster

More JVMs/smaller heaps can outperform fewer JVMs/Larger Heaps. So instead ofhuge heaps, use additional server nodes. Set up a JBoss cluster and balancework between nodes.

JBoss tuning tip8: Don't choose an heap larger then 70% of your OS memory

 

Choose a maximum heap size not more then70% of the memory to avoid excessive page faults and thrashing.

 

JBoss tuning tip 9:  Tunethe Heap ratio

This is one of most important tuning factor: theheap ratio. The heap ratio specifies how the amount of thetotal heap will be partitioned between the young and the tenuredspace.  What happens if you have lots of long lived data (cached data,collections ) ? maybe you're in this situation:

 

The problem here is that the long lived data overflows the tenured generation.When a collection is needed the tenured generation is basically full of livedata. Much of the young generation is also filled with long lived data. Theresult was that a minor collection could not bedone successfully (there wasn't enough room in the tenuredgeneration for the anticipated promotions out of the young generation) so amajor collection was done.

The major collectionworked fine, but the results again was that the tenured generation was full oflong lived data and there was long lived data in the young generation. Therewas also free space in the young generation for more allocations, but the nextcollection was again destined to be a major collection.

This will eventuallybring your application to crawl !!!!!

By decreasing the space in the young generation and putting thatspace into the tenured generation (avalue of NewRatio larger than the default value was chosen), there was enoughroom in the tenured generation to hold all the long lived data and also spaceto support minor collections. This particular application used lots of shortlived objects so after the fix mostly minor collections were done.

NewRatio is a flag that specifies theamount of the total heap that will be partitioned into the young generation.It's the tenured-generation-size / young-generation-size. For example, setting-XX:NewRatio=3 means that the ratio between the young and tenured generation is1:3 

If you want a more precise control over the young generation : NewSize is theinitial size of the young generation, MaxNewSizewillspecify the maximum size of the young generation

What is the recommeded heap ratios ? Setthe tenured generation to be approximately two times the size of the younggeneration. With a 2GB of RAM the recommended sizes are 1200MB for the heap and400MB for the young generation. 
This recommendation is only a starting point, you have to tune fromthere and to do that you have to gather and analyze the garbage collectionstatistics.

 

JBoss tuning tip 10:  Monitorthe free memory with monitors and snapshots 

 

See this tips:
http://www.mastertheboss.com/jboss-howto/47-jboss-jmx/91-jboss-monitor.html
http://www.mastertheboss.com/jboss-howto/47-jboss-jmx/92-jboss-monitor-snapshot.html

JBoss tuning tip 11: Tunethe Operating System

Each operating system sets default tuning parameters differently. For Windowsplatforms, the default settings are usually sufficient. However, the UNIX andLinux operating systems usually need to be tuned appropriately

Solaris tuning parameters: 

Check the following TCP parameters with your sysadmin:

/dev/tcp tcp_time_wait_interval 
/dev/tcp tcp_conn_req_max_q 
/dev/tcp tcp_conn_req_max_q0 
/dev/tcp tcp_ip_abort_interval 
/dev/tcp tcp_keepalive_interval 
/dev/tcp tcp_rexmit_interval_initial 
/dev/tcp tcp_rexmit_interval_max 
/dev/tcp tcp_rexmit_interval_min 
/dev/tcp tcp_smallest_anon_port 
/dev/tcp tcp_xmit_hiwat 
/dev/tcp tcp_recv_hiwat 
/dev/ce instance 
/dev/ce rx_intr_time 

Tip: Use the netstat -s -P tcp command to view all available TCP parameters.

Set TCP-relatedtuning parameters using the ndd command

Example: ndd -set/dev/tcp tcp_conn_req_max_q 16384

Tune /etc/system filesystem

Each socketconnection to JBoss consumes a file descriptor. To optimize socket performance,you may need to configure your operating system to have the appropriate numberof file descriptors.

 

See solaris documentation about this parameters:

set rlim_fd_cur
set rlim_fd_max
set tcp:tcp_conn_hash_size (Solaris 8 and 9)
set ip:ipcl_conn_hash_size (Solaris 10)
set shmsys:shminfo_shmmax Note: This should only be setfor machines that have at least 4 GB RAM or higher.
set autoup
set tune_t_fsflushr

 

Linux tuning parameters:

Since in Linux everything is a file, check the file-max parameter

cat/proc/sys/fs/file-max

setfs.file-max=102642  into /etc/sysctl.conf

 

Raise ulimit with/etc/limits.conf (or ulimit -n for current session)

 

Increase default socket send/receivebuffer

 

sysctl-w net.core.rmem_default=262144
(default socket receive buffer)

sysctl-w net.core.wmem_default=262144
(default socket send buffer)

sysctl-w net.core.rmem_max=262144
(max socket receive buffer)
sysctl -w net.core.wmem_max=262144
(max socket send buffer size)

Optimize MTU. The TCP maximum transfer unit is 1512on the Internet. If you are sending larger packets it's a good idea to increaseMTU size in order to reduce packet fragmentation (especially if you have a slownetwork)

vi/etc/sysconfig/network-scripts/ifcfg-xxx (eth0 for instance)

– add "MTU=9000" (for gigabit ethernet)
– restart the interface (ifdown eth0;ifup eth0)

 

Use Big Memory Pages

Default page size is 4KB (usually too small!)
Check page size with:

$ cat /proc/meminfo

If you see "HugePage_Total,""HugePages_Free" and "Hugepagesize", you can apply thisoptimization

Here's how to do it (2GB Heap Size Example)

$ echo 2147483647 > /proc/sys/kernel/shmmax
$ echo 1000 > /proc/sys/vm/nr_hugepages

In Sun's JVM, add this flag: XX:+UseLargePages

 <

JBoss tuning tip12: Lots of Requests ? check JBoss thread pool

 

JBoss thread pool is defined intoconf/jboss-service.xml

 <mbeancode="org.jboss.util.threadpool.BasicThreadPool"

 name="jboss.system:service=ThreadPool">

 <attributename="Name">JBoss System Threads</attribute>

 <attributename="ThreadGroupName">System Threads</attribute>

 <attributename="KeepAliveTime">60000</attribute>

 <attributename="MaximumPoolSize">10</attribute>

 <attributename="MaximumQueueSize">1000</attribute>

 <attributename="BlockingMode">run</attribute>

 </mbean>

For most applications this defaults willjust work well, however if you are running an application with issues lots ofrequests to jboss (such as EJB invocations) then monitor your thread pool. Openthe Web Console and look for the MBeanjboss.system:service=ThreadPool.

 

 

Start a monitor on the QueueSizeparameter. Have you got a QueueSize whichreaches MaximumPoolSize ? then probably you need to set ahigher MaximumPoolSize pool size attribute

Watchout! Speak at first with your sysadmin andensure that the CPU capacity support the increase in threads.

 

 

Watchout! if your threads make use of JDBCconnections you'll probably need to increase also the JDBC connection poolaccordingly. Also verify that your HTTP connector is enabled to handle thatamount of requests

JBoss tuning tip 13: Check theEmbedded web container

JBoss supports connectors for http,https, and ajp. The configuration file is server.xml and it'sdeployed in the root of JBoss web container (In JBoss 4.2.0 it's: "JBOSS_HOME\server\default\deploy\jboss-web.deployer")

 <Connectorport="8080" address="${jboss.bind.address}"   

 maxThreads="250"maxHttpHeaderSize="8192"

 emptySessionPath="true"protocol="HTTP/1.1"

 enableLookups="false"redirectPort="8443" acceptCount="100"

 connectionTimeout="20000"disableUploadTimeout="true" />

The underlying HTTP connector of JBoss needsto be fine tuned for production settings.  The important parameters are:

maxThreads - This indicates the maximumnumber of threads to be allocated for handling client HTTP requests. Thisfigure corresponds to the concurrent users that are going to access theapplication. Depending on the machine configuration, there is a physical limitbeyond which you will need to do clustering.

acceptCount - This is the number of requestthreads that are put in request queue when all available threads are used. Whenthis exceeds, client machines get a request timeout response.

compression - If you set this attribute to“force”, the content will be compressed by JBoss and will be send to browser.Browser will extract it and display the page on screen. Enabling compressioncan substantially reduce bandwidth requirements of your application.

So how do you know if it's necessary to raise your maxThreadsnumber ? again open the web console and look for the MBeanjboss.web:name=http-127.0.0.1-8080,type=ThreadPool. The key attribute is currentThreadsBusy. If it's about 70-80% ofthe the maxThreads you should consider raising the number ofmaxThreads.  

 Watch out! if youincrease the maxThreads count you need to raise your JBoss Thread poolaccordingly.

 

JBoss tuning tip14:  Turn off JSP Compilation in production

 

JBoss application server regularlychecks whether a JSP requires compilation to a servlet before executing a JSP.In a production server, JSP files won’t change and hence you can configure thesettings for increased performance.

Open the web.xml indeploy/jboss-web.deployer/conf folder. Look for the jsp servlet in the file andmodify the following XML fragment as given below:

<init-param>

 <param-name>development</param-name>

 <param-value>false</param-value>

</init-param>

<init-param>

 <param-name>checkInterval</param-name>

 <param-value>300</param-value>

</init-param>

References:
http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
http://people.redhat.com/alikins/system_tuning.html
http://community.jboss.org/wiki/JBossASTuningSliming