To Bottom |
In this Document
Applies to:Oracle Database - Enterprise Edition - Version 9.2.0.1 to 12.1.0.1 [Release 9.2 to 12.1]Information in this document applies to any platform. Oracle Server Enterprise Edition - Version: 9.2.0.1 to 11.2 PurposeThis note covers the current recommendation for the Real Application Cluster Interconnect and Jumbo Frames ScopeThis article points out the issues surrounding Ethernet Jumbo Frame usage for the Oracle Real Application Cluster (RAC) Interconnect. In Oracle Real Application Clusters, the Cluster Interconnect is designed to run on a dedicated, or stand-alone network. The Interconnect is designed to carry the communication between the nodes in the Cluster needed to check for the Clusters condition and to synchronize the various memory caches used by the database. DetailsConfiguration
In order to make Jumbo Frames work properly for a Cluster Interconnect network, careful configuration in the host, its Network Interface Card and switch level is required:
For example, ifconfig -mtu 9000 followed by ifconfig -a to show the setting completed.
For example, some Intel NIC's require special descriptors and buffers to be configured for Jumbo Frames to work properly.
Failing to properly set these parameters in all nodes of the Cluster and Switches can result in unpredictable errors as well as a degradation in performance. TestingRequest your network and system administrator along with vendors to fully test the configuration using standard tools such as SPRAY or NETCAT and show that there is an improvement not degradation when using Jumbo Frames. Other basic ways to check it's configured correctly on Linux/Unix are using:
[node01] $ traceroute -F node02-priv 9000
traceroute to node02-priv (10.10.10.2), 30 hops max, 9000 byte packets 1 node02-priv (10.10.10.2) 0.232 ms 0.176 ms 0.160 ms [node01] $ traceroute -F node02-priv 9001 traceroute to node02-priv (10.10.10.2), 30 hops max, 9001 byte packets traceroute: sendto: Message too long 1 traceroute: wrote node02-priv 9001 chars, ret=-1 * Note: Due to Oracle Bugzilla 7182 (must have logon privileges) -- also known as RedHat Bugzilla 464044 -- older than EL4.7 traceroute may not work correctly for this purpose.
[node01]$ ping -c 2 -M do -s 8972 node02-priv
PING node02-priv (10.10.10.2) 1472(1500) bytes of data. 1480 bytes from node02-priv (10.10.10.2): icmp_seq=0 ttl=64 time=0.220 ms 1480 bytes from node02-priv (10.10.10.2): icmp_seq=1 ttl=64 time=0.197 ms [node01]$ ping -c 2 -M do -s 8973 node02-priv From node02-priv (10.10.10.1) icmp_seq=0 Frag needed and DF set (mtu = 9000) From node02-priv (10.10.10.1) icmp_seq=0 Frag needed and DF set (mtu = 9000) --- node02-priv ping statistics --- 0 packets transmitted, 0 received, +2 errors
For Solaris platform, the similar ping command is:
$ ping -c 2 -s node02-priv 8972 * Note: Ping reports fragmentation errors, due to exceeding the MTU size. Performance
For RAC Interconnect traffic, devices correctly configured for Jumbo Frame improves performance by reducing the TCP, UDP, and Ethernet overhead that occurs when large messages have to be broken up into the smaller frames of standard Ethernet. Because one larger packet can be sent, inter-packet latency between various smaller packets is eliminated. The increase in performance is most noticeable in scenarios requiring high throughput and bandwidth and when systems are CPU bound.
Known Bugs
In some versions of Linux there are specific bugs in Intel's Ethernet drivers and the UDP code path in conjunction with Jumbo Frames that could affect the performance. Check for and use the latest version of these drivers to be sure you are not running into these older bugs.
Recommendation
There is some complexity involved in configuring Jumbo Frames, which is highly hardware and OS specific. The lack of a specific standard may present OS and hardware bugs. Even with these considerations, Oracle recommends using Jumbo Frames for private Cluster Interconnects.
Since there is no official standard for Jumbo Frames, this configuration should be properly load tested by Customers. Any indication of packet loss, socket buffer or DMA overflows, TX and RX error in adapters should be noted and checked with the hardware and operating system vendors.
Oracle VM didn't support Jumbo in all ovm2 versions and all ovm3.0, however starting with ovm3.1.1 it's supported, refer to: http://www.oracle.com/us/technologies/virtualization/ovm-3-1-whats-new-1634275.pdf
To procedure to change MTU, refer to note 283684.1 - "Changing private network MTU only" ReferencesNOTE:1085885.1 - CRS root.sh Script Failing on Second Node When MTU Larger than 1500NOTE:1166925.1 - Oracle VM: Jumbo Frame on Oracle VM NOTE:300388.1 - Instances Unable To Start If MTU Size Is Different for Cluster_interconnect NOTE:1290585.1 - Solaris: Wrong MTU Size for VIP or HAIP NOTE:283684.1 - How to Modify Private Network Information in Oracle Clusterware BUG:13332363 - BUG 9795321 HAS REAPPEARED IN 11.2.0.3 |
|