UDP Traceroute Over Infiniband Fails On Every Second Attempt (Doc ID 2030052.1)

UDP Traceroute Over Infiniband Fails On Every Second Attempt (Doc ID 2030052.1)

APPLIES TO:

Oracle Exadata Storage Server Software - Version 12.1.2.1.1 and later
Linux OS - Version Oracle Linux 5.0 and later
Information in this document applies to any platform.

SYMPTOMS

 UDP Traceroute Over Infiniband Fails On Every Second Attempt

CHANGES

None:   UDP traceroute over IB invoked

CAUSE

UNIX Traceroute Explained

The in Unix / Linux, traceroute works is by using udp datagrams. 
The outgoing packets will be udp packets with source ip and destination ip address of the Internet router with destination port of 33434. 
Generally the port number keeps on increasing from 33434 as the ttl increases. 

So why does traceroute sometimes drop some packets?
This is a security feature which has been put into place to reduce the unnecessary unreachable replies during a DOS or DDOS attack. 
Due to this feature, the system will only reply to some unreachable messages and the rest are dropped due to rate limit. 
The ratelimit is the minimum timing space between responses in milliseconds.
Due to this behavior you can see that the trace to a router will show a time in the first entry, drop any additional packets received in the defined interval.

Additional information can be found at this URL.

http://babaawesam.com/2013/10/23/traceroute-drops-at-last-hop/

 

To summarize:

Traceroute is a utility to check the path a packet takes across a network and it is a very important tool for troubleshooting.

If you are having connectivity issue to a destination, trace the destination and on the hop the packets start dropping, there is a very good chance that the hop is having congestion.

 

SOLUTION

It seems that ICMP rate-limiting is causing the every second invocation failure.
If you are experiencing dropped packets and want to modify the behavior and prevent it, then modify the ratelimit to 0.
Set in /etc/sysctl.conf file
     net.ipv4.icmp_ratelimit = nnn.
     nnn = minimum space between responses in milliseconds.
     0 = unset (no dropped packets)
     1000 = Linux default (in ms)