Environment Red Hat OpenShift Container Platform 3.x Issue Our ha-proxies have a pretty high restart count (> 300 in a few days). As they do not log much, I can just provide you those outputs. Any ideas how to debug this further? In the logs we see: Raw 1201 08:42:28.395691 1 ratelimiter.go:50] error reloading router: wait: no child processes
Occasionally we report pods which are failing to connect or resolve a service name within the OpenShift cluster. After a few seconds it starts to work again and we have no idea where this is coming from. I'm having weird networking problems in my OpenShift cluster and I see the following messages reported on my nodes Raw Dec 06 09:39:11 ose3-node1 kernel: net_ratelimit: 119 callbacks suppressed Dec 06 09:39:17 ose3-node1 kernel: net_ratelimit: 154 callbacks suppressed
Resolution In Red Hat OpenShift Container Platform clusters with large numbers of routes (greater than the value of net.ipv4.neigh.default.gc_thresh3, which by default is 1024), it's necessary to increase the default values of sysctl variables to allow more entries in the ARP cache. The following sysctl values are suggested for clusters with large numbers of routes: Raw net.ipv4.neigh.default.gc_thresh1 = 8192 net.ipv4.neigh.default.gc_thresh2 = 32768 net.ipv4.neigh.default.gc_thresh3 = 65536
To make these settings permanent across reboots, create a custom tuned profile. Root Cause Red Hat OpenShift Container Platform was overflowing the neighbor cache because the router wanted to talk to so many IP addresses on the local SDN network. The default neighbor (ARP) cache is 1024. So when it overflows and removes entries, it could remove an entry for a packet that was in the queue and the packet would get dropped. Diagnostic Steps Compare ARP table against kernel settings: Raw # ip neigh show | wc -l 1005 # sysctl -a | grep net.ipv4.neigh.default.gc_thresh net.ipv4.neigh.default.gc_thresh1 = 128 net.ipv4.neigh.default.gc_thresh2 = 512 net.ipv4.neigh.default.gc_thresh3 = 1024
In the example above, ARP neighbors is close to threshold setting. Threshold setting would need to be raised in this case. Product(s) Red Hat OpenShift Container Platform Component haproxy kubernetes Tags haproxy kernel kubernetes openshift This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.