iSCSI initiator fails to connect to target when multiple interfaces exist on the same subnet in RHEL

Issue

• Linux sends tcp reset (RST) packets to open iscsi connections
• Connections to iscsi devices fail

• Discovering or logging into a target via one of the interfaces on the same subnet as others results in failures:

  Raw

  # iscsiadm -m discovery -t st -p 192.168.2.2 -I ieth0 
  iscsiadm: connect to 192.168.2.2 timed out
  iscsiadm: connect to 192.168.2.2 timed out
  iscsiadm: connect to 192.168.2.2 timed out
  iscsiadm: connect to 192.168.2.2 timed out
  iscsiadm: connect to 192.168.2.2 timed out
  iscsiadm: connect to 192.168.2.2 timed out
  iscsiadm: connection login retries (reopen_max) 5 exceeded
  iscsiadm: Could not perform SendTargets discovery: encountered connection failure
  

Resolution

• Set sysctl's net.ipv4.conf.all.arp_ignore to 1 to force each interface to only answer ARP requests for its own addresses. Or set net.ipv4.conf.<INTERFACE>.arp_ignore to 1 for each interface that is a path to the iscsi storage.

• For example, in /etc/sysctl.conf:

Raw

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

• Or, to only set it for the relevant interfaces (assuming they are eth0 and eth1):

Raw

net.ipv4.conf.ethX.arp_ignore = 1
net.ipv4.conf.ethX.arp_announce=2

net.ipv4.conf.ethY.arp_ignore = 1
net.ipv4.conf.ethY.arp_announce=2

• Apply:

Raw

# sysctl -p /etc/sysctl.conf

Note: If the system is having RHEL 7 installed, then please enter above sysctl parameters in /etc/sysctl.d/99-sysctl.conf file and then run below command to reread the settings from configuration file:

Raw

# sysctl --system

Root Cause

• This problem can occur if an iSCSI connection is configured to use a specific network interface.  If iscsi packets are delivered to the wrong interface, Linux will respond as if the data was sent to a closed port and send a RST packet.  As long as the recieved data is coming in the wrong interface, the connection will remain broken and cannot be restored.

• This can occur when multiple adapters are connected to the same subnet. By default, Linux will answer all ARP requests for any of the system's IP addresses regardless of which interface the ARP request came from. If a different interface answers the ARP request for the iscsi connection's IP address, traffic will be routed to the wrong interface and the connection will fail.  Turning on arp_ignore forces a network interface to only answer ARP requests for its own addresses.  This prevents the ARP answer from the wrong interface and keeps the iscsi connection functioning.