How do I configure a network interface bandwidth limitation?

https://access.redhat.com/solutions/40824

SOLUTION 已验证 - 已更新 2016年十二月22日14:58 - 

English 

环境

• Red Hat Enterprise Linux
• Networking

问题

• How do I configure a network interface bandwidth limitation?
• How to ratelimit a NIC?
• How can we stop a system or server sending so much LAN traffic?

决议

Inspect Existing qdisc

Ensure the existing queueing discipline on the network interface is pfifo_fast or noqueue:

Raw

# ip link | grep noqueue
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT 
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT 

Raw

# tc -s qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 3485442 bytes 40223 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 

If there is already a classless or classful qdisc applied, this needs to be removed with tc qdisc del, refer to man tc for full syntax of the various qdiscs.

Calculate Ratelimit Parameters

Units are described further on the man tc section PARAMETERS, common values for reference:

Raw

kbit       Kilobits per second
mbit       Megabits per second
gbit       Gigabits per second

kb or k    Kilobytes
mbit       Megabits
mb or m    Megabytes

The rate can be expressed as: rate 20mbit (20 megabit)

The buffer needs to be at least 1 kilobyte per megabit, this buffer is oversized: buffer 256kb (256 kilobytes)

Select how long you wish a packet to sit in the buffer before being dropped: latency 100ms (100 milliseconds)

The exact values for your network will need to be determined through configuration and testing. It is recommended to test with both bandwidth testing tools such as iperf and with the actual production workload to ensure desired outcomes are met.

Apply the Ratelimit to the Interface

We use the Token Bucket Filter to apply the ratelimit:

Raw

# tc qdisc add dev eth0 root tbf rate 20mbit buffer 256kb latency 100ms

Confirm Configuration

We can see qdisc tbf is now applied:

Raw

# tc -s qdisc
qdisc tbf 8001: dev eth0 root refcnt 2 rate 20000Kbit burst 256Kb lat 100.0ms 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 

Test and Confirm

A bandwidth test shows we're reaching roughly our desired bandwidth:

Raw

# iperf -l 1M -w 4M -f m -t 20 -c 172.16.1.7
------------------------------------------------------------
Client connecting to 172.16.1.7, TCP port 5001
TCP window size: 8.00 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 172.16.1.6 port 61900 connected with 172.16.1.7 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.8 sec  53.0 MBytes  21.4 Mbits/sec

The qdisc has been working to limit traffic it submits to the NIC:

Raw

# tc -s qdisc
qdisc tbf 8001: dev eth0 root refcnt 2 rate 20000Kbit burst 256Kb lat 100.0ms 
 Sent 2344194 bytes 1561 pkt (dropped 19, overlimits 130675 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 

To Persist Across Reboots

The methods described at How to make NIC ethtool settings persistent (apply automatically at boot) can also be used to persist these changes.

In RHEL 5, RHEL 6, and RHEL 7 without NetworkManager, write /sbin/ifup-local to apply the ratelimit at interface start.

In RHEL 7 with NetworkManager, write a NM dispatcher script as described in man NetworkManager.

Removal

If you wish to remove the TBF limitation, the qdisc can be deleted with:

Raw

# tc qdisc del dev eth0 root

The interface will revert back to its default queueing discipline:

Raw

# tc -s qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 

Troubleshooting

• Ratelimiting will never be 100% accurate, but within 10% accuracy is a reasonable expectation. Faster ratelimits (eg: gigabit) will likely be more accurate than lower ratelimits (eg: kilobit).
• Note that the TBF can only restrict what we send out. It is possible to ratelimit incoming traffic with moderate accuracy, as described at How to limit the bandwidth of incoming packets.
• If the ratelimit encountered in testing is far off, NIC offloading features may need to be disabled, as described at tc command's transport rate does not seem normal.

根源

The TBF is explained in further detail at Linux Advanced Routing & Traffic Control HOWTO - 9.2. Simple, classless Queueing Disciplines.