https://access.redhat.com/solutions/40824
SOLUTION 已验证 - 已更新 2016年十二月22日14:58 -
环境
- Red Hat Enterprise Linux
- Networking
问题
- How do I configure a network interface bandwidth limitation?
- How to ratelimit a NIC?
- How can we stop a system or server sending so much LAN traffic?
决议
Inspect Existing qdisc
Ensure the existing queueing discipline on the network interface is pfifo_fast
or noqueue
:
# ip link | grep noqueue
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
# tc -s qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 3485442 bytes 40223 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
If there is already a classless or classful qdisc applied, this needs to be removed with tc qdisc del
, refer to man tc
for full syntax of the various qdiscs.
Calculate Ratelimit Parameters
Units are described further on the man tc
section PARAMETERS
, common values for reference:
kbit Kilobits per second
mbit Megabits per second
gbit Gigabits per second
kb or k Kilobytes
mbit Megabits
mb or m Megabytes
The rate can be expressed as: rate 20mbit
(20 megabit)
The buffer needs to be at least 1 kilobyte per megabit, this buffer is oversized: buffer 256kb
(256 kilobytes)
Select how long you wish a packet to sit in the buffer before being dropped: latency 100ms
(100 milliseconds)
The exact values for your network will need to be determined through configuration and testing. It is recommended to test with both bandwidth testing tools such as iperf and with the actual production workload to ensure desired outcomes are met.
Apply the Ratelimit to the Interface
We use the Token Bucket Filter to apply the ratelimit:
# tc qdisc add dev eth0 root tbf rate 20mbit buffer 256kb latency 100ms
Confirm Configuration
We can see qdisc tbf
is now applied:
# tc -s qdisc
qdisc tbf 8001: dev eth0 root refcnt 2 rate 20000Kbit burst 256Kb lat 100.0ms
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
Test and Confirm
A bandwidth test shows we're reaching roughly our desired bandwidth:
# iperf -l 1M -w 4M -f m -t 20 -c 172.16.1.7
------------------------------------------------------------
Client connecting to 172.16.1.7, TCP port 5001
TCP window size: 8.00 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[ 3] local 172.16.1.6 port 61900 connected with 172.16.1.7 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-20.8 sec 53.0 MBytes 21.4 Mbits/sec
The qdisc has been working to limit traffic it submits to the NIC:
# tc -s qdisc
qdisc tbf 8001: dev eth0 root refcnt 2 rate 20000Kbit burst 256Kb lat 100.0ms
Sent 2344194 bytes 1561 pkt (dropped 19, overlimits 130675 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
To Persist Across Reboots
The methods described at How to make NIC ethtool settings persistent (apply automatically at boot) can also be used to persist these changes.
In RHEL 5, RHEL 6, and RHEL 7 without NetworkManager, write /sbin/ifup-local
to apply the ratelimit at interface start.
In RHEL 7 with NetworkManager, write a NM dispatcher script as described in man NetworkManager
.
Removal
If you wish to remove the TBF limitation, the qdisc can be deleted with:
# tc qdisc del dev eth0 root
The interface will revert back to its default queueing discipline:
# tc -s qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
Troubleshooting
- Ratelimiting will never be 100% accurate, but within 10% accuracy is a reasonable expectation. Faster ratelimits (eg: gigabit) will likely be more accurate than lower ratelimits (eg: kilobit).
- Note that the TBF can only restrict what we send out. It is possible to ratelimit incoming traffic with moderate accuracy, as described at How to limit the bandwidth of incoming packets.
- If the ratelimit encountered in testing is far off, NIC offloading features may need to be disabled, as described at tc command's transport rate does not seem normal.
根源
The TBF is explained in further detail at Linux Advanced Routing & Traffic Control HOWTO - 9.2. Simple, classless Queueing Disciplines.