ipset heavy use

http://nemgeek.blogspot.nl/2013/07/ipset-for-heavy-use.html

What is ipset?
According to the official page: "IP sets are a framework inside the Linux 2.4.xand 2.6.x kernel, which can be administered by theipset utility. Dependingon the type, currently an IP set may store IP addresses, (TCP/UDP) port numbersor IP addresses with MAC addresses in a way, which ensures lightning speedwhen matching an entry against a set."
It's worth mentioning that this cool tool is mainly written by Jozsef Kadlecsik, a Hungarian Linux kernel expert.

Why and when to use ipset?
If you have plenty of IP rules in your iptables and their number is growing, one day you are going to experience a heavy performance drop. In practice if you have more than ca. ~1000-1500 rules you should worry about this. Anyway, it's more neater to use ipset above dozens of sets.

How does it work? 
You don't have to know and don't want to know. It's enough to know that it generates hashes from the rules and flipping thru these hashes is so efficient that it doesn't matter how many rules you have, the fastness of searching the whole set remains almost the same.

How to use?


For beginners

Assuming you have a modern .deb based distro, here are some simple steps.
apt-get update
apt-get install ipset
ipset create SET1 hash:net (for example)
ipset add SET1 91.83.231.25 (for example)
ipset add SET1 80.249.172.0/24 (for example)
iptables -I INPUT -m set --match-set SET1 src -j DROP (to drop all matching packets)
To save all your sets:
ipset save > backupfile
To delete:
ipset del SET1 91.83.231.25 - deletes a single line from a set
ipset flush SET1 - deletes a whole set
ipset destroy - deletes all the sets
BEFORE deleting a set you should delete the links in your iptables pointing to your set, e.g.
iptables -D INPUT -m set --match-set SET1 src -j DROP
To see your sets in different ways:
ipset -n list
ipset -t list
ipset list
To check if an IP address exists in a set:
ipset test 10.10.10.10
To restore your sets (assuming that sets in the file don't exist already)
ipset restore < mybackupfile

Some tricks

To create a new ruleset being the type of hash (thats the type because you want ipset, more infohere),append some addresses to it and deny them based on the source IP address.
ipset -N set2 hash
ipset -A set2 10.10.10.0/24
ipset -A set2 80.249.172.62
iptables -A INPUT -m set --myset set2 src -j DROP
To fast delete a rule (don't forget to delete the relevant iptables rule before)
ipset -F set2
ipset -X set2 
or simply:
ipset f  
ipset x
To auto-deny a host that wants to connect to your SSH port is so simple that:
ipset -N denied hash
iptables -A INPUT -p tcp --dport 22 -j SET --add-set denied src
iptables -A INPUT -m set --set denied src -j DROP

To block IP addresses based on geo location (country) here is a simple shellscript:
#!/bin/sh
ipset -N geoblock nethash
for IP in $(wget -O – http://www.ipdeny.com/ipblocks/data/countries/{cn,kr,pk,tw,sg,hk,pe}.zone)
do
ipset -A geoblock $IP
done
iptables -A INPUT -m set –set geoblock src -j DROP

To auto-timeout a rule (and not generate any message if it already exists):
ipset create test hash:ip timeout 10
ipset add --exists test 91.83.231.25 120 (overwriting the default 10 seconds value)

To auto-learn a MAC address: (and define a range)
ipset create test bitmap:ip,mac range 192.168.0.0/24
ipset add test 192.168.0.1,11:11:22:22:11:11
ipset add test 192.168.0.2 (this one will auto-learn)

More advanced WAN/LAN/DMZ firewall example

We define our client (source) IPs and ports they want to communicateto. We define our server IP address and ports. We allow established tcp sessions. Here, things are getting interesting.
We allow all packets coming in my external (internet) interface heading towards to my dmz server ip address and ports. (see dst,dst. That meansdestination IPAND destination port. (Here HTTP and HTTPS and udp only DNS and ping [it will reply the echo].)
Then we allow our LAN clients to access the internet web based on src,dst. (source IP address and destination port). In our case, anyone in the LAN can browse the web but only 192.168.0.10 can use https.
In the last line we allow our trusted administrator to connect to tcp ports 22020 to 22022 anywhere in our system.

ipset n dmzservers hash:ip,port
ipset n mynetworks hash:ip,iface
ipset n lanusers hash:ip,port
ipset n remoteadmin hash:ip,port
ipset a dmzservers 195.195.195.195,http
ipset a dmzservers 195.195.195.195,https
ipset a dmzservers 195.195.195.195,udp:53
ipset a dmzservers 195.195.195.195,icmp:ping 
ipset a mynetworks 192.168.0.0/24,eth0
ipset a mynetworks 8.8.8.0/24,eth1
ipset a mynetworks 195.195.195.193/30,eth2 (these network definitions are not used)
ipset a lanusers 192.168.0.0/24,http
ipset a lanusers 192.168.0.10,https
ipset a remoteadmin 82.112.112.112,tcp:22020-22022
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

iptables -A FORWARD -i $EXTERNAL -m set --match-set dmzservers dst,dst -m state --state NEW -j ACCEPT 
iptables -A FORWARD -i $INTERNAL -m set --match-set lanusers src,dst -m state --state NEW -j ACCEPT
iptables -A FORWARD -i $EXTERNAL -m set --match-set remoteadmin src,dst -m state --state NEW -j ACCEPT

1. 创建一个ipsetipset create $SET_NAME hash:ipfunction build_ipset(){ local SET_NAME="$1" local IFLIST="$2" declare -a IFLIST_ARRAY if [[ $SET_NAME != "" ]]; then ipset -n list $SET_NAME >/dev/null 2>&1 if [[ $? -ne 0 ]]; then # Not exist this set. Create it and add it ipset create $SET_NAME hash:net,iface OIFS=$IFS IFS=',' IFLIST_ARRAY=($IFLIST) IFS=$OIFS for i in "${IFLIST_ARRAY[@]}"; do echo ipset add $SET_NAME 0.0.0.0/1,$i ipset add $SET_NAME 0.0.0.0/1,$i echo ipset add $SET_NAME 128.0.0.0/1,$i ipset add $SET_NAME 128.0.0.0/1,$i done fi else help up set_name is null fi设置规则： -m set --match-set $UP_SET_NAME src,src -m set --match-set $DN_SET_NAME dst,dst" -m set --match-set $UP_SET_NAME dst,dst -m set --match-set $DN_SET_NAME src,src"