Haproxy 透传IP配置方法及测试

1. 环境准备

• golang 运行环境
• docker 运行环境
• vmware 虚拟机(centos7 x64)

2. 测试准备

2.1 启动Haproxy容器方法

2.1.1 拉取官方haproxy镜像

docker pull haproxy:latest

2.1.2 删除旧的容器

如果存在旧的容器，需要停止和删除

docker stop haproxy && docker rm haproxy

2.1.3 编写haproxy配置

新建一个工作目录，在工作目录种新建haproxy.cfg，添加以下内容

global
daemon
ulimit-n 1048576

defaults
mode tcp
retries 3
option redispatch
option abortonclose
option dontlognull
maxconn 100000

timeout server 60m
timeout connect 5s
timeout client 60m
timeout http-request 1m
timeout queue 5s
timeout http-keep-alive 1m
timeout check 1m

#############################
# Add listener config below #
#############################
listen http
        bind *:80
        mode http
        maxconn 20000
        server s1 172.169.18.20:20000

2.1.4 运行配置检查

进入工作目录后
Shell或者Git Bash下使用：

docker run -it --rm --name haproxy-syntax-check \
    -v `pwd`:/usr/local/etc/haproxy:ro \
haproxy:latest -c -f /usr/local/etc/haproxy/haproxy.cfg

CMD或者PS下使用

docker run -it --rm --name haproxy-syntax-check ^
    -v %cd%:/usr/local/etc/haproxy:ro ^
haproxy:latest -c -f /usr/local/etc/haproxy/haproxy.cfg

显示

Configuration file is valid

2.1.5 启动容器

进入工作目录后
Shell或者Git Bash下使用：

docker run -d \
	--name haproxy \
	--hostname haproxy \
	--sysctl net.ipv4.ip_forward=1 \
	--sysctl net.ipv4.tcp_timestamps=0 \
	--sysctl net.ipv4.ip_unprivileged_port_start=0 \
	--restart=always \
	--privileged=true \
	--log-opt max-size=10m \
	--log-opt max-file=10 \
	--ulimit nofile=1048576:1048576 \
	-v /etc/localtime:/etc/localtime:ro \
	-v /etc/timezone:/etc/timezone:ro \
    -v `pwd`:/usr/local/etc/haproxy:ro \
	-p 80:80 \
haproxy:latest

CMD或者PS下使用:

docker run -d ^
	--name haproxy ^
	--hostname haproxy ^
	--sysctl net.ipv4.ip_forward=1 ^
	--sysctl net.ipv4.tcp_timestamps=0 ^
	--sysctl net.ipv4.ip_unprivileged_port_start=0 ^
	--restart=always ^
	--privileged=true ^
	--log-opt max-size=10m ^
	--log-opt max-file=10 ^
	--ulimit nofile=1048576:1048576 ^
	-v /etc/localtime:/etc/localtime:ro ^
	-v /etc/timezone:/etc/timezone:ro ^
    -v %cd%:/usr/local/etc/haproxy:ro ^
	-p 80:80 ^
haproxy:latest

显示

4b1e3cb0afa5ed1c2385944ce439794df4e70c2fb04d365113c22358770c5707

启动成功

2.1.6 更改配置

更改配置后使用如下命令重载配置

docker kill -s HUP haproxy

注意不要更改暴露端口,否则要重新配置haproxy容器端口参数

2.2 Golang Server编写

2.2.1 TCP Server

使用Golang实现简单的TCP服务器 tcp.go

package main

import (
	"log"
	"net"
)

func main() {
	listener, err := net.Listen("tcp", ":20000")
	if err != nil {
		log.Fatalln(err.Error())
	}
	defer listener.Close()
	for {
		conn, err := listener.Accept()
		if err != nil {
			log.Println(err.Error())
			continue
		}
		go func(conn net.Conn) {
			defer func() {
				if err := conn.Close(); err != nil {
					log.Println(err.Error())
				}
			}()
			log.Printf("local addr: %s, remote addr: %s", conn.LocalAddr(), conn.RemoteAddr())
		}(conn)
	}
}

启动TCP服务器

go run tcp.go 

2.2.2 HTTP Server

使用Golang实现简单的TCP服务器 http.go

package main

import (
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		log.Printf("request uri: %s, proto: %s, host: %s, user agent: %s, remote addr: %s, x-forwarded-for: %s",
			r.URL.RequestURI(), r.Proto, r.Host, r.UserAgent(), r.RemoteAddr, r.Header.Get("X-Forwarded-For"))
		w.Write([]byte("OK\n"))
	})
	err := http.ListenAndServe(":20020", nil)
	if err != nil {
		log.Fatalln(err.Error())
	}
}

启动HTTP服务器

go run http.go 

2.3 客户端测试

2.3.1 设置网络

Vmware 虚拟机安装CentOS镜像
按照如下方式设置桥接模式,复制物理网络的连接状态

查询IP

[tony@localhost ~]$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:dc:69:8a:b1  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.169.18.174  netmask 255.255.255.0  broadcast 172.169.18.255
        inet6 fe80::babe:c69c:22a5:6d14  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:23:21:a3  txqueuelen 1000  (Ethernet)
        RX packets 757  bytes 91426 (89.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 94  bytes 11675 (11.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 68  bytes 5916 (5.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68  bytes 5916 (5.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:95:71:3a  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

虚拟机IP为172.169.18.174
本机IP为172.169.18.20

2.3.3 添加hosts（模拟DNS）

在CentOS虚拟机上添加hosts
echo “remote addr: 172.169.18.20:57402” >> /etc/hosts

3. haproxy 参数解释

https://gist.github.com/PiBa-NL/d826e0d6b35bbe4a5fc3

To send the ip addres of the client/webbrowser to the server/webserver behind it there are a few options:
 1- option forwardfor
 2- send-proxy
 3- source 0.0.0.0 usesrc clientip

1- option forwardfor
This is an easy option to configure in haproxy, it does require that http layer7 processing is used 'mode http' and the webserver/ webapplication that wants to log or use the ip of the client must use the http-header 'X-Forwarded-For' to read the clientip.

3.2 send-proxy

• 包含send-proxy(send-proxy-v1), send-proxy-v2, send-proxy-*(其他版本) 等版本
• TCP和HTTP均可用，但是要求服务端升级ProxyProtocol。Golang下可以使用https://github.com/pires/go-proxyproto, 其他语言也有对应的库。
• ProxyProtocol本质是变种TCP。有些应用是通过在TCP协议前面增加IP信息来支持ProxyProtocol,send-proxy协议增加的IP信息为字符串，send-proxy-v2增加的IP信息为字节，性能更优。

3.3 source 0.0.0.0 usesrc clientip

This allows any application and any protocol to be used and see the actual client ip as the origin from the incomming connection.
It does however require to configure IPTABLES or IPFW or other firewall rules to capture reply-traffic, also the haproxy machine must be the defaultroute for the return traffic from the (web-)server.
@LeonanCarvalho

iptables 范例

#!/bin/bash
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

http://cbonte.github.io/haproxy-dconv/2.5/configuration.html#2
source [:] [usesrc { [:] | client | clientip } ]
source [:] [usesrc { [:] | hdr_ip([,]) } ]
source [:] [interface ]
Set the source address for outgoing connections
May be used in sections :

defaults frontend listen backend
yes
yes no
no yes
yes yes
yes
Arguments :
is the IPv4 address HAProxy will bind to before connecting to a
server. This address is also used as a source for health checks.

      The default value of 0.0.0.0 means that the system will select
      the most appropriate address to reach its destination. Optionally
      an address family prefix may be used before the address to force
      the family regardless of the address format, which can be useful
      to specify a path to a unix socket with no slash ('/'). Currently
      supported prefixes are :
        - 'ipv4@' -> address is always IPv4
        - 'ipv6@' -> address is always IPv6
        - 'unix@' -> address is a path to a local unix socket
        - 'abns@' -> address is in abstract namespace (Linux only)
      You may want to reference some environment variables in the
      address parameter, see section 2.3 about environment variables.

is an optional port. It is normally not needed but may be useful
in some very specific contexts. The default value of zero means
the system will select a free port. Note that port ranges are not
supported in the backend. If you want to force port ranges, you
have to specify them on each “server” line.

is the IP address to present to the server when connections are
forwarded in full transparent proxy mode. This is currently only
supported on some patched Linux kernels. When this address is
specified, clients connecting to the server will be presented
with this address, while health checks will still use the address
.

is the optional port to present to the server when connections
are forwarded in full transparent proxy mode (see above).
The default value of zero means the system will select a free
port.

is the name of a HTTP header in which to fetch the IP to bind to.
This is the name of a comma-separated header list which can
contain multiple IP addresses. By default, the last occurrence is
used. This is designed to work with the X-Forwarded-For header
and to automatically bind to the client’s IP address as seen
by previous proxy, typically Stunnel. In order to use another
occurrence from the last one, please see the parameter
below. When the header (or occurrence) is not found, no binding
is performed so that the proxy’s default IP address is used. Also
keep in mind that the header name is case insensitive, as for any
HTTP header.

is the occurrence number of a value to be used in a multi-value
header. This is to be used in conjunction with “hdr_ip()”,
in order to specify which occurrence to use for the source IP
address. Positive values indicate a position from the first
occurrence, 1 being the first one. Negative values indicate
positions relative to the last one, -1 being the last one. This
is helpful for situations where an X-Forwarded-For header is set
at the entry point of an infrastructure and must be used several
proxy layers away. When this value is not specified, -1 is
assumed. Passing a zero here disables the feature.

is an optional interface name to which to bind to for outgoing
traffic. On systems supporting this features (currently, only
Linux), this allows one to bind all traffic to the server to
this interface even if it is not the one the system would select
based on routing tables. This should be used with extreme care.
Note that using this option requires root privileges.
The “source” keyword is useful in complex environments where a specific
address only is allowed to connect to the servers. It may be needed when a
private address must be used through a public gateway for instance, and it is
known that the system cannot determine the adequate source address by itself.

An extension which is available on certain patched Linux kernels may be used
through the “usesrc” optional keyword. It makes it possible to connect to the
servers with an IP address which does not belong to the system itself. This
is called “full transparent proxy mode”. For this to work, the destination
servers have to route their traffic back to this address through the machine
running HAProxy, and IP forwarding must generally be enabled on this machine.

In this “full transparent proxy” mode, it is possible to force a specific IP
address to be presented to the servers. This is not much used in fact. A more
common use is to tell HAProxy to present the client’s IP address. For this,
there are two methods :

• present the client’s IP and port addresses. This is the most transparent
  mode, but it can cause problems when IP connection tracking is enabled on
  the machine, because a same connection may be seen twice with different
  states. However, this solution presents the huge advantage of not
  limiting the system to the 64k outgoing address+port couples, because all
  of the client ranges may be used.

• present only the client’s IP address and select a spare port. This
  solution is still quite elegant but slightly less transparent (downstream
  firewalls logs will not match upstream’s). It also presents the downside
  of limiting the number of concurrent connections to the usual 64k ports.
  However, since the upstream and downstream ports are different, local IP
  connection tracking on the machine will not be upset by the reuse of the
  same session.

This option sets the default source for all servers in the backend. It may
also be specified in a “defaults” section. Finer source address specification
is possible at the server level using the “source” server option. Refer to
section 5 for more information.

In order to work, “usesrc” requires root privileges.
Examples :
backend private
# Connect to the servers using our 192.168.1.200 source address
source 192.168.1.200

backend transparent_ssl1
# Connect to the SSL farm from the client’s source address
source 192.168.1.200 usesrc clientip

backend transparent_ssl2
# Connect to the SSL farm from the client’s source address and port
# not recommended if IP conntrack is present on the local machine.
source 192.168.1.200 usesrc client

backend transparent_ssl3
# Connect to the SSL farm from the client’s source address. It
# is more conntrack-friendly.
source 192.168.1.200 usesrc clientip

backend transparent_smtp
# Connect to the SMTP farm from the client’s source address/port
# with Tproxy version 4.
source 0.0.0.0 usesrc clientip

backend transparent_http
# Connect to the servers using the client’s IP as seen by previous
# proxy.
source 0.0.0.0 usesrc hdr_ip(x-forwarded-for,-1)

4. 测试

4.1 TCP无透传参数

4.1.1 测试过程

• 更改haproxy配置

listen tcp
        bind *:80
        mode tcp
        maxconn 20000
        server h1 172.169.18.20:20000 send_proxy

重载配置，启动server进程

docker kill -s HUP haproxy
go run tcp.go

• 执行虚拟机curl测试

 curl server

• golang日志

2021/05/24 15:56:11 local addr: 172.169.18.20:20000, remote addr: 172.169.18.20:57402

4.1.2 结论

现象： remote addr: 172.169.18.20:57402 显示的是本机地址，并非真正的远程地址
结论: 默认情况下无法透传TCP的远程地址

4.2 HTTP 无透传参数

4.2.1 测试过程

• 更改haproxy配置

listen http
        bind *:80
        mode http
        maxconn 20000
        server s1 172.169.18.20:20000

重载配置，启动server进程

docker kill -s HUP haproxy
go run http.go

• 执行虚拟机curl测试

 curl server

• golang日志

2021/05/24 18:22:07 request uri: /, proto: HTTP/1.1, host: server1, user agent: curl/7.29.0, remote 
addr: 172.169.18.20:63547, x-forwarded-for:

4.2.2 结论

• 现象： x-forwarded-for 为空

• 结论: 默认情况下无法透传HTTP的远程地址

4.3 TCP使用send_proxy参数

4.3.1 测试过程

• 更改并重载haproxy配置

listen tcp
        bind *:80
        mode tcp
        maxconn 20000
        server h1 172.169.18.20:20000 send-proxy

重载配置，启动server进程

docker kill -s HUP haproxy
go run tcp.go

• 执行虚拟机curl请求

 curl server 

• Go日志

2021/05/24 15:56:11 local addr: 172.169.18.20:20000, remote addr: 172.169.18.20:57402

4.3.2 结论

4.4 HTTP使用forward for参数

4.4.1 测试过程

• 更改并重载haproxy配置

listen http
        bind *:80
        mode http
        option forwardfor
        maxconn 20000
        server s1 172.169.18.20:20000

重载配置，启动server进程

docker kill -s HUP haproxy
go run http.go

• 执行虚拟机curl请求

 curl server

• Go日志

2021/05/24 18:57:09 request uri: /, proto: HTTP/1.1, host: server, user agent: curl/7.29.0, remote addr: 172.169.18.20:64634, x-forwarded-for: 172.17.0.1

4.4.2 结论

• 现象： x-forwarded-for 显示出的远端地址（在外网情况下，会显示成外网IP）

• 结论: 使用forwardfor可以通过x-forwad-for Header透传HTTP的远程地址，远程地址不受影响