容器云负载均衡之三：使用iptables对IPVS Direct Routing模式进行端口转换

曾经终败给现在 2022-05-26 10:37 226阅读 0赞

**一、前言**

在cloud环境中，不是所有的application都有使用80端口这种特权端口的权限；但是在IPVS direct routing模式下，VIP监听的端口必须要保持和后端real server上的application监听的端口一致，见文章[http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.rewrite\_ports.html][http_www.austintek.com_LVS_LVS-HOWTO_HOWTO_LVS-HOWTO.rewrite_ports.html]：

![70][]

本文通过iptables来解决这个问题，最终达到IPVS DR模式监听80端口，后端HTTP服务监听8080端口，客户端访问VIP:80可以得到后端8080端口提供的服务。

转载自https://blog.csdn.net/cloudvtech

**二、IPVS DR模式端口转换的实现**

2.1 机器配置

IPVS director： 192.168.166.102
    IPVS real server：192.168.166.103
    VIP：192.168.166.111

2.2 IPVS director的设置

ipvsadm -C  
    ipvsadm -A -t 192.168.166.111 -s rr  
    ipvsadm -a -t  192.168.166.111:80 -r 192.168.166.103:80 -w 1 -g  
      
    ifconfig  ens33:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.0 up  
    route add -host 192.168.166.111  dev ens33:0

/etc/sysctl.conf

net.ipv4.ip_forward = 1
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.conf.ens33.send_redirects = 0
    net.ipv4.conf.all.rp_filter = 0

2.3 IPVS real server的设置

/etc/sysctl.conf

net.ipv4.conf.all.arp_ignore = 1  
    net.ipv4.conf.all.arp_announce = 2  
    net.ipv4.conf.ens33.arp_ignore = 1  
    net.ipv4.conf.ens33.arp_announce = 2

ifconfig lo:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.255 up  
    route add -host 192.168.166.111  dev lo:0

并且启动http服务监听在80端口

2.4 在IPVS director插入如下iptables debug LOG

iptables -t mangle -A INPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix "[INPUT|mangle] "
    iptables -t mangle -A OUTPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix "[OUTPUT|mangle] "

2.5 在IPVS read server插入如下iptables debug LOG

iptables -t mangle -A OUTPUT -p tcp -m tcp --sport 8080 -j LOG --log-prefix "[OUTPUT|mangle] port 8080 "
    iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j LOG --log-prefix "[INPUT|nat] before REDIRECT "
    iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080

2.6 从其它客户端访问VIP的80端口

curl 192.168.166.111:80

可以得到返回：

curl 192.168.166.111 -vvv
    * Rebuilt URL to: 192.168.166.111/
    *   Trying 192.168.166.111...
    * TCP_NODELAY set
    * Connected to 192.168.166.111 (192.168.166.111) port 80 (#0)
    > GET / HTTP/1.1
    > Host: 192.168.166.111
    > User-Agent: curl/7.52.1
    > Accept: */*
    > 
    < HTTP/1.1 403 Forbidden
    < Date: Tue, 01 May 2018 05:37:41 GMT
    < Server: Apache/2.4.6 (CentOS)
    < Last-Modified: Thu, 16 Oct 2014 13:20:58 GMT
    < ETag: "1321-5058a1e728280"
    < Accept-Ranges: bytes
    < Content-Length: 4897
    < Content-Type: text/html; charset=UTF-8
    <

2.7 IPVS director iptables LOG输出

May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=51301 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=51301 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=12788 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=12788 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=43668 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK PSH URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=43668 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK PSH URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=61495 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4027 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=61495 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4027 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=11136 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=11136 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=10635 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4070 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=10635 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4070 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=46543 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK FIN URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=46543 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK FIN URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53957 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0 
    May  1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53957 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0

2.8 IPVS real server iptables LOG输出

May  1 01:37:41 k8s-node2 kernel: [INPUT|nat] before REDIRECT IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=51301 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 
    May  1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
    May  1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=32293 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK URGP=0 
    May  1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=4396 TOS=0x00 PREC=0x00 TTL=64 ID=32294 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK URGP=0 
    May  1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=857 TOS=0x00 PREC=0x00 TTL=64 ID=32297 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK PSH URGP=0 
    May  1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=32298 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK FIN URGP=0

转载自https://blog.csdn.net/cloudvtech

**三、与从IPVS director访问VIP相结合**

可以将本方案和根据文章[《从IPVS DR模式下director不能访问VIP问题的探究》][IPVS DR_director_VIP]提供的解决方案绑定在一起，就可以为kubernetes容器云环境下的IPVS service找到无缝的解决方案：

*  IPVS director可以和后端application混合部署
 *  IPVS director可以监听80特权端口
 *  application只需监听任意无特权端口
 *  客户端可以在包括IPVS director所在的worker node之内的任何位置访问VIP:80服务

转载自https://blog.csdn.net/cloudvtech

[http_www.austintek.com_LVS_LVS-HOWTO_HOWTO_LVS-HOWTO.rewrite_ports.html]: http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.rewrite_ports.html
[70]: /images/20220526/772771f64d174b8c8bcfe5625407f003.png
[IPVS DR_director_VIP]: https://blog.csdn.net/cloudvtech/article/details/80145568