关于LVS的配置

你的名字 2022-05-28 11:12 127阅读 0赞

*  配置`LVS`服务,至少需要四个节点

> [server20.com][] 172.25.23.20  
> [server21.com][] 172.25.23.21  
> [server22.com][] 172.25.23.22  
> [server24.com][] 172.25.23.24  
> 需要关闭各个节点上面的`iptables selinux`,并且禁止`iptables`开机启动;

*  首先在`server22.com`以及`server24.com`上面安装`httpd`服务并且启动,

[root@server22 ~]# yum install httpd -y 
    [root@server22 ~]# /etc/init.d/httpd start
    
    [root@server24 ~]# yum install httpd -y
    [root@server24 ~]# /etc/init.d/httpd start

*  确保浏览器访问可以得到下面的访问结果  
    ![这里写图片描述][70]

![这里写图片描述][70 1]

*  **接下来配置`directory`节点**
 *  首先需要给`directory`上面配置一个`VIP`地址

[root@server21 ~]# ip addr add 172.25.23.100/24 dev eth0 
    [root@server21 ~]# ip addr show eth0
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether 52:54:00:62:3a:95 brd ff:ff:ff:ff:ff:ff
        inet 172.25.23.21/24 brd 172.25.23.255 scope global eth0
        inet 172.25.23.100/24 scope global secondary eth0
        inet6 fe80::5054:ff:fe62:3a95/64 scope link 
           valid_lft forever preferred_lft forever

*  在提供`IP`地址之后,`httpd`服务并不能够正常访问,因为并没有指定合适的`httpd`服务规则和后端的`httpd`服务节点进行绑定;

[root@server21 ~]# ipvsadm -A -t 172.25.23.100:80 -s rr 
    [root@server21 ~]# ipvsadm -a -t 172.25.23.100:80 -r 172.25.23.22:80 -g
    [root@server21 ~]# ipvsadm -a -t 172.25.23.100:80 -r 172.25.23.24:80 -g

*  可以查看新添加的规则

[root@server21 ~]# ipvsadm
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  172.25.23.100:http rr
      -> 172.25.23.22:http            Route   1      0          0         
      -> 172.25.23.24:http            Route   1      0          0

*  接下来查看是否正确进行调度,以及调度到那个节点上面

[root@server21 ~]# ipvsadm
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  172.25.23.100:http rr
      -> 172.25.23.22:http            Route   1      0          2    [表示两个链接]     
      -> 172.25.23.24:http            Route   1      0          2

*  虽然进行了上述配置,但是服务仍然是不能够正常访问的,接下来在`realserver`上面继续进行配置;

[root@server22 ~]# yum install arptables_jf -y
    [root@server22 ~]# arptables -A IN -d 172.25.23.100 -j DROP
    [root@server22 ~]# arptables -A OUT -s 172.25.23.100 -j mangle --mangle-ip-s 172.25.23.22
    [root@server22 ~]# /etc/init.d/arptables_jf save
    Saving current rules to /etc/sysconfig/arptables:          [  OK  ]
    
    [root@server24 ~]#  yum install arptables_jf -y
    [root@server24 ~]# arptables -A IN -d 172.25.23.100 -j DROP
    [root@server24 ~]# arptables -A OUT -s 172.25.23.100 -j mangle --mangle-ip-s 172.25.23.24 
    [root@server24 ~]# /etc/init.d/arptables_jf save
    Saving current rules to /etc/sysconfig/arptables:          [  OK  ]

*  接下来需要在两个网卡上面配置别名

[root@server22 ~]# ifconfig eth0:1 172.25.23.100 netmask 255.255.255.0
    [root@server24 ~]# ifconfig eth0:1 172.25.23.100 netmask 255.255.255.0

*  然后通过浏览器刷新页面  
    ![这里写图片描述][70 2]  
    ![这里写图片描述][70 3]
 *  接下来配置`ldirecotry`用于实现对于后端服务的监控状况检查
 *  首先需要安装软件包

[root@server21 ~]# yum install ldirectord-3.9.5-3.1.x86_64.rpm
    [root@server21 ~]# cp /usr/share/doc/ldirectord-3.9.5/ldirectord.cf /etc/ha.d/

*  修改配置文件

[root@server21 ~]# vim /etc/ha.d/ldirectord.cf 
    更改或者添加
    virtual=172.25.23.100:80
            real=172.25.23.22:80 gate
            real=172.25.23.24:80 gate
            fallback=127.0.0.1:80 gate
            service=http
            scheduler=rr
            #persistent=600
            #netmask=255.255.255.255
            protocol=tcp
            checktype=negotiate
            checkport=80
            request="index.html"
         #  receive="Test Page"
         # virtualhost=www.x.y.z

*  接下来停止某个节点上面的服务,那么用于的调度请求,就不会被调度到非正常的节点上面

[root@server22 ~]# /etc/init.d/httpd stop
    Stopping httpd:                                            [  OK  ]

*  服务请求会交给`server24.com`来进行响应  
    ![这里写图片描述][70 4]
 *  为了防止`ldriectory`的单个节点故障,可以使用`server20.com`以及`server21.com`来配置`ldirectory`的高可用集群,这里是结合高可用来进行配置的

[root@server21 ~]# vim /etc/ha.d/haresources 
    server20.com IPaddr::172.25.23.100/24/eth0 ldirectord httpd

*  同样需要将上面的配置信息复制到另一个节点上面,同时还需要复制`ldirectory`的配置文件

[root@server21 ~]# scp /etc/ha.d/haresources 172.25.23.20:/etc/ha.d/haresources
    haresources                                   100% 5969     5.8KB/s   00:00
    [root@server21 ~]# scp /etc/ha.d/ldirectord.cf 172.25.23.20:/etc/ha.d/
    ldirectord.cf                                 100% 8280     8.1KB/s   00:00

*  同时需要确保节点`server20.com`上面是存在`ldirectory`程序的

[root@server20 ~]# /etc/init.d/ldirectord

*  因为是配置集群服务,所以资源需要交给集群来进行配置,所以这里需要还原资源的状态

[root@server21 ~]# /etc/init.d/ldirectord stop
    Stopping ldirectord... success
    [root@server21 ~]# ip addr del 172.25.23.100/24 dev eth0

*  然后在两个节点上面启动`heartbeat`服务

[root@server20 ~]# /etc/init.d/heartbeat start
    Starting High-Availability services: INFO:  Resource is stopped
    Done.
    
    [root@server21 ~]# /etc/init.d/heartbeat start
    Starting High-Availability services: INFO:  Resource is stopped
    Done.

*  接下来写入服务器故障之后,提示用户的信息

[root@server20 ~]# echo <h2>the server is busying please try later</h2> >/var/www/html/index.html
    [root@server21 ~]# echo <h2>the server is busying please try later</h2> >/var/www/html/index.html

*  查看`heatbeat`管理的集群资源是否正常运行

[root@server20 ~]# /etc/init.d/ldirectord status
    ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 5126
    [root@server20 ~]# /etc/init.d/httpd status
    httpd (pid  5154) is running...
    [root@server20 ~]# ip addr show eth0
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether 52:54:00:03:aa:77 brd ff:ff:ff:ff:ff:ff
        inet 172.25.23.20/24 brd 172.25.23.255 scope global eth0
        inet 172.25.23.100/24 brd 172.25.23.255 scope global secondary eth0
        inet6 fe80::5054:ff:fe03:aa77/64 scope link 
           valid_lft forever preferred_lft forever

*  查看`ipvs`的信息

[root@server20 ~]# ipvsadm
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  localhost:http rr
      -> 172.25.23.22:http            Route   1      0          0         
      -> 172.25.23.24:http            Route   1      0          0

*  接下来关闭节点上面的`httpd`服务

[root@server22 ~]# /etc/init.d/httpd  stop
    Stopping httpd:                                            [  OK  ]
    [root@server24 ~]# /etc/init.d/httpd stop
    Stopping httpd:                                            [  OK  ]

*  接下来通过浏览器查看得到访问失败的页面  
    ![这里写图片描述][70 5]
 *  接下来关闭`heartbeat`并且关闭`ldirectory`来配置`keepalived`集群

root@server20 ~]# /etc/init.d/heartbeat stop
    Stopping High-Availability services: Done.
    
    [root@server20 ~]# chkconfig heartbeat off
    [root@server20 ~]# /etc/init.d/ldirectord stop
    Stopping ldirectord... success
    [root@server20 ~]# chkconfig ldirectord off
    
    root@server20 ~]# /etc/init.d/heartbeat stop
    Stopping High-Availability services: Done.
    
    [root@server21 ~]# chkconfig heartbeat off
    [root@server21 ~]# /etc/init.d/ldirectord stop
    Stopping ldirectord... success
    [root@server21 ~]# chkconfig ldirectord off

\*`keepalived`需要编译进行安装

[root@server20 keepalived-1.2.24]# yum install libnl-devel libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm[这个软件包需要额外下载] -y
    [root@server20 keepalived-1.2.24]# ./configure --prefix=/usr/local/keepalived
    [root@server20 keepalived-1.2.24]# make 
    [root@server20 keepalived-1.2.24]# make install 
    [root@server20 keepalived-1.2.24]# scp -r /usr/local/keepalived/ 172.25.23.21:/usr/local/
    
    [root@server20 keepalived-1.2.24]# ln -sv /usr/local/keepalived/etc/keepalived /etc/
    `/etc/keepalived' -> `/usr/local/keepalived/etc/keepalived'
    
    [root@server20 keepalived-1.2.24]# ln -sv /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
    `/etc/init.d/keepalived' -> `/usr/local/keepalived/etc/rc.d/init.d/keepalived'
    
    [root@server20 keepalived-1.2.24]# chmod +x /etc/init.d/keepalived 
    
    [root@server20 keepalived-1.2.24]# ln -sv  /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
    `/etc/sysconfig/keepalived' -> `/usr/local/keepalived/etc/sysconfig/keepalived'
    
    [root@server20 keepalived-1.2.24]# ln -sv /usr/local/keepalived/sbin/keepalived /sbin/
    `/sbin/keepalived' -> `/usr/local/keepalived/sbin/keepalived'
    
    [root@server20 keepalived-1.2.24]# ln -sv /usr/local/keepalived/etc/ /etc/ 
    `/etc/etc' -> `/usr/local/keepalived/etc/'

*  接下来还需要在`server21.com`上面需要进行配置

[root@server21 ~]# ln -sv /usr/local/keepalived/etc/keepalived /etc/
    `/etc/keepalived' -> `/usr/local/keepalived/etc/keepalived'
    [root@server21 ~]# ln -sv /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
    [root@server21 ~]# chmod +x /etc/init.d/keepalived
    [root@server21 ~]# ln -sv /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
    [root@server21 ~]# ln -sv /usr/local/keepalived/sbin/keepalived /sbin/

*  在两个节点上面尝试启动服务,并且确定是正常的

[root@server21 ~]# /etc/init.d/keepalived start
    Starting keepalived:                                       [  OK  ]
    [root@server21 ~]# /etc/init.d/keepalived stop
    Stopping keepalived:                                       [  OK  ]
    
    [root@server20 keepalived-1.2.24]# /etc/init.d/keepalived start
    Starting keepalived:                                       [  OK  ]
    [root@server20 keepalived-1.2.24]# /etc/init.d/keepalived stop
    Stopping keepalived:                                       [  OK  ]

*  接下来需要修改配置文件

[root@server20 ~]# vim /etc/keepalived/keepalived.conf
    添加或者修改下面的信息:
    
    ! Configuration File for keepalived
    
    global_defs {
       notification_email {
    	root@localhost
       }
       notification_email_from keepalived@server20.com
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface eth0
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
    	172.25.23.100
        }
    }
    
    virtual_server 172.25.23.100 80 {
        delay_loop 6
        lb_algo rr
        lb_kind DR
    #    persistence_timeout 50
        protocol TCP
    
        real_server 172.25.23.22 80 {
            weight 1
            TCP_CHECK {
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
        real_server 172.25.23.24 80 {
            weight 1
            TCP_CHECK {
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
           	}
       }
    }

*  将这个配置文件复制给`server21.com`

[root@server20 ~]# scp /etc/keepalived/keepalived.conf 172.25.23.21:/etc/keepalived/keepalived.conf
    keepalived.conf                                  100% 1037     1.0KB/s   00:00

*  并且进行下面的修改

更改如下:
    ! Configuration File for keepalived
    
    global_defs {
       notification_email {
    	root@localhost
       }
       notification_email_from keepalived@server21.com
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface eth0
        virtual_router_id 51
        priority 50
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
    	172.25.23.100
        }
    }
    
    virtual_server 172.25.23.100 80 {
        delay_loop 6
        lb_algo rr
        lb_kind DR
    #    persistence_timeout 50
        protocol TCP
    
        real_server 172.25.23.22 80 {
            weight 1
            TCP_CHECK {
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
        real_server 172.25.23.24 80 {
            weight 1
            TCP_CHECK {
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
           	}
       }
    }

*  配置`real server`的`httpd`服务正常

[root@server22 ~]# /etc/init.d/httpd  start
    Starting httpd: httpd: apr_sockaddr_info_get() failed for server22.com
    httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
                                                               [  OK  ]
    [root@server24 ~]# /etc/init.d/httpd start
    Starting httpd:                                            [  OK  ]

*  `keepalived`节点配置启动`keepalived`服务

[root@server20 ~]# /etc/init.d/keepalived restart
    Stopping keepalived:                                       [  OK  ]
    Starting keepalived:                                       [  OK  ]
    
    [root@server21 ~]# /etc/init.d/keepalived restart
    Stopping keepalived:                                       [  OK  ]
    Starting keepalived:                                       [  OK  ]

*  查看日志记录的服务启动信息

Apr  4 22:48:13 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:13 server20 Keepalived_vrrp[16094]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.25.23.100
    Apr  4 22:48:13 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:13 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:13 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:13 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:13 server20 Keepalived_healthcheckers[16093]: Netlink reflector reports IP 172.25.23.100 added
    Apr  4 22:48:18 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:18 server20 Keepalived_vrrp[16094]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.25.23.100
    Apr  4 22:48:18 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:18 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:18 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:48:18 server20 Keepalived_vrrp[16094]: Sending gratuitous ARP on eth0 for 172.25.23.100

*  通过浏览器可以正常访问虚拟`IP`提供的服务  
    ![这里写图片描述][70 6]

![这里写图片描述][70 7]

*  通过关闭`server20.com`上面的`keepalived`来观察服务的迁移过程
 *  `server20.com`上面显示服务已经停止

Apr  4 22:57:38 server20 Keepalived_vrrp[16094]: VRRP_Instance(VI_1) removing protocol VIPs.
    Apr  4 22:57:38 server20 Keepalived_healthcheckers[16093]: Netlink reflector reports IP 172.25.23.100 removed
    Apr  4 22:57:38 server20 Keepalived_healthcheckers[16093]: Removing service [172.25.23.22]:80 from VS [172.25.23.100]:80
    Apr  4 22:57:38 server20 Keepalived_healthcheckers[16093]: Removing service [172.25.23.24]:80 from VS [172.25.23.100]:80
    Apr  4 22:57:38 server20 Keepalived_healthcheckers[16093]: Stopped

*  `server21.com`上面显示服务正在配置

Apr  4 22:57:45 server21 Keepalived_vrrp[6283]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.25.23.100
    Apr  4 22:57:45 server21 Keepalived_vrrp[6283]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:57:45 server21 Keepalived_vrrp[6283]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:57:45 server21 Keepalived_vrrp[6283]: Sending gratuitous ARP on eth0 for 172.25.23.100
    Apr  4 22:57:45 server21 Keepalived_vrrp[6283]: Sending gratuitous ARP on eth0 for 172.25.23.100

*  地址已经正常配置

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether 52:54:00:62:3a:95 brd ff:ff:ff:ff:ff:ff
        inet 172.25.23.21/24 brd 172.25.23.255 scope global eth0
        inet 172.25.23.100/32 scope global eth0

*  `ipvsadm`的规则也已经转移过来

*  服务的访问同样是正常的;

[server20.com]: http://server20.com
[server21.com]: http://server21.com
[server22.com]: http://server22.com
[server24.com]: http://server24.com
[70]: /images/20220528/6004b8702108455aba40a28dc8889be6.png
[70 1]: /images/20220528/6f04d1454f9e4a478218a26e73ac6cb9.png
[70 2]: /images/20220528/7776213a39f24cda8ebada6ef3fc44a9.png
[70 3]: /images/20220528/7d4428027a184234bc882da77290dcf1.png
[70 4]: /images/20220528/91baa099fc2743febb57950cd1eac0d2.png
[70 5]: /images/20220528/31f2f9b4de9b4029855e8274c7867084.png
[70 6]: /images/20220528/065679a9fbf745ec8ffac208e48bf9d0.png
[70 7]: /images/20220528/14aa985fb4ac459c93beb6bd9a4b5828.png