keepalived高可用nginx

妖狐艹你老母 2022-01-12 17:45 568阅读 0赞

#### 一、项目概述 ####

负载均衡技术对于一个网站尤其是大型网站的web服务器集群来说是至关重要的！做好负载均衡架构，可以实现故障转移和高可用环境，避免单点故障，保证网站健康持续运行。  
由于业务扩展，网站的访问量不断加大，负载越来越高。现需要在web前端放置nginx负载均衡,同时结合keepalived对前端nginx实现HA高可用。  
介绍下Nginx和keepalive  
1.Nginx

Nginx 是一个很强大的高性能Web和反向代理服务器，它具有很多非常优越的特性：  
Nginx作为负载均衡服务器：Nginx 既可以在内部直接支持 Rails 和 PHP 程序对外进行服务，也可以支持作为 HTTP代理服务器对外进行服务。Nginx采用C进行编写，不论是系统资源开销还是CPU使用效率都比 Perlbal 要好很多。  
2.keepalive

Keepalived是Linux下面实现VRRP备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。二者结合，可以构架出比较稳定的软件LB方案。  
Nginx+keepalive高可用方式有两种：  
1.Nginx+keepalived 主从配置

这种方案，使用一个vip地址，前端使用2台机器，一台做主，一台做备，但同时只有一台机器工作，另一台备份机器在主机器不出现故障的时候，永远处于浪费状态，对于服务器不多的网站，该方案不经济实惠。  
2.Nginx+keepalived 双主配置

这种方案，使用两个vip地址，前端使用2台机器，互为主备，同时有两台机器工作，当其中一台机器出现故障，两台机器的请求转移到一台机器负担，非常适合于当前架构环境。所以在这里就详细介绍下双主模型配置

### 二、项目拓扑图 ###

![image.png][]

### 三、项目规划 ###

<table> 
 <tbody> 
  <tr> 
   <td>操作系统</td> 
   <td>内网IP地址</td> 
   <td>外网IP地址</td> 
   <td colspan="1">主机名</td> 
   <td>部署服务</td> 
  </tr> 
  <tr> 
   <td>Centos7.6</td> 
   <td>192.168.1.230/24</td> 
   <td>172.16.30.230/16</td> 
   <td>LB01-Master</td> 
   <td>nginx,keepalived</td> 
  </tr> 
  <tr> 
   <td>Centos7.6</td> 
   <td>192.168.1.231/24</td> 
   <td>172.16.30.231/16</td> 
   <td>LB02-Backup</td> 
   <td>nginx,keepalived</td> 
  </tr> 
  <tr> 
   <td>Centos7.6</td> 
   <td><br></td> 
   <td>172.16.30.220/16</td> 
   <td>Web-Node01</td> 
   <td>nginx</td> 
  </tr> 
  <tr> 
   <td>Centos7.6</td> 
   <td><br></td> 
   <td>172.16.30.221/16</td> 
   <td colspan="1">Web-Node02</td> 
   <td>nginx</td> 
  </tr> 
 </tbody> 
</table>

**四、操作部署**

1、后端主机部署步骤

1. Web-Node01同步时间

[root@Web-Node01 ~]# ntpdate ntp.aliyun.com

2. Web-Node01安装nginx

[root@Web-Node01 ~]# yum -y install nginx

3. Web-Node01配置nginx服务

[root@Web-Node01 ~]# vim /etc/nginx/conf.d/vancen.conf

server {

listen       80;

server_name  localhost;

root    /usr/share/nginx/html/node1;

location / {

root   html;

index  index.html index.htm;

}

4. Web-Node01编辑主页

[root@Web-Node01 ~]# echo "<h1> Web-Node01 172.16.30.220 </h1>" >/usr/share/nginx/html/node1/index.html

5. Web-Node01重启nginx服务

[root@Web-Node01 ~]# systemctl start nginx

[root@Web-Node01 ~]# systemctl enable nginx

6. Web-Node02同步时间

[root@Web-Node02 ~]# ntpdate ntp.aliyun.com

7. Web-Node02安装nginx服务

[root@Web-Node01 ~]# yum -y install nginx

8. Web-Node02配置虚拟主机

[root@Web-Node02 ~]# vim /etc/nginx/conf.d/vancen.conf

server {

listen       80;

server_name  localhost;

root    /usr/share/nginx/html/node2;

location / {

root   html;

index  index.html index.htm;

}

9. Web-Node02编辑主页

[root@Web-Node02 ~]# echo "<h1> Web-Node02 172.16.30.221 </h1>" >/usr/share/nginx/html/node1/index.html

10. Web-Node02重启nignx服务

[root@Web-Node02 ~]# systemctl start nginx

[root@Web-Node02 ~]# systemctl enable nginx

前端主机部署步骤

1. LB01-Master上同步时间

[root@LB01-Master ~]# ntpdate ntp.aliyun.com

2. LB01-Master上安装nginx,keepalived

[root@LB01-Master ~]# yum -y install nginx keepalived

3. LB01-Master上配置反向代理服务

[root@LB01-Master ~]# vim /etc/nginx/conf.d/proxy_nginx.conf

upstream webserver {

server 172.16.30.220:80;

server 172.16.30.221:80;

}

server {

listen 80;

server_name blog.chenjf.com;

location / {

proxy_pass http://webserver;

}

[root@LB01-Master ~]# vim /etc/nginx/proxy.conf

proxy_redirect default;

proxy_set_header Host $http_host;

proxy_set_header X-Real-IP $remote_addr;

proxy_connect_timeout 30;

proxy_send_timeout 60;

proxy_read_timeout 60;

proxy_buffer_size 32k;

proxy_buffering on;

proxy_buffers 4 128k;

proxy_busy_buffers_size 256k;

proxy_max_temp_file_size 256k;

4. LB01-Master上配置keepalived服务

[root@LB01-Master ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

root@localhost

}

notification_email_from keepalived@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id Node01

vrrp_mcast_group4 224.1.10.66

}

vrrp_script chk_nginx {

script "/etc/keepalived/chk_nginx.sh"

interval 2

weight 2

}

vrrp_instance VI_1 {

state MASTER

interface ens33

virtual_router_id 66

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass QVvYUx0oWw

}

track_script {

chk_nginx

}

virtual_ipaddress {

192.168.1.13/24 dev ens33 label ens33:0

}

notify_master "/etc/keepalived/notify.sh master"

notify_backup "/etc/keepalived/notify.sh backup"

notify_fault "/etc/keepalived/notify.sh fault"

}

5. LB01-Master上编辑邮件报警脚本

[root@LB01-Master ~]# vim /etc/keepalived/notify.sh

#!/bin/bash

# Author: MageEdu <linuxedu@foxmail.com>

# description: An example of notify script

#定义VIP地址

vip=172.16.100.1

#定义邮件接收人

contact='root@localhost'

Notify() {

#邮件标题

mailsubject="`hostname` to be $1: $vip floating"

#邮件正文

mailbody="`date '+%F %H:%M:%S'`: vrrp transition, `hostname` changed to be $1"

#发送邮件

echo $mailbody | mail -s "$mailsubject" $contact

}

case "$1" in

master)

systemctl start nginx

notify master

/etc/rc.d/init.d/haproxy start

exit 0

;;

backup)

systemctl start nginx

notify backup

/etc/rc.d/init.d/haproxy restart

exit 0

;;

fault)

systemctl stop nginx

notify fault

exit 0

;;

echo 'Usage: `basename $0` {master|backup|fault}'

exit 1

;;

esac

6. LB01-Master上配置监控nginx脚本

[root@LB01-Master ~]# vim /etc/keepalived/chk_nginx.sh

#!/bin/bash

status=$(ps -C nginx --no-heading|wc -l)

if [ "${status}" = "0" ]; then

systemctl stop keepalived

7. LB02-backup上同步时间

[root@LB02-Backup ~]# ntpdate ntp.aliyun.com

8. LB02-backup上安装nginx,keepalived

[root@LB02-Backup ~]# yum -y install nginx keepalived

9. LB02-backup上配置反向代理

[root@LB02-Backup ~]# vim /etc/nginx/conf.d/proxy_nginx.conf

upstream webserver {

server 172.16.30.220:80;

server 172.16.30.221:80;

}

server {

listen 80;

server_name blog.chenjf.com;

location / {

proxy_pass http://webserver;

include proxy.conf;

}

[root@LB02-Backup ~]# vim /etc/nginx/proxy.conf

proxy_redirect default;

proxy_set_header Host $http_host;

proxy_set_header X-Real-IP $remote_addr;

proxy_connect_timeout 30;

proxy_send_timeout 60;

proxy_read_timeout 60;

proxy_buffer_size 32k;

proxy_buffering on;

proxy_buffers 4 128k;

proxy_busy_buffers_size 256k;

proxy_max_temp_file_size 256k;

10 LB02-backup上配置keepalived服务

[root@LB02-Backup ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

notification_email {

root@localhost

}

notification_email_from keepalived@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id Node02

vrrp_mcast_group4 224.1.10.66

}

vrrp_script chk_ngx {

script "/etc/keepalived/chk_nginx.sh"

interval 1

weight -5

fall 2

rise 1

}

vrrp_instance VI_1 {

state BACKUP

interface ens33

virtual_router_id 66

priority 98

advert_int 1

authentication {

auth_type PASS

auth_pass QVvYUx0oWw

}

track_script {

chk_ngx

}

virtual_ipaddress {

192.168.1.13/24 dev ens33 label ens33:0

}

notify_master "/etc/keepalived/notify.sh master"

notify_backup "/etc/keepalived/notify.sh backup"

notify_fault "/etc/keepalived/notify.sh fault"

}

11. LB02-backup上编辑邮件报警脚本

[root@LB02-Backup ~]# vim /etc/keepalived/notify.sh

#!/bin/bash

# Author: MageEdu <linuxedu@foxmail.com>

# description: An example of notify script

#定义VIP地址

vip=172.16.100.1

#定义邮件接收人

contact='root@localhost'

Notify() {

#邮件标题

mailsubject="`hostname` to be $1: $vip floating"

#邮件正文

mailbody="`date '+%F %H:%M:%S'`: vrrp transition, `hostname` changed to be $1"

#发送邮件

echo $mailbody | mail -s "$mailsubject" $contact

}

case "$1" in

master)

systemctl start nginx

notify master

/etc/rc.d/init.d/haproxy start

exit 0

;;

backup)

systemctl start nginx

notify backup

/etc/rc.d/init.d/haproxy restart

exit 0

;;

fault)

systemctl stop nginx

notify fault

exit 0

;;

echo 'Usage: `basename $0` {master|backup|fault}'

exit 1

;;

esac

12. LB02-backup上编辑监控nginx脚本

[root@LB02-Backup ~]# vim /etc/keepalived/chk_nginx.sh

#!/bin/bash

status=$(ps -C nginx --no-heading|wc -l)

if [ "${status}" = "0" ]; then

systemctl stop keepalived

13. LB02-backup上启动nginx,keepalived服务，并加入开机自启

[root@LB02-Backup ~]# systemctl start nginx keepalived

[root@LB02-Backup ~]# systemctl enable nginx keepalived

14. LB02-backup上查看keepalived服务状态

[root@LB02-Backup ~]# systemctl status keepalived

● keepalived.service - LVS and VRRP High Availability Monitor

Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)

Active: active (running) since Tue 2019-05-07 17:09:20 CST; 19s ago

Process: 10739 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)

Main PID: 10743 (keepalived)

CGroup: /system.slice/keepalived.service

├─10743 /usr/sbin/keepalived -D

├─10744 /usr/sbin/keepalived -D

└─10745 /usr/sbin/keepalived -D

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: Opening file '/etc/keepalived/keepalived.conf'.

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: Truncating auth_pass to 8 characters

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: VRRP_Instance(VI_1) removing protocol VIPs.

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: Using LinkWatch kernel netlink reflector...

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: VRRP_Instance(VI_1) Entering BACKUP STATE

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: Opening script file /etc/keepalived/notify.sh

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

May 07 17:09:20 LB02-Backup Keepalived_vrrp[10745]: VRRP_Script(chk_ngx) succeeded

May 07 17:09:25 LB02-Backup Keepalived_vrrp[10745]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:09:25 LB02-Backup Keepalived_vrrp[10745]: Opening script file /etc/keepalived/notify.sh

May 07 17:09:30 LB02-Backup Keepalived_vrrp[10745]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:09:30 LB02-Backup Keepalived_vrrp[10745]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.1.13

May 07 17:09:30 LB02-Backup Keepalived_vrrp[10745]: Sending gratuitous ARP on ens33 for 192.168.1.13

15. LB01-Master上启动nginx,keepalived服务，并加入开机自启

[root@LB01-Master ~]# systemctl start nginx keepalived

[root@LB01-Master ~]# systemctl enable nginx keepalived

16. LB01-Master上查看keepalived服务状态

[root@LB01-Master ~]# systemctl status keepalived

● keepalived.service - LVS and VRRP High Availability Monitor

Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)

Active: active (running) since Tue 2019-05-07 17:11:23 CST; 30s ago

Main PID: 36639 (keepalived)

CGroup: /system.slice/keepalived.service

├─36639 /usr/sbin/keepalived -D

├─36640 /usr/sbin/keepalived -D

└─36641 /usr/sbin/keepalived -D

May 07 17:11:25 LB01-Master Keepalived_vrrp[36641]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:11:25 LB01-Master Keepalived_vrrp[36641]: Opening script file /etc/keepalived/notify.sh

May 07 17:11:30 LB01-Master Keepalived_vrrp[36641]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:11:30 LB01-Master Keepalived_vrrp[36641]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.1.13

May 07 17:11:30 LB01-Master Keepalived_vrrp[36641]: Sending gratuitous ARP on ens33 for 192.168.1.13

**四、测试步骤**

1. LB01-Master上停止nginx服务，查看是keepalived状态

[root@LB01-Master ~]# systemctl stop nginx

[root@LB01-Master ~]# systemctl status keepalived

● keepalived.service - LVS and VRRP High Availability Monitor

Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)

Active: inactive (dead) since Tue 2019-05-07 17:12:41 CST; 974ms ago

Main PID: 36639 (code=exited, status=0/SUCCESS)

CGroup: /system.slice/keepalived.service

May 07 17:11:30 LB01-Master Keepalived_vrrp[36641]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.1.13

May 07 17:11:30 LB01-Master Keepalived_vrrp[36641]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:12:40 LB01-Master Keepalived[36639]: Stopping

May 07 17:12:40 LB01-Master systemd[1]: Stopping LVS and VRRP High Availability Monitor...

May 07 17:12:40 LB01-Master Keepalived_vrrp[36641]: VRRP_Instance(VI_1) sent 0 priority

May 07 17:12:40 LB01-Master Keepalived_vrrp[36641]: VRRP_Instance(VI_1) removing protocol VIPs.

May 07 17:12:41 LB01-Master systemd[1]: Stopped LVS and VRRP High Availability Monitor.

2. LB02-Backup上查看keepalived状态

[root@LB02-Backup ~]# systemctl status keepalived

● keepalived.service - LVS and VRRP High Availability Monitor

Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)

Active: active (running) since Tue 2019-05-07 17:09:20 CST; 3min 49s ago

Process: 10739 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)

Main PID: 10743 (keepalived)

CGroup: /system.slice/keepalived.service

├─10743 /usr/sbin/keepalived -D

├─10744 /usr/sbin/keepalived -D

└─10745 /usr/sbin/keepalived -D

May 07 17:12:41 LB02-Backup Keepalived_vrrp[10745]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:12:41 LB02-Backup Keepalived_vrrp[10745]: Opening script file /etc/keepalived/notify.sh

May 07 17:12:46 LB02-Backup Keepalived_vrrp[10745]: Sending gratuitous ARP on ens33 for 192.168.1.13

May 07 17:12:46 LB02-Backup Keepalived_vrrp[10745]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.1.13

May 07 17:12:46 LB02-Backup Keepalived_vrrp[10745]: Sending gratuitous ARP on ens33 for 192.168.1.13

3. LB02-Backup上查看VIP地址

[root@LB02-Backup ~]# ifconfig

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

inet 192.168.1.231  netmask 255.255.255.0  broadcast 192.168.1.255

inet6 fe80::657b:51c8:2bf4:e92c  prefixlen 64  scopeid 0x20<link>

ether 00:0c:29:b9:86:01  txqueuelen 1000  (Ethernet)

RX packets 122639  bytes 29532328 (28.1 MiB)

RX errors 0  dropped 0  overruns 0  frame 0

TX packets 18981  bytes 1982039 (1.8 MiB)

TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

inet 192.168.1.13  netmask 255.255.255.0  broadcast 0.0.0.0

ether 00:0c:29:b9:86:01  txqueuelen 1000  (Ethernet)

**五、客户端访问后端主机**

1. 客户端访问http://blog.chenjf.com

[root@localhost ~]# for i in {1..10};do curl http://blog.chenjf.com ;done

2. 在访问过程中启动LB01-Master 上的nginx服务查看客户端访问是否中断

[root@localhost ~]# for i in {1..66};do curl http://blog.chenjf.com ;done

[root@LB01-Master ~]# systemctl start nginx keepalived

转载于:https://blog.51cto.com/10802692/2391767

[image.png]: https://s1.51cto.com/images/20190509/1557392029924379.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=