Redis+Keepalived高可用方案详细分析

骑猪看日落 2021-12-20 12:31 312阅读 0赞

**背景**

目前，Redis集群的官方方案还处在开发测试中，未集成到稳定版中。且目前官方开发中的Redis Cluster提供的功能尚不完善（可参考官方网站或[http://www.redisdoc.com/en/latest/topic/cluster-spec.html][http_www.redisdoc.com_en_latest_topic_cluster-spec.html]），在生产环境中不推荐使用。通过调研发现市面上要实现采用单一的IP来访问，大多采用keepalived实现redis的双机热备作为过渡方案。

**环境部署**

环境介绍：  
Master: 192.168.1.218 redis，keepalived  
Slave: 192.168.1.219 redis，keepalived  
Virtural IP Address (VIP): 192.168.1.220

设计思路：

两个redis server主从备份。提供redis 服务高可用；两个keepalived 服务主从备份，提供VIP 服务的高可用。

1)每台redis server分别有主，从两个配置文件(redis\_master.conf, redis\_slave.conf)，通过启动脚本启动服务，启动脚本会检测这个redis集群中的其他服务器的角色，如果有master 服务存在，则以slave角色启动，否则将自己以master服务器启动；

2) keepalived 监控脚本，定时（频率：每秒一次）检测当前服务器是否获取集群VIP，如果获取集群VIP，则将本服务器上的redis服务器设置为master。同时将远端的其他redis服务器设置为slave；保证获取集群VIP的redis 服务器角色为master，其他的设置为slave。

3) keepalived 监控脚本，还会自动检测当前redis服务器是否正常，如果连续两次检测异常，则停掉本本机的keepalived服务，释放集群VIP，让去漂移到其他可以提供redis 服务的服务器上；

4)当 Master 与 Slave 均运作正常时, Master负责服务，Slave负责同步数据；当 Master 挂掉，Slave 正常时, Slave接管服务，同时关闭主从复制功能；当 Master 恢复正常，则从Slave同步数据，同步数据之后关闭主从复制功能，恢复Master身份，于此同时Slave等待Master同步数据完成之后，恢复Slave身份。然后依次循环。

实施步骤：

\----创建专用用户

useradd -g develop redisadmin  
echo Hisun@1125|passwd --stdin redisadmin

说明：以下部署过程都是在root(或具备sudo权限的账号)账户下进行。

\----安装配置redis

1.下载redis源码

wget [http://download.redis.io/releases/redis-2.8.3.tar.gz][http_download.redis.io_releases_redis-2.8.3.tar.gz]

2.安装redis

tar -zxvf redis-2.8.3.tar.gz

cd redis-2.8.3

\#reds的安装可以不用执行configure

make

\#测试

make test

\#\#\#\#在速度较慢的机器上执行make test可能出现下列错误，无影响

\#\*\*\* \[err\]: Test replication partial resync: no backlog in tests/integration/replication-psync.tcl

3.配置redis

\#创建redis主目录

mkdir -p /usr/local/redis-2.8.3/\{bin,conf,logs\}

cp -a -R -p src/redis-server /usr/local/redis-2.8.3/bin/

cp -a -R -p src/redis-cli /usr/local/redis-2.8.3/bin/

cp -a -R -p src/redis-benchmark /usr/local/redis-2.8.3/bin/

cp -a -R -p src/redis-sentinel /usr/local/redis-2.8.3/bin/

cp -a -R -p src/redis-check-dump /usr/local/redis-2.8.3/bin/

cp -a -R -p src/redis-check-aof /usr/local/redis-2.8.3/bin/

\#创建redis启动脚本

vi /usr/local/redis-2.8.3/redis-start.sh

\#\#\#\#以下为master上的配置，slave上的配置只需要修改对应的LOCALIP和REMOTEIP即可。

![复制代码][copycode.gif]

#!/bin/bash
    REDISPATH=/usr/local/redis-2.8.3
    REDISCLI=$REDISPATH/bin/redis-cli
    LOGFILE=$REDISPATH/logs/redis-state.log
    LOCALIP=192.168.1.218
    REMOTEIP=192.168.1.219
    
    
    REMOTEREDISROLE=`$REDISCLI -h $REMOTEIP info | grep "role"`
    if grep "role:master" <<< $REMOTEREDISROLE ; then
            #start as slave
            $REDISPATH/bin/redis-server $REDISPATH/conf/redis_slave.confif [ "$?" == "0" ];then
                    echo "[INFO]`date +%F/%H:%M:%S` :$LOCALIP start as slave successful." >> $LOGFILEelseecho "[ERROR]`date +%F/%H:%M:%S` :$LOCALIP start as slave error." >> $LOGFILE
            fi
    else#start as master
            $REDISPATH/bin/redis-server $REDISPATH/conf/redis_master.confif [ "$?" == "0" ];then
                    echo "[INFO]`date +%F/%H:%M:%S` :$LOCALIP start as master successful." >> $LOGFILEelseecho "[ERROR]`date +%F/%H:%M:%S` :$LOCALIP start as master error." >> $LOGFILE
            fi
    fi

![复制代码][copycode.gif]

\#创建redis关闭脚本

vi /usr/local/redis-2.8.3/redis-stop.sh

\#\#\#\#以下为master上的配置，slave上的配置相同。

#!/bin/bash
    REDISPATH=/usr/local/redis-2.8.3
    LOGFILE=$REDISPATH/logs/redis-state.log
    kill -9 `ps -ef|grep '/bin/redis-server'|grep -v grep|awk  '{print $2}'`
    if [ "$?" == "0" ];then
            echo "[INFO]`date +%F/%H:%M:%S` :redis shutdown completed!" >> $LOGFILE
    elseecho "[ERROR]`date +%F/%H:%M:%S` :redis is not started." >> $LOGFILE
    fi

#创建redis配置文件

cp -a -R -p redis.conf /usr/local/redis-2.8.3/conf/redis\_master.conf

cp -a -R -p redis.conf /usr/local/redis-2.8.3/conf/redis\_slave.conf

\#修改redis\_master.conf对应配置项：

![复制代码][copycode.gif]

####192.168.1.218主服务器redis_master.conf对应配置项######
    #daemonize no
    daemonize yes
    #bind 127.0.0.1
    bind 192.168.1.218
    logfile "/usr/local/redis-2.8.3/logs/redis.log"
    #其他配置依据实际生产环境修改
    ########################################################
    ####192.168.1.219从服务器redis_master.conf对应配置项######
    #daemonize no
    daemonize yes
    #bind 127.0.0.1
    bind 192.168.1.219
    logfile "/usr/local/redis-2.8.3/logs/redis.log"
    #其他配置依据实际生产环境修改
    ########################################################

![复制代码][copycode.gif]

\#修改redis\_slave.conf对应配置项：

![复制代码][copycode.gif]

####192.168.1.218主服务器redis_slave.conf对应配置项#######
    #daemonize no
    daemonize yes
    #bind 127.0.0.1
    bind 192.168.1.218
    logfile "/usr/local/redis-2.8.3/logs/redis.log"
    # slaveof <masterip> <masterport>
    slaveof 192.168.1.219 6379
    #其他配置依据实际生产环境修改
    ########################################################
    ####192.168.1.219从服务器redis_slave.conf对应配置项#######
    #daemonize no
    daemonize yes
    #bind 127.0.0.1
    bind 192.168.1.219
    logfile "/usr/local/redis-2.8.3/logs/redis.log"
    # slaveof <masterip> <masterport>
    slaveof 192.168.1.218 6379
    #其他配置依据实际生产环境修改
    ########################################################

![复制代码][copycode.gif]

\#修改redis的属主和权限

![复制代码][copycode.gif]

chmod –R 750 /usr/local/redis-2.8.3/

chown –R redisadmin:develop /usr/local/redis-2.8.3/

\----安装配置keepalived  
  
1.下载keepalived源码Release 1.2.9

注意：最新版为1.2.10测试过程中出错.  
wget [http://www.keepalived.org/software/keepalived-1.2.9.tar.gz][http_www.keepalived.org_software_keepalived-1.2.9.tar.gz]  
2.安装keepalived

需要安装以下依赖包： make gcc libpopt-dev libnl-dev libcurl4-openssl-dev popt openssl  
cd

tar –zxvf keepalived-1.2.9.tar.gz

cd keepalived-1.2.9

./configure --prefix=/usr/local/keepalived

make && make install

3.配置keepalived

\#在Master和Slave上创建如下配置文件（可根据实际情况调整）：

mv /usr/local/keepalived/etc/keepalived/keepalived.conf /usr/local/keepalived/etc/keepalived/keepalived.conf-bak  
vim /usr/local/keepalived/etc/keepalived/keepalived.conf

![复制代码][copycode.gif]

vrrp_script chk_redis {
            script "/usr/local/keepalived/etc/keepalived/scripts/chk_redis.sh"   ###监控脚本
            interval 2                                        ###监控时间
    }
    vrrp_instance VI_1 {
            state MASTER                            ###设置为MASTER
            interface eth3                          ###监控网卡,依据实际情况来定  
            virtual_router_id 51
            priority 101                            ###权重值
            advert_int 1 
            authentication {
                         auth_type PASS             ###加密
                         auth_pass redis            ###密码
            }
            track_script {
                    chk_redis                       ###执行上面定义的chk_redis
            }
            virtual_ipaddress {
                 192.168.1.220                         ###VIP
            }
    }

![复制代码][copycode.gif]

在Master和Slave上创建监控Redis的脚本  
mkdir /usr/local/keepalived/etc/keepalived/scripts  
vi /usr/local/keepalived/etc/keepalived/scripts/chk\_redis.sh

\#\#\#\#以下为master上的配置，slave上的配置只需要修改对应的LOCALIP和REMOTEIP即可。

#!/bin/bash
    REDISPATH=/usr/local/redis-2.8.3
    REDISCLI=$REDISPATH/bin/redis-cli
    LOGFILE=$REDISPATH/logs/redis-state.log
    LOCALIP=192.168.1.218
    REMOTEIP=192.168.1.219
    VIP=192.168.1.220
    
    VIPALIVE=`ip a | grep "$VIP"`
    if [ "$VIPALIVE" == "" ]; then
    	echo "[info]:"`date`" keepalived server is pengding or stop" >> $LOGFILE
    else
    	echo "bbb" >> $LOGFILE
    	#check local service is running
    	if [ "`$REDISCLI –h $LOCALIP –p 6379 PING`" == "PONG" ]; then
    		# check local redis server role.
    		REDISROLE=`$REDISCLI info | grep "role"`
    		if grep "role:slave" <<< $REDISROLE ; then
    			#change local redis server as master 
    			echo "[info1]:"`date`" Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
    			$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
    
    			#change remoting redis server as slave
    			REMOTEREDISROLE=`$REDISCLI -h $REMOTEIP info | grep "role"`
    			if grep "role:master" <<< $REMOTEREDISROLE ; then
    				echo "[info2]:"`date`" Run remote server SLAVEOF cmd ..." >> $LOGFILE
    				$REDISCLI -h $REMOTEIP SLAVEOF $LOCALIP 6379 >> $LOGFILE  2>&1
    			fi
    		else
    			REMOTEREDISROLE=`$REDISCLI -h $REMOTEIP info | grep "role"`
    			if grep "role:master" <<< $REMOTEREDISROLE ; then
    				echo "[info3]:"`date`" Run remote server SLAVEOF cmd ..." >> $LOGFILE
    				$REDISCLI -h $REMOTEIP SLAVEOF $LOCALIP 6379 >> $LOGFILE  2>&1
    			fi
    		fi	
    	else
    		echo "[warn]:"`date`"  redis server($LOCALIP) is not health..." >> $LOGFILE
    		sleep 1
    		if [ "`$REDISCLI –h $LOCALIP –p 6379 PING`" != "PONG" ]; then
    			echo "[error]:"`date`"  redis server($LOCALIP) will be stop..." >> $LOGFILE
    			service keepalived stop
    		fi
    	fi
    fi

![复制代码][copycode.gif]

**重要：**将相应的配置文件放到相应的地方.

\#首先在2台服务器上设置keepalived的启动文件：

cp -a -R -p /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/keepalived

chmod 750 /etc/rc.d/init.d/keepalived

chown root /etc/rc.d/init.d/keepalived  
\#然后在2台服务器上创建配置文件的链接：

mkdir /etc/keepalived/

ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

![复制代码][copycode.gif]

**系统测试**

注意:一定要先启动redis，再启动keealived，否则redis\_check.sh会将keepalived自动关闭。

脚本创建完成以后，我们开始按照如下流程进行测试：  
1.启动Master上的Redis  
/usr/local/redis-2.8.3/redis-start.sh

\#关闭时，直接杀死进程或执行以下脚本

/usr/local/redis-2.8.3/redis-stop.sh

2.启动Slave上的Redis  
/usr/local/redis-2.8.3/redis-start.sh

\#关闭时，直接杀死进程或执行以下脚本

\#/usr/local/redis-2.8.3/redis-stop.sh

3.启动Master上的Keepalived  
/etc/rc.d/init.d/keepalived start

\#关闭方法

\#/etc/rc.d/init.d/keepalived stop

4.启动Slave上的Keepalived  
/etc/rc.d/init.d/keepalived start

\#关闭方法

\#/etc/rc.d/init.d/keepalived stop

\-----------------------------------------------------------------------------

**Centos开机自启动redis**

*  修改redis.conf，打开后台运行选项：

# By default Redis does not run as a daemon. Use 'yes' if you need it. # Note that Redis will write a pid file in /var/run/redis.pid when daemonized. daemonize yes

*  编写脚本，vim /etc/init.d/redis:

# chkconfig: 2345 10 90
    # description: Start and Stop redis
    
    PATH=/usr/local/bin:/sbin:/usr/bin:/bin
    
    REDISPORT=6379 #实际环境而定 EXEC=/usr/local/redis/src/redis-server #实际环境而定 REDIS_CLI=/usr/local/redis/src/redis-cli #实际环境而定 PIDFILE=/var/run/redis.pid CONF="/usr/local/redis/redis.conf" #实际环境而定 case "$1" in start) if [ -f $PIDFILE ] then echo "$PIDFILE exists, process is already running or crashed." else echo "Starting Redis server..." $EXEC $CONF fi if [ "$?"="0" ] then echo "Redis is running..." fi ;; stop) if [ ! -f $PIDFILE ] then echo "$PIDFILE exists, process is not running." else PID=$(cat $PIDFILE) echo "Stopping..." $REDIS_CLI -p $REDISPORT SHUTDOWN while [ -x $PIDFILE ] do echo "Waiting for Redis to shutdown..." sleep 1 done echo "Redis stopped" fi ;; restart|force-reload) ${0} stop ${0} start ;; *) echo "Usage: /etc/init.d/redis {start|stop|restart|force-reload}" >&2 exit 1 esac

*  执行权限：

chmod +x /etc/init.d/redis

*  开机自启动：

# 尝试启动或停止redis
    service redis start
    service redis stop
    
    # 开启服务自启动
    chkconfig redis on

**参考资料：**

**[http://luyx30.blog.51cto.com/1029851/1350832][http_luyx30.blog.51cto.com_1029851_1350832]**

http://www.cnblogs.com/davidwang456/p/3525090.html

[http://patrick-tang.blogspot.com/2012/06/redis-keepalived-failover-system.html][http_patrick-tang.blogspot.com_2012_06_redis-keepalived-failover-system.html]

[http://deidara.blog.51cto.com/400447/302402][http_deidara.blog.51cto.com_400447_302402]

[http://my.oschina.net/guol/blog/182491][http_my.oschina.net_guol_blog_182491]

[http://shiguanghui.iteye.com/blog/2001499][http_shiguanghui.iteye.com_blog_2001499]

http://my.oschina.net/indestiny/blog/197272

转载于:https://www.cnblogs.com/liubei/p/RedisK.html

[http_www.redisdoc.com_en_latest_topic_cluster-spec.html]: http://www.redisdoc.com/en/latest/topic/cluster-spec.html
[http_download.redis.io_releases_redis-2.8.3.tar.gz]: http://download.redis.io/releases/redis-2.8.3.tar.gz
[copycode.gif]: /images/20211220/a84e7bd911f4490bbb686863c0c727cf.png
[http_www.keepalived.org_software_keepalived-1.2.9.tar.gz]: http://www.keepalived.org/software/keepalived-1.2.9.tar.gz
[http_luyx30.blog.51cto.com_1029851_1350832]: http://luyx30.blog.51cto.com/1029851/1350832
[http_patrick-tang.blogspot.com_2012_06_redis-keepalived-failover-system.html]: http://patrick-tang.blogspot.com/2012/06/redis-keepalived-failover-system.html
[http_deidara.blog.51cto.com_400447_302402]: http://deidara.blog.51cto.com/400447/302402
[http_my.oschina.net_guol_blog_182491]: http://my.oschina.net/guol/blog/182491
[http_shiguanghui.iteye.com_blog_2001499]: http://shiguanghui.iteye.com/blog/2001499