多台宿主机使用docker部署redis集群

朴灿烈づ我的快乐病毒、 2022-03-11 02:00 634阅读 0赞

# 前言 #

在Centos7.5上多台宿主机使用docker部署redis集群的过程，redis集群一般需要6台redis服务器,使用docker可以节省服务器资源.第一次发博客，有什么不对的地方还请大家指出。

# 集群架构 #

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>IP</th> 
   <th>redis集群ip(并非为redis容器的ip)</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>test（部署用）</td> 
   <td>192.168.0.35</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>redis1</td> 
   <td>192.168.0.71</td> 
   <td>192.168.0.71:6379    192.168.0.71:6380</td> 
  </tr> 
  <tr> 
   <td>redis2</td> 
   <td>192.168.0.72</td> 
   <td>192.168.0.72:6379   192.168.0.72:6380</td> 
  </tr> 
  <tr> 
   <td>redis3</td> 
   <td>192.168.0.73</td> 
   <td>192.168.0.73:6379   192.168.0.73:6380</td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <thead> 
  <tr> 
   <th>主</th> 
   <th>从</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>redis1: 6379</td> 
   <td>redis2 : 6380</td> 
  </tr> 
  <tr> 
   <td>redis2 : 6379</td> 
   <td>redis3 : 6380</td> 
  </tr> 
  <tr> 
   <td>redis3 : 6379</td> 
   <td>redis1 : 6380</td> 
  </tr> 
 </tbody> 
</table>

# docker环境部署 #

安装docker

[root@test ~]# yum -y install docker-engine

启动docker

[root@test ~]# systemctl start docker

我的docker版本

[root@lianxi-0001 ~]# docker version 
    Client:
     Version:      1.12.1
     API version:  1.24
     Go version:   go1.6.3
     Git commit:   23cf638
     Built:        
     OS/Arch:      linux/amd64

下载官方centos镜像

[root@test ~]# docker pull centos

制作redis镜像，wokdir下需要redis-4.0.8源码包解压后的文件夹，可用的repo文件和Dockerfile

[root@test ~]# mkdir workdir
    [root@test ~]# ls workdir/
    CentOS-Base.repo  Dockerfile  local.repo  redis-4.0.8
    [root@test ~]# vim workdir/Dockerfile
    FROM centos
    RUN rm -rf /etc/yum.repos.d/*
    ADD *.repo /etc/yum.repos.d/
    ADD redis-4.0.8 /root/redis-4.0.8
    WORKDIR /root/redis-4.0.8
    RUN yum -y install make  gcc
    RUN make && make install
    RUN mkdir -p /var/log/redis /var/lib/redis/ /etc/redis/
    CMD ["redis-server", "/etc/redis/redis.conf"]
    [root@test ~]# docker build -t redis workdir/

制作私有镜像仓库并上传redis镜像

[root@test ~]# vim /etc/docker/daemon.json
    { 
       "insecure-registries": ["192.168.0.35:5000"]
    }
    [root@test ~]# systemctl restart docker
    [root@test ~]# docker pull registry
    [root@test ~]# docker run -itd -p 5000:5000 registry
    [root@test ~]# docker tag redis 192.168.0.35:5000/redis:latest
    [root@test ~]# docker push 192.168.0.35:5000/redis:latest

部署redis节点的docker 使用ansible部署

[root@test ~]# vim /etc/hosts
    192.168.0.71 redis1
    192.168.0.72 redis2
    192.168.0.73 redis3
    [root@lianxi-0001 ~]# yum -y install ansible
    [root@lianxi-0001 ~]# vim /etc/ansible/ansible.cfg
    [defaults]
    inventory      = /etc/ansible/hosts
    host_key_checking = False
    [root@lianxi-0001 ~]# vim /etc/ansible/hosts
    [redis]
    redis[1:3]
    [root@test ~]# ansible redis --list-hosts #提前先部署秘钥
      hosts (3):
        redis1
        redis2
        redis3
    [root@test ~]# ansible redis -m shell -a 'yum -y install docker-engine;systemctl start docker'
    [root@test ~]# ansible redis -m copy -a 'src=/etc/docker/daemon.json dest=/etc/docker/'
    [root@test ~]# ansible redis -m shell -a 'systemctl restart docker;
    docker pull 192.168.0.35:5000/redis:latest;
    docker tag 192.168.0.35:5000/redis:latest redis' [root@test ~]# ansible redis -m shell -a 'docker images'    #检查每个节点是否有redis镜像

# redis部署 #

创建redis容器挂载目录

[root@test ~]# ansible redis -m shell -a 'mkdir -p /var/redis/63{79,80}/{conf,logs,data}'

创建redis模板配置文件

[root@test ~]# vim redis.conf.tem
    protected-mode no
    bind 0.0.0.0                     
    port ${PORT}                         
    daemonize no                         #关闭以守护进程运行
    dir /var/lib/redis
    logfile /var/log/redis/redis.log
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    cluster-announce-ip ${IP}           #指定redis集群ip,即为宿主机ip
    cluster-announce-port ${PORT}
    cluster-announce-bus-port 1${PORT}

使用envsubst命令修改模板文件后分发到各个节点(也可以使用ansible-playbook完成)

[root@test ~]# for ip in 192.168.0.7{1..3}
    do 
    for port in 63{ 79,80} 
    do 
    scp redis.conf.tem $ip: 
    ssh $ip "export IP=$ip PORT=$port;envsubst < redis.conf.tem > /var/redis/${port}/conf/redis.conf"
    done
    done

在节点上检查结构

[root@redis1 ~]# tree /var/redis/
    /var/redis/
    ├── 6379
    │   ├── conf
    │   │   └── redis.conf
    │   ├── data
    │   └── logs
    │      
    └── 6380
        ├── conf
        │   └── redis.conf
        ├── data
        └── logs

检查redis.conf文件,如查看redis1上的配置文件

[root@test ~]# ansible redis1 -m shell -a 'cat /var/redis/6379/conf/redis.conf'
    protected-mode no
    bind 192.168.0.0          
    port 6379
    daemonize no
    dir /var/lib/redis
    logfile /var/log/redis/redis.log
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    cluster-announce-ip 192.168.0.71
    cluster-announce-port 6379
    cluster-announce-bus-port 16379

创建redis实例  
\-p 端口映射,需要把每个实例的63(79|80) 和163(79|80)端口都映射出来  
\-v 指定目录映射  
–name 给容器命名  
\-d 容器放入后台

[root@test ~]# for i in 63{79,80}
    do  
    ansible redis -m shell -a "docker run -itd -p $i:$i -p 1$i:1$i -v /var/redis/$i/logs:/var/log/redis -v /var/redis/$i/data:/var/lib/redis/ -v /var/redis/$i/conf/redis.conf:/etc/redis/redis.conf --name redis$i redis" 
    done

验证redis实例 随便进入一台redis服务器,进入redis容器内部,执行完命令后看见6个PONG则验证成功

[root@test ~]# ssh redis1
    [root@redis1 ~]# docker exec -it redis6379 bash
    [root@857e857386ee redis-4.0.8]# for ip in 192.168.0.7{1..3}
    do 
    for port in 63{ 79,80}
    do
    redis-cli -h $ip -p $port <<eof ping eof
    done
    done
    
    PONG
    PONG
    PONG
    PONG
    PONG
    PONG

# 搭建redis集群 #

安装ruby环境(也可使用docker里的ruby镜像)  
创建3个主redis节点分别为每个redis服务器上的6379端口

[root@test ~]# yum -y install ruby rubygems ruby-devel
    [root@test ~]# gem install redis-3.2.1.gem
    [root@test ~]# cp workdir/redis-4.0.8/src/redis-trib.rb /sbin/
    [root@test ~]# redis-trib.rb create 192.168.0.7{1..3}:6379 
    >>> Creating cluster
    >>> Performing hash slots allocation on 3 nodes...
    Using 3 masters:
    192.168.0.71:6379
    192.168.0.72:6379
    192.168.0.73:6379
    M: f2db485427801702c2c6b0fab6c161f7dcbd2c57 192.168.0.71:6379
       slots:0-5460 (5461 slots) master
    M: 53cde4277d80fb07cf9138787ba01f4f1128470f 192.168.0.72:6379
       slots:5461-10922 (5462 slots) master
    M: d2760a19b13d774e8325e529201e568814c8f5ed 192.168.0.73:6379
       slots:10923-16383 (5461 slots) master
    Can I set the above configuration? (type 'yes' to accept): yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join..
    >>> Performing Cluster Check (using node 192.168.0.71:6379)
    M: f2db485427801702c2c6b0fab6c161f7dcbd2c57 192.168.0.71:6379
       slots:0-5460 (5461 slots) master
       0 additional replica(s)
    M: 53cde4277d80fb07cf9138787ba01f4f1128470f 192.168.0.72:6379
       slots:5461-10922 (5462 slots) master
       0 additional replica(s)
    M: d2760a19b13d774e8325e529201e568814c8f5ed 192.168.0.73:6379
       slots:10923-16383 (5461 slots) master
       0 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.

为每个主节点添加在不同redis服务器的从节点  
格式为

[root@test ~]# redis-trib.rb add-node --slave --master-id f2db485427801702c2c6b0fab6c161f7dcbd2c57 192.168.0.72:6380 192.168.0.71:6379
    [root@test ~]# redis-trib.rb add-node --slave --master-id 53cde4277d80fb07cf9138787ba01f4f1128470f 192.168.0.73:6380 192.168.0.71:6379
    [root@test ~]# redis-trib.rb add-node --slave --master-id d2760a19b13d774e8325e529201e568814c8f5ed 192.168.0.71:6380 192.168.0.71:6379

检查集群状态

[root@test ~]# redis-trib.rb check 192.168.0.71:6379
    >>> Performing Cluster Check (using node 192.168.0.71:6379)
    M: f2db485427801702c2c6b0fab6c161f7dcbd2c57 192.168.0.71:6379
       slots:0-5460 (5461 slots) master
       1 additional replica(s)
    S: 6e8dc744a314a142e1f3494e304e94b3bd112c13 192.168.0.73:6380
       slots: (0 slots) slave
       replicates 53cde4277d80fb07cf9138787ba01f4f1128470f
    S: 2c0abf5fc5e9bc11ae3a4919cb2e9d0377f6d43b 192.168.0.71:6380
       slots: (0 slots) slave
       replicates d2760a19b13d774e8325e529201e568814c8f5ed
    M: 53cde4277d80fb07cf9138787ba01f4f1128470f 192.168.0.72:6379
       slots:5461-10922 (5462 slots) master
       1 additional replica(s)
    S: bd20aea7765ebb6283476e68c5a2ecd65ddbc47d 192.168.0.72:6380
       slots: (0 slots) slave
       replicates f2db485427801702c2c6b0fab6c161f7dcbd2c57
    M: d2760a19b13d774e8325e529201e568814c8f5ed 192.168.0.73:6379
       slots:10923-16383 (5461 slots) master
       1 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    
    [root@test ~]# redis-cli -c -h 192.168.0.71 -p 6379 <<eof
    > cluster info
    > eof
    cluster_state:ok
    cluster_slots_assigned:16384
    cluster_slots_ok:16384
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:6
    cluster_size:3
    cluster_current_epoch:3
    cluster_my_epoch:1
    cluster_stats_messages_ping_sent:1158
    cluster_stats_messages_pong_sent:1156
    cluster_stats_messages_sent:2314
    cluster_stats_messages_ping_received:1151
    cluster_stats_messages_pong_received:1158
    cluster_stats_messages_meet_received:5
    cluster_stats_messages_received:2314

集群搭建成功

个人认为最重要的为指定redis集群ip为宿主机ip,不然集群搭建时会报错