redis 集群简介

深碍√TFBOYSˉ_ 2022-12-21 10:52 128阅读 0赞

备注:测试版本 redis 4.0.9

### 文章目录 ###

*  一.Cluster模式介绍
 *  二.redis三主三从集群搭建
 *   *  2.1 配置文件
     *  2.2 启动redis服务
     *  2.3 安装ruby
     *  2.4 初始化集群
 *  参考

# 一.Cluster模式介绍 #

sentinel模式基本可以满足一般生产的需求，具备高可用性。但是当数据量过大到一台服务器存放不下的情况时，主从模式或sentinel模式就不能满足需求了，这个时候需要对存储的数据进行分片，将数据存储到多个Redis实例中。cluster模式的出现就是为了解决单机Redis容量有限的问题，将Redis的数据根据一定的规则分配到多台机器。

cluster可以说是sentinel和主从模式的结合体，通过cluster可以实现主从和master重选功能，所以如果配置两个副本三个分片的话，就需要六个Redis实例。因为Redis的数据是根据一定规则分配到cluster的不同机器的，当数据量过大时，可以新增机器进行扩容。

使用集群，只需要将redis配置文件中的cluster-enable配置打开即可。每个集群中至少需要三个主数据库才能正常运行，新增节点非常方便。

cluster集群特点：  
1.多个redis节点网络互联，数据共享  
2.所有的节点都是一主一从（也可以是一主多从），其中从不提供服务，仅作为备用  
3.不支持同时处理多个key（如MSET/MGET），因为redis需要把key均匀分布在各个节点上，  
4.并发量很高的情况下同时创建key-value会降低性能并导致不可预测的行为  
5.支持在线增加、删除节点  
6.客户端可以连接任何一个主节点进行读写

# 二.redis三主三从集群搭建 #

<table> 
 <thead> 
  <tr> 
   <th>服务器类别</th> 
   <th>IP</th> 
   <th>端口</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>主</td> 
   <td>10.31.1.123</td> 
   <td>7000</td> 
  </tr> 
  <tr> 
   <td>从</td> 
   <td>10.31.1.123</td> 
   <td>7001</td> 
  </tr> 
  <tr> 
   <td>主</td> 
   <td>10.31.1.124</td> 
   <td>7002</td> 
  </tr> 
  <tr> 
   <td>从</td> 
   <td>10.31.1.124</td> 
   <td>7003</td> 
  </tr> 
  <tr> 
   <td>主</td> 
   <td>10.31.1.125</td> 
   <td>7004</td> 
  </tr> 
  <tr> 
   <td>从</td> 
   <td>10.31.1.125</td> 
   <td>7005</td> 
  </tr> 
 </tbody> 
</table>

服务器及端口配置如上，此时 123、124、125三台服务器已经安装redis

## 2.1 配置文件 ##

创建目录

-- 123
    mkdir -p /usr/local/redis/7000/conf
    mkdir -p /usr/local/redis/7000/log
    mkdir -p /usr/local/redis/7001/conf
    mkdir -p /usr/local/redis/7001/log
    
    -- 124 
    mkdir -p /usr/local/redis/7002/conf
    mkdir -p /usr/local/redis/7002/log
    mkdir -p /usr/local/redis/7003/conf
    mkdir -p /usr/local/redis/7003/log
    
    -- 124
    mkdir -p /usr/local/redis/7004/conf
    mkdir -p /usr/local/redis/7004/log
    mkdir -p /usr/local/redis/7005/conf
    mkdir -p /usr/local/redis/7005/log

10.31.1.123上主库端口7000的配置文件：  
protected-mode no 、\#\# requirepass 123456 这两个我测试过程中，会影响到后面的集群初始化，如果需要提高安全等级，可以在配置完集群之后，再开启

## SHOW REDIS LOGO ##
    always-show-logo yes
    
    ## LOCAL CONNECT
    bind 10.31.1.123
    protected-mode no
    
    ## GENERAL ##
    daemonize yes
    supervised no
    pidfile /usr/local/redis/7000/redis.pid
    port 7000
    timeout 0
    tcp-keepalive 300
    loglevel notice
    logfile "/usr/local/redis/7000/log/redis.log"
    databases 16
    
    ## SNAPSHOTTING ##
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /usr/local/redis/7000/
    
    ## REPLICATION ##
    #slaveof MASTER_IP 6380
    masterauth 123456
    slave-serve-stale-data yes
    slave-read-only yes
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    slave-priority 100
    
    ## SECURITY ##
    ## requirepass 123456
    
    ## LIMITS ##
    #maxclients 10000
    #maxmemory <bytes>
    
    ## APPEND ONLY MODE ##
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    aof-use-rdb-preamble no
    
    ## LUA SCRIPTING ##
    lua-time-limit 5000
    
    ## SLOW LOG ##
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    
    ## EVENT NOTIFICATION ##
    notify-keyspace-events ""
    
    ## ADVANCED CONFIG ##
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes
    
    ## EXPIRED DATA AND ELIMINATION STRATEGY ##
    slave-lazy-flush no
    lazyfree-lazy-eviction no
    lazyfree-lazy-expire no
    lazyfree-lazy-server-del no
    
    ## DELAY MONITORING ##
    latency-monitor-threshold 0
    
    ## cluster ##
    cluster-enabled yes
    cluster-config-file /usr/local/redis/7000/conf/nodes.conf
    cluster-node-timeout 5000

10.31.1.123上从库端口7001的配置文件：  
区别两点:  
1.端口号从7000改为7001

## SHOW REDIS LOGO ##
    always-show-logo yes
    
    ## LOCAL CONNECT
    bind 10.31.1.123
    protected-mode no
    
    ## GENERAL ##
    daemonize yes
    supervised no
    pidfile /usr/local/redis/7001/redis.pid
    port 7001
    timeout 0
    tcp-keepalive 300
    loglevel notice
    logfile "/usr/local/redis/7001/log/redis.log"
    databases 16
    
    ## SNAPSHOTTING ##
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /usr/local/redis/7001/
    
    ## REPLICATION ##
    #slaveof MASTER_IP 6380
    masterauth 123456
    slave-serve-stale-data yes
    slave-read-only yes
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    slave-priority 100
    
    ## SECURITY ##
    ## requirepass 123456
    
    ## LIMITS ##
    #maxclients 10000
    #maxmemory <bytes>
    
    ## APPEND ONLY MODE ##
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    aof-use-rdb-preamble no
    
    ## LUA SCRIPTING ##
    lua-time-limit 5000
    
    ## SLOW LOG ##
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    
    ## EVENT NOTIFICATION ##
    notify-keyspace-events ""
    
    ## ADVANCED CONFIG ##
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes
    
    ## EXPIRED DATA AND ELIMINATION STRATEGY ##
    slave-lazy-flush no
    lazyfree-lazy-eviction no
    lazyfree-lazy-expire no
    lazyfree-lazy-server-del no
    
    ## DELAY MONITORING ##
    latency-monitor-threshold 0
    
    ## cluster ##
    cluster-enabled yes
    cluster-config-file /usr/local/redis/7001/conf/nodes.conf
    cluster-node-timeout 5000

10.31.1.124、10.31.1.125两台上的配置文件同10.31.1.123，只需要改对应的端口号即可

## 2.2 启动redis服务 ##

-- 123
    nohup redis-server /usr/local/redis/7000/conf/7000.conf &
    nohup redis-server /usr/local/redis/7001/conf/7001.conf &
    -- 124
    nohup redis-server /usr/local/redis/7002/conf/7002.conf &
    nohup redis-server /usr/local/redis/7003/conf/7003.conf &
    -- 125
    nohup redis-server /usr/local/redis/7004/conf/7004.conf &
    nohup redis-server /usr/local/redis/7005/conf/7005.conf &

查看redis服务是否启动  
如下所示，则表示均已经启动成功了

[root@10-31-1-123 conf]# ps -ef | grep redis
    root     22166     1  0 16:21 ?        00:00:00 redis-server 10.31.1.123:7000 [cluster]
    root     22318     1  0 16:24 ?        00:00:00 redis-server 10.31.1.123:7001 [cluster]
    root     22329 18004  0 16:24 pts/0    00:00:00 grep --color=auto redis
    
    [root@10-31-1-124 conf]# ps -ef | grep redis
    root      9971     1  0 16:22 ?        00:00:00 redis-server 10.31.1.124:7002 [cluster]
    root     10124     1  0 16:25 ?        00:00:00 redis-server 10.31.1.124:7003 [cluster]
    root     10132  8437  0 16:25 pts/0    00:00:00 grep --color=auto redis
    
    [root@10-31-1-125 conf]# ps -ef | grep redis
    root      5725     1  0 16:22 ?        00:00:00 redis-server 10.31.1.125:7004 [cluster]
    root      5740     1  0 16:26 ?        00:00:00 redis-server 10.31.1.125:7005 [cluster]
    root      5745  5651  0 16:26 pts/0    00:00:00 grep --color=auto redis

## 2.3 安装ruby ##

10.31.1.123上执行即可

yum -y install zlib ruby rubygems
    gem install redis

报错:

[root@10-31-1-123 ~]# gem install redis
    Fetching: redis-4.2.2.gem (100%)
    ERROR:  Error installing redis:
            redis requires Ruby version >= 2.3.0.
    [root@10-31-1-123 ~]#

参考如下解决方案:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
    curl -sSL https://get.rvm.io | bash -s stable
    find / -name rvm -print
    source /usr/local/rvm/scripts/rvm
    rvm list known
    rvm install 2.5.1
    rvm use 2.5.1
    rvm use 2.5.1 --default
    rvm remove 2.0.0
    ruby --version
    gem install redis

## 2.4 初始化集群 ##

/usr/local/src/redis-4.0.9/src/redis-trib.rb create --replicas 1 10.31.1.123:7000 10.31.1.123:7001 10.31.1.124:7002 10.31.1.124:7003 10.31.1.125:7004 10.31.1.125:7005

测试记录:

[root@10-31-1-123 local]# /usr/local/src/redis-4.0.9/src/redis-trib.rb create --replicas 1 10.31.1.123:7000 10.31.1.123:7001 10.31.1.124:7002 10.31.1.124:7003 10.31.1.125:7004 10.31.1.125:7005
    >>> Creating cluster
    >>> Performing hash slots allocation on 6 nodes...
    Using 3 masters:
    10.31.1.123:7000
    10.31.1.124:7002
    10.31.1.125:7004
    Adding replica 10.31.1.124:7003 to 10.31.1.123:7000
    Adding replica 10.31.1.125:7005 to 10.31.1.124:7002
    Adding replica 10.31.1.123:7001 to 10.31.1.125:7004
    M: 247b6072aa240ba66703403466c159158dd880a4 10.31.1.123:7000
       slots:0-5460 (5461 slots) master
    S: 47257b5838640c6736cf3b5eb6eaf87d0f11fece 10.31.1.123:7001
       replicates df1747df706f01a6a671f390af9f56659874b959
    M: c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 10.31.1.124:7002
       slots:5461-10922 (5462 slots) master
    S: e7f72bdf6efb013017c3942855507bffac4ed979 10.31.1.124:7003
       replicates 247b6072aa240ba66703403466c159158dd880a4
    M: df1747df706f01a6a671f390af9f56659874b959 10.31.1.125:7004
       slots:10923-16383 (5461 slots) master
    S: 14e1ff83ef179a2a5661b11de4e0412993eb466b 10.31.1.125:7005
       replicates c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea
    Can I set the above configuration? (type 'yes' to accept): yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join...
    >>> Performing Cluster Check (using node 10.31.1.123:7000)
    M: 247b6072aa240ba66703403466c159158dd880a4 10.31.1.123:7000
       slots:0-5460 (5461 slots) master
       1 additional replica(s)
    S: e7f72bdf6efb013017c3942855507bffac4ed979 10.31.1.124:7003
       slots: (0 slots) slave
       replicates 247b6072aa240ba66703403466c159158dd880a4
    S: 14e1ff83ef179a2a5661b11de4e0412993eb466b 10.31.1.125:7005
       slots: (0 slots) slave
       replicates c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea
    M: df1747df706f01a6a671f390af9f56659874b959 10.31.1.125:7004
       slots:10923-16383 (5461 slots) master
       1 additional replica(s)
    M: c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 10.31.1.124:7002
       slots:5461-10922 (5462 slots) master
       1 additional replica(s)
    S: 47257b5838640c6736cf3b5eb6eaf87d0f11fece 10.31.1.123:7001
       slots: (0 slots) slave
       replicates df1747df706f01a6a671f390af9f56659874b959
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    [root@10-31-1-123 local]#

下面根据输出信息详细介绍集群创建的过程。

首先redis-trib.rb会以客户端的形式尝试连接所有的节点，并发送ping命令以确定节点能够正常服务。如果有任何节点无法连接，则创建失败。同时发送info命令获取每个节点的运行ID以及是否开启了集群功能（即cluster\_enabled为1）。

准备就绪后集群会向每个节点发送cluster meet命令，格式为cluster meet ip port，这个命令用来告诉当前节点指定ip和port上在运行的节点也是集群的一部分，从而使得6个节点最终可以归入一个集群。

然后redis-trib.rb会分配主从库节点，分配的原则是尽量保证每个主库运行在不同的IP地址上，同时每个从库和主库均不运行在同一IP地址上，以保证系统的容灾能力。分配结果如下：

Using 3 masters:
    10.31.1.123:7000
    10.31.1.124:7002
    10.31.1.125:7004
    Adding replica 10.31.1.124:7003 to 10.31.1.123:7000
    Adding replica 10.31.1.125:7005 to 10.31.1.124:7002
    Adding replica 10.31.1.123:7001 to 10.31.1.125:7004

主从分配完成后，会为每个主库分配插槽，分配插槽的过程其实就是分配哪些键归哪些节点负责。之后对每个要成为从库的节点发送cluster replicate 主库运行ID，来将当前节点转换成从库并复制指定运行ID的主库节点。

此时整个集群的过程即创建完成，使用redis命令行客户端连接任意一个节点执行cluster nodes可以获得集群中的所有节点信息，如在10.31.1.123执行：

[root@10-31-1-123 ~]# redis-cli -h 10.31.1.123 -p 7000 cluster nodes
    e7f72bdf6efb013017c3942855507bffac4ed979 10.31.1.124:7003@17003 slave 247b6072aa240ba66703403466c159158dd880a4 0 1604483299146 4 connected
    14e1ff83ef179a2a5661b11de4e0412993eb466b 10.31.1.125:7005@17005 slave c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 0 1604483300149 6 connected
    df1747df706f01a6a671f390af9f56659874b959 10.31.1.125:7004@17004 master - 0 1604483300550 5 connected 10923-16383
    c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 10.31.1.124:7002@17002 master - 0 1604483299000 3 connected 5461-10922
    247b6072aa240ba66703403466c159158dd880a4 10.31.1.123:7000@17000 myself,master - 0 1604483300000 1 connected 0-5460
    47257b5838640c6736cf3b5eb6eaf87d0f11fece 10.31.1.123:7001@17001 slave df1747df706f01a6a671f390af9f56659874b959 0 1604483301150 5 connected
    [root@10-31-1-123 ~]#

# 参考 #

1.https://blog.csdn.net/miss1181248983/article/details/90056960  
2.https://blog.csdn.net/wzy0623/article/details/82706614