（Redis系列三）centos7实现redis集群数据迁移

谁践踏了优雅 2022-05-08 10:42 303阅读 0赞

本次迁移是把一个三主0从的数据迁移至三主三从  
伪集群：172-22-14-88，三个节点6379\\6380\\6381  
真集群：  
172-22-14-105 两个节点 6379\\6380  
172-22-14-106 两个节点 6379\\6380  
172-22-14-107 两个节点 6379\\6380

先看下三主0从的结构，都是集群  
主机为172-22-14-88  
\[root@host-172-22-14-88 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes  
4b53748f587e80f7f9209062eba94a2945b0799d 172.22.14.88:6379 myself,master - 0 0 1 connected 0-5460  
157fe9a2f93fd27ebefc661cbac2e3de5713c16a 172.22.14.88:6380 master - 0 1539226455397 2 connected 5461-10922  
806e842e6adf53ee82e155d0bd9805bdf158b68c 172.22.14.88:6381 master - 0 1539226454394 3 connected 10923-16383

\[root@host-172-22-14-88 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes|grep master  
4b53748f587e80f7f9209062eba94a2945b0799d 172.22.14.88:6379 myself,master - 0 0 1 connected 0-5460  
157fe9a2f93fd27ebefc661cbac2e3de5713c16a 172.22.14.88:6380 master - 0 1539226482573 2 connected 5461-10922  
806e842e6adf53ee82e155d0bd9805bdf158b68c 172.22.14.88:6381 master - 0 1539226481570 3 connected 10923-16383

\[root@host-172-22-14-88 cluster-conf\]\# ls  
6379 6380 6381

本次迁移尝试直接复制aof和rdb数据文件  
![在这里插入图片描述][70]

再查看真集群的结构  
\[root@host-172-22-14-105 ~\]\# cd /data/soft/redis-3.2.12/cluster-conf/  
\[root@host-172-22-14-105 cluster-conf\]\# ls  
6379 6380  
\[root@host-172-22-14-105 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -c -h 172.22.14.105 -p 6379  
172.22.14.105:6379> keys \*  
(empty list or set) 发现真集群中没数据  
172.22.14.105:6379> exit  
\[root@host-172-22-14-105 cluster-conf\]\#  
\[root@host-172-22-14-105 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes  
![三主三从][70 1]  
\[root@host-172-22-14-105 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes|grep master  
![在这里插入图片描述][70 2]  
\[root@host-172-22-14-105 cluster-conf\]\#

以下是我的尝试，第一种方法失败，大家没有兴趣的话请直接跳到第二种方法

\*\*

## 第一种方法：思路是直接复制aof和rdb文件进行替换 ##

\*\*

先关掉该服务器的主节点  
\[root@host-172-22-14-105 cluster-conf\]\# cat 6379/redis\_6379.pid | xargs kill  
\[root@host-172-22-14-105 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes  
Could not connect to Redis at 127.0.0.1:6379: Connection refused  
连不上了,查看主从是否切换  
\[root@host-172-22-14-105 cluster-conf\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6380 cluster nodes  
![在这里插入图片描述][70 3]

主从已完成切换，那么把伪集群的6379数据复制过来，不过先备份旧数据  
\[root@host-172-22-14-105 cluster-conf\]\# cd 6379  
\[root@host-172-22-14-105 6379\]\# ls  
appendonly.aof dump.rdb nodes-6379.conf redis.conf redis.log  
\[root@host-172-22-14-105 6379\]\# cp appendonly.aof appendonly.aof.bak  
\[root@host-172-22-14-105 6379\]\# cp dump.rdb dump.rdb.bak  
\[root@host-172-22-14-105 6379\]\# ls  
appendonly.aof appendonly.aof.bak dump.rdb dump.rdb.bak nodes-6379.conf redis.conf redis.log  
\[root@host-172-22-14-105 6379\]\#  
\[root@host-172-22-14-105 6379\]\# ll  
总用量 92  
\-rw-r–r-- 1 root root 0 10月 9 18:35 appendonly.aof  
\-rw-r–r-- 1 root root 0 10月 11 11:16 appendonly.aof.bak  
\-rw-r–r-- 1 root root 77 10月 11 11:10 dump.rdb  
\-rw-r–r-- 1 root root 77 10月 11 11:16 dump.rdb.bak  
\-rw-r–r-- 1 root root 757 10月 9 18:40 nodes-6379.conf  
\-rw-r–r-- 1 root root 46808 10月 9 18:34 redis.conf  
\-rw-r–r-- 1 root root 25230 10月 11 11:10 redis.log

**登录172-22-14-88进行复制**  
\[root@host-172-22-14-88 6379\]\# ll  
总用量 36160  
\-rw-r–r-- 1 root root 35824026 10月 11 11:15 appendonly.aof  
\-rw-r–r-- 1 root root 58273 10月 11 11:12 dump.rdb  
\-rw-r–r-- 1 root root 349 9月 17 16:55 nodes.conf  
\-rw-r–r-- 1 root root 7 9月 21 11:46 redis\_6379.pid  
\-rw-r–r-- 1 root root 48099 9月 29 15:38 redis.conf  
\-rw-r–r-- 1 root root 1074932 10月 11 11:12 redis.log

\[root@host-172-22-14-88 6379\]\# scp appendonly.aof [root@172.22.14.105][root_172.22.14.105]:/data/soft/redis-3.2.12/cluster-conf/6379/  
The authenticity of host ‘172.22.14.105 (172.22.14.105)’ can’t be established.  
ECDSA key fingerprint is 40:62:8e:d2:4b??a6:dd:43:c9:26:7d:1b:bc:d5:af.  
Are you sure you want to continue connecting (yes/no)? yes  
Warning: Permanently added ‘172.22.14.105’ (ECDSA) to the list of known hosts.  
[root@172.22.14.105][root_172.22.14.105]’s password:  
appendonly.aof 100% 34MB 34.2MB/s 00:00  
\[root@host-172-22-14-88 6379\]\# scp dump.rdb [root@172.22.14.105][root_172.22.14.105]:/data/soft/redis-3.2.12/cluster-conf/6379/  
[root@172.22.14.105][root_172.22.14.105]’s password:  
dump.rdb 100% 57KB 56.9KB/s 00:00

**登录172-22-14-105查看，并开启redis的6379节点**  
\[root@host-172-22-14-105 6379\]\# ll  
总用量 35136  
\-rw-r–r-- 1 root root 35824026 10月 11 11:19 appendonly.aof  
\-rw-r–r-- 1 root root 0 10月 11 11:16 appendonly.aof.bak  
\-rw-r–r-- 1 root root 58275 10月 11 11:19 dump.rdb  
\-rw-r–r-- 1 root root 77 10月 11 11:16 dump.rdb.bak  
\-rw-r–r-- 1 root root 757 10月 9 18:40 nodes-6379.conf  
\-rw-r–r-- 1 root root 46808 10月 9 18:34 redis.conf  
\-rw-r–r-- 1 root root 25230 10月 11 11:10 redis.log

开启节点  
\[root@host-172-22-14-105 6379\]\# /data/soft/redis-3.2.12/bin/redis-server /data/soft/redis-3.2.12/cluster-conf/6379/redis.conf &  
\[1\] 19313  
查看集群状态  
\[root@host-172-22-14-105 6379\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes  
![在这里插入图片描述][70 4]  
查看105服务器的redis状态，可以发现6379为slave,6380为master  
\[root@host-172-22-14-105 6379\]\# /data/soft/redis-3.2.12/bin/redis-cli -p 6379 cluster nodes|grep 105  
![在这里插入图片描述][70 5]  
登录6379节点查看数据,发现还是为空。排查原因发现是被集群自动清空异常数据  
\[root@host-172-22-14-105 6379\]\# /data/soft/redis-3.2.12/bin/redis-cli -c -h 172.22.14.105 -p 6379  
172.22.14.105:6379> keys \*  
(empty list or set)

\[root@host-172-22-14-105 6379\]\# ll  
总用量 100  
\-rw-r–r-- 1 root root 0 10月 11 11:21 appendonly.aof  
\-rw-r–r-- 1 root root 0 10月 11 11:16 appendonly.aof.bak  
\-rw-r–r-- 1 root root 77 10月 11 11:22 dump.rdb  
\-rw-r–r-- 1 root root 77 10月 11 11:16 dump.rdb.bak  
\-rw-r–r-- 1 root root 819 10月 11 11:21 nodes-6379.conf  
\-rw-r–r-- 1 root root 6 10月 11 11:21 redis\_6379.pid  
\-rw-r–r-- 1 root root 46808 10月 9 18:34 redis.conf  
\-rw-r–r-- 1 root root 29738 10月 11 11:22 redis.log

\*\*

## 第二种方法，采用开源工具redis-migrate-tool来进行迁移 ##

\*\*  
使用 开源工具 redis-migrate-tool来进行迁移,开源项目地址：[https://github.com/vipshop/redis-migrate-tool][https_github.com_vipshop_redis-migrate-tool]  
先下载redis-migrate-tool  
\[root@host-172-22-14-105 ~\]\# cd /data/soft/  
\[root@host-172-22-14-105 ~\]\# git clone [https://github.com/vipshop/redis-migrate-tool.git][https_github.com_vipshop_redis-migrate-tool.git]  
\[root@host-172-22-14-105 soft\]\# cd redis-migrate-tool/  
\[root@host-172-22-14-105 redis-migrate-tool\]\# ls  
ChangeLog [configure.ac][] dep LICENSE m4 [Makefile.am][] nodes [README.md][] rmt.conf src test  
\[root@host-172-22-14-105 redis-migrate-tool\]\# autoreconf -fvi  
![在这里插入图片描述][70 6]

\[root@host-172-22-14-105 redis-migrate-tool\]\# ./configure  
![在这里插入图片描述][70 7]  
\[root@host-172-22-14-105 redis-migrate-tool\]\# make  
\[root@host-172-22-14-105 redis-migrate-tool\]\# src/redis-migrate-tool -h  
This is redis-migrate-tool-0.1.0  
ok,到这里工具安装好了

接下去拷贝数据过来  
\[root@host-172-22-14-88 cluster-conf\]\# scp 6379/dump.rdb [root@172.22.14.105][root_172.22.14.105]:/data/soft/20181011data/dump6379.rdb  
\[root@host-172-22-14-88 cluster-conf\]\# scp 6380/dump.rdb [root@172.22.14.105][root_172.22.14.105]:/data/soft/20181011data/dump6380.rdb  
\[root@host-172-22-14-88 cluster-conf\]\# scp 6381/dump.rdb [root@172.22.14.105][root_172.22.14.105]:/data/soft/20181011data/dump6381.rdb

登录105服务器  
\[root@host-172-22-14-105 20181011data\]\# ls  
dump6379.rdb dump6380.rdb dump6381.rdb

\[root@host-172-22-14-105 redis-migrate-tool\]\# vim rmt.conf  
![在这里插入图片描述][70 8]  
改为：  
\[source\]  
type: rdb file  
servers :

*  /data/soft/20181011data/dump6379.rdb
 *  /data/soft/20181011data/dump6380.rdb
 *  /data/soft/20181011data/dump6381.rdb

\[target\]  
type: redis cluster  
servers:  
\-127.0.0.1:6379

\[common\]  
listen: 0.0.0.0:8888

执行迁移  
\[root@host-172-22-14-105 redis-migrate-tool\]\# src/redis-migrate-tool -c rmt.conf -o log -d

状态查看：通过redis-cli连接redis-migrate-tool监控的端口，运行info命令  
\[root@host-172-22-14-105 ~\]\# /data/soft/redis-3.2.12/bin/redis-cli -h 172.22.14.105 -p 8888  
![在这里插入图片描述][70 9]

然后查看日志查看是否完成导入  
\[root@host-172-22-14-105 redis-migrate-tool\]\# cat log  
\[2018-10-11 14:35:33.987\] rmt\_core.c:525 Nodes count of source group : 3  
\[2018-10-11 14:35:33.987\] rmt\_core.c:526 Total threads count : 4  
\[2018-10-11 14:35:33.987\] rmt\_core.c:527 Read threads count assigned: 1  
\[2018-10-11 14:35:33.987\] rmt\_core.c:528 Write threads count assigned: 3  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2444 Total threads count in fact: 3  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2445 Read threads count in fact: 0  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2446 Write threads count in fact: 3  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2488 write thread(0):  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2494 /data/soft/20181011data/dump6381.rdb  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2488 write thread(1):  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2494 /data/soft/20181011data/dump6380.rdb  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2488 write thread(2):  
\[2018-10-11 14:35:33.990\] rmt\_core.c:2494 /data/soft/20181011data/dump6379.rdb  
\[2018-10-11 14:35:33.991\] rmt\_core.c:2551 migrate job is running…  
\[2018-10-11 14:35:33.994\] rmt\_redis.c:6601 Rdb file for node\[/data/soft/20181011data/dump6381.rdb\] parsed finished, use: 0 s.  
\[2018-10-11 14:35:33.994\] rmt\_redis.c:6601 Rdb file for node\[/data/soft/20181011data/dump6380.rdb\] parsed finished, use: 0 s.  
\[2018-10-11 14:35:33.998\] rmt\_redis.c:6601 Rdb file for node\[/data/soft/20181011data/dump6379.rdb\] parsed finished, use: 0 s.

通过日志可以发现都finished了，ok，再通过keys \* 检查一下  
\[root@host-172-22-14-105 ~\]\# /data/soft/redis-3.2.12/bin/redis-cli -c -h 172.22.14.105 -p 6379  
172.22.14.105:6379> keys \*

正式完成迁移

[70]: /images/20220508/2e51f71e430047608edaa7cae1999e04.png
[70 1]: /images/20220508/f76c984e77294a8091c61e0fa79015b3.png
[70 2]: /images/20220508/427e470117464c2ab417b24a517bfdf3.png
[70 3]: /images/20220508/ebbdf007e11b415eb9201aa2782dbb6d.png
[root_172.22.14.105]: mailto:root@172.22.14.105
[70 4]: /images/20220508/27db89c84a4f4d3397324017b92f978f.png
[70 5]: /images/20220508/2d2aba8137834d5bbab10823a27f37ef.png
[https_github.com_vipshop_redis-migrate-tool]: https://github.com/vipshop/redis-migrate-tool
[https_github.com_vipshop_redis-migrate-tool.git]: https://github.com/vipshop/redis-migrate-tool.git
[configure.ac]: http://configure.ac
[Makefile.am]: http://Makefile.am
[README.md]: http://README.md
[70 6]: /images/20220508/32b44bd770154930b4508ee206d17b6c.png
[70 7]: /images/20220508/7b917068d3824535b45b820f32a6a7e6.png
[70 8]: /images/20220508/801361f1fb6542f08a727be2aeb3897e.png
[70 9]: /images/20220508/66e7095c6e744568958c5d5138a7893b.png