REdis主从复制之repl_backlog

清疚 2022-01-22 05:21 150阅读 0赞

# **目录** #

目录 1

1. 前言 1

2. 配置项 1

3. redisServer 2

4. feedReplicationBacklog-写repl\_backlog 3

5. addReplyReplicationBacklog-读repl\_backlog 4

# **1. 前言** #

注意，repl\_backlog只针对**部分复制**（Partial Replication），而非全量复制。

本文内容基于redis-5.0.5（截至2019/6/6的最新版本），本文深入介绍REdis主从复制的部分复制核心要素repl\_backlog，与其相关的配置直接影响主从间的稳定性，对提升集群的稳定性十分重要。

注意REdis的主节点把所有从节点也当作一个Client看待，正常的数据同步并不涉及repl\_backlog。当从节点断开重连，这个时候repl\_backlog的作用就体现出来了。截至到5.0.5版本，从节点重启用不上repl\_backlog，原因是从节点没有保存repl\_backlog的信息，无法实现部分同步，但可少量改动REdis源代码，实现从节点重启后的部分复制。

正常情况下，主节点会往从节点连接缓冲区写一份数据，同时往repl\_backlog也写一份数据，所有从节点共享同一份repl\_backlog，因此可以考虑repl\_backlog配置大一点，以容忍从节点更长时间失联。

从节点向主节点发送命令PSYNC，触发部分复制。有关REdis主从复制的细节，请参见《REdis复制研究》。

# **2. 配置项** #

REdis的复制分全量复制和部分复制，全量复制是个很重的过程，而部分复制则是轻量的，部分复制实际是一个增量复制。

REdis的主节点创建和维护一个环形缓冲复制队列（即repl\_backlog），从节点部分复制（增量复制）的数据均来自于repl\_backlog。

主节点只有一个repl\_backlog，所有从节点共享，直接相关的配置项有两个：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:116.15pt;"> 配置项名 </td> 
 <td style="vertical-align:top;width:309.95pt;"> 配置项说明 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:116.15pt;"> repl-backlog-size </td> 
 <td style="vertical-align:top;width:309.95pt;"> 环形缓冲复制队列大小，可不带单位，但同时支持单位：b、k、kb、m、mb、g、gb，单位不区分大小写，其中k、m、g间的计算倍数是1000，而kb、mb和gb的计算倍数是1024。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:116.15pt;"> repl-backlog-ttl </td> 
 <td style="vertical-align:top;width:309.95pt;"> 环形缓冲复制队列存活时长（所有slaves不可用时，保留repl_backlog多长时间，单位：秒） </td> 
 </tr> 
 </tbody> 
</table>

# **3. redisServer** #

结构体redisServer是REdis的第一核心结构，**repl\_backlog**是它的组成成员。

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> struct&nbsp;redisServer&nbsp;{ &nbsp;&nbsp;/*&nbsp;My&nbsp;current&nbsp;replication&nbsp;offset&nbsp;*/ &nbsp;&nbsp;long&nbsp;long&nbsp;master_repl_offset; &nbsp;&nbsp;/*&nbsp;Accept&nbsp;offsets&nbsp;up&nbsp;to&nbsp;this&nbsp;for&nbsp;replid2.&nbsp;*/ &nbsp;&nbsp;long&nbsp;long&nbsp;second_replid_offset; &nbsp;&nbsp;/*&nbsp;Replication&nbsp;backlog&nbsp;for&nbsp;partial&nbsp;syncs&nbsp;*/ &nbsp;&nbsp;char&nbsp;*repl_backlog;&nbsp;//&nbsp;环形缓冲复制队列 &nbsp;&nbsp;/*&nbsp;Backlog&nbsp;circular&nbsp;buffer&nbsp;size&nbsp;*/ &nbsp;&nbsp;long&nbsp;long&nbsp;repl_backlog_size;&nbsp;//&nbsp;环形缓冲复制队列容量 &nbsp;&nbsp;/*&nbsp;Backlog&nbsp;actual&nbsp;data&nbsp;length&nbsp;*/ &nbsp;&nbsp;long&nbsp;long&nbsp;repl_backlog_histlen;&nbsp;//&nbsp;环形缓冲复制队列已用大小（影响是否能部分复制） &nbsp;&nbsp;/*&nbsp;Backlog&nbsp;circular&nbsp;buffer&nbsp;current&nbsp;offset, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;that&nbsp;is&nbsp;the&nbsp;next&nbsp;byte&nbsp;will'll&nbsp;write&nbsp;to.*/ &nbsp;&nbsp;//&nbsp;实际上谈不上空闲，因为总是环绕覆盖写， &nbsp;&nbsp;//&nbsp;理解为最新数据的截止位置更为合适，更新的数据总是从这里开始写入到repl_backlog中。 &nbsp;&nbsp;long&nbsp;long&nbsp;repl_backlog_idx;&nbsp;//&nbsp;环形缓冲复制队列空闲起始位置（写从这里开始） &nbsp;&nbsp;/*&nbsp;Replication&nbsp;"master&nbsp;offset"&nbsp;of&nbsp;first &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;byte&nbsp;in&nbsp;the&nbsp;replication&nbsp;backlog&nbsp;buffer.*/ &nbsp;&nbsp;long&nbsp;long&nbsp;repl_backlog_off;&nbsp;//&nbsp;数据在环形缓冲复制队列的起始位置（读从这里开始） &nbsp;&nbsp;/*&nbsp;Time&nbsp;without&nbsp;slaves&nbsp;after&nbsp;the&nbsp;backlog &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;gets&nbsp;released.&nbsp;*/ &nbsp;&nbsp;time_t&nbsp;repl_backlog_time_limit;&nbsp;//&nbsp;环形缓冲复制队列生存时长 &nbsp;&nbsp;/*&nbsp;We&nbsp;have&nbsp;no&nbsp;slaves&nbsp;since&nbsp;that&nbsp;time. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Only&nbsp;valid&nbsp;if&nbsp;server.slaves&nbsp;len&nbsp;is&nbsp;0.&nbsp;*/ &nbsp;&nbsp;time_t&nbsp;repl_no_slaves_since;&nbsp;//&nbsp;无可用从节点的发生时间 &nbsp;&nbsp;/*&nbsp;Min&nbsp;number&nbsp;of&nbsp;slaves&nbsp;to&nbsp;write.&nbsp;*/ &nbsp;&nbsp;int&nbsp;repl_min_slaves_to_write;&nbsp;//&nbsp;最小需写的从节点数 &nbsp;&nbsp;/*&nbsp;Max&nbsp;lag&nbsp;of&nbsp;&lt;count&gt;&nbsp;slaves&nbsp;to&nbsp;write.&nbsp;*/ &nbsp;&nbsp;int&nbsp;repl_min_slaves_max_lag; &nbsp;&nbsp;/*&nbsp;Number&nbsp;of&nbsp;slaves&nbsp;with&nbsp;lag&nbsp;&lt;=&nbsp;max_lag.&nbsp;*/ &nbsp;&nbsp;int&nbsp;repl_good_slaves_count; &nbsp;&nbsp;/*&nbsp;Send&nbsp;RDB&nbsp;to&nbsp;slaves&nbsp;sockets&nbsp;directly.&nbsp;*/ &nbsp;&nbsp;int&nbsp;repl_diskless_sync;&nbsp;//&nbsp;不落磁盘（无盘）往从节点发送RDB（全量复制） &nbsp;&nbsp;/*&nbsp;Delay&nbsp;to&nbsp;start&nbsp;a&nbsp;diskless&nbsp;repl&nbsp;BGSAVE.&nbsp;*/ &nbsp;&nbsp;//&nbsp;无盘复制时，延迟指定的时长，以等待更多的从节点 &nbsp;&nbsp;int&nbsp;repl_diskless_sync_delay; }; </td> 
 </tr> 
 </tbody> 
</table>

# **4. feedReplicationBacklog-写repl\_backlog** #

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> /*&nbsp;Add&nbsp;data&nbsp;to&nbsp;the&nbsp;replication&nbsp;backlog. &nbsp;*&nbsp;This&nbsp;function&nbsp;also&nbsp;increments&nbsp;the&nbsp;global&nbsp;replication&nbsp;offset&nbsp;stored&nbsp;at &nbsp;*&nbsp;server.master_repl_offset,&nbsp;because&nbsp;there&nbsp;is&nbsp;no&nbsp;case&nbsp;where&nbsp;we&nbsp;want&nbsp;to&nbsp;feed &nbsp;*&nbsp;the&nbsp;backlog&nbsp;without&nbsp;incrementing&nbsp;the&nbsp;offset.&nbsp;*/ //&nbsp;主要被replicationFeedSlaves调用 //&nbsp;写len长的数据ptr到repl_backlog中 //&nbsp;repl_backlog是一个环形buffer，不存在溢出的问题，策略是最新数据覆盖最老数据。 //&nbsp;如果参数len大于repl_backlog_size， //&nbsp;则repl_backlog没有实际意义， //&nbsp;因为无法存储一份完整数据。 void&nbsp;feedReplicationBacklog(void&nbsp;*ptr,&nbsp;size_t&nbsp;len)&nbsp;{ &nbsp;&nbsp;unsigned&nbsp;char&nbsp;*p&nbsp;=&nbsp;ptr; &nbsp;&nbsp;server.master_repl_offset&nbsp;+=&nbsp;len; &nbsp; &nbsp;&nbsp;/*&nbsp;This&nbsp;is&nbsp;a&nbsp;circular&nbsp;buffer,&nbsp;so&nbsp;write&nbsp;as&nbsp;much&nbsp;data&nbsp;we&nbsp;can&nbsp;at&nbsp;every &nbsp;&nbsp;&nbsp;*&nbsp;iteration&nbsp;and&nbsp;rewind&nbsp;the&nbsp;"idx"&nbsp;index&nbsp;if&nbsp;we&nbsp;reach&nbsp;the&nbsp;limit.&nbsp;*/ &nbsp;&nbsp;//&nbsp;因为repl_backlog是环形buffer， &nbsp;&nbsp;//&nbsp;剩余的空间可能容纳不了len长的数据， &nbsp;&nbsp;//&nbsp;当不够时，就需要环绕从头开始写， &nbsp;&nbsp;//&nbsp;因此这里需while循环。 &nbsp;&nbsp;while(len)&nbsp;{ &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;repl_backlog_size为repl_backlog的容量大小， &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;由配置项决定repl_backlog_size值决定， &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;repl_backlog_idx是repl_backlog空闲区域的起始位置， &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;这两个值相减得到repl_backlog可用大小。 &nbsp;&nbsp;&nbsp;&nbsp;size_t&nbsp;thislen&nbsp;=&nbsp;server.repl_backlog_size&nbsp;-&nbsp;server.repl_backlog_idx; &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;如果thislen大于len，则表示足够容纳 &nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;(thislen&nbsp;&gt;&nbsp;len)&nbsp;thislen&nbsp;=&nbsp;len; &nbsp;&nbsp;&nbsp;&nbsp;memcpy(server.repl_backlog+server.repl_backlog_idx,p,thislen); &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;空闲位置往后挪动 &nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_idx&nbsp;+=&nbsp;thislen; &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;如果空闲位置达到容量大小，则环绕回去从0开始 &nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;(server.repl_backlog_idx&nbsp;==&nbsp;server.repl_backlog_size) &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_idx&nbsp;=&nbsp;0; &nbsp;&nbsp;&nbsp;&nbsp;len&nbsp;-=&nbsp;thislen; &nbsp;&nbsp;&nbsp;&nbsp;p&nbsp;+=&nbsp;thislen; &nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;repl_backlog_histlen记录了repl_backlog中的数据大小&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_histlen&nbsp;+=&nbsp;thislen; &nbsp;&nbsp;} &nbsp;&nbsp;//&nbsp;修正存储在repl_backlog中的数据大小， &nbsp;&nbsp;//&nbsp;它不可能超过repl_backlog_size值。 &nbsp;&nbsp;if&nbsp;(server.repl_backlog_histlen&nbsp;&gt;&nbsp;server.repl_backlog_size) &nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_histlen&nbsp;=&nbsp;server.repl_backlog_size; &nbsp;&nbsp;/*&nbsp;Set&nbsp;the&nbsp;offset&nbsp;of&nbsp;the&nbsp;first&nbsp;byte&nbsp;we&nbsp;have&nbsp;in&nbsp;the&nbsp;backlog.&nbsp;*/&nbsp;&nbsp; &nbsp;&nbsp;server.repl_backlog_off&nbsp;=&nbsp;server.master_repl_offset&nbsp;- &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_histlen&nbsp;+&nbsp;1; } </td> 
 </tr> 
 </tbody> 
</table>

# **5. addReplyReplicationBacklog-读repl\_backlog** #

当主节点判断可部分复制时，会记录如下日志：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> Partial&nbsp;resynchronization&nbsp;request&nbsp;from&nbsp;%s&nbsp;accepted.&nbsp;Sending&nbsp;%lld&nbsp;bytes&nbsp;of&nbsp;backlog&nbsp;starting&nbsp;from&nbsp;offset&nbsp;%lld. </td> 
 </tr> 
 </tbody> 
</table>

给从节点的响应头为“**+CONTINUE** **replid**\\r\\n”或“+CONTINUE\\r\\n”。函数addReplyReplicationBacklog的实现：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> /*&nbsp;Feed&nbsp;the&nbsp;slave&nbsp;'c'&nbsp;with&nbsp;the&nbsp;replication&nbsp;backlog&nbsp;starting&nbsp;from&nbsp;the &nbsp;*&nbsp;specified&nbsp;'offset'&nbsp;up&nbsp;to&nbsp;the&nbsp;end&nbsp;of&nbsp;the&nbsp;backlog.&nbsp;*/ //&nbsp;被masterTryPartialResynchronization调用 //&nbsp;而masterTryPartialResynchronization被syncCommand调用（对应命令PSYNC）。 //&nbsp;从repl_backlog取数据发给slave， //&nbsp;数据的开始位置由offset指定。 long&nbsp;long&nbsp;addReplyReplicationBacklog(client&nbsp;*c,&nbsp;long&nbsp;long&nbsp;offset)&nbsp;{ &nbsp;&nbsp;long&nbsp;long&nbsp;j,&nbsp;skip,&nbsp;len; &nbsp; &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Replica&nbsp;request&nbsp;offset:&nbsp;%lld",&nbsp;offset); &nbsp;&nbsp;//&nbsp;repl_backlog_histlen为0， &nbsp;&nbsp;//&nbsp;表示repl_backlog中无数据。 &nbsp;&nbsp;if&nbsp;(server.repl_backlog_histlen&nbsp;==&nbsp;0)&nbsp;{ &nbsp;&nbsp;&nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Backlog&nbsp;history&nbsp;len&nbsp;is&nbsp;zero"); &nbsp;&nbsp;&nbsp;&nbsp;return&nbsp;0; &nbsp;&nbsp;} &nbsp; &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Backlog&nbsp;size:&nbsp;%lld", &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_size); &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;First&nbsp;byte:&nbsp;%lld", &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_off); &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;History&nbsp;len:&nbsp;%lld", &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_histlen); &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Current&nbsp;index:&nbsp;%lld", &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_idx); &nbsp; &nbsp;&nbsp;/*&nbsp;Compute&nbsp;the&nbsp;amount&nbsp;of&nbsp;bytes&nbsp;we&nbsp;need&nbsp;to&nbsp;discard.&nbsp;*/ &nbsp;&nbsp;skip&nbsp;=&nbsp;offset&nbsp;-&nbsp;server.repl_backlog_off; &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Skipping:&nbsp;%lld",&nbsp;skip); &nbsp; &nbsp;&nbsp;/*&nbsp;Point&nbsp;j&nbsp;to&nbsp;the&nbsp;oldest&nbsp;byte,&nbsp;that&nbsp;is&nbsp;actually&nbsp;our &nbsp;&nbsp;&nbsp;*&nbsp;server.repl_backlog_off&nbsp;byte.&nbsp;*/ &nbsp;&nbsp;j&nbsp;=&nbsp;(server.repl_backlog_idx&nbsp;+ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(server.repl_backlog_size-server.repl_backlog_histlen))&nbsp;% &nbsp;&nbsp;&nbsp;&nbsp;server.repl_backlog_size; &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Index&nbsp;of&nbsp;first&nbsp;byte:&nbsp;%lld",&nbsp;j); &nbsp; &nbsp;&nbsp;/*&nbsp;Discard&nbsp;the&nbsp;amount&nbsp;of&nbsp;data&nbsp;to&nbsp;seek&nbsp;to&nbsp;the&nbsp;specified&nbsp;'offset'.&nbsp;*/ &nbsp;&nbsp;j&nbsp;=&nbsp;(j&nbsp;+&nbsp;skip)&nbsp;%&nbsp;server.repl_backlog_size; &nbsp; &nbsp;&nbsp;/*&nbsp;Feed&nbsp;slave&nbsp;with&nbsp;data.&nbsp;Since&nbsp;it&nbsp;is&nbsp;a&nbsp;circular&nbsp;buffer&nbsp;we&nbsp;have&nbsp;to &nbsp;&nbsp;&nbsp;*&nbsp;split&nbsp;the&nbsp;reply&nbsp;in&nbsp;two&nbsp;parts&nbsp;if&nbsp;we&nbsp;are&nbsp;cross-boundary.&nbsp;*/ &nbsp;&nbsp;len&nbsp;=&nbsp;server.repl_backlog_histlen&nbsp;-&nbsp;skip; &nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;Reply&nbsp;total&nbsp;length:&nbsp;%lld",&nbsp;len); &nbsp;&nbsp;while(len)&nbsp;{ &nbsp;&nbsp;&nbsp;&nbsp;long&nbsp;long&nbsp;thislen&nbsp;= &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;((server.repl_backlog_size&nbsp;-&nbsp;j)&nbsp;&lt;&nbsp;len)&nbsp;? &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(server.repl_backlog_size&nbsp;-&nbsp;j)&nbsp;:&nbsp;len; &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;serverLog(LL_DEBUG,&nbsp;"[PSYNC]&nbsp;addReply()&nbsp;length:&nbsp;%lld",&nbsp;thislen); &nbsp;&nbsp;&nbsp;&nbsp;addReplySds(c,sdsnewlen(server.repl_backlog&nbsp;+&nbsp;j,&nbsp;thislen)); &nbsp;&nbsp;&nbsp;&nbsp;len&nbsp;-=&nbsp;thislen; &nbsp;&nbsp;&nbsp;&nbsp;j&nbsp;=&nbsp;0; &nbsp;&nbsp;} &nbsp;&nbsp;return&nbsp;server.repl_backlog_histlen&nbsp;-&nbsp;skip; } </td> 
 </tr> 
 </tbody> 
</table>