Greenplum数据库源码学习——FTS简介

太过爱你忘了你带给我的痛 2022-09-04 13:48 320阅读 0赞

## 简介 ##

FTS(Fault Tolerance Service)是greenplum提供的对于子节点的故障检测与恢复的服务。FTS是一个隶属于master的子进程，通过定期轮询每个primary的状态来获取每个primary-mirror组的状态。该进程只在master上存在，进程名为ftsprobe process。  
![在这里插入图片描述][5fcb39837abb4f948ac0861966c32c47.png]  
这里需要注意的一点是，FTS并不直接连接mirror，对于mirror的状态，FTS也是通过primary来获取。Primary则通过wal-sender进程的状态来获取mirror存活和同步状态。  
FTS在满足如下三个条件的时候会触发轮询

1.  到时间了，gp\_fts\_probe\_interval
2.  用户手动执行select gp\_request\_fts\_probe\_scan()
3.  查询执行过程中发现节点异常  
    轮询过程图如下：  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzbWFydGtpbGxlcg_size_16_color_FFFFFF_t_70]

## FTS进程工作流程 ##

FTS的进程名为ftsprobe process，后台进程标记为BGWORKER\_SHMEM\_ACCESS和BGWORKER\_BACKEND\_DATABASE\_CONNECTION，也就是FTS进程可以访问后台进程共享内存，可以连接数据库。后台进程启动时机由下面的枚举类型标识，BgWorkerStart\_DtxRecovering意味着进程可以在postmaster启动后，在分布式事务恢复前启动。这里的FTS的进程就是这样的进程。

Static BackgroundWorker PMAuxProcList[MaxPMAuxProc] = { 
    	{ "ftsprobe process",
    	 BGWORKER_SHMEM_ACCESS | BGWORKER_BACKEND_DATABASE_CONNECTION,
    	 BgWorkerStart_DtxRecovering, /* no need to wait dtx recovery */
    	 0, /* restart immediately if ftsprobe exits with non-zero code */
    	 FtsProbeMain, { 0}, { 0}, 0, 0,
    	 FtsProbeStartRule},

Greenplum sets up a group of GP background processes through an array structure PMAuxProcList. A entry in that struct represents a GP background process.  
Two functions pointers are important members of the BackgroundWorker structure. One points to main entry function of the GP background process. The other points to the function that determine if the process should be started or not. For FTS, these two functions are FtsProbeMain() and FtsProbeStartRule(), respectively. This is hard-coded in postmaster.c.  
\#define MaxPMAuxProc 6  
static BackgroundWorker PMAuxProcList\[MaxPMAuxProc\]  
In Postmaster, we will check the following condition:  
Gp\_entry\_postmaster && Gp\_role == GP\_ROLE\_DISPATCH  
The FTS probe process is started when the condition is true.  
In the initialization phase, we register one BackgroundWorker entry for each GP background process into postmaster’s private structure BackgroundWorkerList. When we do this, the above condition is checked to decide if FTS should be registered there or not. The reader may want to check load\_auxiliary\_libraries() for more detail.  
Later, the postmaster tries to start the processes that have beenregistered in the BackgroundWorkerList, which includes the FTS probe process. If first attempt to start a particular process fails, or a process goes down for some reason and needs to be brought up again, postmaster restarts it in its main loop. Every iteration, it checks the status of these processes and acts accordingly.

### FTS共享内存初始化 ###

FTS共享内存初始化在共享内存创建pid\_t大小内存，并初始化为0.

static volatile pid_t *shmFtsProbePID;
    void FtsProbeShmemInit(void) { 
    	if (IsUnderPostmaster)
    		return;
    	shmFtsProbePID = (volatile pid_t*)ShmemAlloc(sizeof(*shmFtsProbePID));
    	*shmFtsProbePID = 0;
    }

### FTS进程主函数FtsProbeMain ###

对于SIGHUP信号设置sigHupHandler信号处理函数，将got\_SIGHUP设置为true，设置Latch。表示需要读取postgresql.conf。

static volatile sig_atomic_t got_SIGHUP = false;
    static void sigHupHandler(SIGNAL_ARGS) { 
    	got_SIGHUP = true;
    	if (MyProc) SetLatch(&MyProc->procLatch);
    }

对于SIGINT信号设置sigIntHandler信号处理函数，将probe\_requested设置为true，设置Latch。表示需要重新触发FTS扫描。

static volatile bool probe_requested = false;
    static void sigIntHandler(SIGNAL_ARGS) { 
    	probe_requested = true;
    	if (MyProc) SetLatch(&MyProc->procLatch);
    }

对于SIGSEGV程序错误信号设置CdbProgramErrorHandler信号处理函数，

void CdbProgramErrorHandler(SIGNAL_ARGS) { 
        int			save_errno = errno;
        char       *pts = "process";
    	if (!pthread_equal(main_tid, pthread_self())) { 
    #ifndef _WIN32
    		write_stderr("\nUnexpected internal error: Master %d received signal %d in worker thread %lu (forwarding signal to main thread)\n\n", MyProcPid, postgres_signal_arg, (unsigned long)pthread_self());
    #else
    		write_stderr("\nUnexpected internal error: Master %d received signal %d in worker thread %lu (forwarding signal to main thread)\n\n", MyProcPid, postgres_signal_arg, (unsigned long)pthread_self().p);
    #endif
    		/* Only forward if the main thread isn't quick-dying. */
    		if (!in_quickdie) pthread_kill(main_tid, postgres_signal_arg);
    
    		/* * Don't exit the thread when we reraise SEGV/BUS/ILL signals to the OS. * This thread will die together with the main thread after the OS reraises * the signal. This is to ensure that the dumped core file contains the call * stack on this thread for later debugging. */
    		if (!(gp_reraise_signal &&(postgres_signal_arg == SIGSEGV || postgres_signal_arg == SIGILL || postgres_signal_arg == SIGBUS))) { 
    			pthread_exit(NULL);
    		}
    		return;
    	}
    
        if (Gp_role == GP_ROLE_DISPATCH)
            pts = "Master process";
        else if (Gp_role == GP_ROLE_EXECUTE)
            pts = "Segment process";
        else
            pts = "Process";
        errno = save_errno;
        StandardHandlerForSigillSigsegvSigbus_OnMainThread(pts, PASS_SIGNAL_ARGS);
    }

Fts进程主函数FtsProbeMain主要流程就是设置信号处理函数，放开信号屏蔽，初始化连接，调用FtsLoop函数。

bool am_ftsprobe = false;
    void FtsProbeMain(Datum main_arg) { 
    	*shmFtsProbePID = MyProcPid;  // 向共享内存中设置进程pid
    	am_ftsprobe = true;           // 设置fts标志位启动
    	/* reread postgresql.conf if requested */
    	pqsignal(SIGHUP, sigHupHandler); pqsignal(SIGINT, sigIntHandler);
    	/* CDB: Catch program error signals. Save our main thread-id for comparison during signals. */
    	main_tid = pthread_self();
    #ifdef SIGSEGV
    	pqsignal(SIGSEGV, CdbProgramErrorHandler);
    #endif
    	/* We're now ready to receive signals */
    	BackgroundWorkerUnblockSignals();
    	/* Connect to our database */
    	BackgroundWorkerInitializeConnection(DB_FOR_COMMON_ACCESS, NULL); // DB_FOR_COMMON_ACCESS定义为postgres
    	/* main loop */
    	FtsLoop();
    	/* One iteration done, go away */
    	proc_exit(0);
    }

## FTS故障检测 ##

对于每个primary-mirror组来说一般会有如下几种状态。

1.  primary正常，mirror正常
2.  primary正常，mirror异常
3.  primary异常，mirror正常
4.  primary异常，mirror异常  
    Greenplum目前是一主一从架构，所以对于第四种primary-mirror都出故障的情况是解决不了的，在这种情况下只能通过人工干预解决。对于第一种都正常的情况也不需要做任何处理。所以下面对二三两种情况做详细介绍。

### 故障1：primary挂掉 ###

这是最常见，也是高可用解决的最主要的问题。一旦FTS发现某个primary已经宕机，此时，如果Mirror是同步的，则将把对应的Mirror Promote成primary，并更新catalog。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzbWFydGtpbGxlcg_size_16_color_FFFFFF_t_70 1]  
Promote后catalog可以看到如下更新。  
![在这里插入图片描述][df4d7c1751524784a4034ded530416bd.png]  
通过role可以看出，mirror成了primary，preferred\_role没有变，同时原primary被标记成了mirror，但status是d。mode也被标记成了n。

### 故障2：mirror挂掉 ###

如果mirror挂掉了，意味着primary与mirror的复制不可能同步了，所以primary会hang住，直到FTS来通知primary执行Sync-off来关闭同步复制。关闭同步复制的方法即将synchronous\_standby\_names设置成空。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzbWFydGtpbGxlcg_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][60257633a42c47ebaebf580d52ffeb0b.png]  
可以看到catalog表的变化，mode已经变成不同步，mirror的状态也标记成down。

注意：FTS在轮询的时候，**如果Primary节点发现Mirror存活并且复制方式是非同步，则会强制设置为同步复制，即更新synchronous\_standby\_names为’\*’**。

FTS相关GUC  
![在这里插入图片描述][4b507229d6284f028c696703681f6653.png]

[5fcb39837abb4f948ac0861966c32c47.png]: /images/20220829/7553479e956a4c6699b96bc41d951e76.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzbWFydGtpbGxlcg_size_16_color_FFFFFF_t_70]: /images/20220829/052c65cfe99449ea8408531e9bf1467a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzbWFydGtpbGxlcg_size_16_color_FFFFFF_t_70 1]: /images/20220829/f9721acd94284d15875f59045acfa757.png
[df4d7c1751524784a4034ded530416bd.png]: /images/20220829/2aec67609cb74b9a8731da210f78a181.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzbWFydGtpbGxlcg_size_16_color_FFFFFF_t_70 2]: /images/20220829/41d49ebe8328424993c96a125b9e4d9a.png
[60257633a42c47ebaebf580d52ffeb0b.png]: /images/20220829/4b84f6dd3eee4d409dafe36dd1dc3dd6.png
[4b507229d6284f028c696703681f6653.png]: /images/20220829/30a45a43f6c9428cb6fde8c13a93aa29.png