vertica-三节点集群，一台机器故障，所有文件丢失。的恢复

青旅半醒 2022-07-21 01:27 113阅读 0赞

数据文件夹全部丢失，此时在节点上重装软件，数据库识别该失败节点，但是无法启动或者删除该节点，因为缺少核心的媒介spread.conf 文件

重启这个节点报错

"\{0\}. Error was: \{1\}.".format(repr(msg), e))

ATReceiveFailure\_Init: Problem json decoding message '\{"status": null, "content": \{"special\_environment": null\}, "error\_type": null, "error\_message": null, "exec\_stack": null\}'. Error was: None is not a valid status.

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ sudo mkdir EmBigData\_dev

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ sudo chown -R dbadmin:verticadba EmBigData\_dev/

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ cd EmBigData\_dev/

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ sudo mkdir v\_embigdata\_dev\_node0003\_catalog

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ sudo chown -R dbadmin:verticadba v\_embigdata\_dev\_node0003\_catalog/

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ cd v\_embigdata\_dev\_node0003\_catalog/

正常节点向其传输spread.conf 文件

\[ [dbadmin@n3][dbadmin_n3] v\_embigdata\_dev\_node0003\_catalog\]$ scp spread.conf 172.16.57.26:/opt/EmBigData\_dev/v\_embigdata\_dev\_node0003\_catalog/

spread.conf                                                                                                               100%  403     0.4KB/s   00:00

问题节点查看spread.conf

\[ [dbadmin@n1][dbadmin_n1] v\_embigdata\_dev\_node0003\_catalog\]$ ll

total 4

\-rw-r--r-- 1 dbadmin verticadba 403 Jul 12 09:24 spread.conf

尝试重启问题节点，多次执行以下步骤，执行成功

Nodes UP: v\_embigdata\_dev\_node0002, v\_embigdata\_dev\_node0003

Nodes DOWN: v\_embigdata\_dev\_node0001 (may be still initializing).

It is suggested that you continue waiting.

Do you want to continue waiting? (yes/no)  \[yes\] yes

总结一下：

这里可以看到我们在节点1里面，却配置了节点3的v\_embigdata\_dev\_node0003\_catalog，节点1依然启动成功！！

这里发现vertica有一个识别节点的spread.conf文件，这个文件在每个节点都一样的

可以看看问题节点在重启成功后的文件结构

\[ [dbadmin@n1][dbadmin_n1] EmBigData\_dev\]$ ll

total 36

\-rw-r--r--   1 dbadmin verticadba  7164 Jul 12 09:30 dbLog

drwx------   9 dbadmin verticadba  4096 Jul 12 09:40 v\_embigdata\_dev\_node0001\_catalog

drwxrwx--- 502 dbadmin verticadba 20480 Jul 12 09:31 v\_embigdata\_dev\_node0001\_data

drwxr-xr-x   2 dbadmin verticadba  4096 Jul 12 09:24 v\_embigdata\_dev\_node0003\_catalog

这里v\_embigdata\_dev\_node0001\_catalog和v\_embigdata\_dev\_node0001\_data全部重新生成了

查看日志，了解vertica恢复步骤

2016-07-12 09:24:48.426 unknown:0x7f7352307720 \[Txn\] <INFO> Looking for catalog at: /opt/EmBigData\_dev/v\_embigdata\_dev\_node0001\_catalog/Catalog

2016-07-12 09:24:48.426 unknown:0x7f7352307720 \[Init\] <INFO> Catalog not loaded from /opt/EmBigData\_dev/v\_embigdata\_dev\_node0001\_catalog, probably being bootstrapped or empty

2016-07-12 09:24:48.426 unknown:0x7f7352307720 \[Comms\] <INFO> Spread domain socket will be opened in directory /opt/vertica/spread/tmp

2016-07-12 09:24:48.426 unknown:0x7f7352307720 \[Comms\] <INFO> About to launch spread with ' /opt/vertica/spread/sbin/spread -c /opt/EmBigData\_dev/v\_embigdata\_dev\_node0001\_catalog/spread.conf'

2016-07-12 09:24:48.427 unknown:0x7f7352307720 \[Comms\] <INFO> forked spread pid=33233, wrote pidfile /opt/EmBigData\_dev/v\_embigdata\_dev\_node0001\_catalog/spread.pid

可以看到spread.conf在集群之间传递

...............

2016-07-12 09:25:00.021 Spread Client:0x7c4eb30 \[VMPI\] <INFO> We are invited to join the cluster

2016-07-12 09:25:00.021 Spread Client:0x7c4eb30 \[Init\] <INFO> Startup \[Waiting for Cluster Invite\] Invited

2016-07-12 09:25:00.021 Spread Client:0x7c4eb30-fff0000000000bca \[Txn\] <INFO> Begin Txn: fff0000000000bca ' Installing New Catalog'

2016-07-12 09:25:00.021 Spread Client:0x7c4eb30-fff0000000000bca \[Catalog\] <INFO> installNewCatalog: Received new catalog, replacing current TRANSACTION-fff0000000000bca catalog (old version=0, new version 0x35f2d)

...............

2016-07-12 09:25:24.026 Spread Client:0x7c4eb30 \[Txn\] <INFO> Commit Complete: Txn: a0000000000001 at epoch 0xa93c

2016-07-12 09:25:24.026 Spread Client:0x7c4eb30 \[Txn\] <INFO>  Joining DB group with catalog version 220973

2016-07-12 09:25:24.026 Spread Client:0x7c4eb30 \[Comms\] <INFO> joinDBGroup: Node v\_embigdata\_dev\_node0001 (\#r0-10\#N172016057026) joining cluster for DB EmBigData\_dev

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Comms\] <INFO> nodeToState map:

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Comms\] <INFO>    v\_embigdata\_dev\_node0001 : INITIALIZING

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Comms\] <INFO>   v\_embigdata\_dev\_node0002 : UP

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Comms\] <INFO>   v\_embigdata\_dev\_node0003 : UP

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Recover\] <INFO> State change for node v\_embigdata\_dev\_node0001: INITIALIZING; catalog 220973

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Comms\] <INFO> Saw membership message 4352 (0x1100) on V:EmBigData\_dev

2016-07-12 09:25:24.037 Spread Client:0x7c4eb30 \[Comms\] <INFO> DB Group changed

...............

2016-07-12 09:25:24.688 DistCall Dispatch:0x7f72a07b6bb0 <LOG> @v\_embigdata\_dev\_node0001: 00000/3298: Event Posted: Event Code:6 Event Id:1 Event Severity: Informational \[6\] PostedTimestamp: 2016-07-12 09:25:24.688094 ExpirationTimestamp: 2084-07-30 12:39:31.688094 EventCodeDescription: Node State Change ProblemDescription: Changing node v\_embigdata\_dev\_node0001 startup state to RECOVERING DatabaseName: EmBigData\_dev Hostname: n1

2016-07-12 09:25:24.688 DistCall Dispatch:0x7f72a07b6bb0 \[Recover\] <INFO>  Changing node v\_embigdata\_dev\_node0001 startup state from INITIALIZING to RECOVERING

2016-07-12 09:25:24.688 Spread Client:0x7c4eb30 \[Comms\] <INFO> nodeToState map:

2016-07-12 09:25:24.688 Spread Client:0x7c4eb30 \[Comms\] <INFO>    v\_embigdata\_dev\_node0001 : RECOVERING

2016-07-12 09:25:24.688 Spread Client:0x7c4eb30 \[Comms\] <INFO>   v\_embigdata\_dev\_node0002 : UP

2016-07-12 09:25:24.688 Spread Client:0x7c4eb30 \[Comms\] <INFO>   v\_embigdata\_dev\_node0003 : UP

[dbadmin_n1]: mailto:dbadmin@n1
[dbadmin_n3]: mailto:dbadmin@n3