hbase-1.1.2：某个表区域的状态处于regions in transition(RIT)的troubleshooting

柔光的暖阳◎ 2022-05-31 00:20 301阅读 0赞

【发现RIT问题】

hbase某个表区域的状态处于Regions In Transition，即RIT

处于RIT的region是 6f6f4f8204ccaa0b71df08591de155ba 在master角色的web UI查看到具体RIT信息如下  
Region  State  RIT time (ms)  
6f6f4f8204ccaa0b71df08591de155ba  ks\_namespace:active\_users,9c1be2,1512918712031.6f6f4f8204ccaa0b71df08591de155ba. state=OFFLINE, ts=Thu Jan 25 09:15:31 CST 2018 (1657s ago), server=kmr-5b9c18fc-gn-7b3518df-core-1-006.ksc.com,16020,1513713156134  1657561  
\# RIT的持续时间1657561ms，超过了27分钟！  
  
【排查过程】

（1）使用hdfs dfs -ls -R /apps/hbase/data | grep -i --color split查看，结果也没发现正在被split的Region

（2）到HDFS上查看此region是否有数据，结果是有数据的(若无数据则进一步处理)

\[hdfs@hbase-master1 yarn\]$ hdfs dfs -ls -R /apps/hbase/data/data/ks\_namespace/ | grep --color 6f6f4f8204ccaa0b71df08591de155ba  
drwxr-xr-x - hbase hdfs 0 2017-12-20 03:23 /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba  
\-rw-r--r-- 3 hbase hdfs 105 2017-12-10 23:11 /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba/.regioninfo  
drwxr-xr-x - hbase hdfs 0 2017-12-19 05:26 /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba/cf  
\-rw-r--r-- 3 hbase hdfs 386450467 2017-12-19 05:26 /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba/cf/cfaa112cc5f449e0890411a35966825d  
drwxr-xr-x - hbase hdfs 0 2017-12-22 21:11 /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba/recovered.edits  
\-rw-r--r-- 3 hbase hdfs 0 2017-12-22 21:11 /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba/recovered.edits/206932.seqid  
  
  
（3）检查hadoop fsck检查是否有损坏的文件块，结果是健康的(若不健康则在进一步处理)  
\[hdfs@hbase-master1 yarn\]$ hadoop fsck /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba  
DEPRECATED: Use of this script to execute hdfs command is deprecated.  
Instead use the hdfs command for it.  
  
Connecting to namenode via http://hbase-master1.ksc.com:50070/fsck?ugi=hdfs&path=%2Fapps%2Fhbase%2Fdata%2Fdata%2Fks\_namespace%2Factive\_users%2F6f6f4f8204ccaa0b71df08591de155ba  
FSCK started by hdfs (auth:SIMPLE) from /172.31.0.10 for path /apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba at Thu Jan 25 09:33:56 CST 2018  
...Status: HEALTHY  
 Total size:  386450572 B  
 Total dirs:  3  
 Total files:  3  
 Total symlinks:  0  
 Total blocks (validated):  4 (avg. block size 96612643 B)  
 Minimally replicated blocks:  4 (100.0 %)  
 Over-replicated blocks:  0 (0.0 %)  
 Under-replicated blocks:  0 (0.0 %)  
 Mis-replicated blocks:  0 (0.0 %)  
 Default replication factor:  3  
 Average block replication:  3.0  
 Corrupt blocks:  0  
 Missing replicas:  0 (0.0 %)  
 Number of data-nodes:  8  
 Number of racks:  1  
FSCK ended at Thu Jan 25 09:33:56 CST 2018 in 2 milliseconds  
  
The filesystem under path '/apps/hbase/data/data/ks\_namespace/active\_users/6f6f4f8204ccaa0b71df08591de155ba' is HEALTHY  
  
  
【解决】尝试删除zookeeper下的此region节点  
\# 进到zkCli，删掉RIT节点region-in-transition下的区域名  
\[zk: localhost:2181(CONNECTED) 3\] ls /hbase-unsecure/region-in-transition  
\[6f6f4f8204ccaa0b71df08591de155ba\]  
\[zk: localhost:2181(CONNECTED) 4\] rmr /hbase-unsecure/region-in-transition/6f6f4f8204ccaa0b71df08591de155ba  
\[zk: localhost:2181(CONNECTED) 5\] ls /hbase-unsecure/region-in-transition   
\[\]  
\[zk: localhost:2181(CONNECTED) 6\] ls /hbase-unsecure/region-in-transition  
\[\]  
  
\#  重启[Active HBase Master][]之后(master会重建RIT和region信息)，再到 hbase shell验证一下，结果正常显示  
hbase(main):007:0> scan 'ks\_namespace:active\_users', LIMIT => 2  
ROW COLUMN+CELL   
 (null) column=cf:confidence, timestamp=10000, value=1.0   
 (null) column=cf:label, timestamp=10000, value=\\xE4\\xB8\\xAD\\xE5\\x9B\\xBD,\\xE4\\xB8\\x8A\\xE6\\xB5\\xB7,\\xE4  
 \\xB8\\x8A\\xE6\\xB5\\xB7,,\\xE7\\xA7\\xBB\\xE5\\x8A\\xA8   
 \---- column=cf:confidence, timestamp=10000, value=1.0   
 \---- column=cf:label, timestamp=10000, value=\\xE4\\xB8\\xAD\\xE5\\x9B\\xBD,\\xE8\\xBE\\xBD\\xE5\\xAE\\x81,\\xE8  
 \\x90\\xA5\\xE5\\x8F\\xA3,,\\xE8\\x81\\x94\\xE9\\x80\\x9A

2 row(s) in 0.2300 seconds

【延伸】

具体的RIT产生机制可参考这哥们的博文：http://blog.csdn.net/LW\_GHY/article/details/60780065

[Active HBase Master]: http://120.92.115.148:28291/#