Hbase在hdfs上的archive目录占用空间过大

缺乏、安全感 2022-05-27 07:27 875阅读 0赞

hbase版本：1.1.2 hadoop版本：2.7.3

Hbase在hdfs上的目录/apps/hbase/data/archive占用空间过大，导致不停地发出hdfs空间使用率告警。  
  
【问题】  
告警信息 alert: datanode\_storage is triggered 告警信息表明某个或某些data node 的HDFS存储空间使用率已超过阈值(我们设置的是80%)，需要清理。  
\[hdfs@master-2 root\]$ hdfs dfs -du -h /apps/hbase/data/archive/data/  
19.1 M /apps/hbase/data/archive/data/default  
12.6 T /apps/hbase/data/archive/data/good\_namespace\# 此目录占用过多空间  
\[hdfs@master-2 root\]$ hdfs dfs -du -h /apps/hbase/data/archive/data/default  
19.1 M /apps/hbase/data/archive/data/default/KYLIN\_YEDCQ82BF3  
\[hdfs@master-2 root\]$ hdfs dfs -du -h /apps/hbase/data/archive/data/good\_namespace  
4.8 M /apps/hbase/data/archive/data/good\_namespace/url\_history\_30  
1.3 M /apps/hbase/data/archive/data/good\_namespace/url\_history\_7  
5.8 M /apps/hbase/data/archive/data/good\_namespace/user\_statistic  
12.6 T /apps/hbase/data/archive/data/good\_namespace/users\# 是这张表占用了过多空间  
90.8 M /apps/hbase/data/archive/data/good\_namespace/weekly\_stat  
23.5 G /apps/hbase/data/archive/data/good\_namespace/android\_active\_user\_info  
  
【分析】  
查遍了HDFS集群上所有可能发生数据臃肿的地方，例如oldWALs、.Trash，并清理了相关的文件，效果甚微。  
后来想到会不会是因为备份数据的归档导致的空间占用，于是去查看hbase相关的表做的快照snapshot,果然有一张大表的快照。  
  
【解决】  
清理占用空间最多的表快照，只保留最新的，删除旧的  
\# 查询发现/apps/hbase/data/archive/data目录下的每个子目录分别对应着hbase表的快照  
hbase(main):002:0> list\_snapshots  
SNAPSHOT TABLE + CREATION TIME  
KYLIN\_YEDCQ82BF3\_snapshot\_20180315 KYLIN\_YEDCQ82BF3 (Thu Mar 15 18:19:56 +0800 2018)  
url\_history\_30-snapshot\_20180313 good\_namespace:url\_history\_30 (Tue Mar 13 16:41:54 +0800 2018)  
url\_history\_7-snapshot\_20180313 good\_namespace:url\_history\_7 (Tue Mar 13 17:07:04 +0800 2018)  
user\_statistic\_Snapshot\_20180209 good\_namespace:user\_statistic (Fri Feb 09 18:01:21 +0800 2018)  
user\_statistic\_snapshot\_20180313 good\_namespace:user\_statistic (Tue Mar 13 16:36:06 +0800 2018)  
users\_snapshot\_20180209 good\_namespace:users (Fri Feb 09 17:26:51 +0800 2018)  
users\_snapshot\_20180313 good\_namespace:users (Tue Mar 13 15:39:20 +0800 2018)  
users\_snapshot\_20180408 good\_namespace:users (Sun Apr 08 16:16:32 +0800 2018)  
weekly\_stat\_snapshot\_20180313 good\_namespace:weekly\_stat (Tue Mar 13 16:17:33 +0800 2018)  
android\_active\_user\_info\_Snapshot\_20180212 good\_namespace:android\_active\_user\_info (Mon Feb 12 11:35:33 +0800 2018)  
10 row(s) in 0.2800 seconds  
  
=> \["KYLIN\_YEDCQ82BF3\_snapshot\_20180315", "url\_history\_30-snapshot\_20180313", "url\_history\_7-snapshot\_20180313", "user\_statistic\_Snapshot\_20180209", "user\_statistic\_snapshot\_20180313", "users\_snapshot\_20180209", "users\_snapshot\_20180313", "users\_snapshot\_20180408", "weekly\_stat\_snapshot\_20180313", "android\_active\_user\_info\_Snapshot\_20180212"\]  
hbase(main):003:0>  
  
\# 先删除旧的快照，再创建最新的快照，只保留最新的快照  
hbase(main):002:0> list\_snapshots  
SNAPSHOT TABLE + CREATION TIME  
KYLIN\_YEDCQ82BF3\_snapshot\_20180315 KYLIN\_YEDCQ82BF3 (Thu Mar 15 18:19:56 +0800 2018) url\_history\_30\_snapshot\_20180412 good\_namespace:url\_history\_30 (Thu Apr 12 15:05:08 +0800 2018) url\_history\_7-snapshot\_20180412 good\_namespace:url\_history\_7 (Thu Apr 12 15:06:10 +0800 2018)  
user\_statistic\_snapshot\_20180412 good\_namespace:user\_statistic (Thu Apr 12 14:56:33 +0800 2018) users\_snapshot\_20180412 good\_namespace:users (Thu Apr 12 14:59:40 +0800 2018) weekly\_stat\_snapshot\_20180412 good\_namespace:weekly\_stat (Thu Apr 12 15:03:20 +0800 2018)  
android\_active\_user\_info\_20180412 good\_namespace:android\_active\_user\_info (Thu Apr 12 15:02:15 +0800 2018) android\_active\_user\_info\_Snapshot\_20180212 good\_namespace:android\_active\_user\_info (Mon Feb 12 11:35:33 +0800 2018) 8 row(s) in 0.1540 seconds  
  
=> \["KYLIN\_YEDCQ82BF3\_snapshot\_20180315", "url\_history\_30\_snapshot\_20180412", "url\_history\_7-snapshot\_20180412", "user\_statistic\_snapshot\_20180412", "users\_snapshot\_20180412", "weekly\_stat\_snapshot\_20180412", "android\_active\_user\_info\_20180412", "android\_active\_user\_info\_Snapshot\_20180212"\]  
  
\# 再次查看hdfs该目录下的空间占用情况：成功释放掉快照相关文件占用的空间  
\[hdfs@master-2 root\]$ hdfs dfs -du -h /apps/hbase/data/archive/data/  
19.1 M /apps/hbase/data/archive/data/default  
119.5 G /apps/hbase/data/archive/data/good\_namespace\# 此目录占用空间从12.6Tb下降到119.5Gb