用happybase查询hbase时出现ThriftServer报错

快来打我* 2022-05-26 05:54 332阅读 0赞

【运行环境】

java SE 1.8.0\_101，hadoop-2.7.3，HBase-1.1.2，Zookeeper-3.4.6，python2.7，happybase1.1.0

【作业报错信息】

IOError: IOError(message='org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 69992 actions: IOException: 69992 times, \\n\\tat org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:234)\\n\\tat org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:214)\\n\\tat org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1623)\\n\\tat org.apache.hadoop.hbase.client.HTable.batch(HTable.java:909)\\n\\tat org.apache.hadoop.hbase.client.HTable.batch(HTable.java:923)\\n\\tat org.apache.hadoop.hbase.client.HTable.get(HTable.java:884)\\n\\tat org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.getRowsWithColumnsTs(ThriftServerRunner.java:1108)\\n\\tat org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.getRowsWithColumns(ThriftServerRunner.java:1064)\\n\\tat sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\\n\\tat com.sun.proxy.$Proxy11.getRowsWithColumns(Unknown Source)\\n\\tat org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$getRowsWithColumns.getResult(Hbase.java:4266)\\n\\tat org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$getRowsWithColumns.getResult(Hbase.java:4250)\\n\\tat org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\\n\\tat org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\\n\\tat org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)\\n\\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\\n\\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\\n\\tat java.lang.Thread.run(Thread.java:745)\\n')  
Thu Apr 19 06:28:20 2018 <Greenlet at 0x7f9710c08e10: import\_user\_profile(<happybase.table.Table name='all\_users:city\_sales'>, \['P\_2CAD1AC8878528C06AAB6EF859AB6DD8', 'P\_2CAD1B1A, \['basic:pc\_label', 'pref:shopping)> failed with IOError  
  
【日志信息】  
查看\{HBASE\_LOG\_DIR\}/hbase 目录下的日志文件 hbase-root-thrift-YOUR\_HOSTNAME-master-1.log，找到作业报错时间附近的日志消息  
2018-04-19 16:07:16,284 WARN \[thrift-worker-0\] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"  
2018-04-19 16:07:21,456 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:07:21,903 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:07:23,779 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 135571 actions to finish  
2018-04-19 16:07:26,322 WARN \[hconnection-0x26338bf1-shared--pool2-t75\] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"  
2018-04-19 16:07:31,463 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:07:31,913 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:07:33,780 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 118358 actions to finish  
2018-04-19 16:07:34,380 INFO \[thrift-worker-0\] client.AsyncProcess: \#1, waiting for 135571 actions to finish  
2018-04-19 16:07:41,465 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:07:41,922 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:07:43,792 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 118358 actions to finish  
2018-04-19 16:07:44,386 INFO \[thrift-worker-0\] client.AsyncProcess: \#1, waiting for 135571 actions to finish  
2018-04-19 16:07:51,466 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:07:51,928 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:07:53,800 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 118358 actions to finish  
2018-04-19 16:07:54,416 INFO \[thrift-worker-0\] client.AsyncProcess: \#1, waiting for 135571 actions to finish  
2018-04-19 16:08:01,470 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:08:01,934 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:08:03,803 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 52007 actions to finish  
2018-04-19 16:08:04,419 INFO \[thrift-worker-0\] client.AsyncProcess: \#1, waiting for 135571 actions to finish  
2018-04-19 16:08:11,479 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:08:11,940 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:08:13,803 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 52007 actions to finish  
2018-04-19 16:08:14,428 INFO \[thrift-worker-0\] client.AsyncProcess: \#1, waiting for 135571 actions to finish  
2018-04-19 16:08:21,481 INFO \[thrift-worker-3\] client.AsyncProcess: \#1, waiting for 83858 actions to finish  
2018-04-19 16:08:21,943 INFO \[thrift-worker-2\] client.AsyncProcess: \#1, waiting for 65870 actions to finish  
2018-04-19 16:08:23,809 INFO \[thrift-worker-1\] client.AsyncProcess: \#1, waiting for 52007 actions to finish  
  
【日志解读】  
thrift-worker就是thriftServer的线程。  
client.AsyncProcess: \#1, waiting for 65870 actions to finish 是说此客户端在等待进程号65870的作业完成。  
再结合正在运行的大查询作业，发现有一个客户端程序正在 scan 一张Hbase的表。  
可以推测出，出错原因就是thriftServer线程数不足。  
  
【解决】增加thriftServer线程数量  
\# 在两个hmaster都做ThriftServer启动参数配置：最小线程数量300个，最大线程数500个  
cd \{your\_hbase\_home\}/bin && ./hbase-daemon.sh start thrift \--threadpool -m 300 -w 500  
ssh \{hmaster\_ip\} "cd \{your\_hbase\_home\}/bin/ && ./hbase-daemon.sh start thrift \--threadpool -m 300 -w 500"  
\# 检查刚才重启的thrift进程，查看设置是否生效  
\[root@tony-cluster-master-001 bin\]\# ps aux | grep -v grep | grep thrift  
root 27239 0.0 0.0 113132 1556 pts/3 S 11:58 0:00 bash \{your\_hbase\_home\}/bin/hbase-daemon.sh --config \{your\_hbase\_home\}/bin/../conf foreground\_start thrift --threadpool -m 300 -w 500  
root 27253 104 0.7 10324400 244132 pts/3 Sl 11:58 0:04 /usr/java/oracle-jdk/bin/java -Dproc\_thrift -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Dhdp.version=2.6.1.0-129 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/mnt/log/hbase/hs\_err\_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/mnt/log/hbase/gc.log-201804201158 -Dhbase.log.dir=/mnt/log/hbase -Dhbase.log.file=hbase-root-thrift-kmr-a20125dd-gn-a05044c6-master-1-001.ksc.com.log -Dhbase.home.dir=\{your\_hbase\_home\}/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.thrift.ThriftServer \--threadpool -m 300 -w 500 start  
\# 启动参数生效！

重新运行python客户端的作业，成功完成。