Scala操作Hbase空指针异常java.lang.NullPointerException处理

布满荆棘的人生 2022-05-08 11:14 332阅读 0赞

> Hbase版本：Hortonworks Hbase 1.1.2

--------------------

### 问题描述： ###

使用Scala操作Hbase时，发生空指针异常（java.lang.RuntimeException: java.lang.NullPointerException），异常如下：

18/09/13 20:28:51 ERROR Executor: Exception in task 1.0 in stage 361.0 (TID 18925)
    java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
        at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
        at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1255)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1161)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1449)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1052)
        at scala.OperateMysqlTable$$anonfun$main$1$$anonfun$apply$3$$anonfun$apply$4.apply(AccompanyAnalyze.scala:177)
    	at scala.OperateMysqlTable$$anonfun$main$1$$anonfun$apply$3$$anonfun$apply$4.apply(AccompanyAnalyze.scala:161)
        at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
        at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1954)
    	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1954)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:99)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:399)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:552)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1191)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1158)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
        ... 22 more

--------------------

### 问题分析： ###

该异常为客户端去zookeeper获取Hbase相关信息时，没有找到，抛出空指针异常

--------------------

### 问题原因： ###

不同发行商的hbase的在zookeeper的znode设置不同  
Hortonworks的hbase在zookeeper中znode路径为/hbase-unsecure  
![在这里插入图片描述][e7af24812db8da70f305fa90df66a5b5.png]  
查看该路径下hbase的相关信息

[zk: localhost:2181(CONNECTED) 1] ls /hbase-unsecure
    [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, region-in-transition, online-snapshot, acl, switch, master, running, recovering-regions, draining, namespace, hbaseid, table]

--------------------

### 解决办法： ###

我使用的是Hortonworks的Hbase，所以要在Hbase的conf中添加如下配置信息

conf.set("zookeeper.znode.parent", "/hbase-unsecure")

如果是Cloudera的Hbase，添加配置信息如下：

conf.set("zookeeper.znode.parent", "/hbase")

添加后，重新执行程序，Hbase表创建成功，问题解决

**整个Hbase测试代码如下：**

object HbaseTest {
      def main(args: Array[String]): Unit = {
        val conf = HBaseConfiguration.create()
        conf.set("hbase.zookeeper.quorum", "192.168.1.45,192.168.1.46,192.168.1.47")
        conf.set("hbase.zookeeper.property.clientPort", "2181")
        conf.set("zookeeper.znode.parent", "/hbase-unsecure")   //新增znode配置
        conf.set("hbase.master", "192.168.1.47:16000")
        val connection = ConnectionFactory.createConnection(conf)
        val tableName = "house_info"
        val admin = connection.getAdmin
        def createTable(tableName: String, colFamily: List[String]): Unit = {
          if(admin.tableExists(TableName.valueOf(tableName))){
            println(tableName+"已存在！")
          }else
          {
            val descriptor = new HTableDescriptor(TableName.valueOf(tableName))
            colFamily.foreach(x => descriptor.addFamily(new HColumnDescriptor(x)))
            admin.createTable(descriptor)
            println(tableName+"创建成功")
            admin.close()
          }
        }
        val colFamilyList=List("xiaoqu_info", "house_info")
        createTable(tableName,colFamilyList)
      }
    }

[e7af24812db8da70f305fa90df66a5b5.png]: /images/20220508/6644333f1f20455493b2a84f6df9cb63.png