org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test

绝地灬酷狼 2022-06-01 01:08 359阅读 0赞

执行命令`create 'iw:test11',{NAME=>'i',VERSIONS=>1,COMPRESSION=>'SNAPPY'}`  
hbase提示下面的异常

2018-01-16 14:07:43,308 INFO  [RS_OPEN_REGION-dashuju172:16020-0] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 2c52b5a2d05ba1abe8503c904c1f54d9, NAME => 'iw:test11,,1516082862364.2c52b5a2d05ba1abe8503c904c1f54d9.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 4
    2018-01-16 14:07:43,377 INFO  [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open iw:test11,,1516082862364.2c52b5a2d05ba1abe8503c904c1f54d9.
    2018-01-16 14:07:43,398 ERROR [RS_OPEN_REGION-dashuju172:16020-1] handler.OpenRegionHandler: Failed open of region=iw:test11,,1516082862364.2c52b5a2d05ba1abe8503c904c1f54d9., starting to roll back the global memstore size.
    org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test.
    	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:91)
    	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6300)
    	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6251)
    	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6218)
    	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6189)
    	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6145)
    	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6096)
    	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
    	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
    	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    	at java.lang.Thread.run(Thread.java:745)

如果我执行`create 'iw:test12',{NAME=>'i',VERSIONS=>1}`,日志是这样的

2018-01-16 14:08:18,432 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=455.85 KB, freeSize=370.80 MB, max=371.25 MB, blockCount=10, accesses=190, hits=180, hitRatio=94.74%, , cachingAccesses=190, cachingHits=180, cachingHitsRatio=94.74%, evictions=209, evicted=0, evictedPerRun=0.0
    2018-01-16 14:09:55,773 INFO  [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.RSRpcServices: Open iw:test12,,1516082994955.18e0a7886db8f6a37f55a1c29f1f34fd.
    2018-01-16 14:09:55,814 INFO  [StoreOpener-18e0a7886db8f6a37f55a1c29f1f34fd-1] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=10, currentSize=466792, freeSize=388817048, maxSize=389283840, heapSize=466792, minSize=369819648, minFactor=0.95, multiSize=184909824, multiFactor=0.5, singleSize=92454912, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=true, prefetchOnOpen=false
    2018-01-16 14:09:55,815 INFO  [StoreOpener-18e0a7886db8f6a37f55a1c29f1f34fd-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000
    2018-01-16 14:09:55,863 INFO  [RS_OPEN_REGION-dashuju172:16020-2] regionserver.HRegion: Onlined 18e0a7886db8f6a37f55a1c29f1f34fd; next sequenceid=2
    2018-01-16 14:09:55,868 WARN  [sync.0] wal.FSHLog: Too many consecutive RollWriter requests, it's a sign of the total number of live datanodes is lower than the tolerable replicas.
    2018-01-16 14:09:55,870 INFO  [PostOpenDeployTasks:18e0a7886db8f6a37f55a1c29f1f34fd] regionserver.HRegionServer: Post open deploy tasks for iw:test12,,1516082994955.18e0a7886db8f6a37f55a1c29f1f34fd.
    2018-01-16 14:09:55,875 INFO  [PostOpenDeployTasks:18e0a7886db8f6a37f55a1c29f1f34fd] hbase.MetaTableAccessor: Updated row iw:test12,,1516082994955.18e0a7886db8f6a37f55a1c29f1f34fd. with server=dashuju172,16020,1516080795128

执行`hadoop checknative`,snappy是支持的。

18/01/16 14:23:42 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
    18/01/16 14:23:42 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
    Native library checking:
    hadoop:  true /home/hadoop/application/hadoop-2.6.4/lib/native/libhadoop.so.1.0.0
    zlib:    true /lib64/libz.so.1
    snappy:  true /home/hadoop/application/hadoop-2.6.4/lib/native/libsnappy.so.1
    lz4:     true revision:99
    bzip2:   false 
    openssl: true /usr/lib64/libcrypto.so

按照[Hbase 笔记(11) 性能优化][Hbase _11_]中在hbase-site.xml添加

<property>
         <name>hbase.regionserver.codecs</name>
         <value>snappy</value>
    </property>

hbase的异常如下：

2018-01-16 14:34:41,241 WARN  [main] util.CompressionTest: Can't instantiate codec: snappy
    org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
    	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
    	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:68)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:690)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:499)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2633)
    	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)
    	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
    	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2650)
    Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
    	at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
    	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
    	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
    	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
    	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
    	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:303)
    	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
    	... 13 more
    2018-01-16 14:34:41,244 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
    java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2635)
    	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)
    	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
    	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2650)
    Caused by: java.lang.reflect.InvocationTargetException
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2633)
    	... 5 more
    Caused by: java.io.IOException: Compression codec snappy not supported, aborting RS construction
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:691)
    	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:499)
    	... 10 more

最终找到了解决办法，在$HBASE\_HOME/lib/native中创建软链接

ln -s /home/hadoop/application/hadoop/lib/native Linux-amd64-64

并且注意环境变量中添加

export HBASE_LIBRARY_PATH=/home/hadoop/application/hbase/lib/native/Linux-amd64-64

[Hbase _11_]: http://blog.csdn.net/lihui6636/article/details/41119915