改写Cassandra自带的SSTableExport类相关报错信息及解决办法

Dear 丶 2021-09-28 15:58 192阅读 0赞

下面这些错误是在改写Cassandra自带的SSTableExport类的过程中遇到的错误信息及解决办法，相关代码在GitHub上，有兴趣的话可以查看，地址：[https://github.com/WanZhang1/SSTableExport][https_github.com_WanZhang1_SSTableExport]

### 1、加载schema ###

Exception in thread "main" java.lang.RuntimeException: Attempting to load already loaded column family user_train.return_record
    	at org.apache.cassandra.config.Schema.load(Schema.java:396)
    	at org.apache.cassandra.config.Schema.load(Schema.java:120)
    	at org.apache.cassandra.config.Schema.load(Schema.java:105)
    	at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:537)
    	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:250)
    解决办法：
     DatabaseDescriptor.loadSchemas();// 加载schema
    这句代码放到循环外面

### 2、hdfs拒绝连接 ###

Exception in thread "main" java.net.ConnectException: Call From node2/192.168.3.102 to 192.168.3.7:8020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    Caused by: java.net.ConnectException: 拒绝连接
    解决办法：
    修改hdfs端口8020为9000

### 3、连接HA集群hadoop报错 ###

Exception in thread "main" java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; Host Details : local host is: "bogon/127.0.0.1"; destination host is: "192.168.3.7":9000;
    Caused by: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length
    
    解决办法：
    确认Hadoop中的active节点地址

### 4、jdk版本问题 ###

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/conf/Configuration : Unsupported major.minor version 52.0
    	at java.lang.ClassLoader.defineClass1(Native Method)
    	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
    	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
    	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    	at org.apache.cassandra.tools.SSTableExport2.getOutputStream(SSTableExport2.java:260)
    	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:240)
    	解决办法：
    	jdk版本的问题，1.6不行
    	换成1.8 ok

### 5、写Hdfs报错 ###

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Ticker
    	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:336)
    	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:303)
    	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1827)
    	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:709)
    	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:659)
    	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:570)
    	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3538)
    	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3528)
    	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3370)
    	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
    	at org.apache.cassandra.tools.SSTableExport2.getOutputStream(SSTableExport2.java:261)
    	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:240)
    Caused by: java.lang.ClassNotFoundException: com.google.common.base.Ticker
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 12 more

解决办法：
    	<dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>13.0-rc1</version>
    </dependency>

### 6、写hdfs报错 ###

Exception in thread "main" org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
    	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
    	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
    	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
    	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
    	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
    	at org.apache.cassandra.tools.SSTableExport2.getOutputStream(SSTableExport2.java:261)
    	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:240)
    	解决办法：
    	 <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>3.1.1</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>3.1.1</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-hdfs</artifactId>
                <version>3.1.1</version>
            </dependency>

### 7、snappy错误 ###

java.lang.reflect.InvocationTargetException
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
    	at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
    	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
    	at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:46)
    	at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:56)
    	at org.apache.cassandra.io.compress.SnappyCompressor.<clinit>(SnappyCompressor.java:38)
    	at org.apache.cassandra.config.CFMetaData.<clinit>(CFMetaData.java:76)
    	at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79)
    	at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:441)
    	at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:117)
    	at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:394)
    Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
    	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    	at java.lang.System.loadLibrary(System.java:1122)
    	at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
    	... 15 more
    Exception in thread "main" java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: SnappyCompressor.create() threw an error: java.lang.NoClassDefFoundError Could not initialize class org.xerial.snappy.Snappy
    	at org.apache.cassandra.config.CFMetaData.fromSchemaNoColumns(CFMetaData.java:1234)
    	at org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1247)
    	at org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:294)
    	at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:275)
    	at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:159)
    	at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:537)
    	at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:394)
    Caused by: org.apache.cassandra.config.ConfigurationException: SnappyCompressor.create() threw an error: java.lang.NoClassDefFoundError Could not initialize class org.xerial.snappy.Snappy
    	at org.apache.cassandra.io.compress.CompressionParameters.createCompressor(CompressionParameters.java:131)
    	at org.apache.cassandra.io.compress.CompressionParameters.<init>(CompressionParameters.java:62)
    	at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:55)
    	at org.apache.cassandra.config.CFMetaData.fromSchemaNoColumns(CFMetaData.java:1224)
    	... 6 more
    Caused by: java.lang.reflect.InvocationTargetException
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.cassandra.io.compress.CompressionParameters.createCompressor(CompressionParameters.java:108)
    	... 9 more
    Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    	at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:46)
    	... 14 more

解决办法：
    增加运行参数
    -Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib -Dorg.xerial.snappy.tempdir=/tmp

### 8、其他 ###

Cassandra自带的export类依赖Cassandra数据库，所以需要cassandra.yaml配置文件，如果指定的数据文件不属于当前数据库，会报错 The provided column family is not part of this cassandra database: keysapce = %s, column family = %s

### 9、jar包冲突 ###

遇到jar包冲突的问题，spring boot(2.1.4.RELEASE)依赖 snakeyaml 1.23，cassandra(1.1.5)依赖 snakeyaml 1.6，并且高版本不兼容低版本的，去掉哪个版本都会报错

解决办法：

spring boot需要用来加载yml配置文件，Cassandra需要用来加载yaml配置文件，尝试了不同版本的snakeyaml 都不行，甚至降低spring boot的版本也不行，降太低的话代码本身也报错了，最后取折中办法，把springboot的配置文件改为properties文件方式，问题得到解决。

[https_github.com_WanZhang1_SSTableExport]: https://github.com/WanZhang1/SSTableExport