改写Cassandra自带的SSTableExport类相关报错信息及解决办法

下面这些错误是在改写Cassandra自带的SSTableExport类的过程中遇到的错误信息及解决办法，相关代码在GitHub上，有兴趣的话可以查看，地址：https://github.com/WanZhang1/SSTableExport

 

1、加载schema

Exception in thread "main" java.lang.RuntimeException: Attempting to load already loaded column family user_train.return_record
	at org.apache.cassandra.config.Schema.load(Schema.java:396)
	at org.apache.cassandra.config.Schema.load(Schema.java:120)
	at org.apache.cassandra.config.Schema.load(Schema.java:105)
	at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:537)
	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:250)
解决办法：
 DatabaseDescriptor.loadSchemas();// 加载schema
这句代码放到循环外面

2、hdfs拒绝连接

Exception in thread "main" java.net.ConnectException: Call From node2/192.168.3.102 to 192.168.3.7:8020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
Caused by: java.net.ConnectException: 拒绝连接
解决办法：
修改hdfs端口8020为9000

3、连接HA集群hadoop报错

Exception in thread "main" java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; Host Details : local host is: "bogon/127.0.0.1"; destination host is: "192.168.3.7":9000;
Caused by: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length

解决办法：
确认Hadoop中的active节点地址

4、jdk版本问题

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/conf/Configuration : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	at org.apache.cassandra.tools.SSTableExport2.getOutputStream(SSTableExport2.java:260)
	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:240)
	解决办法：
	jdk版本的问题，1.6不行
	换成1.8 ok

5、写Hdfs报错

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Ticker
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:336)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:303)
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1827)
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:709)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:659)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:570)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3538)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3528)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3370)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
	at org.apache.cassandra.tools.SSTableExport2.getOutputStream(SSTableExport2.java:261)
	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:240)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Ticker
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 12 more

解决办法：
	<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>13.0-rc1</version>
</dependency>

6、写hdfs报错

Exception in thread "main" org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
	at org.apache.cassandra.tools.SSTableExport2.getOutputStream(SSTableExport2.java:261)
	at org.apache.cassandra.tools.SSTableExport2.main(SSTableExport2.java:240)
	解决办法：
	 <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>3.1.1</version>
        </dependency>

7、snappy错误

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
	at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
	at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:46)
	at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:56)
	at org.apache.cassandra.io.compress.SnappyCompressor.<clinit>(SnappyCompressor.java:38)
	at org.apache.cassandra.config.CFMetaData.<clinit>(CFMetaData.java:76)
	at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79)
	at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:441)
	at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:117)
	at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:394)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
	... 15 more
Exception in thread "main" java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: SnappyCompressor.create() threw an error: java.lang.NoClassDefFoundError Could not initialize class org.xerial.snappy.Snappy
	at org.apache.cassandra.config.CFMetaData.fromSchemaNoColumns(CFMetaData.java:1234)
	at org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1247)
	at org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:294)
	at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:275)
	at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:159)
	at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:537)
	at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:394)
Caused by: org.apache.cassandra.config.ConfigurationException: SnappyCompressor.create() threw an error: java.lang.NoClassDefFoundError Could not initialize class org.xerial.snappy.Snappy
	at org.apache.cassandra.io.compress.CompressionParameters.createCompressor(CompressionParameters.java:131)
	at org.apache.cassandra.io.compress.CompressionParameters.<init>(CompressionParameters.java:62)
	at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:55)
	at org.apache.cassandra.config.CFMetaData.fromSchemaNoColumns(CFMetaData.java:1224)
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cassandra.io.compress.CompressionParameters.createCompressor(CompressionParameters.java:108)
	... 9 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
	at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:46)
	... 14 more

解决办法：
增加运行参数
-Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib -Dorg.xerial.snappy.tempdir=/tmp

8、其他

Cassandra自带的export类依赖Cassandra数据库，所以需要cassandra.yaml配置文件，如果指定的数据文件不属于当前数据库，会报错 The provided column family is not part of this cassandra database: keysapce = %s, column family = %s

9、jar包冲突

遇到jar包冲突的问题，spring boot(2.1.4.RELEASE)依赖 snakeyaml 1.23，cassandra(1.1.5)依赖 snakeyaml 1.6，并且高版本不兼容低版本的，去掉哪个版本都会报错

解决办法：

spring boot需要用来加载yml配置文件，Cassandra需要用来加载yaml配置文件，尝试了不同版本的snakeyaml 都不行，甚至降低spring boot的版本也不行，降太低的话代码本身也报错了，最后取折中办法，把springboot的配置文件改为properties文件方式，问题得到解决。