解决：Spark-HBASE Error Caused by: java.lang.IllegalStateException: unread block data

最新推荐文章于 2024-07-22 16:21:11 发布

jxj21

最新推荐文章于 2024-07-22 16:21:11 发布

阅读量5.5k

点赞数

分类专栏： spark hive hadoop habase

本文链接：https://blog.csdn.net/jxj21/article/details/71503979

版权

在Spark-2.1.0-bin-hadoop2.7环境中，通过Spark-SQL连接HBase时遇到'java.lang.IllegalStateException: unread block data'错误。通过在spark-env.sh中添加HIVE_HOME、HBASE_HOME等环境变量，并在hive-site.xml中配置相关jar路径，问题仍未解决。最终，通过在spark-sql命令行使用--jars选项指定相关HBase库解决该问题，避免每次运行时手动输入长串jar路径。这提供了一种更便捷的方式来连接HBase和Spark进行数据操作。

摘要由CSDN通过智能技术生成

参考：1 http://stackoverflow.com/questions/31701273/setting-spark-classpaths-on-ec2-spark-driver-extraclasspath-and-spark-executor

参考：2 http://stackoverflow.com/questions/34901331/spark-hbase-error-java-lang-illegalstateexception-unread-block-data

参考：3 http://blog.csdn.net/wzq294328238/article/details/48054525

1 基本环境：spark-2.1.0-bin-hadoop2.7,apache-hive-1.2.2,Hbase 1.3.1

2 问题描述：通过spark-sql连接hbase出现错误，Caused by: java.lang.IllegalStateException: unread block data。

spark-env.sh:

export HIVE_HOME=/root/apache-hive-1.2.2-bin
export HBASE_HOME=/root/hbase-1.3.1
export SCALA_HOME=/usr/share/scala
export JAVA_HOME=/usr/java/jdk1.7.0_79
export SPARK_MASTER_IP=centos01
export SPARK_WORKER_MEMORY=4G
export HADOOP_CONF_DIR=/root/hadoop-2.7.3/etc/hadoop
export JAVA_OPTS="-server -Xms800m -Xmx800m -XX:PermSize=64M -XX:MaxNewSize=256m -XX:MaxPermSize=128m -Djava.awt.headless=true"
#export SPARK_CLASSPATH=$HIVE_HOME/lib:$SPARK_CLASSPATH
export SPARK_MEM=512M
export LD_LIBRARY_PATH=/root/hadoop-2.7.3/lib/native

hive-site.xml:

<configuration>

<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://centos03:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>

<property>

<name>javax.jdo.option.ConnectionUserName</name>
<value>...</value>
<description>username to use against metastore database</description>
</property>

<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>...</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>file:///root/apache-hive-1.2.2-bin/lib/mysql-connector-java-5.1.41-bin.jar,file:///root/hbase-1.3.1/lib/activation-1.1.jar,file:///root/hbase-1.3.1/lib/aopalliance-1.0.jar,file:///root/hbase-1.3.1/lib/apacheds-i18n-2.0.0-M15.jar,file:///root/hbase-1.3.1/lib/apache