Spark snappy

Spark Snappy 问题

In my experience, if you cd into the /sparkDir/conf and rename the spark-env.sh.template to spark-env.sh, and then set the JAVA_OPTSand hadoop_DIR, it works.

You will also have to edit this /etc/profile line:

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH

hive on spark：
[Error 30017]: Skipping stats aggregation by error org.apache.hadoop.hive.ql.metadata.HiveException: [Error 30001]: StatsPublisher cannot be initialized. There was a error in the initialization of StatsPublisher, and retrying might help. If you dont want the query to fail because accurate statistics could not be collected, set hive.stats.reliable=false
Table tmp.app_fin_assign_debt stats: [numFiles=2, numRows=-1, totalSize=80885949, rawDataSize=-1]

yarn logs 发现：
16/06/12 17:43:14 ERROR jdbc.JDBCStatsPublisher: Error during JDBC connection to jdbc:mysql://127.0.0.1:3306;databaseName=TempStatsStore;create=true.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Cannot load connection class because of underlying exception: 'java.lang.NumberFormatException: For input string: "3306;databaseName=TempStatsStore;create=true"'.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)

连接Hive元数据库 配置hive-site
<property>
<name>hive.stats.jdbcdriver</name>
<value>com.mysql.jdbc.Driver</value>
<description>The JDBC driver for the database that stores temporary hive statistics.</description>
</property>

<property>
<name>hive.stats.dbconnectionstring</name>
<value>jdbc:mysql://******:3306;databaseName=TempStatsStore;create=true</value>
<description>The default connection string for the database that stores temporary hive statistics.</description>
</property>