编译hadoop

最新推荐文章于 2021-02-15 23:05:34 发布

flyingqr

最新推荐文章于 2021-02-15 23:05:34 发布

阅读量2k

点赞数

分类专栏： hadoop hive

hadoop hive 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp

$ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse

$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp

$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

出现报错：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

原因是hadoop从网站下载时，是32bit，而系统是64bit，需要下载源码本地编译，然后将编译好的库放入到环境中

源码安装hadoop

mvn package -Pdist,native -DskipTests -Dtar

HIVE debug模式

./hive -hiveconf hive.root.logger=DEBUG,console

Hadoop debug模式

export HADOOP_ROOT_LOGGER=DEBUG,console

执行hive出错：

14/05/29 11:03:57 [main]: ERROR metastore.MetaStoreDirectSql: Self-test query [select "DB_ID" from "DBS"] failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" from "DBS"".
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)

死活找不到问题，换了HIVE版本，好了

从 apache-hive-0.13.0-bin该文apache-hive-0.12.0-bin

启动 hive --service hwi

报错

FATAL hwi.HWIServer: HWI WAR file not found at /usr/local/hadoop/hive/usr/local/hadoop/hive/lib/hive-hwi-0.12.0.war

在hive-site.xml中写入：

<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.12.0.war</value>
<description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </descr
iption>
</property>

浏览器连接：

http://10.58.49.255:9999/hwi/

启动php hive

hive --service hiveserver >/dev/null 2>/dev/null &

spawn-fcgi -a 127.0.0.1 -p 3334 -C 10 -u www-data -f /usr/bin/php-cgi

jps 显示进程

hdfs getconf -namenodes

显示namenodes

编译hadoop java源码，替换

share/hadoop/common/hadoop-common-2.4.0.jar

问题：

Problem accessing /hwi/. Reason:

    No Java compiler available

cp lib/ant.jar hive/lib/

cp lib/ant-launcher.jar hive/lib/

cp /usr/lib/jvm/java-6-oracle/lib/tools.jar /usr/lib/jvm/java-6-oracle/jre/lib/

    Unable to find a javac compiler;
com.sun.tools.javac.Main is not on the classpath.
Perhaps JAVA_HOME does not point to the JDK.
It is currently set to "/usr/lib/jvm/java-6-oracle/jre"

cp /usr/lib/jvm/java-6-oracle/lib/tools.jar lib/

<name>dfs.http.address</name>
<value>bspspo:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>bspspo:50090</value>
</property>
<property>
<name>dfs.namenode.backup.address</name>
<value>bspspo:9002</value>
</property>
<property>
<name>dfs.namenode.backup.http-address</name>
<value>bspspo:9003</value>
</property>