hive 单节点环境搭建指南 1.2.1版本

1、本文讲述在基于hadoop2.6.0 基础上搭建hive，关于hadoop2.6.0集群搭建，详见我另一片文章。

      linux环境为centos，数据库版本为mysql5.6，关于mysql在linux下搭建，详见我另一片文章。

2、步骤。

      1、检查hadoop是否启动，注意此处没有通过yarn去配置，也没有配置zookeeper。

            进入hadoop文件下，启动hadop，通过jps查看。正常如下图：

      

           执行：hdfs dfs -ls /    命令，查看文件系统是否正常，如果报错，显示连接异常，注意，文件系统可能已经损坏，需要重新格式化。

      2、下载hive文件，进行解压缩。

           tar zxf apache-hive-1.2.1-bin.tar.gz

mv apache-hive-1.2.1-bin hive-1.2.1

3、修改系统环境变量，vim /etc/profile

export HIVE_HOME=/usr/local/hadoop/hive-1.2.1 

PATH=${PATH}:${HIVE_HOME}/bin

4、创建hive-env.sh文件

cp hive-env.sh.template hive-env.sh

5、修改hive配置文件，hive-site.xml

注意，默认没有hive-site.xml，需要通过模板去创建

#cp hive-default.xml.template hive-site.xml
#vim hive-site.xml
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive</value>
    <description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>数据库用户名</value>
    <description>Username to use against metastore database</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>数据库密码</value>
    <description>password to use against metastore database</description>
</property>

#如果不配置下面的部分会产生错误1.
<property>
    <name>hive.exec.local.scratchdir</name>
    <value>自定义目录</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>自定义目录</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
    <name>hive.querylog.location</name>
    <value>自定义目录</value>
    <description>Location of Hive run time structured log file</description>
</property>
<property>
   <name>hive.server2.logging.operation.log.location</name>
    <value>自定义目录/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
</property>

    6、日志文件配置

     <span class="hljs-variable" style="box-sizing: border-box;"></span><pre name="code" class="html">#cp hive-log4j.properties.template hive-log4j.properties
#vim hive-log4j.properties
hive.log.dir=自定义目录/log/

   7、在HDFS上建立/tmp和/user/hive/warehouse目录，并赋予组用户写权限。

   HADOOP_HOME/bin/hadoop fs -<span class="hljs-keyword" style="box-sizing: border-box;">mkdir</span>       <pre name="code" class="html">HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

8、mysql配置

   <pre name="code" class="html">#创建数据库
mysql> create database hive;
#赋予访问权限
mysql> grant all privileges on hive.* to root@localhost identified by '密码' with grant option;
mysql> flush privileges;
#将JDBC复制到Hive库目录用于java程序与mysql的连接
cp mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar /usr/local/apache-hive-1.1.0-bin/lib/

注意：通过客户端去创建数据库时候，编码格式选择latin，否则表创建长度会超过768byte,报错。

   9、删除hadoop 中的jline-0.94.jar  或者其他版本的雷同，否则会冲突

  <pre name="code" class="html">1.Delete jline from the Hadoop lib directory (it's only pulled in transitively from ZooKeeper).
2.export HADOOP_USER_CLASSPATH_FIRST=true

  10、启动

     hive

</pre><pre code_snippet_id="1868197" snippet_file_name="blog_20160906_13_2737496" name="code" class="html">  报错信息：<pre name="code" class="html">Logging initialized using configuration in file:/usr/local/apache-hive-1.1.0-bin/conf/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:472)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    at org.apache.hadoop.fs.Path.initialize(Path.java:206)
    at org.apache.hadoop.fs.Path.<init>(Path.java:172)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:515)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:458)
    ... 8 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    at java.net.URI.checkPath(URI.java:1804)
    at java.net.URI.<init>(URI.java:752)
    at org.apache.hadoop.fs.Path.initialize(Path.java:203)
    ... 11 more

解决办法：

    <a target=_blank href="http://stackoverflow.com/questions/28536340/hive-shell-not-opening-when-i-have-hive-site-xml" style="box-sizing: border-box;">hive shell not opening when I have hive-site.xml</a>

 错误2：

   <pre name="code" class="html">Logging initialized using configuration in file:/usr/local/apache-hive-1.1.0-bin/conf/hive-log4j.properties
[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.TerminalFactory.create(TerminalFactory.java:101)
    at jline.TerminalFactory.get(TerminalFactory.java:158)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
    at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:773)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:715)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:230)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
    at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:773)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:715)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

解决办法：

    Hive has upgraded <span class="hljs-keyword" style="box-sizing: border-box;">to</span> Jline2 <span class="hljs-keyword" style="box-sizing: border-box;">but</span> jline <span class="hljs-number" style="box-sizing: border-box;">0.94</span> exists <span class="hljs-keyword" style="box-sizing: border-box;">in</span> <span class="hljs-keyword" style="box-sizing: border-box;">the</span> Hadoop lib.

  删除Jline相关jar，在hadoop安装目录中。

错误3：

<pre name="code" class="html"><pre name="code" class="html">hive create table Execution Error,return code 1,we do not suport retries at...

 这个问题是由于<span style="font-family:Source Code Pro, monospace;color:#3f3f3f;"><span style="box-sizing: border-box; padding: 2px 4px; font-size: 12.6px; white-space: nowrap; border-radius: 0px; line-height: 26px; background-color: rgba(128, 128, 128, 0.0745098);">hive</span></span>的元数据存储<a target=_blank href="http://lib.csdn.net/base/14" class="replace_word" title="MySQL知识库" target="_blank" style="box-sizing: border-box; padding: 2px 4px; border-radius: 0px;">MySQL</a><span style="box-sizing: border-box; padding: 2px 4px; border-radius: 0px;">配置不当引起的，可以这样解决：</span>

  alter database hive <span class="hljs-property" style="box-sizing: border-box;">character</span> <span class="hljs-keyword" style="box-sizing: border-box;">set</span> latin1;

 注意，不能用utf字符集，长度会超过限制。

最后：测试

      CREATE TABLE poke(koo INT)；

      SHOW TABLES;

      可以看到结果。

Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql)

新的版本 可能会存在hive 初始化存储数据未初始化，需要先执行

schematool -initSchema   -dbType mysql 

在执行 ./hive