据说hive-0.14可以进行insert,update操作了,于是决定先下载下来体验一把。
开始之前先部署一个hadoop-2.4.0伪分布。怎么部署网上一堆
第一步:获取源码
从github上把hive clone下来:https://github.com/apache/hive.git。
注意:git获取的是目前正在更新的最新的分支,直接编译的话问题很多,有可能编译不通过。
所以在clone完hive后记得先同步切换成branch-0.14分支再进行编译
第二步:编译
进入hive主目录,运行:
mvn clean install -U -Dmaven.test.skip=true -Phadoop-2,dist
打包:
mvn package -Dmaven.test.skip=true
会在packaging文件夹下生成目标文件夹:apache-hive-0.14.0-SNAPSHOT-bin,这个就是我们要部署的hive文件夹
第三步:配置hive
进入hive/conf,配置conf
cp hive-env.sh.template hive-env.sh
cp hive-default.xml.template hive-site.xml
cp hive-log4j.properties.template hive-log4j.properties
--------------------------------------------------------
1)编辑hive-env.sh(配置mysql来存储元数据)
# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/home/hadoop/hadoop-2.4.0
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/home/hadoop/hive-0.14/conf
# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/home/hadoop/hive-0.14/lib
--------------------------------------------------------
2)编辑hive-site.xml
#设置存储元数据的mysql地址
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.2.48:3306/hive?createDatabaseIfNoExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
#设置jdbc驱动
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
#设置mysql访问用户
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>Username to use against metastore database</description>
</property>
#设置mysql访问密码
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
<description>password to use against metastore database</description>
</property>
然后在mysql中建立hive数据库,以及hive用户,
注意:建立的hive数据库必须为latin1字符集,不然会报错,如果这步没有设,后面也可以改
--------------------------------------------------------
3)编辑/etc/profile
export HIVE_HOME=/home/hadoop/hive-0.14
PATH=$PATH:/sbin:/usr/sbin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$HADOOP_HOME/bin:$SQOOP_HOME/bin:$HIVE_HOME/bin
然后:source /etc/profile
第四步:准备就绪,启动hive
hive --service metastore &
hive --service hiveserver2 &
启动报错:
Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
... 55 more
mysql的jdbc包没有放到lib下
建表的时候报错:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: An exception was thrown while adding/validating class(es) : Specified key was too long; max key length is 767 bytes
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.Util.getInstance(Util.java:384)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
这个就是之前说的,在mysql中建hive数据库的时候没有设置字符集,把它改成latin1就没问题了
alter database hive character set latin1;
下篇测试insert、update、delete操作