Upgrade json-serde-xxx jar in Apache Hive-1.2.1-CSDN博客

1.背景:

在迁移ETL至ALiYun EMR环境，使用我们定制的Hive-1.2.1-emr版本过程中，业务脚本需要使用json-serde-1.1.9.9-SNAPSHOT-jar-with-dependencies.jar

2.抛错如下:

Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe.

http://stackoverflow.com/questions/26644351/cannot-validate-serde-org-openx-data-jsonserde-jsonserde

3.解决版本:

升级json-serde jar为最新版

1.Configure mvn

[root@hadoopnn-01 hadoop]# export MAVEN_HOME=/hadoop/maven
[root@hadoopnn-01 hadoop]# export PATH=$MAVEN_HOME/bin:$PATH

2.Download json-serde source code

[root@hadoopnn-01 hadoop]# mkdir test
[root@hadoopnn-01 hadoop]# cd test
[root@hadoopnn-01 test]# git clone https://github.com/rcongiu/Hive-JSON-Serde.git

3.Compile json-serde

[root@hadoopnn-01 test]# cd Hive-JSON-Serde/
[root@hadoopnn-01 Hive-JSON-Serde]# mvn -Pcdh5 clean package
.................
.................
[INFO] Building jar: /hadoop/test/Hive-JSON-Serde/json-udf/target/json-udf-1.3.8-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] openx-json-serde ................................... SUCCESS [  1.627 s]
[INFO] json ............................................... SUCCESS [  4.308 s]
[INFO] json-cdh4-shim ..................................... SUCCESS [  3.250 s]
[INFO] json-cdh5-shim ..................................... SUCCESS [  2.800 s]
[INFO] json-serde-main .................................... SUCCESS [  6.131 s]
[INFO] json-udf ........................................... SUCCESS [  1.701 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.445 s
[INFO] Finished at: 2016-10-04T11:43:33+08:00
[INFO] Final Memory: 76M/335M
[INFO] ------------------------------------------------------------------------

4.sz json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar

[root@hadoopnn-01 Hive-JSON-Serde]# cd json-serde/target/
[root@hadoopnn-01 target]# ll
total 180
drwxr-xr-x 2 root root  4096 Oct  4 11:43 archive-tmp
drwxr-xr-x 3 root root  4096 Oct  4 11:43 classes
-rw-r--r-- 1 root root 37935 Oct  4 11:43 json-serde-1.3.8-SNAPSHOT.jar
-rw-r--r-- 1 root root 83787 Oct  4 11:43 json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
-rw-r--r-- 1 root root 29997 Oct  4 11:43 json-serde-1.3.8-SNAPSHOT-tests.jar
drwxr-xr-x 2 root root  4096 Oct  4 11:43 maven-archiver
drwxr-xr-x 2 root root  4096 Oct  4 11:43 surefire
drwxr-xr-x 2 root root  4096 Oct  4 11:43 surefire-reports
drwxr-xr-x 4 root root  4096 Oct  4 11:43 test-classes

[root@hadoopnn-01 target]# sz json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
rz
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar...
  100%      81 KB      81 KB/sec    00:00:01       0 Errors