2.Hive系列之docker-compose部署升级总结

1. 基于docker-compose快速部署3.1.2

目前https://github.com/big-data-europe/docker-hive最新的hive版本是2.3.2，无奈，只有基于大佬写的改造升级至3.1.2，或许很快大佬会合并，到时候大家只需要运行docker-compose即可

1.1 克隆代码

# 未合并前可以克隆我的仓库地址
git clone https://github.com/SJshenjian/docker-hive.git
# 合并后克隆地址
git clone https://github.com/big-data-europe/docker-hive.git

1.2 Dockerfile构建镜像(未合并前执行该步骤)

进入到Dockerfile与docker-compose.yml所在目录后，执行以下命令构建镜像bde2020/hive:3.1.2-postgresql-metastore

docker build -t bde2020/hive:3.1.2-postgresql-metastore .

1.3 运行docker-compose.yml文件

运行以下命令启动服务，第一次运行需要下载镜像，耐心等待

docker-compose up -d

1.4 服务验证

加载数据至Hive中并查询

$ docker-compose exec hive-server bash
# /opt/hive/bin/beeline -u jdbc:hive2://
> CREATE TABLE pokes (foo INT, bar STRING);
> LOAD DATA LOCAL INPATH '/opt/hive/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
> SELECT * FROM pokes;

好了，现在可以愉快的学习Hive了，下一章，我将介绍，升级过程中遇到的问题及如何排查解决的

欢迎关注公众号算法小生与我沟通交流

2. ocker-compose部署升级总结

2.1 版本号修改

对于升级而言，我们最先考虑的是docker hub中有的较新的版本，然后我们需要简单了解下hadoop2与hadoop3的区别，首先明确的是端口号有所改变，如下图所示

2.2 Hive镜像构建

刚刚我们修改了Hive为bde2020/hive:3.1.2-postgresql-metastore，仓库中并没有，我们就要考虑修改Dockerfile自行构建镜像了

现在可以尝试构建镜像了,并启动

docker build -t bde2020/hive:3.1.2-postgresql-metastore .

2.3 apt-get update 404 Not Found

哇偶，构建出错

原因：/etc/apt/sources.list中源存在问题，我们修改为目前可用的源即可

RUN sed -i 's/^.*$/deb http:\/\/deb.debian.org\/debian\/ buster main\ndeb-src http:\/\/deb.debian.org\/debian\/ buster main\ndeb http:\/\/deb.debian.org\/debian-security\/ buster\/updates main\ndeb-src http:\/\/deb.debian.org\/debian-security\/ buster\/updates main/g' /etc/apt/sources.list

2.4 entrypoint.sh文件不存在

再次尝试构建镜像了,并启动，发现如下错误

打开entrypoint.sh文件，发现报错，我们按照提示加上注释即可

2.5 版本冲突与格式错误

/opt/hive/conf/hive-env.sh: line 37: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 45: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 46: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 49: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 52: $'\r': command not found
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)

我们将hive-env.sh格式改为LF，unix格式

Dockerfile中加入这两段解决log冲突问题

# solve log version conflict
RUN cp /opt/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar /opt/hive/lib/
RUN rm -rf /opt/hive/lib/guava-19.0.jar

再次构建尝试运行

2.6 url链接问题

看起来日志都很正常了，但是我们执行任何hive操作时，报如下错误:

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

控制台没有输出错误日志，我们查看其他地方是由有hive.log

# find / -name hive.log
/tmp/root/hive.log

查看日志，可以发现，url非法了哎，这是因为docker在默认创建网络的时候，会给添加_default后缀导致url非法

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Got exception: java.net.URISyntaxException Illegal character in hostname at index 49: thrift://docker-hive-hive-metastore-1.docker-hive_default:9083
        at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:168) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:267) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:182) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:94) ~[hive-exec-3.1.2.jar:3.1.2]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_232]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_232]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_232]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_232]
        at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4299) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4367) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4347) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:960) ~[hive-exec-3.1.2.jar:3.1.2]
        ... 16 mor

我们修改docker-compose.yaml，指定网络名称即可

# 版本由3改为3.5
version: "3.5"
# solve java.net.URISyntaxException Illegal character in hostname at index 49: thrift://docker-hive-hive-metastore-1.docker-hive_default:9083
networks:
  default:
    name: docker-hive-default

先down后up，再次执行show databases;正常了哎, 也可以看到网络docker-hive-default

(base) PS D:\Java\新建文件夹\docker-hive> docker network ls
NETWORK ID     NAME                  DRIVER    SCOPE
b53167dc0bb1   bridge                bridge    local
6ae196a5dfd6   docker-hive-default   bridge    local
1582f3dcbd8b   docker-hive_default   bridge    local
3e85deffd6b1   host                  host      local
8740d8865907   none                  null      local

现在即使通过beeline连接也是OK的，完结

欢迎关注公众号算法小生，与我沟通交流