hue+docker 安装部署

前期服务

hive相关服务
    bin/hive --service metastore &
    bin/hive --service hiveserver2 &


连接的端口是10000,hiveserver2服务
 <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
    <description>Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'binary'.</description>
  </property>

docker安装部署

基本操作
官网文档
若本来有docker
则先卸载再安装

$ sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

$ sudo yum install -y yum-utils

$ sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

$ sudo yum install docker-ce docker-ce-cli containerd.io

$ yum list docker-ce --showduplicates | sort -r

docker-ce.x86_64  3:18.09.1-3.el7                     docker-ce-stable
docker-ce.x86_64  3:18.09.0-3.el7                     docker-ce-stable
docker-ce.x86_64  18.06.1.ce-3.el7                    docker-ce-stable
docker-ce.x86_64  18.06.0.ce-3.el7                    docker-ce-stable

sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io

$ sudo systemctl start docker

$ sudo docker run hello-world

参考连接

yum install -y yum-utils device-mapper-persistent-data lvm2

源配置

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

查看可安装docker版本

yum list docker-ce.x86_64  --showduplicates | sort -r 

hue安装部署

官方文档

docker run -it -p 8888:8888 gethue/hue:latest

后台运行
docker run -itd -p 8888:8888 gethue/hue:latest(或者用镜像id)
docker images

退出容器
ctrl+p+q

docker stop $CONTAINER_ID
docker start $CONTAINER_ID
docker restart $CONTAINER_ID 

hue使用方法

浏览器访问server:8888
第一次是创建用户名和密码

hue配置hive

进入容器
sudo docker exec -it <containerID> /bin/bash 

退出
exit

1.查看自己文件路径

/usr/share/hue/desktop/conf/hue.ini

2.docker文件挂载

sudo docker exec -it <containerID> /bin/bash 

docker run --name huehue -itd -p 8888:8888 gethue/hue:latest(或者用镜像id)  -v /bigdata/binfile/hue/hue.ini:/usr/share/hue/desktop/conf/hue.ini

第一次尝试报错：
容器一段时间挂掉
查询资料原因为 版本不匹配
参考文档

3.第二方法
修改hue容器内部文件，这个方法不合理，因为原则上容器不应该产生内容

容器没有vi，vim命令，我们用cp覆盖的方式

docker ps -a
docker exec -it xxx /bin/bash
进去容器，查找hue.ini路径
博主路径:/usr/share/hue/desktop/conf/hue.ini

docker cp <containerId>:/usr/share/hue/desktop/conf/hue.ini   /hue/hue.ini

修改

数据源配置
[[[hive]]]
name=Hive
interface=hiveserver2


[beeswax]
hive_server_host=node1
hive_server_port=10000
hive_conf_dir=/usr/hive-2.3.7/conf

覆盖

docker cp  /hue/hue.ini   <containerId>:/usr/share/hue/desktop/conf/hue.ini 

hue出现问题

经过上述操作后，出现

    user_id = _get_user_session_key(request)
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/contrib/auth/__init__.py", line 61, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/contrib/sessions/backends/base.py", line 57, in __getitem__
    return self._session[key]
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/contrib/sessions/backends/base.py", line 207, in _get_session
    self._session_cache = self.load()
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/contrib/sessions/backends/db.py", line 35, in load
    expire_date__gt=timezone.now()
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 374, in get
    num = len(clone)
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 232, in __len__
    self._fetch_all()
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/share/hue/build/env/local/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
OperationalError: database is locked

修改hue.ini

  [[database]]
    # Database engine is typically one of:
    # postgresql_psycopg2, mysql, sqlite3 or oracle.
    #
    # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name.
    # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes.
    # Note for Oracle, you can use the Oracle Service Name by setting "port=0" and then "name=<host>:<port>/<service_name>".
    # Note for MariaDB use the 'mysql' engine.
    engine=mysql
    host=bigdatamaster
    port=3306
    user=hue
    password=123456
    name=hue   【数据库名】

# 新建数据库
mysql -uroot -p
# 输入密码
# 创建 hue数据库
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
# 创建 hue用户   密码 123456   % 允许外网访问（localhost仅服务器访问）
create user 'hue'@'%' identified by '123456';

# 授予 hue用户在外网通过密码对 hue数据库全部权限
grant all privileges on `hue`.* to 'hue'@'%' identified by '123456';

# MySQL 权限放在 cache（grant表）中，所以必须重新重新加载
flush privileges;

进入容器hue初始化

/usr/share/hue/build/env/bin/hue syncdb

/usr/share/hue/build/env/bin/hue migrate

人生总不是一帆风顺

错误

Thrift version configured by property thrift_version might be too high. Request failed with 
"Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, 
username:hue, configuration:{hive.server2.proxy.user=xxx})" (code OPEN_SESSION): None

看翻译意思好像是thrift version太高了,我都以为要降版本了
去查hue.ini

他说试试7就试试7   thrift_version=7

之后重新cp复制过去，重启容器docker restart

最终结果

参考文档，有问题可以查查这里
https://www.jianshu.com/p/087404300cda

---

cannot access hdfs

• 原因1：没有hdfs超级用户权限

修改${hadoop_home}/etc/hadoop/core-site.xml
<property>
<name>hadoop.proxyuser.hue.hosts</name>
  <value>*</value>
 </property>
<property>
<name>hadoop.proxyuser.hue.groups</name>
<value>*</value>
</property>


修改${hadoop_home}/etc/hadoop/hdfs-site.xml

</property>
<property>  <!--设置为true，可以在浏览器中IP+port查看-->
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>

• 端口没有配置正确

针对hadoop-3 
修改${hadoop_home}/etc/hadoop/hdfs-site.xml
<property>
  <name>dfs.namenode.http-address</name>
  <value>0.0.0.0:9870</value>
  <description>
    The address and the base port where the dfs namenode web ui will listen on.
  </description>
</property>



如果是hadoop-2,就无需修改,默认是50070
修改hue.ini

```bash
[hadoop]

  # Configuration for HDFS NameNode
  # ------------------------------------------------------------------------
  [[hdfs_clusters]]
    # HA support by using HttpFs

    [[[default]]]
      # Enter the filesystem uri
      fs_defaultfs=hdfs://xxxxx:2020

      # NameNode logical name.
      ## logical_name=

      # Use WebHdfs/HttpFs as the communication mechanism.
      # Domain should be the NameNode or HttpFs host.
      # Default port is 14000 for HttpFs.(hadoop3就是9870)
       webhdfs_url=http://xxxxxxx:50070/webhdfs/v1

      # Change this if your HDFS cluster is Kerberos-secured
      ## security_enabled=false

个人博客:https://blog.jathamcloud.top/
博主公众号
求关注