一、Hue的安装部署
- Hue版本:cdh5.3.6-hue-3.7.0
需要编译才能使用
- 需要联网
依赖(针对不同Linux版本)
ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libtidy (for unit tests only) libxml2-devel libxslt-devel mvn (from maven package or maven3 tarball) mysql mysql-devel openldap-devel python-devel sqlite-devel openssl-devel (for version 7+)
- 检查系统有没有这些包
sudo rpm -qa | grep package_name
- yum 安装
sudo yum install ant asciidoc cyrus-sasl-devel gcc gcc-c++ krb5-devel libtidy libxml2-devel libxslt-devel mvn mysql-devel openldap-devel python-devel sqlite-devel openssl-devel -y
- 编译
- 下载解压:
tar -zxvf hue-3.7.0-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/
- 进入hue家目录:
cd /opt/cdh-5.3.6/hue-3.7.0-cdh5.3.6
- 编译:make apps
- 卸载open-jdk
- 查看:
sudo rpm -qa | grep java
- 卸载:
sudo rpm -e --nodeps java_cup-0.10k-5.el6.x86_64 java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 tzdata-java-2016j-1.el6.noarch java-1.7.0-openjdk-1.7.0.121-2.6.8.1.el6_8.x86_64 java-1.7.0-openjdk-devel-1.7.0.121-2.6.8.1.el6_8.x86_64
- 查看:
- 下载解压:
配置
- hue的全局配置文件:desktop/conf/hue.ini
配置文件中所有的标记启用以中括号表示
[desktop] secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn<qW5o # Webserver listens on this address and port http_host=hadoop-senior01.ibeifeng.com http_port=8888 # Time zone name time_zone=Asia/Shanghai
- 使用
- 启动:
build/env/bin/supervisor
- web ui:hadoop-senior01.ibeifeng.com:8888
- 启动:
二、 Hue框架集成
hadoop-HDFS集成
HDFS的配置
hdfs-site.xml
<!-- 启用WebHdfs. --> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property>
core-site.xml
<!-- Hue配置 --> <property> <name>hadoop.proxyuser.hue.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hue.groups</name> <value>*</value> </property>
重启hdfs
hue的配置:hue的全局配置文件:
desktop/conf/hue.ini
[[hdfs_clusters]] fs_defaultfs=hdfs://hadoop-senior01.ibeifeng.com:8020 webhdfs_url=http://hadoop-senior01.ibeifeng.com:50070/webhdfs/v1 hadoop_conf_dir=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop hadoop_hdfs_home=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6 hadoop_bin=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/bin
- 通过网页查看HDFS
Hadoop-yarn集成
配置hue [[yarn_clusters]]
resourcemanager_host=hadoop-senior01.ibeifeng.com resourcemanager_port=8032 submit_to=True resourcemanager_api_url=http://hadoop-senior01.ibeifeng.com:8088 proxy_api_url=http://hadoop-senior01.ibeifeng.com:8088 history_server_api_url=http://hadoop-senior01.ibeifeng.com:19888
- 测试一个mr程序:
bin/yarn jar /opt/datas/wc2.jar org.apache.hadoop.wordcount.WordCount /input /hue
Hive的集成
- hive的metastore
- Derby存储:只能启动一个实例
- 本地MySQL:MySQL与hive在同一台机器上
- 远程的MySQL
- a) 修改远程机器的hive配置文件hive-site.xml
<property>
<name>hive.metastore.uris</name>
<value></value>
</property>
- b) 启动远程metastore
bin/hive --service metastore
后台启动:nohup bin/hive --service metastore &
- a) 修改远程机器的hive配置文件hive-site.xml
- hive的配置
- hiveserver2两种模式:binary和http,默认为binary
- 开启hiveserver2:bin/hiveserver2
hue的配置 [beeswax]
hive_server_host=hadoop-senior01.ibeifeng.com hive_server_port=10000 hive_conf_dir=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf hive_home_dir=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6
- Hue上查看:
- hive的metastore
RDBMS的集成,配置hue
sqlite [[[sqlite]]]
nice_name=SQLite name=/opt/cdh-5.3.6/hue-3.7.0-cdh5.3.6/desktop/desktop.db engine=sqlite
MySql
nice_name="My SQL DB" engine=mysql host=hadoop-senior01.ibeifeng.com port=3306 user=root password=123456
oozie的集成
hue [liboozie]
oozie_url=http://hadoop-senior01.ibeifeng.com:11000/oozie
启动Oozie存在问题,无法检测到sharelib
- 原因:同一家公司出的,有默认的检索地址
解决:oozie-site.xml
<property> <name>oozie.service.WorkflowAppService.system.libpath</name> <value>/user/oozie/share/lib</value> </property>
重新生成sharelib
bin/oozie-setup.sh sharelib create -fs hdfs://hadoop-senior01.ibeifeng.com:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz
- 重启oozie:
bin/oozied.sh start
三、 Hue中运行workflow
- 一个节点对应一个.xml文件
- 创建一个workflow,名称为test2
- 在中间添加一个Mapreduce节点mr-node
- mr-node节点中主要配置属性
- 作业XML中hueworkflow.xml只保留了节点中的内容
- 运行结束后可以看到结果