impala+hue

16 篇文章 0 订阅
1 篇文章 0 订阅

下载impala 和 hue

http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_vd_cdh_package_tarball.html

https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.5.0/RPMS/x86_64/

impala 
impala-catalog 
impala-state-store 
impala-server

hue


rpm -ivh impala-xxxx.rpm --force --nodeps  (不安装所有依赖)

否则会有下面的报错:

<span style="font-family:Consolas;background-color: rgb(255, 255, 255);">error: Failed dependencies:
        bigtop-utils >= 0.7 is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hadoop is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hadoop-hdfs is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hadoop-yarn is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hadoop-mapreduce is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hbase is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hive >= 0.12.0+cdh5.1.0 is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        zookeeper is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        hadoop-libhdfs is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        avro-libs is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        parquet is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        sentry >= 1.3.0+cdh5.1.0 is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        sentry is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        /lib/lsb/init-functions is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
        libhdfs.so.0.0.0()(64bit) is needed by impala-2.3.0+cdh5.5.0+0-1.cdh5.5.0.p0.15.el7.x86_64
</span>

wget -O /etc/yum.repos.d/ http://www.apache.org/dist/bigtop/stable/repos/centos7/bigtop.repo

yum install bigtop-utils

yum install lsb


#添加impala_home

vi /etc/profile

export JAVA_HOME=/usr/java/latest
export CLASSPATH=.:%JAVA_HOME%/lib/dt.jar:%JAVA_HOME%/lib/tools.jar:%JAVA_HOME%/lib/jdbc.jar
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/usr/hadoop/latest
export HADOOP_PREFIX=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HIVE_HOME=/usr/hive/latest
export PATH=$HIVE_HOME/bin:$PATH

#impala
export IMPALA_HOME=/usr/lib/impala

export IMPALA_CONF_DIR=/etc/impala/conf


:wq

source /etc/profile

在各节点装好对应的服务后拷贝或者软链接core-site.xml,hdfs-site.xml,hbase-stie.xml到impala的conf目录

ln -s $HADOOP_CONF_DIR/core-site.xml $IMPALA_CONF_DIR/core-site.xml
ln -s $HADOOP_CONF_DIR/hdfs-site.xml $IMPALA_CONF_DIR/hdfs-site.xml
ln -s $HIVE_HOME/conf/hive-site.xml $IMPALA_CONF_DIR/hive-site.xml


下面是 $IMPALA_HOME/lib 中需要解决的死链接。

./hbase-annotations.jar
./hbase-common.jar
./hive-shims.jar
./hbase-client.jar
./hbase-protocol.jar
./hive-beeline.jar
./hadoop-yarn-server-applicationhistoryservice.jar
./hive-ant.jar
./avro.jar
./hadoop-annotations.jar
./hadoop-auth.jar
./hive-common.jar
./hadoop-aws.jar
./hadoop-common.jar
./hadoop-hdfs.jar
./hadoop-mapreduce-client-common.jar
./hive-exec.jar
./hadoop-mapreduce-client-core.jar
./hive-serde.jar
./hadoop-mapreduce-client-jobclient.jar
./hive-hbase-handler.jar
./hive-metastore.jar
./hadoop-mapreduce-client-shuffle.jar
./hive-service.jar
./hadoop-yarn-api.jar
./hive-shims-scheduler.jar
./hadoop-yarn-client.jar
./hadoop-yarn-common.jar
./hadoop-yarn-server-common.jar
./hadoop-yarn-server-nodemanager.jar
./hive-shims-common.jar
./hadoop-yarn-server-resourcemanager.jar
./hadoop-yarn-server-web-proxy.jar
./zookeeper.jar
./sentry-core-common.jar
./sentry-core-model-db.jar
./sentry-core-model-search.jar
./libhdfs.so
./sentry-policy-common.jar
./sentry-policy-db.jar
./sentry-provider-cache.jar
./parquet-hadoop-bundle.jar
./sentry-provider-common.jar
./sentry-provider-db-sh.jar
./sentry-provider-file.jar
./sentry-binding-hive.jar
./libhadoop.so
./libhadoop.so.1.0.0
./libhdfs.so.0.0.0

从官网下载相应的tar包,从中解压出需要的jar库文件放到 $IMPALA_HOME/lib/cdh5_lib/ 下面

http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_vd_cdh_package_tarball.html

然后全部加上链接

ln --force -s $HADOOP_HOME/lib/native/libhadoop.so $IMPALA_HOME/lib/libhadoop.so
ln --force -s $HADOOP_HOME/lib/native/libhdfs.so $IMPALA_HOME/lib/libhdfs.so

ln --force -s $IMPALA_HOME/lib/cdh5_lib/avro-1.7.6-cdh5.5.2.jar $IMPALA_HOME/lib/avro.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-exec-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-exec.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-annotations-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-annotations.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-hbase-handler-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-hbase-handler.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-auth-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-auth.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-metastore-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-metastore.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-aws-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-aws.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-serde-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-serde.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-common-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-service-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-service.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-hdfs-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-hdfs.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-shims-0.23-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-shims.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-mapreduce-client-common-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-mapreduce-client-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-shims-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-shims.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-mapreduce-client-core.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-shims-common-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-shims-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-mapreduce-client-jobclient.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-shims-scheduler-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-shims-scheduler.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-mapreduce-client-shuffle.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/parquet-hadoop-bundle-1.5.0-cdh5.5.2.jar $IMPALA_HOME/lib/parquet-hadoop-bundle.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-api-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-api.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-binding-hive-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-binding-hive.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-client-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-client.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-core-common-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-core-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-common-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-core-model-db-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-core-model-db.jar

ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-server-applicationhistoryservice.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-core-model-search-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-core-model-search.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-server-common-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-server-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-policy-common-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-policy-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-server-nodemanager-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-server-nodemanager.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-policy-db-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-policy-db.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-server-resourcemanager.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-provider-cache-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-provider-cache.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hadoop-yarn-server-web-proxy-2.6.0-cdh5.5.2.jar $IMPALA_HOME/lib/hadoop-yarn-server-web-proxy.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-provider-common-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-provider-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-ant-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-ant.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-provider-db-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-provider-db-sh.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-beeline-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-beeline.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/sentry-provider-file-1.5.1-cdh5.5.2.jar $IMPALA_HOME/lib/sentry-provider-file.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hive-common-1.1.0-cdh5.5.2.jar $IMPALA_HOME/lib/hive-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/zookeeper-3.4.5-cdh5.5.2.jar $IMPALA_HOME/lib/zookeeper.jar

ln --force -s $IMPALA_HOME/lib/cdh5_lib/hbase-annotations-1.0.0-cdh5.5.2.jar $IMPALA_HOME/lib/hbase-annotations.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hbase-common-1.0.0-cdh5.5.2.jar $IMPALA_HOME/lib/hbase-common.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hbase-client-1.0.0-cdh5.5.2.jar $IMPALA_HOME/lib/hbase-client.jar
ln --force -s $IMPALA_HOME/lib/cdh5_lib/hbase-protocol-1.0.0-cdh5.5.2.jar $IMPALA_HOME/lib/hbase-protocol.jar

最后需要把jdbc driver 放到 /var/lib/impala 或者使用链接放到这里(如下),否则 impala-catalog服务启动不了。
ln -s $JAVA_HOME/lib/jdbc.jar /var/lib/impala/jdbc.jar

在hdfs-site.xml中增加以下内容: setting for locality function of impala.
<!--impala-->
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hadoop-hdfs/dn._PORT</value>
</property>
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.file-block-storage-locations.timeout.millis</name>
<value>10000</value>
</property>
在/etc/run/下创建hadoop-hdfs文件夹,所属用户hadoop。
mkdir /var/run/hadoop-hdfs
编辑/etc/default/impala配置文件
IMPALA_CATALOG_SERVICE_HOST=namenode
IMPALA_STATE_STORE_HOST=namenode
HADOOP_CONF_DIR=$IMPALA_CONF_DIR
HIVE_CONF_DIR=$IMPALA_CONF_DIR

service impala-state-store start 
service impala-catalog start 
service impala-server start
OK!



  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值