Impala安装

最新推荐文章于 2024-08-07 09:43:18 发布

清风狂刀

最新推荐文章于 2024-08-07 09:43:18 发布

阅读量892

点赞数

本文链接：https://blog.csdn.net/weixin_43865893/article/details/86688760

版权

MySQL或PostgreSQL，充当Impala和Hive的Metastore数据库。
安装和配置Hive Metastore是Impala的要求。没有Metastore数据库，Impala不起作用
集群环境：

IP	主机名	角色
192.168.11.70	hnode1	主节点
192.168.11.71	hnode2	从节点
192.168.11.72	hnode3	从节点

下载

本文适用于以tar方式安装的hadoop，并以RPM方式安装impala
下载地址：
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.13.1/RPMS/x86_64/
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.13.1/RPMS/noarch/

bigtop-utils-0.7.0+cdh5.13.1+0-1.cdh5.13.1.p0.3.el6.noarch.rpm       
sentry-1.5.1+cdh5.13.1+417-1.cdh5.13.1.p0.3.el6.noarch.rpm
impala-server-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
impala-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm            
impala-shell-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
impala-catalog-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm    
impala-state-store-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
impala-udf-devel-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
impala-debuginfo-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm

安装

主节点安装

yum install redhat-lsb
yum install -y python-setuptools 
rpm -ivh bigtop-utils-0.7.0+cdh5.13.1+0-1.cdh5.13.1.p0.3.el6.noarch.rpm
rpm -ivh --nodeps sentry-1.5.1+cdh5.13.1+417-1.cdh5.13.1.p0.3.el6.noarch.rpm  #impala依赖
rpm -ivh --nodeps impala-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm  #因为hadoop是通过tar安装的，所以要强制安装，否则会提示找不到关联
rpm -ivh impala-state-store-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-catalog-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-server-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-shell-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-udf-devel-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-debuginfo-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm

从节点安装

yum install -y python-setuptools 
rpm -ivh bigtop-utils-0.7.0+cdh5.13.1+0-1.cdh5.13.1.p0.3.el6.noarch.rpm
rpm -ivh --nodeps sentry-1.5.1+cdh5.13.1+417-1.cdh5.13.1.p0.3.el6.noarch.rpm
rpm -ivh --nodeps impala-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-server-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-udf-devel-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-shell-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm
rpm -ivh impala-debuginfo-2.10.0+cdh5.13.1+0-1.cdh5.13.1.p0.4.el6.x86_64.rpm

查看安装后的目录

find / -name impala

[root@hnode1 impala]# find / -name impala
/var/run/impala
/var/lib/alternatives/impala
/var/lib/impala
/var/log/impala
/opt/modules/impala
/usr/lib/sentry/lib/impala
/usr/lib/debug/usr/lib/impala
/usr/lib/impala
/etc/alternatives/impala
/etc/default/impala
/etc/impala

impala默认的安装目录为 /usr/lib/impala，jar包地址为/usr/lib/impala/lib/
由于我的Hadoop、Hive、Kudu等环境都是通过tar包命令行方式安装的，因此需要修复Impala默认的链接

删除默认不存在的软连接

rm -rf /usr/lib/impala/lib/avro*.jar
rm -rf /usr/lib/impala/lib/hadoop-*.jar
rm -rf /usr/lib/impala/lib/hive-*.jar
rm -rf /usr/lib/impala/lib/hbase-*.jar
rm -rf /usr/lib/impala/lib/parquet-hadoop-bundle.jar
rm -rf /usr/lib/impala/lib/sentry-*.jar
rm -rf /usr/lib/impala/lib/zookeeper.jar
rm -rf /usr/lib/impala/lib/libhadoop.so
rm -rf /usr/lib/impala/lib/libhadoop.so.1.0.0
rm -rf /usr/lib/impala/lib/libhdfs.so
rm -rf /usr/lib/impala/lib/libhdfs.so.0.0.0

建立新的软连接

建议利用脚本运行，自己设一下各个home即可，软连后的名字不能改

export HADOOP_HOME=/opt/modules/hadoop-2.6.0-cdh5.13.1
export HIVE_HOME=/opt/modules/hive-1.1.0-cdh5.13.1
export SENTRY_HOME=/usr/lib/sentry
export HBASE_HOME=/opt/modules/hbase-1.2.0-cdh5.13.1

#avro
ln -s $HADOOP_HOME/share/hadoop/common/lib/avro-1.7.6-cdh5.13.1.jar /usr/lib/impala/lib/avro.jar
#hadoop
ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.13.1.jar      /usr/lib/impala/lib/hadoop-annotations.jar 
ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.13.1.jar   /usr/lib/impala/lib/hadoop-auth.jar
ln -s $HADOOP_HOME/share/hadoop/tools/lib/hadoop-aws-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-aws.jar
ln -s $HADOOP_HOME/share/hadoop/tools/lib/hadoop-azure-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-azure-datalake.jar
ln -s $HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-common.jar 
ln -s $HADOOP_HOME/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.13.1.jar    /usr/lib/impala/lib/hadoop-hdfs.jar
ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar 
ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-yarn-api.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-yarn-client.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-yarn-common.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-yarn-server-common.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.13.1.jar  /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.13.1.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar 
#hive
ln -s $HIVE_HOME/lib/hive-ant-1.1.0-cdh5.13.1.jar  /usr/lib/impala/lib/hive-ant.jar
ln -s $HIVE_HOME/lib/hive-beeline-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-beeline.jar 
ln -s $HIVE_HOME/lib/hive-cli-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-cli.jar
ln -s $HIVE_HOME/lib/hive-common-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-common.jar
ln -s $HIVE_HOME/lib/hive-exec-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-exec.jar
ln -s $HIVE_HOME/lib/hive-hbase-handler-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-hbase-handler.jar
#ln -s $HIVE_HOME/lib/hive-hcatalog-core-1.1.0.jar /usr/lib/impala/lib/hive-hcatalog-core.jar
#ln -s $HIVE_HOME/lib/hive-hcatalog-server-extensions-1.1.0.jar /usr/lib/impala/lib/hive-hcatalog-server-extensions.jar
ln -s $HIVE_HOME/lib/hive-metastore-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-metastore.jar
ln -s $HIVE_HOME/lib/hive-serde-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-serde.jar
ln -s $HIVE_HOME/lib/hive-service-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-service.jar
ln -s $HIVE_HOME/lib/hive-shims-common-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-shims-common.jar
ln -s $HIVE_HOME/lib/hive-shims-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-shims.jar
ln -s $HIVE_HOME/lib/hive-shims-scheduler-1.1.0-cdh5.13.1.jar /usr/lib/impala/lib/hive-shims-scheduler.jar
#hbase
ln -s $HBASE_HOME/lib/hbase-annotations-1.2.0-cdh5.13.1.jar /usr/lib/impala/lib/hbase-annotations.jar
ln -s $HBASE_HOME/lib/hbase-client-1.2.0-cdh5.13.1.jar /usr/lib/impala/lib/hbase-client.jar
ln -s $HBASE_HOME/lib/hbase-common-1.2.0-cdh5.13.1.jar /usr/lib/impala/lib/hbase-common.jar
ln -s $HBASE_HOME/lib/hbase-protocol-1.2.0-cdh5.13.1.jar /usr/lib/impala/lib/hbase-protocol.jar
#parquet-hadoop-bundle
ln -s $HIVE_HOME/lib/parquet-hadoop-bundle-1.5.0-cdh5.13.1.jar /usr/lib/impala/lib/parquet-hadoop-bundle.jar
#sentry
ln -s $SENTRY_HOME/lib/sentry-binding-hive-conf-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-binding-hive-conf.jar
ln -s $SENTRY_HOME/lib/sentry-binding-hive-follower-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-binding-hive-follower.jar
ln -s $SENTRY_HOME/lib/sentry-binding-hive-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-binding-hive.jar
ln -s $SENTRY_HOME/lib/sentry-core-common-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-core-common.jar
ln -s $SENTRY_HOME/lib/sentry-core-model-db-1.5.1-cdh5.13.1.jar  /usr/lib/impala/lib/sentry-core-model-db.jar
ln -s $SENTRY_HOME/lib/sentry-core-model-kafka-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-core-model-kafka.jar
ln -s $SENTRY_HOME/lib/sentry-core-model-search-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-core-model-search.jar
ln -s $SENTRY_HOME/lib/sentry-hdfs-common-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-hdfs-common.jar
ln -s $SENTRY_HOME/lib/sentry-policy-common-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-policy-common.jar
ln -s $SENTRY_HOME/lib/sentry-policy-db-1.5.1-cdh5.13.1.jar  /usr/lib/impala/lib/sentry-policy-db.jar
ln -s $SENTRY_HOME/lib/sentry-policy-kafka-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-policy-kafka.jar
ln -s $SENTRY_HOME/lib/sentry-policy-search-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-policy-search.jar
ln -s $SENTRY_HOME/lib/sentry-provider-cache-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-provider-cache.jar
ln -s $SENTRY_HOME/lib/sentry-provider-common-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-provider-common.jar
ln -s $SENTRY_HOME/lib/sentry-provider-db-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-provider-db-sh.jar
ln -s $SENTRY_HOME/lib/sentry-provider-file-1.5.1-cdh5.13.1.jar /usr/lib/impala/lib/sentry-provider-file.jar
#zookeeper
ln -s $HIVE_HOME/lib/zookeeper-3.4.5-cdh5.13.1.jar /usr/lib/impala/lib/zookeeper.jar

ln -s $HADOOP_HOME/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so
ln -s $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0
ln -s $HADOOP_HOME/lib/native/libhdfs.so /usr/lib/impala/lib/libhdfs.so

配置Hadoop

复制${HADOOP_HOME}/etc/hadoop/hdfs-site.xml到/etc/impala/conf目录
修改hdfs-site.xml

<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>
<property>
    <name>dfs.domain.socket.path</name>
    <value>/var/run/hadoop/dn._PORT</value>
</property>
<property>
  <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
  <value>true</value>
</property>
<property>
   <name>dfs.client.use.legacy.blockreader.local</name>
   <value>false</value>
</property>
<property>
   <name>dfs.datanode.data.dir.perm</name>
   <value>750</value>
</property>
<property>
   <name>dfs.block.local-path-access.user</name>
   <value>hadoop</value>
</property>
<property>
   <name>dfs.client.file-block-storage-locations.timeout</name>
   <value>3000</value>
</property>

重启HDFS，注意需要创建/var/run/hadoop/目录，并且具有写权限。

配置impala

修改配置文件

修改bigtop-utils ：

 vim /etc/default/bigtop-utils

修改JDK_HOME

 export JAVA_HOME=/opt/modules/jdk1.8.0_171/

修改impala默认配置

 vim /etc/default/impala

修改如下内容：

IMPALA_CATALOG_SERVICE_HOST=192.168.11.70
IMPALA_STATE_STORE_HOST=192.168.11.70

MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar  #如果这个路径没有mysql的jar包，需要上传或修改位置
HIVE_HOME=/opt/modules/hive-1.1.0-cdh5.13.1
HADOOP_CONF_DIR=/opt/modules/hadoop-2.6.0-cdh5.13.1/etc/hadoop

修改启动文件

impala默认以impala用户执行，可以在三个文件中修改 /etc/init.d/impala-state-store、/etc/init.d/impala-server、/etc/init.d/impala-catalog
修改内容：

SVC_USER=”hadoop”

install -d -m 0755 -o hadoop -g hadoop /var/run/impala 1>/dev/null 2>&1 || :

修改日志目录权限：

chmod -R 777 /var/log/impala/

配置impala配置文件

cd /etc/impala/conf

通过软连接的方式配置impala，也可以将对应的文件拷贝过来

ln -s /opt/modules/hive-1.1.0-cdh5.13.1/conf/hive-site.xml hive-site.xml
ln -s /opt/modules/hive-1.1.0-cdh5.13.1/conf/hive-env.sh hive-env.sh
ln -s /opt/modules/hadoop-2.6.0-cdh5.13.1/etc/hadoop/core-site.xml core-site.xml
ln -s /opt/modules/hadoop-2.6.0-cdh5.13.1/etc/hadoop/hdfs-site.xml hdfs-site.xml

启动Master

service impala-state-store start

service impala-catalog start

启动Slave

service impala-server start

查看日志

启动后，在/var/log/impala目录下会产生对应服务的日志

连接测试

到任意一台Slave节点上，执行impala-shell命令，进入impala命令行：

impala-shell

[hadoop@hnode1 init.d]$ impala-shell
Starting Impala Shell without Kerberos authentication
Connected to hnode1:21000
Server version: impalad version 2.10.0-cdh5.13.1 RELEASE (build 1e4b23c4eb52dac95c5be6316f49685c41783c51)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.1 (1e4b23c) built on Thu Nov  9 08:29:47 PST 2017)

Want to know what version of Impala you're connected to? Run the VERSION command to
find out!
***********************************************************************************
[hnode1:21000] >

查看
在这里插入图片描述