安装cdh 版的impala

Impala cannot find com.mysql.jdbc.Driver
I fixed this by copying mysql-connector-java-5.1.25-bin.jar to /var/lib/impala - the startup script was telling the classpath to look here for the connector jar for some reason.


Cloudera Impala 安装笔记3一impala安装
时间 2014-02-13 18:25:00  博客园-所有随笔区
原文  http://www.cnblogs.com/chengyeliang/p/3548558.html
主题 Cloudera Linux命令
安装impala之前，确认满足Cloudera Impala Requirements中要求的所有条件：


Supported Operating Systems
Supported CDH Versions
Hive Metastore and Related Configuration
Java Dependencies
Packages and Repositories
Networking Configuration Requirements
Hardware Requirements
User Account Requirements
Impala官方安装wiki地址：


http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_install.html


确保你的包管理工具中加了impala需要的库（在该链接中描述 http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_prereqs.html#prereqs ）
安装impala各组件的命令：
$ sudo yum install impala             # Binaries for daemons
$ sudo yum install impala-server      # Service start/stop script
$ sudo yum install impala-state-store # Service start/stop script
$ sudo yum install impala-catalog     # Service start/stop script
3. 复制hive-site.xml, core-site.xml, and hdfs-site.xml到impala的配置文件夹中，默认目录在/etc/impala/conf如果没有此文件夹就创建。


4. 安装impala-shell


$ sudo yum install impala-shell
5.    完成impala 要求的配 置。


Post-Installation Configuration for Impala


强制Mandatory 配置(Short-Circuit Reads)


开启Block Location Tracking可以允许impala能够从文件系统中直接读取本地数据,而不需要像平常一样与datanode通信,从而可以提升查询的速度。


依据你所安装的CDH的版本不一样，配置也是不一样的，此处我仅整理我对应的CDH4.3正确的配置，实际安装中要采用对应版本的配置。


在所有的impala的节点上，修改hdfs-site.xml文件，添加下列属性：
<property>
   <name>dfs.client.read.shortcircuit</name>
   <value>true</value>
</property>
<property>
   <name>dfs.domain.socket.path</name>
   <value>/var/run/hadoop-hdfs/dn._PORT</value>
</property>
<property>
   <name>dfs.client.file-block-storage-locations.timeout</name>
   <value>3000</value>
</property>
<property>
   <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
   <value>true</value>
</property>
2. if /var/run/hadoop-hdfs 是组可写的，那么确保他的组是 root 。（更改该目录的属组）


3.   复制hive-site.xml, core-site.xml, and hdfs-site.xml到impala的配置文件夹中，默认目录在/etc/impala/conf。


4. 上述配置结束后，重启所有datanodes。


强制Mandatory 配置(Block Location Tracking)


开启Block Location metadata允许impala知道数据块在哪块磁盘上，这样可以更好的利用下面的磁盘。（如果不开启这个设置，impala默认是不会启动）。


对于每一个Datanode，添加下列属性到hdfs-site.xml 文件：
<property>
  <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
  <value>true</value>
</property>
2. 复制hive-site.xml, core-site.xml, and hdfs-site.xml到impala的配置文件夹中，默认目录在/etc/impala/conf。


3. 上述配置结束后，重启所有datanodes。


配置impala 启动的一些操作


Impala服务启动的时候会加载/etc/default/impala文件中的配置，这里包含了很多impala要用到的资源的信息。例如，典型的classpath变量，你还需要加入或者修改一些信息，比如statstore的地址信息。


IMPALA_STATE_STORE_HOST=127.0.0.1
IMPALA_STATE_STORE_PORT=24000
IMPALA_BACKEND_PORT=22000
IMPALA_LOG_DIR=/var/log/impala


IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} "
IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE
_STORE_PORT}"
IMPALA_SERVER_ARGS=" \
    -log_dir=${IMPALA_LOG_DIR} \
    -state_store_port=${IMPALA_STATE_STORE_PORT} \
    -use_statestore \
    -state_store_host=${IMPALA_STATE_STORE_HOST} \
    -be_port=${IMPALA_BACKEND_PORT}"


ENABLE_CORE_DUMPS=false
LIBHDFS_OPTS=-Djava.library.path=/usr/lib/impala/lib
MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
IMPALA_BIN=/usr/lib/impala/sbin
IMPALA_HOME=/usr/lib/impala
HIVE_HOME=/home/godp/hive-0.9.0-cdh4.1.0
# HBASE_HOME=/usr/lib/hbase
IMPALA_CONF_DIR=/etc/impala/conf
HADOOP_CONF_DIR=/etc/impala/conf
HIVE_CONF_DIR=/etc/impala/conf
# HBASE_CONF_DIR=/etc/impala/conf
修改完这些信息之后，重启所有impala相关的服务。


$ sudo service impala-server restart
$ sudo service impala-state-store restart
$ sudo service impala-catalog restart
至此，安装结束






1、所有节点安装文档：
1.1、/etc/default/impala 配置如下
IMPALA_CATALOG_SERVICE_HOST=192.168.0.131
IMPALA_STATE_STORE_HOST=192.168.0.131
IMPALA_STATE_STORE_PORT=24000
IMPALA_BACKEND_PORT=22000
IMPALA_LOG_DIR=/var/log/impala


IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} "
IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}"
IMPALA_SERVER_ARGS=" \
    -log_dir=${IMPALA_LOG_DIR} \
    -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
    -state_store_port=${IMPALA_STATE_STORE_PORT} \
    -use_statestore \
    -state_store_host=${IMPALA_STATE_STORE_HOST} \
    -be_port=${IMPALA_BACKEND_PORT}"


ENABLE_CORE_DUMPS=false


LIBHDFS_OPTS=-Djava.library.path=/usr/lib/impala/lib
#MYSQL_CONNECTOR_JAR=/opt/isu/hive/lib/mysql-connector-java-5.1.10.jar
MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java-5.1.10.jar
IMPALA_BIN=/usr/lib/impala/sbin
IMPALA_HOME=/usr/lib/impala
HIVE_HOME=/opt/isu/hive
# HBASE_HOME=/usr/lib/hbase
IMPALA_CONF_DIR=/etc/impala/conf
HADOOP_CONF_DIR=/etc/impala/conf
HIVE_CONF_DIR=/etc/impala/conf
# HBASE_CONF_DIR=/etc/impala/conf
1.2、/var/lib/impala 
添加 mysql-connector-java-5.1.10.jar
1.3、/etc/impala/conf
添加 core-site.xml，hdfs-site.xml，hive-site.xml
其中
[isuhadoop@isunode001 conf]$ more hdfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> 
 <property> 
      <name>dfs.name.dir</name> 
           <value>/opt/isu/data/namenode2</value> 
      </property> 
  <property> 
      <name>dfs.data.dir</name> 
           <value>/data1_4T/dfs,/data2_4T/dfs,/data3_4T/dfs</value> 
  </property> 
  <property> 
      <name>dfs.replication</name> 
           <value>3</value> 
  </property>
  <property>
     <name>dfs.client.read.shortcircuit</name>
     <value>true</value>
  </property>
  <property>
     <name>dfs.domain.socket.path</name>
     <value>/var/run/hadoop-hdfs/dn._PORT</value>
  </property>
  <property>
     <name>dfs.client.file-block-storage-locations.timeout.millis</name>
     <value>3000000</value>
  </property>
  <property>
     <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
     <value>true</value>
  </property> 
</configuration> 


[isuhadoop@isunode001 conf]$ 


2主机节点启动的服务：
$ sudo service impala-server restart
$ sudo service impala-state-store restart
$ sudo service impala-catalog restart


3从节点启动的服务：
$ sudo service impala-server restart


-----------------------
实际转载的问题
1、java_home 应该在/etc/default/bigtop-utils 文件中配置
 sudo vi /etc/default/bigtop-utils
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


# Override JAVA_HOME detection for all bigtop packages
# export JAVA_HOME
export JAVA_HOME=/opt/isu/jdk
# Provide a colon-delimited list of directories to search for native Java libraries (e.g. libjvm.so)
2、impala-catalog 服务没有启起来，报错如下：
[isuhadoop@isunode002 impala]$ more catalogd.ERROR
Log file created at: 2016/08/02 17:48:39
Running on machine: isunode002
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0802 17:48:39.910905  5922 logging.cc:120] stderr will be logged to this file.
E0802 17:48:40.236948  5922 catalog.cc:70] java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/UnknownDBException
        at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:91)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.UnknownDBException
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 1 more
loadFileSystems error:
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)


3、错
[isuhadoop@isunode035 ~]$ sudo service impala-server restart
/etc/init.d/impala-server: line 34: /lib/lsb/init-functions: No such file or directory
/etc/init.d/impala-server: line 111: killproc: command not found
/etc/init.d/impala-server: line 118: log_failure_msg: command not found
/etc/init.d/impala-server: line 103: log_success_msg: command not found




sudo yum -y install redhat-lsb




******************************************************

namenode和DataNode 安装的和DataNode安装的几乎差不多，

只是不用启动

$ sudo service impala-server restart

启动

$ sudo service impala-state-store restart
$ sudo service impala-catalog restart

即可

DataNode
1、Eguan)@)$
su - root
sed '108 aisuhadoop        ALL=(ALL)       NOPASSWD: ALL' -i /etc/sudoers
2、
scp -r /opt/isu/impala root@192.168.200.198:/opt/isu/
3、


sudo rpm -ivh /opt/isu/impala/bigtop-jsvc-debuginfo-0.6.0+cdh5.5.1+766-1.cdh5.5.1.p0.14.el6.x86_64.rpm
sudo rpm -ivh /opt/isu/impala/bigtop-utils-0.7.0+cdh5.5.1+0-1.cdh5.5.1.p0.15.el6.noarch.rpm
sudo rpm -ivh /opt/isu/impala/bigtop-jsvc-0.6.0+cdh5.5.1+766-1.cdh5.5.1.p0.14.el6.x86_64.rpm


sudo rpm -ivh /opt/isu/impala/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17.el6.x86_64.rpm --force --nodeps
sudo rpm -ivh /opt/isu/impala/impala-server-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17.el6.x86_64.rpm
4、
scp /etc/default/impala root@isunode005:/etc/default/impala 1
5、
scp /var/lib/impala/mysql-connector-java-5.1.10.jar root@isunode005:/var/lib/impala/
6、
scp -r /etc/impala/conf/* root@isunode005:/etc/impala/conf/
7、
scp /etc/default/bigtop-utils root@isunode005:/etc/default/


8、
在目标主机上
sudo rm -rf /usr/lib/impala/lib/*
9、
scp -r /usr/lib/impala/lib/* root@isunode005:/usr/lib/impala/lib/ 1


10
在目标主机上
sudo service impala-server restart