1、使用/bin/bootstrap_development.sh脚本设置机器环境。
在bootstrap_development.sh中,主要执行三个另外的脚本,分别是bootstrap_system.sh、impala-config.sh和buildall.sh。
2、bootstrap_system.sh脚本介绍
主要用于构建Impala开发环境,包括安装依赖与环境配置,官方说在ubuntu16.04能够完美执行
执行bootstrap_system.sh脚本,
- 首先判断操作系统类型,脚本只检测两种系统以及是否在docker中执行,分别为redhat、ubuntu和docker
- 在本次源码编译中,使用ubuntu16.04作为编译环境,所以在判断为ubuntu16.04之后,开始安装相关依赖(installing build tools)
- 脚本执行过程中遇见的问题:Setting up postgresql-common (173ubuntu0.2) ... dpkg: error processing package postgresql-common (--configure): subprocess installed post-installation script returned error exit status 10 Errors were encountered while processing: postgresql-common E: Sub-process /usr/bin/dpkg returned an error code (1)网上方法解决办法:https://blog.csdn.net/ax7399/article/details/78210051 (不能解决)尝试源码编译安装postgresql:https://blog.csdn.net/m0_38026518/article/details/93588035 (成功)(base) vadmin@vadmin:~/postgresql-11.4{content}nbsp;psql --version psql (PostgreSQL) 11.5 (Ubuntu 11.5-3.pgdg16.04+1)
- 安装完依赖之后,需要连接postgresql数据库,由于postgresql是从源码安装的,所以下面的某些配置文件位置和通过apt install方式安装的不一样,需要进行修改:
- 比如:postgresql数据库启动方式的改变#ubuntu sudo service postgresql start ubuntu pg_ctl start数据库启动之后,默认进程是在tmp目录下的,就像这样子:(base) vadmin@vadmin:~/impala/impala/bin{content}nbsp;pg_ctl start waiting for server to start....2019-11-04 11:12:11.039 CST [16697] LOG: listening on IPv4 address "127.0.0.1", port 5432 2019-11-04 11:12:11.073 CST [16697] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" 2019-11-04 11:12:11.188 CST [16698] LOG: database system was shut down at 2019-11-04 11:11:22 CST 2019-11-04 11:12:11.235 CST [16697] LOG: database system is ready to accept connections done server started而在连接数据库时,ubuntu系统会在var/run/postgresql目录下查找,如果存在这个问题,应该在postgresql安装目录/usr/local/pgsql/data/下的postgresql.conf文件下修改配置,进行如下修改:#unix_socket_directories = '/tmp' # comma-separated list of directories unix_socket_directories = '/var/run/postgresql'
- 此脚本编译打包hadoop-lzo和impala-lzo,hadoop-lzo我这边没有使用源码打包,是直接从maven仓库上下的,这个文件在impala-config.sh文件中会使用到,具体路径看一下impala-conig.sh文件中用到的hadoop-lzo的路径就可以了。
3、impala-config.sh文件是变量配置,执行命令./impala-config.sh输出大概是这样子:
(base) vadmin@vadmin:~/impala/impala/bin{content}nbsp;./impala-config.sh IMPALA_HOME = /home/vadmin/impala/impalaHADOOP_HOME = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT/HADOOP_CONF_DIR = /home/vadmin/impala/impala/fe/src/test/resourcesHADOOP_INCLUDE_DIR = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//includeHADOOP_LIB_DIR = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//libMINI_DFS_BASE_DATA_DIR = /home/vadmin/impala/impala/cdh-6-hdfs-dataHIVE_HOME = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hive-2.1.1-cdh6.x-SNAPSHOTHIVE_CONF_DIR = /home/vadmin/impala/impala/fe/./src/test/resourcesHIVE_SRC_DIR = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hive-2.1.1-cdh6.x-SNAPSHOT/srcHBASE_HOME = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hbase-2.1.0-cdh6.x-SNAPSHOT/HBASE_CONF_DIR = /home/vadmin/impala/impala/fe/src/test/resourcesSENTRY_HOME = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/sentry-2.1.0-cdh6.x-SNAPSHOTSENTRY_CONF_DIR = /home/vadmin/impala/impala/fe/src/test/resourcesRANGER_HOME = /home/vadmin/impala/impala/toolchain/cdp_components-1352353/ranger-1.2.0.7.1.0.0-33-adminRANGER_CONF_DIR = /home/vadmin/impala/impala/fe/src/test/resources MINIKDC_HOME = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/llama-minikdc-1.0.0THRIFT_HOME = /home/vadmin/impala/impala/toolchain/thrift-0.9.3-p7HADOOP_LZO = /home/vadmin/impala/impala/../hadoop-lzoIMPALA_LZO = /home/vadmin/impala/impala/../Impala-lzoCLASSPATH = /home/vadmin/impala/impala/../hadoop-lzo/build/hadoop-lzo-0.4.13.jar:/home/vadmin/impala/impala/fe/src/test/resources:/home/vadmin/impala/impala/fe/target/classes:/home/vadmin/impala/impala/fe/target/dependency:LIBHDFS_OPTS = -Djava.library.path=/home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native/ -XX:ErrorFile=/home/vadmin/impala/impala/logs/hs_err_pid%p.log -XX:MaxPermSize=128mJAVA_HOME = /usr/lib/jvm/java-8-openjdk-amd64LD_LIBRARY_PATH = /usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/local/cuda-8.0/lib64:/usr/lib/x86_64-linux-gnu::/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native:/home/vadmin/impala/impala/toolchain/snappy-1.1.4/lib:/home/vadmin/impala/impala/../Impala-lzo/build:/home/vadmin/impala/impala/toolchain/gcc-4.9.2/lib64LD_PRELOAD = :/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjsig.soPOSTGRES_JDBC_DRIVER = /home/vadmin/impala/impala/fe/target/dependency/postgresql-42.2.5.jarIMPALA_TOOLCHAIN = /home/vadmin/impala/impala/toolchainMETASTORE_DB = HMS_home_vadmin_impala_impalaDOWNLOAD_CDH_COMPONENTS = trueIMPALA_MAVEN_OPTIONS = IMPALA_TOOLCHAIN_HOST = native-toolchain.s3.amazonaws.comCDH_BUILD_NUMBER = 1474057CDH_COMPONENTS_HOME = /home/vadmin/impala/impala/toolchain/cdh_components-1474057CDP_BUILD_NUMBER = 1352353CDP_COMPONENTS_HOME = /home/vadmin/impala/impala/toolchain/cdp_components-1352353IMPALA_HADOOP_VERSION = 3.0.0-cdh6.x-SNAPSHOTIMPALA_HIVE_VERSION = 2.1.1-cdh6.x-SNAPSHOTIMPALA_HBASE_VERSION = 2.1.0-cdh6.x-SNAPSHOTIMPALA_SENTRY_VERSION = 2.1.0-cdh6.x-SNAPSHOTIMPALA_KUDU_VERSION = 1.11.0-cdh6.x-SNAPSHOTIMPALA_KUDU_JAVA_VERSION= 1.11.0-cdh6.x-SNAPSHOTIMPALA_RANGER_VERSION = 1.2.0.7.1.0.0-33
4、buildall.sh文件
执行命令,需要下载python相关依赖,这边更换pip国内源,在impala/infra/python/deps/pip_download.py,清华源:https://pypi.tuna.tsinghua.edu.cn
在下载python相关依赖的时候,有一个包名有问题,原来是python_dateutil == 2.5.2,应该改为:python-dateutil == 2.5.2
![00b9086e60232df68b7bf59ba517ce94.png](https://i-blog.csdnimg.cn/blog_migrate/aa65c73b61964db3194efe4d6888f395.jpeg)