hadoop源码_Impala源码编译

１、使用/bin/bootstrap_development.sh脚本设置机器环境。

在bootstrap_development.sh中，主要执行三个另外的脚本，分别是bootstrap_system.sh、impala-config.sh和buildall.sh。

２、bootstrap_system.sh脚本介绍

主要用于构建Impala开发环境，包括安装依赖与环境配置，官方说在ubuntu16.04能够完美执行

执行bootstrap_system.sh脚本，

• 首先判断操作系统类型，脚本只检测两种系统以及是否在docker中执行，分别为redhat、ubuntu和docker
• 在本次源码编译中，使用ubuntu16.04作为编译环境，所以在判断为ubuntu16.04之后，开始安装相关依赖(installing build tools)
• 脚本执行过程中遇见的问题:Setting up postgresql-common (173ubuntu0.2) ... dpkg: error processing package postgresql-common (--configure): subprocess installed post-installation script returned error exit status 10 Errors were encountered while processing: postgresql-common E: Sub-process /usr/bin/dpkg returned an error code (1)网上方法解决办法：https://blog.csdn.net/ax7399/article/details/78210051　(不能解决)尝试源码编译安装postgresql：https://blog.csdn.net/m0_38026518/article/details/93588035　(成功)(base) vadmin@vadmin:~/postgresql-11.4{content}nbsp;psql --version psql (PostgreSQL) 11.5 (Ubuntu 11.5-3.pgdg16.04+1)
• 安装完依赖之后，需要连接postgresql数据库，由于postgresql是从源码安装的，所以下面的某些配置文件位置和通过apt install方式安装的不一样，需要进行修改：
• 比如：postgresql数据库启动方式的改变#ubuntu sudo service postgresql start ubuntu pg_ctl start数据库启动之后，默认进程是在tmp目录下的，就像这样子：(base) vadmin@vadmin:~/impala/impala/bin{content}nbsp;pg_ctl start waiting for server to start....2019-11-04 11:12:11.039 CST [16697] LOG: listening on IPv4 address "127.0.0.1", port 5432 2019-11-04 11:12:11.073 CST [16697] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" 2019-11-04 11:12:11.188 CST [16698] LOG: database system was shut down at 2019-11-04 11:11:22 CST 2019-11-04 11:12:11.235 CST [16697] LOG: database system is ready to accept connections done server started而在连接数据库时，ubuntu系统会在var/run/postgresql目录下查找，如果存在这个问题，应该在postgresql安装目录/usr/local/pgsql/data/下的postgresql.conf文件下修改配置，进行如下修改：#unix_socket_directories = '/tmp' # comma-separated list of directories unix_socket_directories = '/var/run/postgresql'
• 此脚本编译打包hadoop-lzo和impala-lzo，hadoop-lzo我这边没有使用源码打包，是直接从maven仓库上下的，这个文件在impala-config.sh文件中会使用到，具体路径看一下impala-conig.sh文件中用到的hadoop-lzo的路径就可以了。

３、impala-config.sh文件是变量配置，执行命令./impala-config.sh输出大概是这样子：

(base) vadmin@vadmin:~/impala/impala/bin{content}nbsp;./impala-config.sh IMPALA_HOME             = /home/vadmin/impala/impalaHADOOP_HOME             = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT/HADOOP_CONF_DIR         = /home/vadmin/impala/impala/fe/src/test/resourcesHADOOP_INCLUDE_DIR      = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//includeHADOOP_LIB_DIR          = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//libMINI_DFS_BASE_DATA_DIR  = /home/vadmin/impala/impala/cdh-6-hdfs-dataHIVE_HOME               = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hive-2.1.1-cdh6.x-SNAPSHOTHIVE_CONF_DIR           = /home/vadmin/impala/impala/fe/./src/test/resourcesHIVE_SRC_DIR            = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hive-2.1.1-cdh6.x-SNAPSHOT/srcHBASE_HOME              = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/hbase-2.1.0-cdh6.x-SNAPSHOT/HBASE_CONF_DIR          = /home/vadmin/impala/impala/fe/src/test/resourcesSENTRY_HOME             = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/sentry-2.1.0-cdh6.x-SNAPSHOTSENTRY_CONF_DIR         = /home/vadmin/impala/impala/fe/src/test/resourcesRANGER_HOME             = /home/vadmin/impala/impala/toolchain/cdp_components-1352353/ranger-1.2.0.7.1.0.0-33-adminRANGER_CONF_DIR         = /home/vadmin/impala/impala/fe/src/test/resources MINIKDC_HOME            = /home/vadmin/impala/impala/toolchain/cdh_components-1474057/llama-minikdc-1.0.0THRIFT_HOME             = /home/vadmin/impala/impala/toolchain/thrift-0.9.3-p7HADOOP_LZO              = /home/vadmin/impala/impala/../hadoop-lzoIMPALA_LZO              = /home/vadmin/impala/impala/../Impala-lzoCLASSPATH               = /home/vadmin/impala/impala/../hadoop-lzo/build/hadoop-lzo-0.4.13.jar:/home/vadmin/impala/impala/fe/src/test/resources:/home/vadmin/impala/impala/fe/target/classes:/home/vadmin/impala/impala/fe/target/dependency:LIBHDFS_OPTS            =  -Djava.library.path=/home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native/ -XX:ErrorFile=/home/vadmin/impala/impala/logs/hs_err_pid%p.log -XX:MaxPermSize=128mJAVA_HOME               = /usr/lib/jvm/java-8-openjdk-amd64LD_LIBRARY_PATH         = /usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/:/usr/local/cuda-8.0/lib64:/usr/lib/x86_64-linux-gnu::/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/vadmin/impala/impala/toolchain/cdh_components-1474057/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native:/home/vadmin/impala/impala/toolchain/snappy-1.1.4/lib:/home/vadmin/impala/impala/../Impala-lzo/build:/home/vadmin/impala/impala/toolchain/gcc-4.9.2/lib64LD_PRELOAD              = :/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjsig.soPOSTGRES_JDBC_DRIVER    = /home/vadmin/impala/impala/fe/target/dependency/postgresql-42.2.5.jarIMPALA_TOOLCHAIN        = /home/vadmin/impala/impala/toolchainMETASTORE_DB            = HMS_home_vadmin_impala_impalaDOWNLOAD_CDH_COMPONENTS = trueIMPALA_MAVEN_OPTIONS    = IMPALA_TOOLCHAIN_HOST   = native-toolchain.s3.amazonaws.comCDH_BUILD_NUMBER        = 1474057CDH_COMPONENTS_HOME     = /home/vadmin/impala/impala/toolchain/cdh_components-1474057CDP_BUILD_NUMBER        = 1352353CDP_COMPONENTS_HOME     = /home/vadmin/impala/impala/toolchain/cdp_components-1352353IMPALA_HADOOP_VERSION   = 3.0.0-cdh6.x-SNAPSHOTIMPALA_HIVE_VERSION     = 2.1.1-cdh6.x-SNAPSHOTIMPALA_HBASE_VERSION    = 2.1.0-cdh6.x-SNAPSHOTIMPALA_SENTRY_VERSION   = 2.1.0-cdh6.x-SNAPSHOTIMPALA_KUDU_VERSION     = 1.11.0-cdh6.x-SNAPSHOTIMPALA_KUDU_JAVA_VERSION= 1.11.0-cdh6.x-SNAPSHOTIMPALA_RANGER_VERSION   = 1.2.0.7.1.0.0-33  

4、buildall.sh文件

执行命令，需要下载python相关依赖，这边更换pip国内源，在impala/infra/python/deps/pip_download.py，清华源:https://pypi.tuna.tsinghua.edu.cn

在下载python相关依赖的时候，有一个包名有问题，原来是python_dateutil == 2.5.2​，应该改为：python-dateutil == 2.5.2​