Centos7 + Apache Ranger 2.4.0 编译(docker方式)

目录

一、Ranger简介

1、组件列表

2、支持的数据引擎服务

二、主机环境准备

1、关闭防火墙

2、关闭SELINUX

3、安装docker

4、下载Ranger源码包

5、下载Maven安装包

三、编译Ranger源码

1、修改官方包中的build_ranger_using_docker.sh

2、运行脚本编译

3、编译检查


一、Ranger简介

        Apache Ranger提供一个集中式安全管理框架, 并解决授权和审计。它可以对Hadoop生态的组件如HDFS、Yarn、Hive、Hbase等进行细粒度的数据访问控制。通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问权限。

1、组件列表

#

Service Name

Listen Port

Core Ranger Service

1ranger6080/tcpY (ranger engine - 3.0.0-SNAPSHOT version)
2ranger-postgres5432/tcpY (ranger datastore)
3ranger-solr8983/tcpY (audit store)
4ranger-zk2181/tcpY (used by solr)
5ranger-usersync-Y (user/group synchronization from Local Linux/Mac)
6ranger-kms9292/tcpN (needed only for Encrypted Storage / TDE)
7ranger-tagsync-N (needed only for Tag Based Policies to be sync from ATLAS)

2、支持的数据引擎服务

#

Service Name

Listen Port

Service Description

1Hadoop

8088/tcp
9000/tcp

Apache Hadoop 3.3.0
Protected by Apache Ranger's Hadoop Plugin

2HBase16000/tcp
16010/tcp
16020/tcp
16030/tcp
Apache HBase 2.4.6
Protected by Apache Ranger's HBase Plugin
3

Hive

10000/tcpApache Hive 3.1.2
Protected by Apache Ranger's Hive Plugin
4Kafka6667/tcpApache Kafka 2.8.1
Protected by Apache Ranger's Kafka Plugin
5Knox8443/tcpApache Knox 1.4.0
Protected by Apache Ranger's Knox Plugin

二、主机环境准备

1、关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

2、关闭SELINUX

sed -i.bak$DATE '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
setenforce 0

3、安装docker

yum install -y docker
systemctl start docker
systemctl enable docker

4、下载Ranger源码包

Apache Ranger官网没有可以直接部署的安装包,必须通过源码进行编译。

官网地址:Apache Ranger - Download Apache Ranger?

wget https://www.apache.org/dist/ranger/2.4.0/apache-ranger-2.4.0.tar.gz --no-check-certificate

5、下载Maven安装包

wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate

三、编译Ranger源码

1、修改官方包中的build_ranger_using_docker.sh

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License

#This script creates the Docker image (if not already created) and runs maven in the container
#1. Install Docker
#2. Checkout Ranger source and go to the root directory
#3. Run this script. If host is linux, then run this script as "sudo $0 ..."
#4. If you are running on Mac, then you don't need to use "sudo"
#5. To delete the image, run "[sudo] docker rmi ranger_dev"

#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn  <build params>
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true 
#Notes: To remove build image manually, run "docker rmi ranger_dev" or "sudo docker rmi ranger_dev"

default_command="mvn -Pall -DskipTests=true clean compile package install"
build_image=0
if [ "$1" = "-build_image" ]; then
    build_image=1
    shift
fi

params=$*
if [ $# -eq 0 ]; then
    params=$default_command
fi

image_name="ranger_dev"
remote_home="$HOME"
container_name="--name ranger_build"

if [ ! -d security-admin ]; then
    echo "ERROR: Run the script from root folder of source. e.g. $HOME/git/ranger"
    exit 1
fi

images=`docker images | cut -f 1 -d " "`
[[ $images =~ $image_name ]] && found_image=1 || build_image=1

if [ $build_image -eq 1 ]; then
    echo "Creating image $image_name ..."
    docker rmi -f $image_name

docker build -t $image_name - <<Dockerfile
FROM centos:centos7.6.1810

RUN mkdir /tools
WORKDIR /tools

#Install default services
RUN yum install -y wget git gcc bzip2 fontconfig python3 java-1.8.0-openjdk-devel.x86_64
RUN ln -sf /usr/bin/python3 /usr/bin/python

ENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk/
ENV PATH $JAVA_HOME/bin:$PATH

RUN wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate

RUN tar xfz apache-maven-3.9.4-bin.tar.gz
RUN ln -sf /tools/apache-maven-3.9.4 /tools/maven

ENV  PATH /tools/maven/bin:$PATH
ENV MAVEN_OPTS "-Xmx2048m -XX:MaxPermSize=512m"

RUN mkdir -p /scripts

RUN echo "#!/bin/bash" > /scripts/mvn.sh
RUN echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh

RUN chmod -R 777 /scripts
RUN chmod -R 777 /tools

ENTRYPOINT ["/scripts/mvn.sh"]
Dockerfile

fi

src_folder=`pwd`
LOCAL_M2="$HOME/.m2"
mkdir -p $LOCAL_M2
set -x

docker run --rm  -v "${src_folder}:/ranger" -w "/ranger" -v "${LOCAL_M2}:${remote_home}/.m2" $container_name $image_name $params

说明:
考虑实验运行环境为centos7.6,且国内有部分外国源访问不到,所以做了一定修改和裁剪,本次实验中,使用root用户运行该脚本,容器内使用root用户进行编译操作。
1、原脚本的基础镜像为centos:lastest,其对应Centos 8.1,现修改为centos:centos7.6.1810;
2、原脚本安装jkd8时,使用AWS s3's docker-assets里的jdk-8u101-linux-x64.rpm,现修改为使用centos自带的openjdk1.8,即java-1.8.0-openjdk-devel.x86_64;
3、原脚本未安装python3,最终编译时会找不到python3包而报错,现增加安装python3,同时设置默认使用python3,即RUN ln -sf /usr/bin/python3 /usr/bin/python
4、原脚本安装maven时,使用ADD来获取apache-maven-3.6.3-bin.tar.gz并校验包,现修改为使用wget获得最新的apache-maven-3.9.4-bin.tar.gz,且不做额外的包正确性校验,即wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificatewget
5、原脚本的启动脚本创建并使用了非root用户builder,但会与后面运行容器时映射本地卷组有权限限制,考虑到只是临时编译使用,剔除所有builder用户的内容,包含gosu安装、用户创建、用户判断等,只保留echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh
6、原脚本中${remote_home}为空值,会将运行该脚本的用户Home下的.m2映射到容器内根目录的.m2,现修改为容器内工作用户的Home目录,即remote_home="$HOME"

2、运行脚本编译

chmod +x build_ranger_using_docker.sh
./build_ranger_using_docker.sh

说明:
参照脚本使用说明
#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn  <build params>
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true 

3、编译检查

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 2.4.0:
[INFO] 
[INFO] ranger ............................................. SUCCESS [ 12.567 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [ 13.553 s]
[INFO] Credential Support ................................. SUCCESS [ 14.914 s]
[INFO] Audit Component .................................... SUCCESS [01:09 min]
[INFO] ranger-plugin-classloader .......................... SUCCESS [  9.662 s]
[INFO] Common library for Plugins ......................... SUCCESS [02:03 min]
[INFO] ranger-intg ........................................ SUCCESS [ 40.185 s]
[INFO] Installer Support Component ........................ SUCCESS [  8.196 s]
[INFO] Credential Builder ................................. SUCCESS [ 12.157 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 33.355 s]
[INFO] Key Management Service ............................. SUCCESS [01:40 min]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 52.109 s]
[INFO] HBase Security Plugin .............................. SUCCESS [01:25 min]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 36.159 s]
[INFO] Hive Security Plugin ............................... SUCCESS [ 41.491 s]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [  9.255 s]
[INFO] Knox Security Plugin ............................... SUCCESS [ 21.750 s]
[INFO] Storm Security Plugin .............................. SUCCESS [ 16.017 s]
[INFO] YARN Security Plugin ............................... SUCCESS [ 13.554 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 12.752 s]
[INFO] Ranger Util ........................................ SUCCESS [ 11.776 s]
[INFO] Unix Authentication Client ......................... SUCCESS [ 11.990 s]
[INFO] User Group Synchronizer Util ....................... SUCCESS [  6.909 s]
[INFO] Security Admin Web Application ..................... SUCCESS [08:54 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [01:17 min]
[INFO] SOLR Security Plugin ............................... SUCCESS [01:18 min]
[INFO] NestedStructure Security Plugin .................... SUCCESS [ 24.474 s]
[INFO] NiFi Security Plugin ............................... SUCCESS [ 12.265 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 11.211 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 24.201 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [ 14.920 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [02:08 min]
[INFO] Ldap Config Check Tool ............................. SUCCESS [ 11.640 s]
[INFO] Unix Authentication Service ........................ SUCCESS [ 11.348 s]
[INFO] KMS Security Plugin ................................ SUCCESS [01:13 min]
[INFO] Tag Synchronizer ................................... SUCCESS [ 45.784 s]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [  9.535 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [01:23 min]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 42.092 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 23.710 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [ 10.665 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 10.838 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 22.091 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 28.752 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 52.920 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [ 26.065 s]
[INFO] ranger-examples .................................... SUCCESS [  0.272 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 11.692 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [  5.863 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 10.167 s]
[INFO] sample-client ...................................... SUCCESS [ 11.777 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [  6.742 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 35.518 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 41.615 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [03:02 min]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 53.693 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 14.680 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [03:33 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [ 41.171 s]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 22.381 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 37.204 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [02:26 min]
[INFO] Unix Native Authenticator .......................... SUCCESS [  4.438 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  49:17 min
[INFO] Finished at: 2023-08-07T10:43:31Z
[INFO] ------------------------------------------------------------------------

在target目录可以看到生成的程序包:

-rw-r--r-- 1 root root 579387182 Aug  7 18:42 ranger-2.4.0-admin.tar.gz
-rw-r--r-- 1 root root  43729654 Aug  7 18:43 ranger-2.4.0-atlas-plugin.tar.gz
-rw-r--r-- 1 root root  34172214 Aug  7 18:43 ranger-2.4.0-elasticsearch-plugin.tar.gz
-rw-r--r-- 1 root root  39122941 Aug  7 18:42 ranger-2.4.0-hbase-plugin.tar.gz
-rw-r--r-- 1 root root  37684529 Aug  7 18:42 ranger-2.4.0-hdfs-plugin.tar.gz
-rw-r--r-- 1 root root  37478412 Aug  7 18:42 ranger-2.4.0-hive-plugin.tar.gz
-rw-r--r-- 1 root root  56846325 Aug  7 18:42 ranger-2.4.0-kafka-plugin.tar.gz
-rw-r--r-- 1 root root 195376717 Aug  7 18:43 ranger-2.4.0-kms.tar.gz
-rw-r--r-- 1 root root  51454934 Aug  7 18:42 ranger-2.4.0-knox-plugin.tar.gz
-rw-r--r-- 1 root root  36625366 Aug  7 18:43 ranger-2.4.0-kylin-plugin.tar.gz
-rw-r--r-- 1 root root     34201 Aug  7 18:43 ranger-2.4.0-migration-util.tar.gz
-rw-r--r-- 1 root root  43393403 Aug  7 18:42 ranger-2.4.0-ozone-plugin.tar.gz
-rw-r--r-- 1 root root  57425250 Aug  7 18:43 ranger-2.4.0-presto-plugin.tar.gz
-rw-r--r-- 1 root root  16563346 Aug  7 18:43 ranger-2.4.0-ranger-tools.tar.gz
-rw-r--r-- 1 root root     36915 Aug  7 18:42 ranger-2.4.0-solr_audit_conf.tar.gz
-rw-r--r-- 1 root root  38256335 Aug  7 18:42 ranger-2.4.0-solr-plugin.tar.gz
-rw-r--r-- 1 root root  36860763 Aug  7 18:43 ranger-2.4.0-sqoop-plugin.tar.gz
-rw-r--r-- 1 root root   6376456 Aug  7 18:43 ranger-2.4.0-src.tar.gz
-rw-r--r-- 1 root root  51760282 Aug  7 18:42 ranger-2.4.0-storm-plugin.tar.gz
-rw-r--r-- 1 root root  31046503 Aug  7 18:42 ranger-2.4.0-tagsync.tar.gz
-rw-r--r-- 1 root root  20128101 Aug  7 18:42 ranger-2.4.0-usersync.tar.gz
-rw-r--r-- 1 root root  35792990 Aug  7 18:42 ranger-2.4.0-yarn-plugin.tar.gz

参考文档:

Ranger Installation Guide - Ranger - Apache Software Foundation

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

snipercai

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值