Ranger 0.5.4-SNAPSHOT CDH 5.8.4 HDFS Plugin Hive Plugin

组件版本信息：

---

Hadoop 版本：cdh-5.8.4
Hive 版本: 1.1.0-cdh5.8.4
Ranger 版本: ranger-0.5.4-SNAPSHOT
Solr 版本: 5.2.1
Jdk 版本：1.7.80

前提条件：
cdh-5.8.4 需要修复 HDFS-6826，否则Ranger HDFS Plugin编译无法通过
本地mvn仓库编译安装修改后hdfs依赖包：
[cloudera@quickstart hadoop-2.6.0-cdh5.8.4-src]$ mvn clean install package -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcdh.mr1.version=2.6.0-mr1-cdh5.8.4 -Dcdh.hadoop.version=2.6.0-cdh5.8.4 -Dcdh.hbase.version=1.2.0-cdh5.8.4 -Dcdh.zookeeper.version=3.4.5-cdh5.8.4 -Phadoop-2 -Dmaven.test.skip=true
编译Ranger：
JDK 版本：
[cloudera@quickstart ~]$ java -version
java version "1.7.0_80"
[cloudera@quickstart ~]$ git clone https://github.com/apache/ranger.git
[cloudera@quickstart ranger]$ git checkout ranger-0.5

POM文件修改下面Hadoop,Hive相关内容:
+ <hadoop-auth.version>2.6.0-cdh5.8.4</hadoop-auth.version>
+ <hadoop-common.version>2.6.0-cdh5.8.4</hadoop-common.version>
+ <hadoop-common-kms.version>2.6.0-cdh5.8.4</hadoop-common-kms.version>
+ <hadoop.version>2.6.0-cdh5.8.4</hadoop.version>
+ <hive.version>1.1.0-cdh5.8.4</hive.version>

[cloudera@quickstart ranger]$ mvn -Drat.skip=true -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 compile install package assembly:assembly

---

安装Solr记录审计日志：
官网建议要求
JDK 1.7, Apache Solr 5.2
Solr memory and cpu intensive
32GB RAM, 1TB free space

Solr - Standalone
172.17.8.4

[cloudera@quickstart ~] tar -cvf ranger.tar.gz ranger/*
[cloudera@quickstart ~] scp ranger.tar.gz [email protected]:/home/xxxxxxx/
[xxxxxxx@xxxx-dp-dev05-84 ~]$ tar -xvf ranger.tar.gz
[xxxxxxx@xxxx-dp-dev05-84 ~]$ cd ranger/security-admin/contrib/solr_for_audit_setup/
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ pwd
/home/xxxxxxx/ranger/security-admin/contrib/solr_for_audit_setup

编辑自动安装配置：
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ vim install.properties
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ cat install.properties | grep -v "^#" | grep -v "^$"
注意下面配置项：
JAVA_HOME=/usr/java/jdk1.7.0_80
SOLR_USER=solr
SOLR_INSTALL=true
SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
SOLR_INSTALL_FOLDER=/opt/solr
SOLR_RANGER_HOME=/opt/solr/ranger_audit_server
SOLR_RANGER_PORT=6083
SOLR_DEPLOYMENT=standalone
SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data
SOLR_LOG_FOLDER=/var/log/solr/ranger_audits
SOLR_MAX_MEM=2g

自动安装：
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ sudo ./setup.sh
Fri Oct 11 17:30:41 CST 2019|INFO|Downloading solr from http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
--2019-10-11 17:30:41-- http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
Resolving archive.apache.org (archive.apache.org)... 163.172.17.199
Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 148849766 (142M) [application/x-gzip]
Saving to: ‘solr-5.2.1.tgz’

100%[=========================================================================================>] 148,849,766 3.41MB/s in 4m 49s

2019-10-11 17:35:31 (503 KB/s) - ‘solr-5.2.1.tgz’ saved [148849766/148849766]

Fri Oct 11 17:35:33 CST 2019|INFO|Installed Solr in /opt/solr
Fri Oct 11 17:35:33 CST 2019|INFO|Configuring standalone instance
Fri Oct 11 17:35:33 CST 2019|INFO|Copying Ranger Audit Server configuration to /opt/solr/ranger_audit_server
df: ‘/opt/solr/ranger_audit_server/data’: No such file or directory
Fri Oct 11 17:35:35 CST 2019|INFO|Done configuring Solr for Apache Ranger Audit
Fri Oct 11 17:35:35 CST 2019|INFO|Solr HOME for Ranger Audit is /opt/solr/ranger_audit_server
Fri Oct 11 17:35:35 CST 2019|INFO|Data folder for Audit logs is /opt/solr/ranger_audit_server/data
Fri Oct 11 17:35:35 CST 2019|INFO|To start Solr run /opt/solr/ranger_audit_server/scripts/start_solr.sh
Fri Oct 11 17:35:35 CST 2019|INFO|To stop Solr run /opt/solr/ranger_audit_server/scripts/stop_solr.sh
Fri Oct 11 17:35:35 CST 2019|INFO|After starting Solr for RangerAudit, it will listen at 6083. E.g http://xxxx-dp-dev05-84.jpushoa.com:6083
Fri Oct 11 17:35:35 CST 2019|INFO|Configure Ranger to use the following URL http://xxxx-dp-dev05-84.jpushoa.com:6083/solr/ranger_audits
########## Done ###################
Created file /opt/solr/ranger_audit_server/install_notes.txt with instructions to start and stop
###################################

[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ cat /opt/solr/ranger_audit_server/install_notes.txt
Solr installation notes for Ranger Audits.

Note: Don't edit this file. It will be over written if you run ./setup.sh again.

You have installed Solr in standalone mode.

Note: In production deployment, it is recommended to run in SolrCloud mode with at least 2 nodes and replication factor 2

Start and Stoping Solr:
Login as user solr or root and the run the below commands to start or stop Solr:

To start Solr run: /opt/solr/ranger_audit_server/scripts/start_solr.sh
To stop Solr run: /opt/solr/ranger_audit_server/scripts/stop_solr.sh

After starting Solr for RangerAudit, Solr will listen at 6083. E.g http://xxxx-dp-dev05-84.jpushoa.com:6083

Configure Ranger to use the following URL http://xxxx-dp-dev05-84.jpushoa.com:6083/solr/ranger_audits

Solr HOME for Ranger Audit is /opt/solr/ranger_audit_server

DATA FOLDER: /opt/solr/ranger_audit_server/data

Make sure you have enough disk space for index. In production, it is recommended to have at least 1TB free.

[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ sudo /opt/solr/ranger_audit_server/scripts/start_solr.sh
Running this script as solr...
This account is currently not available.
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ sudo useradd solr
useradd: user 'solr' already exists
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ sudo vim /etc/passwd
solr:x:597:600:Solr:/var/lib/solr:/bin/bash
[xxxxxxx@xxxx-dp-dev05-84 solr_for_audit_setup]$ sudo /opt/solr/ranger_audit_server/scripts/start_solr.sh
Running this script as solr...

Started Solr server on port 6083 (pid=21030). Happy searching!

网址：http://172.17.8.4:6083

配置Ranger时，注意修改下面配置项：
Configuring Ranger Admin and Ranger Plugins
Ranger Admin and Ranger Plugins need the URL to Solr collection. Check the install_notes.txt for the appropriate value. The sample URL is:

http://${SOLR_HOST}:6083/solr/ranger_audits (Replace ${SOLR_HOST} with the server were Solr is installed.

For Ranger Admin, configure the following properties in install.properties:

#Source for Audit DB
# * audit_db is solr or db
audit_store=solr

# * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits
audit_solr_urls=http://localhost:6083/solr/ranger_audits

 

For all plugins, configure the following properties in install.properties

XAAUDIT.SOLR.ENABLE=true
XAAUDIT.SOLR.URL=http://localhost:6083/solr/ranger_audits

(replace localhost with the Solr host)

---

安装Ranger Admin(管理权限策略):

ranger admin 安装(172.17.8.94)：
[xxxxxxx@xxxx-dp-dev05-84 target]$ scp ranger-0.5.4-SNAPSHOT-admin.tar.gz 172.17.8.94:/home/xxxxxxx/
登录：172.17.8.94
[xxxxxxx@xxxx-dp-dev06-894 ~]$ tar -xvf ranger-0.5.4-SNAPSHOT-admin.tar.gz
[xxxxxxx@xxxx-dp-dev06-894 ~]$ cd /usr/local/
[xxxxxxx@xxxx-dp-dev06-894 local]$ pwd
/usr/local
[xxxxxxx@xxxx-dp-dev06-894 local]$ sudo mv ~/ranger-0.5.4-SNAPSHOT-admin ./
[xxxxxxx@xxxx-dp-dev06-894 local]$ sudo chown -R root:root ranger-0.5.4-SNAPSHOT-admin/
[xxxxxxx@xxxx-dp-dev06-894 local]$ sudo ln -s ranger-0.5.4-SNAPSHOT-admin ranger-admin
[xxxxxxx@xxxx-dp-dev06-894 local]$ cd /usr/local/ranger-admin/

建立ranger数据库管理员：
CREATE USER 'rangerdba'@'localhost' IDENTIFIED BY 'rangerdba';
GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost' WITH GRANT OPTION;
CREATE USER 'rangerdba'@'%' IDENTIFIED BY 'rangerdba';
GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

[xxxxxxx@xxxx-dp-dev06-894 ranger-admin]$ sudo mv ~/mysql-connector-java-5.1.37.jar /usr/local/ranger-0.5.4-SNAPSHOT-admin/
[xxxxxxx@xxxx-dp-dev06-894 ranger-admin]$ sudo chown root:root mysql-connector-java-5.1.37.jar
配置文件：
[xxxxxxx@xxxx-dp-dev06-894 ranger-admin]$ sudo cat install.properties | grep -v "^#" | grep -v "^$"
DB_FLAVOR=MYSQL
SQL_CONNECTOR_JAR=/usr/local/ranger-admin/mysql-connector-java-5.1.37.jar
db_root_user=rangerdba
db_root_password=rangerdba
db_host=172.17.8.48:3306
db_name=ranger
db_user=rangeradmin
db_password=******
audit_store=solr
audit_solr_urls=http://172.17.8.4:6083/solr/ranger_audits
policymgr_external_url=http://localhost:6080

dba_script.py
db_setup.py
添加：
os.environ['JAVA_HOME'] = '/usr/java/jdk1.7.0_80'
/usr/local/ranger-0.5.4-SNAPSHOT-admin/db/mysql/create_dbversion_catalog.sql
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Invalid default value for