Apache Sentry手动安装、使用手册

Sentry手动安装、使用手册

1 Sentry简介

Apache Sentry 是Cloudera公司发布的一个Hadoop开源组件,截止目前还是Apache的孵化项目,它提供了细粒度级、基于角色的授权以及多租户的管理模式。Sentry当前可以和Hive/Hcatalog、Apache Solr 和Cloudera Impala集成,未来会扩展到其他的Hadoop组件,例如HDFS和HBase.

2 准备

2.1 环境说明

1 Sentry安装采用rpm包的方式.

2 Hadoop版本为hadoop-2.5.0-cdh5.3.3,Hive版本为hive-0.13.1-cdh5.3.3,Sentry版本为sentry-1.4.0-cdh5.3.3

3 Sentry下载地址:

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.3.3/RPMS/noarch/

 

2.1.1虚拟机里Linux系统版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14    |

+-----------+

注:

1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar

2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享

 

3 安装

3.1安装Mysql

1)安装mysql相关服务

rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm时报如下错误:

file/usr/share/mysql/charsets/macroman.xml from install of MySQL-server-5.6.14-1.el6.x86_64conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64 file/usr/share/mysql/charsets/swe7.xml from install ofMySQL-server-5.6.14-1.el6.x86_64 conflicts with file from packagemysql-libs-5.1.71-1.el6.x86_64

rpm -q mysql-libs-5.1.71-1.el6.x86_64

rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

ARANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !

Youwill find that password in '/root/.mysql_secret'.(生成mysql root密码)

Youmust change that password on your first connect,

noother statement but 'SET PASSWORD' will be accepted.

rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

service mysql start

#这里密码 来自/root/.mysql_secret

mysql -uroot -p9RNrbk9O

 

#首次执行会提示修改mysql root密码

SET PASSWORD=PASSWORD('root');

#创建Hive数据库为Hive的元数据库

create database hive;

GRANT all ON hive.* TO root@'%' IDENTIFIED BY 'root';

#创建sentry数据库为sentry元数据库

create database sentry;

CREATE USER sentry IDENTIFIED BY 'sentry';

GRANT all ON sentry.* TO sentry@'%' IDENTIFIED BY'sentry';

flush privileges;

 

3.2 安装Hive

1)解压Hive压缩包,并配置环境变量

cd /root

tar –zxvf hive-0.13.1-cdh5.3.3.tar.gz

vi ~/.bash_profile

exportHIVE_HOME=/root/hive-0.13.1-cdh5.3.3

#追加HIVE执行文件路径到PATH内

exportPATH=$PATH:$HIVE_HOME/bin

2) 拷贝Mysql驱动到Hive的lib目录下:

cp     /root/mysql-connector-java-5.1.31-bin.jar/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar

 

3)配置Hive的conf.详细的hive-site.xml配置内容见下:

<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
 <name>javax.jdo.option.ConnectionURL</name>
 <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
  <description>JDBC connect string for aJDBC metastore</description>
</property>
 
<property>
 <name>javax.jdo.option.ConnectionDriverName</name>
 <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for aJDBC metastore</description>
</property>
 
<property>
 <name>javax.jdo.PersistenceManagerFactoryClass</name>
 <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
  <description>class implementing the jdopersistence</description>
</property>
 
<property>
  <name>javax.jdo.option.DetachAllOnCommit</name>
  <value>true</value>
  <description>detaches all objects fromsession so that they can be used after transaction iscommitted</description>
</property>
 
<property>
 <name>javax.jdo.option.NonTransactionalRead</name>
  <value>true</value>
  <description>reads outside oftransactions</description>
</property>
 
<property>
 <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
  <description>username to use againstmetastore database</description>
</property>
 
<property>
 <name>javax.jdo.option.ConnectionPassword</name>
  <value>root</value>
  <description>password to use againstmetastore database</description>
</property>
 
<property>
 <name>javax.jdo.option.Multithreaded</name>
  <value>true</value>
  <description>Set this to true ifmultiple threads access metastore through JDO concurrently.</description>
</property>
 
</configuration>


3.3 安装Sentry

1)安装Sentry相关服务

rpm -ivh --nodepssentry-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm

rpm -ivh --nodeps sentry-hdfs-plugin-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm

rpm -ivh --nodepssentry-store-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm

 

2)替换Sentry内的Hadoop、Hive、Impala、Hbase、Zookeeper、Parquet、Avro等jar包

rm -rf /usr/lib/sentry/lib/hive*.jar

rm -rf /usr/lib/sentry/lib/hadoop*.jar

rm -rf /usr/lib/sentry/lib/zookeeper*.jar

rm -rf /usr/lib/sentry/lib/avro*.jar

rm -rf /usr/lib/sentry/lib/server/hive-beeline.jar

cp ~/SentryLibs/* /usr/lib/sentry/lib/

注:这里的Sentrylibs里的jar包是通过/usr/lib/sentry/lib里的文件名分别在Hadoop、Hive、Hbase等里的安装目录里一一找到的.

mv /root/SentryLibs/hive-beeline.jar/usr/lib/sentry/lib/server/  

#拷贝mysql驱动到sentry的lib目录内 

cp/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar/usr/lib/sentry/lib/

#如果装了Impala,此步可忽略

rpm -ivh bigtop-utils-0.7.0+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6.noarch.rpm

 

3) 配置Sentry,详细配置见sentry-site.xml内容如下:

<?xmlversion="1.0" encoding="UTF-8"?>
<configuration>
    <property>
       <name>sentry.service.security.mode</name>
        <value>none</value>
    </property>
    <property>
       <name>sentry.service.admin.group</name>
        <value>impala,hive</value>
    </property>
    <property>
       <name>sentry.service.allow.connect</name>
        <value>impala,hive</value>
    </property>
    <property>
        <name>sentry.verify.schema.version</name>
        <value>true</value>
    </property>
    <property>
   <name>sentry.service.server.rpc-address</name>
    <value>data1</value>
    </property>
    <property>
   <name>sentry.service.server.rpc-port</name>
    <value>8038</value>
    </property>
    <property>
       <name>sentry.store.jdbc.url</name>
       <value>jdbc:mysql://localhost:3306/sentry</value>
    </property>
    <property>
       <name>sentry.store.jdbc.driver</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
       <name>sentry.store.jdbc.user</name>
        <value>sentry</value>
    </property>
    <property>
       <name>sentry.store.jdbc.password</name>
        <value>sentry</value>
    </property>
    <property>
       <name>sentry.hive.server</name>
        <value>data1</value>
    </property>
    <property>
       <name>sentry.store.group.mapping</name>
       <value>org.apache.sentry.provider.common.HadoopGroupMappingService</value>
    </property>
</configuration>


 

4)初始化Sentry元数据

sentry --command schema-tool --conffile/etc/sentry/conf/sentry-site.xml --dbType mysql --initSchema

                  ..........................

                  No rows affected (0.094 seconds)

                  No rows affected (0.015 seconds)

                  No rows affected (0.075 seconds)

                  1 row affected (0.007 seconds)

                  Closing: 0: jdbc:mysql://localhost:3306/sentry

                  Initialization script completed

                  Sentry schemaTool completed

 

3.4 安装问题

暂无.

4 配置

Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr

注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr.

4.1 Hive集成Sentry配置

1) 拷贝Sentry相关jar包到Hive的lib目录内

 cp /usr/lib/sentry/lib/sentry*.jar$HIVE_HOME/lib/

 cp /usr/lib/sentry/lib/shiro-core-*.jar$HIVE_HOME/lib/

 

2)  Hive conf内新增Sentry-site.xml,配置内容见下:

<?xmlversion="1.0" encoding="UTF-8"?>
<configuration>
 
<property>
     <name>sentry.service.security.mode</name>
     <value>none</value>
</property>
<property>
     <name>sentry.service.server.principal</name>
     <value>hivemeta/centos64.cloudera.com@HS2.CLOUDERA.COM</value>
</property>
    <property>
       <name>sentry.service.client.server.rpc-port</name>
       <value>8038</value>
     </property>
     <property>
       <name>sentry.service.client.server.rpc-address</name>
       <value>data1</value>
     </property>
     <property>
       <name>sentry.service.client.server.rpc-connection-timeout</name>
       <value>200000</value>
     </property>
     <property>
              <name>sentry.hive.provider</name>
              <value>org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider</value>
      </property>
     <property>
              <name>sentry.hive.provider.backend</name>
              <value>org.apache.sentry.provider.db.SimpleDBProviderBackend</value>
      </property>
     <property>
              <name>sentry.hive.server</name>
              <value>server1</value>
      </property>
      <property>
              <name>sentry.metastore.service.users</name>
              <value>root</value>
      </property>
     <property>
              <name>sentry.hive.testing.mode</name>
              <value>true</value>
      </property>
</configuration>


 

3)  修改Hive conf内的hive-site.xml,修改成如下:

<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<configuration>
<!--########################################################################################-->
<!--################################ sentry for metastore############################################# -->
<!--
<property>
    <name>hive.metastore.client.impl</name>
   <value>org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient</value>
    <description>Sets custom Hivemetastore client which Sentry uses to filter out metadata.</description>
</property>
-->
<property>
    <name>hive.metastore.execute.setugi</name>
    <value>true</value>
    <description>
      In unsecure mode, setting this propertyto true will cause the metastore to execute DFS operations using
      the client's reported user and grouppermissions. Note that this property must be set on
      both the client and server sides. Furthernote that its best effort.
      If client sets its to true and serversets it to false, client setting will be ignored.
    </description>
 </property>
<property>
  <name>hive.metastore.pre.event.listeners</name>
 <value>org.apache.sentry.binding.metastore.MetastoreAuthzBinding</value>
</property>
 
<property>
 <name>hive.metastore.event.listeners</name>
 <value>org.apache.sentry.binding.metastore.SentryMetastorePostEventListener</value>
</property>
 
 
 
<property>
     <name>hive.server2.enable.impersonation</name>
     <value>true</value>
</property>
<property>
   <name>hive.security.authorization.task.factory</name>
              <value>org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl</value>
</property>
<property>
   <name>hive.server2.session.hook</name>
   <value>org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook</value>
</property>
<property>
   <name>hive.sentry.conf.url</name>
    <value>file:///root/hive-0.13.1-cdh5.3.3/conf/sentry-site.xml</value>
</property>
<!--
<property>
  <name>hive.metastore.uris</name>
 <value>thrift://data1:9083</value>
  <description>Thrift URI for the remotemetastore. Used by metastore client to connect to remotemetastore.</description>
</property>
-->
<property>
 <name>javax.jdo.option.ConnectionURL</name>
 <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
  <description>JDBC connect string for aJDBC metastore</description>
</property>
 
<property>
 <name>javax.jdo.option.ConnectionDriverName</name>
 <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for aJDBC metastore</description>
</property>
<property>
  <name>javax.jdo.PersistenceManagerFactoryClass</name>
 <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
  <description>class implementing the jdopersistence</description>
</property>
<property>
 <name>javax.jdo.option.DetachAllOnCommit</name>
  <value>true</value>
  <description>detaches all objects fromsession so that they can be used after transaction iscommitted</description>
</property>
<property>
 <name>javax.jdo.option.NonTransactionalRead</name>
  <value>true</value>
  <description>reads outside of transactions</description>
</property>
<property>
 <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
  <description>username to use againstmetastore database</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>root</value>
  <description>password to use againstmetastore database</description>
</property>
<property>
 <name>javax.jdo.option.Multithreaded</name>
  <value>true</value>
  <description>Set this to true ifmultiple threads access metastore through JDO concurrently.</description>
</property>
</configuration>


 

4)  验证Sentry的权限控制,详见 5 使用:

 

 

4.2 Impala集成Sentry配置

1)  Impala的conf里新增sentry配置,/etc/impala/conf/sentry-site.xml内容如下:

<?xmlversion="1.0" encoding="UTF-8"?>
<configuration>
<property>
  <name>sentry.service.client.server.rpc-port</name>
   <value>8038</value>
</property>
<property>
  <name>sentry.service.client.server.rpc-address</name>
   <value>data1</value>
</property>
<property>
   <name>sentry.service.client.server.rpc-connection-timeout</name>
   <value>200000</value>
</property>
<property>
  <name>sentry.service.security.mode</name>
   <value>none</value>
</property>
</configuration>



2)  修改impala的参数

vi/etc/default/impala

IMPALA_CATALOG_ARGS 参数里追加 -sentry_config=/etc/impala/conf/sentry-site.xml

IMPALA_SERVER_ARGS 参数里追加 -server_name=server1\

   -sentry_config=/etc/impala/conf/sentry-site.xml

5 使用

5.1 Hive集成Sentry权限验证

1) 启动sentry服务,hive的metastore和hiveserver2服务

sentry --command service--conffile /etc/sentry/conf/sentry-site.xml

nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &

nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &

#hiveserver2里新增hive和test用户,通过beeline方式访问.

groupadd hive; useradd hive -ghive;passwd hive (type hive123)

groupadd test; useradd test -gtest;passwd test (type test123)

beeline -u "jdbc:hive2://data1:10000"-n hive -p test

 

2)验证整体思路及场景概述

思路:

1 HIVE用户属于管理员组,服务整个server的权限

2 Test用户创建两个数据库test_only(all权限),test_select_only(仅有select权限)

具体验证:

1 通过show databases和use database验证访问权限

2 通过test用户的use test_only能drop table,create table验证all权限

3 通过test用户的use test_select_only不能drop table验证仅有select权限

 

#场景一 root用户连入,尝试执行DDL操作.(无权操作)

[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n root

                  scan complete in 17ms

                  Connecting to jdbc:hive2://data1:10000

                  Connected to: Apache Hive (version 0.13.1-cdh5.3.3)

                  Driver: Hive JDBC (version 0.13.1-cdh5.3.3)

                  Transaction isolation: TRANSACTION_REPEATABLE_READ

                  Beeline version 0.13.1-cdh5.3.3 by Apache Hive

0:jdbc:hive2://data1:10000>create database sensitive;

        Error: Error while compiling statement: FAILED:SemanticException No valid privileges

        Required privileges for this query: Server=server1->action=*;(state=42000,code=40000)

        0: jdbc:hive2://data1:10000> create role admin_role;

        Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.SentryAccessDeniedException: Access denied to root (state=08S01,code=1)

0:jdbc:hive2://data1:10000> !q

        Closing: 0: jdbc:hive2://data1:10000

 

#场景二 hive用户连入,对用户hive、test进行授权.

 

[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n hive

                  scan complete in 5ms

                  Connecting to jdbc:hive2://data1:10000

                  Connected to: Apache Hive (version 0.13.1-cdh5.3.3)

                  Driver: Hive JDBC (version 0.13.1-cdh5.3.3)

                  Transaction isolation: TRANSACTION_REPEATABLE_READ

                  Beeline version 0.13.1-cdh5.3.3 by Apache Hive

0:jdbc:hive2://data1:10000>create role admin_role;

        No rows affected (1.383 seconds)

GRANT ALL ONSERVER server1 TO ROLE admin_role;

GRANT ROLEadmin_role TO GROUP hive;

create roletest_role;

GRANT ALL ONDATABASE test_only TO ROLE test_role;

GRANT ROLEtest_role TO GROUP test;

GRANT SELECT ONDATABASE test_select_only TO ROLE test_role;

#场景三 test用户连入,对数据库admin_only、test_select_only进行权限验证.

beeline -u"jdbc:hive2://data1:10000" -n test

0:jdbc:hive2://data1:10000> showdatabases;

                  +----------------+--+

                  | database_name  |

                  +----------------+--+

                  | admin_only     |

                  | default        |

                  | test_only      |

                  +----------------+--+

                  3 rows selected (0.721 seconds)

0:jdbc:hive2://data1:10000>use admin_only;

                  Error: Error while compiling statement: FAILED:SemanticException No valid privileges

                  Required privileges for this query:Server=server1->Db=admin_only->Table=*->action=insert;Server=server1->Db=admin_only->Table=*->action=select;(state=42000,code=40000)           

0:jdbc:hive2://data1:10000>use test_select_only;

                  No rows affected (0.313 seconds)

0:jdbc:hive2://data1:10000>show tables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | select_only  |

                  +--------------+--+

                  1 row selected (0.337 seconds)

0:jdbc:hive2://data1:10000>drop table select_only;

                  Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.DDLTask.MetaException(message:hive.metastore.execute.setugi can't be false in nonsecure mode) (state=08S01,code=1)

    #这里需要在HIVE服务端添加如下参数

                  <property>

                           <name>hive.metastore.execute.setugi</name>

                           <value>true</value>

                  </property>

                  

                  4 rows selected (2.9 seconds)

0:jdbc:hive2://data1:10000>use test_select_only;

                   Norows affected (0.46 seconds)

0:jdbc:hive2://data1:10000>show tables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | select_only  |

                  +--------------+--+

                  1 row selected (0.507 seconds)

0:jdbc:hive2://data1:10000>drop table select_only;

                  Error: Error while compilingstatement: FAILED: SemanticException No valid privileges

                  Required privileges for this query:Server=server1->Db=test_select_only->Table=select_only->action=*;(state=42000,code=40000)

#场景四 test用户连入,对数据库test_only进行验证,拥有所有权限:可以查看、删除表

0: jdbc:hive2://data1:10000>usetest_only;

                  No rows affected (0.819 seconds)

0:jdbc:hive2://data1:10000> showtables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | test_itself  |

                  +--------------+--+

                  1 row selected (0.426 seconds)

0:jdbc:hive2://data1:10000>drop table test_itself;

                  No rows affected (6.336 seconds)

                  

0: jdbc:hive2://data1:10000> create table test_newone ( ont string);

                   Norows affected (1.377 seconds)

0:jdbc:hive2://data1:10000>show tables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | test_newone  |

                  +--------------+--+

5.2 Impala集成Sentry权限验证

1)  进入impala-shell,执行权限验证

create role supervisor;

grant all on server to rolesupervisor;

grant role supervisor to groupimpala;

GRANT ALL ON SERVER server1 TOROLE any_operation;

GRANT ROLE any_operation TOGROUP hive;

grant ALL ON database EDA TOtest_role;

grant select ON databasepriselect TO test_role;

 

2)  切换至test用户测试impala权限:

 

6 总结

仔细跟踪log,具体问题具体分析.


阅读更多
版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/shenliang1985/article/details/50463432
文章标签: Apache Sentry CDH
个人分类: 大数据
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

不良信息举报

Apache Sentry手动安装、使用手册

最多只允许输入30个字

加入CSDN,享受更精准的内容推荐,与500万程序员共同成长!
关闭
关闭