配置YARN队列权限由Ranger管理

配置YARN队列权限由Ranger管理

背景

黑龙江项目对YARN的队列容量成功配置后，还需要对该队列的访问权限进行配置，这样就能限制其他租户使用该队列。在OCDP集群中，YARN的队列访问控制由Ranger组件完成。

问题

ranger对hdfs，hive，hbase管控都没有问题，对队列的管控要不全能访问，要不全不能访问，一定是ranger哪个地方配置不对

解决

配置

a. 在Ambari中打开Ranger配置界面（Ranger->Configs->Advanced->YARN Ranger Plugin），打开YARN Ranger Plugin。保存配置并重启受影响的组件。

图1 配置Ranger YARN Plugin

b. 在Ambari中打开YARN配置界面（YARN->Configs->Advanced->Custom ranger-yarn-security），配置属性ranger.add-yarn-authorization为false，如下图所示。这样配置的目的是禁用YARN本身的ACL权限控制，YARN队列的权限控制由Ranger统一管理。

图2 禁用YARN自身ACL权限控制

c. 打开Ranger界面，在Access Manager->Resource Based Policies下，选择YARN->HDP_yarn，创建新策略(Add New Policy)。配置策略名，需要设定访问控制的队列，在用户和用户组权限中选择用户，或者组，并赋予相应的权限，如下图所示。

图3 Ranger新增YARN权限控制

按照上述的配置完成后，即实现了Hadoop集群中多租户资源分配和调度，即：不同的租户根据各自配置的队列获取集群资源；并且对各队列的资源具有访问控制，未被授权的租户不能获取该队列的资源。

yarn队列配置

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_queue=ocdp,oc_ai_app
yarn.scheduler.capacity.root.default.acl_submit_applications=ocdp,oc_ai_app
yarn.scheduler.capacity.root.default.capacity=10
yarn.scheduler.capacity.root.default.maximum-capacity=20
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.oc_ai.acl_administer_queue=ocdp,oc_ai
yarn.scheduler.capacity.root.oc_ai.acl_submit_applications=oc_ai
yarn.scheduler.capacity.root.oc_ai.capacity=40
yarn.scheduler.capacity.root.oc_ai.maximum-capacity=70
yarn.scheduler.capacity.root.oc_ai.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.oc_ai.ordering-policy=fifo
yarn.scheduler.capacity.root.oc_ai.priority=0
yarn.scheduler.capacity.root.oc_ai.state=RUNNING
yarn.scheduler.capacity.root.oc_ai.user-limit-factor=1
yarn.scheduler.capacity.root.oc_ai_app.acl_administer_queue=ocdp,oc_ai_app
yarn.scheduler.capacity.root.oc_ai_app.acl_submit_applications=oc_ai_app
yarn.scheduler.capacity.root.oc_ai_app.capacity=10
yarn.scheduler.capacity.root.oc_ai_app.maximum-capacity=70
yarn.scheduler.capacity.root.oc_ai_app.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.oc_ai_app.ordering-policy=fifo
yarn.scheduler.capacity.root.oc_ai_app.priority=0
yarn.scheduler.capacity.root.oc_ai_app.state=RUNNING
yarn.scheduler.capacity.root.oc_ai_app.user-limit-factor=1
yarn.scheduler.capacity.root.oc_ai_released.acl_administer_queue=ocdp,oc_ai_released
yarn.scheduler.capacity.root.oc_ai_released.acl_submit_applications=oc_ai_released
yarn.scheduler.capacity.root.oc_ai_released.capacity=10
yarn.scheduler.capacity.root.oc_ai_released.maximum-capacity=30
yarn.scheduler.capacity.root.oc_ai_released.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.oc_ai_released.ordering-policy=fifo
yarn.scheduler.capacity.root.oc_ai_released.priority=0
yarn.scheduler.capacity.root.oc_ai_released.state=RUNNING
yarn.scheduler.capacity.root.oc_ai_released.user-limit-factor=1
yarn.scheduler.capacity.root.oc_guoxin.acl_administer_queue=ocdp,oc_guoxin
yarn.scheduler.capacity.root.oc_guoxin.acl_submit_applications=oc_guoxin
yarn.scheduler.capacity.root.oc_guoxin.capacity=10
yarn.scheduler.capacity.root.oc_guoxin.maximum-capacity=30
yarn.scheduler.capacity.root.oc_guoxin.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.oc_guoxin.ordering-policy=fifo
yarn.scheduler.capacity.root.oc_guoxin.priority=0
yarn.scheduler.capacity.root.oc_guoxin.state=RUNNING
yarn.scheduler.capacity.root.oc_guoxin.user-limit-factor=1
yarn.scheduler.capacity.root.oc_guoxin_normal.acl_administer_queue=ocdp,oc_guoxin_normal
yarn.scheduler.capacity.root.oc_guoxin_normal.acl_submit_applications=oc_guoxin_normal
yarn.scheduler.capacity.root.oc_guoxin_normal.capacity=10
yarn.scheduler.capacity.root.oc_guoxin_normal.maximum-capacity=15
yarn.scheduler.capacity.root.oc_guoxin_normal.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.oc_guoxin_normal.ordering-policy=fifo
yarn.scheduler.capacity.root.oc_guoxin_normal.priority=0
yarn.scheduler.capacity.root.oc_guoxin_normal.state=RUNNING
yarn.scheduler.capacity.root.oc_guoxin_normal.user-limit-factor=1
yarn.scheduler.capacity.root.oc_telecom.acl_administer_queue=ocdp,oc_telecom
yarn.scheduler.capacity.root.oc_telecom.acl_submit_applications=oc_telecom
yarn.scheduler.capacity.root.oc_telecom.capacity=10
yarn.scheduler.capacity.root.oc_telecom.maximum-capacity=20
yarn.scheduler.capacity.root.oc_telecom.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.oc_telecom.ordering-policy=fifo
yarn.scheduler.capacity.root.oc_telecom.priority=0
yarn.scheduler.capacity.root.oc_telecom.state=RUNNING
yarn.scheduler.capacity.root.oc_telecom.user-limit-factor=1
yarn.scheduler.capacity.root.priority=0
yarn.scheduler.capacity.root.queues=default,oc_ai,oc_ai_app,oc_ai_released,oc_guoxin,oc_guoxin_normal,oc_telecom

注意

1.图2 禁用YARN自身ACL权限控制，要关闭使权限控制完全有ranger管控

2.图3 Ranger新增YARN权限控制，ranger 队列名要使用yarn.scheduler.capacity.<root.oc_guoxin>.capacity即root.oc_guoxin或者root.default而不是 oc_guoxin或者default，否者权限管控是不生效的