[TOC]
上一份工作主要负责大数据平台的建设,在这个过程中积累了一些Hadoop生态组件的搭建和使用笔记,由于时间关系,不打算去修改其中的错别字和排版问题,直接释出原始笔记。
前置条件
我所在的集群有三台服务其,对应的host分别为master,slave1,slave2。hadoop服务的安装分部为
机器host | 组件情况 |
---|---|
master | namenode、datanode、journalnode、resourcemanager、nodemanager、jobhistoryserver |
slave1 | namenode、datanode、journalnode、resourcemanager、nodemanager |
slave2 | datanode、journalnode、nodemanager |
kerberos相关
首先我们要安装好kerberos,kerberos的安装搭建参考链接 https://www.cnblogs.com/niceshot/p/13216455.html
给hadoop各组件创建kerberos账号
进入kerberos的admin.local后,依次执行下述命令
//组件web服务的princial
addprinc -randkey HTTP/master@TEST.COM
addprinc -randkey HTTP/slave1@TEST.COM
addprinc -randkey HTTP/slave2@TEST.COM
//namenode的princial
addprinc -randkey nn/master@TEST.COM
addprinc -randkey nn/slave1@TEST.COM
//datanode的princial
addprinc -randkey dn/master@TEST.COM
addprinc -randkey dn/slave1@TEST.COM
addprinc -randkey dn/slave2@TEST.COM
//journalnode的princial
addprinc -randkey jn/master@TEST.COM
addprinc -randkey jn/slave1@TEST.COM
addprinc -randkey jn/slave2@TEST.COM
//resourcemanager 的princial
addprinc -randkey rm/master@TEST.COM
addprinc -randkey rm/slave1@TEST.COM
//nodemanager的principal
addprinc -randkey nm/master@TEST.COM
addprinc -randkey nm/slave1@TEST.COM
addprinc -randkey nm/slave2@TEST.COM
//job hisotry server的princial
addprinc -randkey jhs/master@TEST.COM
将这些账号做成keytab
同样是在admin.local中,将上述账号认证信息做成keytab
ktadd -k /opt/keytab_store/http.service.keytab HTTP/master@TEST.COM
ktadd -k /opt/keytab_store/http.service.keytab HTTP/slave1@TEST.COM
ktadd -k /opt/keytab_store/http.service.keytab HTTP/slave2@TEST.COM
ktadd -k /opt/keytab_store/nn.service.keytab nn/master@TEST.COM
ktadd -k /opt/keytab_store/nn.service.keytab nn/slave1@TEST.COM
ktadd -k /opt/keytab_store/dn.service.keytab dn/master@TEST.COM
ktadd -k /opt/keytab_store/dn.service.keytab dn/slave1@TEST.COM
ktadd -k /opt/keytab_store/dn.service.keytab dn/slave2@TEST.COM
ktadd -k /opt/keytab_store/jn.service.keytab jn/master@TEST.COM
ktadd -k /opt/keytab_store/jn.service.keytab jn/slave1@TEST.COM
ktadd -k /opt/keytab_store/jn.service.keytab jn/slave2@TEST.COM
ktadd -k /opt/keytab_store/rm.service.keytab rm/master@TEST.COM
ktadd -k /opt/keytab_store/rm.service.keytab rm/slave1@TEST.COM
ktadd -k /opt/keytab_store/nm.service.keytab nm/master@TEST.COM
ktadd -k /opt/keytab_store/nm.service.keytab nm/slave1@TEST.COM
ktadd -k /opt/keytab_store/nm.service.keytab nm/slave2@TEST.COM
ktadd -k /opt/keytab_store/jhs.service.keytab jhs/master@TEST.COM
多个账号可以做到一个keytab中去,上述的命令做了多个文件,不同组件角色的单独放到了一个keytab文件中。其实内部网络,可以把所有的hadoop相关组件做成一个大的keytab文件,降低配置复杂性。
将上述的keytab文件,分发到集群所有机器
core-site.xml
关键配置
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
<property>
<name>hadoop.security.auth_to_local</name>
<value>
RULE:[2:$1/$2@$0]([ndj]n/.*@TEST.COM)s/.*/hdfs/
RULE:[2:$1/$2@$0]([rn]m/.*@TEST.COM)s/.*/yarn/
RULE:[2:$1/$2@$0](jhs/.*@TEST.COM)s/.*/mapred/
DEFAULT
</value>
</property>
上述配置的意思是 在整个集群中费用kerberos作为安全