【安全】Apache HDFS 上配置 kerberos

九师兄

已于 2023-06-25 14:14:28 修改

阅读量1.9k

点赞数

分类专栏：未解决的专栏合集文章标签：安全 hadoop kerberos

于 2018-11-02 10:25:48 首次发布

本文为博主九师兄（QQ:541711153 欢迎来探讨技术）原创文章，未经允许博主不允许转载。

本文链接：https://blog.csdn.net/qq_21383435/article/details/83651443

版权

未解决的专栏合集专栏收录该内容

44 篇文章 1 订阅

订阅专栏

文章目录

5. 测试启动
问题集锦

4.1 创建认证规则
在 Kerberos 安全机制里，一个 principal 就是 realm 里的一个对象，一个 principal 总是和一个密钥（secret key）成对出现的。

这个 principal 的对应物可以是 service，可以是 host，也可以是 user，对于 Kerberos 来说，都没有区别。

Kdc(Key distribute center) 知道所有 principal 的 secret key，但每个 principal 对应的对象只知道自己的那个 secret key 。这也是“共享密钥“的由来。

对于 hadoop，principals 的格式为 username/fully.qualified.domain.name@YOUR-REALM.COM。

通过 yum 源安装的 cdh 集群中，NameNode 和 DataNode 是通过 hdfs 启动的，故为集群中每个服务器节点添加两个principals：hdfs、HTTP。

在 KCD server 上（这里是 cdh1）创建 hdfs principal：

[root@cdh-server2 ~]# kadmin.local -q "addprinc -randkey hdfs/cdh-server1@YONG.COM"
Authenticating as principal root/admin@YONG.COM with password.
WARNING: no policy specified for hdfs/cdh-server1@YONG.COM; defaulting to no policy
Principal "hdfs/cdh-server1@YONG.COM" created.
[root@cdh-server2 ~]# kadmin.local -q "addprinc -randkey hdfs/cdh-server2@YONG.COM"
[root@cdh-server2 ~]# kadmin.local -q "addprinc -randkey hdfs/cdh-server3@YONG.COM"
[root@cdh-server2 ~]#

-randkey 标志没有为新 principal 设置密码，而是指示 kadmin 生成一个随机密钥。之所以在这里使用这个标志，是因为此 principal 不需要用户交互。它是计算机的一个服务器帐户。

创建 HTTP principal：

kadmin.local -q "addprinc -randkey HTTP/cdh-server1@YONG.COM"
kadmin.local -q "addprinc -randkey HTTP/cdh-server2@YONG.COM"
kadmin.local -q "addprinc -randkey HTTP/cdh-server3@YONG.COM"

创建完成后，查看：

[root@cdh-server2 ~]# kadmin.local -q "listprincs"
Authenticating as principal root/admin@YONG.COM with password.
HTTP/cdh-server1@YONG.COM
HTTP/cdh-server2@YONG.COM
HTTP/cdh-server3@YONG.COM
K/M@YONG.COM
admin/admin@YONG.COM
hdfs/cdh-server1@YONG.COM
hdfs/cdh-server2@YONG.COM
hdfs/cdh-server3@YONG.COM
kadmin/admin@YONG.COM
kadmin/cdh-server2@YONG.COM
kadmin/changepw@YONG.COM
kiprop/cdh-server2@YONG.COM
krbtgt/YONG.COM@YONG.COM
root/root@YONG.COM
[root@cdh-server2 ~]#

4.2 创建keytab文件
keytab 是包含 principals 和加密 principal key 的文件。

keytab 文件对于每个 host 是唯一的，因为 key 中包含 hostname。keytab 文件用于不需要人工交互和保存纯文本密码，实现到 kerberos 上验证一个主机上的 principal。

因为服务器上可以访问 keytab 文件即可以以 principal 的身份通过 kerberos 的认证，所以，keytab 文件应该被妥善保存，应该只有少数的用户可以访问

创建包含 hdfs principal 和 host principal 的 hdfs keytab：

xst -norandkey -k hdfs.keytab hdfs/fully.qualified.domain.name host/fully.qualified.domain.name
创建包含 mapred principal 和 host principal 的 mapred keytab：

xst -norandkey -k mapred.keytab mapred/fully.qualified.domain.name host/fully.qualified.domain.name

注意：
上面的方法使用了xst的norandkey参数，有些kerberos不支持该参数。
当不支持该参数时有这样的提示：Principal -norandkey does not exist.，需要使用下面的方法来生成keytab文件。

或者

[root@cdh-server2 ~]# xst -norandkey -k hdfs.keytab hdfs/fully.qualified.domain.name host/fully.qualified.domain.name
-bash: xst: command not found

在 dh-server2 节点，即 KDC server 节点上执行下面命令：

[root@cdh-server2 ~]# cd /var/kerberos/krb5kdc
[root@cdh-server2 krb5kdc]# ll
total 40
-rw------- 1 root root    19 Nov  1 19:08 kadm5.acl
-rw------- 1 root root   458 Nov  1 19:03 kdc.conf
-rw------- 1 root root 24576 Nov  2 09:39 principal
-rw------- 1 root root  8192 Nov  1 20:16 principal.kadm5
-rw------- 1 root root     0 Nov  1 20:16 principal.kadm5.lock
-rw------- 1 root root     0 Nov  2 09:39 principal.ok


[root@cdh-server2 krb5kdc]# kadmin.local -q "xst  -k hdfs-unmerged.keytab  hdfs/cdh-server1@YONG.COM"
[root@cdh-server2 krb5kdc]# kadmin.local -q "xst  -k hdfs-unmerged.keytab  hdfs/cdh-server2@YONG.COM"
[root@cdh-server2 krb5kdc]# kadmin.local -q "xst  -k hdfs-unmerged.keytab  hdfs/cdh-server3@YONG.COM"
 
[root@cdh-server2 krb5kdc]# kadmin.local -q "xst  -k HTTP.keytab  HTTP/cdh-server1@YONG.COM"
[root@cdh-server2 krb5kdc]# kadmin.local -q "xst  -k HTTP.keytab  HTTP/cdh-server2@YONG.COM"
[root@cdh-server2 krb5kdc]# kadmin.local -q "xst  -k HTTP.keytab  HTTP/cdh-server3@YONG.COM"

[root@cdh-server2 krb5kdc]# ll
total 48
-rw------- 1 root root  1454 Nov  2 09:44 HTTP.keytab
-rw------- 1 root root  1454 Nov  2 09:44 hdfs-unmerged.keytab
-rw------- 1 root root    19 Nov  1 19:08 kadm5.acl
-rw------- 1 root root   458 Nov  1 19:03 kdc.conf
-rw------- 1 root root 24576 Nov  2 09:44 principal
-rw------- 1 root root  8192 Nov  1 20:16 principal.kadm5
-rw------- 1 root root     0 Nov  1 20:16 principal.kadm5.lock
-rw------- 1 root root     0 Nov  2 09:44 principal.ok
[root@cdh-server2 krb5kdc]#

这样，就会在 /var/kerberos/krb5kdc/ 目录下生成 hdfs-unmerged.keytab 和 HTTP.keytab 两个文件，接下来使用 ktutil 合并者两个文件为 hdfs.keytab。

[root@cdh-server2 krb5kdc]# ktutil
ktutil:  rkt hdfs-unmerged.keytab
ktutil:  rkt HTTP.keytab
ktutil:  wkt hdfs.keytab
ktutil:  exit
[root@cdh-server2 krb5kdc]# ll
total 52
-rw------- 1 root root  1454 Nov  2 09:44 HTTP.keytab
-rw------- 1 root root  1454 Nov  2 09:44 hdfs-unmerged.keytab
-rw------- 1 root root  2906 Nov  2 09:49 hdfs.keytab
-rw------- 1 root root    19 Nov  1 19:08 kadm5.acl
-rw------- 1 root root   458 Nov  1 19:03 kdc.conf
-rw------- 1 root root 24576 Nov  2 09:44 principal
-rw------- 1 root root  8192 Nov  1 20:16 principal.kadm5
-rw------- 1 root root     0 Nov  1 20:16 principal.kadm5.lock
-rw------- 1 root root     0 Nov  2 09:44 principal.ok
[root@cdh-server2 krb5kdc]#

使用 klist 显示 hdfs.keytab 文件列表：

[root@cdh-server2 krb5kdc]#  klist -ket  hdfs.keytab
Keytab name: FILE:hdfs.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (aes128-cts-hmac-sha1-96)
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (des3-cbc-sha1)
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (arcfour-hmac)
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (camellia256-cts-cmac)
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (camellia128-cts-cmac)
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (des-hmac-sha1)
   2 11/02/18 09:49:51 hdfs/cdh-server1@YONG.COM (des-cbc-md5)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (aes128-cts-hmac-sha1-96)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (des3-cbc-sha1)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (arcfour-hmac)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (camellia256-cts-cmac)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (camellia128-cts-cmac)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (des-hmac-sha1)
   2 11/02/18 09:49:51 hdfs/cdh-server2@YONG.COM (des-cbc-md5)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (aes128-cts-hmac-sha1-96)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (des3-cbc-sha1)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (arcfour-hmac)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (camellia256-cts-cmac)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (camellia128-cts-cmac)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (des-hmac-sha1)
   2 11/02/18 09:49:51 hdfs/cdh-server3@YONG.COM (des-cbc-md5)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (aes128-cts-hmac-sha1-96)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (des3-cbc-sha1)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (arcfour-hmac)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (camellia256-cts-cmac)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (camellia128-cts-cmac)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (des-hmac-sha1)
   2 11/02/18 09:49:51 HTTP/cdh-server1@YONG.COM (des-cbc-md5)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (aes128-cts-hmac-sha1-96)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (des3-cbc-sha1)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (arcfour-hmac)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (camellia256-cts-cmac)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (camellia128-cts-cmac)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (des-hmac-sha1)
   2 11/02/18 09:49:51 HTTP/cdh-server2@YONG.COM (des-cbc-md5)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (aes128-cts-hmac-sha1-96)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (des3-cbc-sha1)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (arcfour-hmac)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (camellia256-cts-cmac)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (camellia128-cts-cmac)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (des-hmac-sha1)
   2 11/02/18 09:49:51 HTTP/cdh-server3@YONG.COM (des-cbc-md5)
[root@cdh-server2 krb5kdc]#

验证是否正确合并了key，使用合并后的keytab，分别使用hdfs和host principals来获取证书。

[root@cdh-server2 krb5kdc]# kinit -k -t hdfs.keytab hdfs/cdh-server1@YONG.COM
[root@cdh-server2 krb5kdc]# kinit -k -t hdfs.keytab HTTP/cdh-server1@YONG.COM
[root@cdh-server2 krb5kdc]#

如果出现错误：kinit: Key table entry not found while getting initial credentials，
则上面的合并有问题，重新执行前面的操作。

4.3 部署kerberos keytab文件

拷贝 hdfs.keytab 文件到其他节点的 /etc/hadoop/conf 目录

[root@cdh-server2 krb5kdc]# scp hdfs.keytab root@cdh-server1:/etc/hadoop-2.7.4/conf
[root@cdh-server2 krb5kdc]# scp hdfs.keytab root@cdh-server2:/etc/hadoop-2.7.4/conf
[root@cdh-server2 krb5kdc]# scp hdfs.keytab root@cdh-server3:/etc/hadoop-2.7.4/conf

并设置权限，分别在 cdh-server1、cdh-server2、cdh-server3 上执行：

[root@cdh-server2 krb5kdc]# ssh cdh-server1  "chown hdfs:hadoop /etc/hadoop-2.7.4/conf/hdfs.keytab ;chmod 400 /etc/hadoop-2.7.4/conf/hdfs.keytab"
[root@cdh-server2 krb5kdc]# ssh cdh-server2  "chown hdfs:hadoop /etc/hadoop-2.7.4/conf/hdfs.keytab ;chmod 400 /etc/hadoop-2.7.4/conf/hdfs.keytab"
[root@cdh-server2 krb5kdc]# ssh cdh-server3  "chown hdfs:hadoop /etc/hadoop-2.7.4/conf/hdfs.keytab ;chmod 400 /etc/hadoop-2.7.4/conf/hdfs.keytab"
[root@cdh-server2 krb5kdc]#

由于 keytab 相当于有了永久凭证，不需要提供密码(如果修改kdc中的principal的密码，则该keytab就会失效)，所以其他用户如果对该文件有读权限，就可以冒充 keytab 中指定的用户身份访问 hadoop，所以 keytab 文件需要确保只对 owner 有读权限(0400)

4.4 命令测试

测试keytab可用性，如果啥都没有，那就是对了

[root@cdh-server1 soft]# kinit -k -t /etc/hadoop-2.7.4/conf/hdfs.keytab hdfs/cdh-server1@YONG.COM
[root@cdh-server1 soft]#

4.5 写个测试类测试一下


import org.apache.hadoop.conf.Configuration;

import java.io.IOException;

/**
 * @Author: chuanchuan.lcc
 * @CreateDate: 2018/11/23 AM10:53
 * @Version: 1.0
 * @Description: java类作用描述：
 */
public class LoginTest {
    public static void main(String[] args) throws IOException {

        String krb5ConfPath = args[0];
        String principal = args[1];
        String keytab = args[2];

        System.out.println("参数：krb5ConfPath "+krb5ConfPath);
        System.out.println("参数：principal "+principal);
        System.out.println("参数：keytab "+keytab);


        UserLogginUtil.setKrb5Conf(krb5ConfPath);

        KerberosEntity kerberosEntity = new KerberosEntity();
        kerberosEntity.setKrb5Conf(krb5ConfPath);
        kerberosEntity.setPrincipal(principal);
        kerberosEntity.setKeytab(keytab);

        Configuration configuration = new Configuration();

        UserLogginUtil.login(kerberosEntity,configuration);

        System.out.println("登陆成功");
    }
}


import lombok.Data;

/**
 * @Author: chuanchuan.lcc
 * @CreateDate: 2018/11/23 AM10:49
 * @Version: 1.0
 * @Description: java类作用描述：
 */
@Data
public class KerberosEntity {
    /**
     * krb5 配置信息
     */
    private String krb5Conf;
    /**
     * 服务的keytab文件
     */
    private String keytab;

    /**
     * 服务的principal
     */
    private String principal;
    /**
     * 是否需要更新登陆
     */
    private boolean needRenewTicket;

    /**
     * 最大票据更新时间
     */
    private long maxRenewTicketLife;

}


    public static void login(KerberosEntity kerberosEntity,Configuration conf) throws IOException{
        // 检查必须的参数信息
        Preconditions.checkArgument(StringUtils.isNotBlank(kerberosEntity.getKeytab()),"must specify the principal");
        Preconditions.checkArgument(StringUtils.isNotBlank(kerberosEntity.getPrincipal()),"must specify the keytab");
        Preconditions.checkArgument(StringUtils.isNotBlank(kerberosEntity.getKrb5Conf()),"must specify the krb5Conf");
        // 校验krb5Conf
        setKrb5Conf(kerberosEntity.getKrb5Conf());
        // 校验keytab
        File keytabConfigFile = new File(kerberosEntity.getKeytab());
        if(!keytabConfigFile.exists() || !keytabConfigFile.isFile()){
            throw new IOException(String.format("the specified hdfs-keytab file-%s does not exist, or is not a file",
                    kerberosEntity.getKeytab()));
        }
        // 设置配置信息，并以相应的principal和 principal登陆
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("hadoop.security.authorization", "true");
        UserGroupInformation.setConfiguration(conf);

        String keytab = keytabConfigFile.getAbsolutePath();
        try {
            UserGroupInformation.loginUserFromKeytab(kerberosEntity.getPrincipal(),keytab);
          
            }

        } catch (IOException e) {
            LOGGER.warn("failed to logIn in the principal-{} and keyTab-{},error:{}",
                    kerberosEntity.getPrincipal(),keytab, ExceptionUtils.getStackTrace(e));
            throw e;
        }
    }

打成带依赖的jar包 上传到安装kerberos client的机器上，然后执行

[root@cdh-server1 soft]# java -cp hadoop_kerberos-1.0-SNAPSHOT-jar-with-dependencies.jar com.lcc.hadoop.kerberos.LoginTest  /etc/krb5.conf   hdfs/cdh-server1@YONG.COM   /etc/hadoop-2.7.4/conf/hdfs.keytab
log4j:WARN No appenders could be found for logger (com.lcc.hadoop.kerberos.UserLogginUtil).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
登录成功

没有抛出任何异常，可以看到，现在kerberos是可以登录成功的。

4.5 修改 hdfs 配置文件

4.5.1 常规配置

在集群中所有节点的 core-site.xml 文件中添加下面的配置:

<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>
 
<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

在集群中所有节点的 hdfs-site.xml 文件中添加下面的配置：


        <property>
          <name>dfs.block.access.token.enable</name>
          <value>true</value>
        </property>
        <property>
          <name>dfs.datanode.data.dir.perm</name>
          <value>700</value>
        </property>
        <property>
          <name>dfs.namenode.keytab.file</name>
          <value>/etc/hadoop-2.7.4/conf/hdfs.keytab</value>
        </property>
        <property>
          <name>dfs.namenode.kerberos.principal</name>
          <value>hdfs/_HOST@YONG.COM</value>
        </property>
        <property>
          <name>dfs.namenode.kerberos.https.principal</name>
          <value>HTTP/_HOST@YONG.COM</value>
        </property>
                <property>
          <name>dfs.datanode.address</name>
          <value>0.0.0.0:1004</value>
        </property>
        <property>
          <name>dfs.datanode.http.address</name>
          <value>0.0.0.0:1006</value>
        </property>
        <property>
          <name>dfs.datanode.keytab.file</name>
          <value>/etc/hadoop-2.7.4/conf/hdfs.keytab</value>
        </property>
        <property>
          <name>dfs.datanode.kerberos.principal</name>
          <value>hdfs/_HOST@YONG.COM</value>
        </property>
        <property>
          <name>dfs.datanode.kerberos.https.principal</name>
          <value>HTTP/_HOST@YONG.COM</value>
        </property>

4.5.2 可选配置

如果想开启 SSL，请添加（本文不对这部分做说明）：

<property>
  <name>dfs.http.policy</name>
  <value>HTTPS_ONLY</value>
</property>

4.5.2 可选配置

如果 HDFS 配置了 QJM HA，则需要添加（另外，你还要在 zookeeper 上配置 kerberos）：

<property>
  <name>dfs.journalnode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>
<property>
  <name>dfs.journalnode.kerberos.principal</name>
  <value>hdfs/_HOST@JAVACHEN.COM</value>
</property>
<property>
  <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
  <value>HTTP/_HOST@JAVACHEN.COM</value>
</propert

4.5.3 可选配置

如果配置了 WebHDFS，则添加：

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
 
<property>
  <name>dfs.web.authentication.kerberos.principal</name>
  <value>HTTP/_HOST@JAVACHEN.COM</value>
</property>
 
<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>

4.5.4 注意点

配置中有几点要注意的：

dfs.datanode.address表示 data transceiver RPC server 所绑定的 hostname 或 IP 地址，如果开启 security，端口号必须小于 1024(privileged port)，否则的话启动 datanode 时候会报 Cannot start secure cluster without privileged resources 错误
principal 中的 instance 部分可以使用 _HOST 标记，系统会自动替换它为全称域名
如果开启了 security, hadoop 会对 hdfs block data(由 dfs.data.dir 指定)做 permission check，方式用户的代码不是调用hdfs api而是直接本地读block data，这样就绕过了kerberos和文件权限验证，管理员可以通过设置 dfs.datanode.data.dir.perm 来修改 datanode 文件权限，这里我们设置为700

5. 测试启动

然后启动start-all.sh遇到第2个问题。hdfs/localhost@YONG.COM 这个localhost不知道是从哪里来的？
解决方法请看 错误2

重启报错 错误4，根据解决方法添加配置。

重启，NameNode启动成功，但是dataNode没有启动成功。

问题集锦

错误1

kinit: Client 'hdfs@YONG.COM' not found in Kerberos database while getting initial credentials

错误2

ava.io.IOException: Login failure for hdfs/localhost@YONG.COM from keytab /etc/hadoop-2.7.4/conf/hdfs.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

原因：
我的配置文件中

[root@cdh-server1 soft]# vi  hadoop-2.7.4/etc/hadoop/core-site.xml
 		<property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>

解决：
修改为主机名

 	<property>
                <name>fs.defaultFS</name>
                <value>hdfs://cdh-server1:9000</value>
        </property>

问题3

[root@cdh-server3 ~]# klist
klist: No credentials cache found (filename: /tmp/krb5cc_0)

问题4：

2018-11-23 17:56:39,871 ERROR org.apache.hadoop.http.HttpServer2: WebHDFS and security are enabled, but configuration property 'dfs.web.authentication.kerberos.principal' is not set.
2018-11-23 17:56:40,013 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab /etc/hadoop-2.7.4/conf/hdfs.keytab, for principal ${dfs.web.authentication.kerberos.principal}
2018-11-23 17:56:40,021 WARN org.mortbay.log: failed org.apache.hadoop.hdfs.web.AuthFilter: javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration
2018-11-23 17:56:40,021 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@1e0f9063{/,file:/root/soft/hadoop-2.7.4/share/hadoop/hdfs/webapps/hdfs}
javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration

原因：
未知
提示：
r日志里有下面这句话。但是我没有开启，不知道为啥还报错，经测试，把这个加上去，就不报错了

WebHDFS and security are enabled, but configuration property 'dfs.web.authentication.kerberos.principal' is not set.

解决：
添加配置

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
 
<property>
  <name>dfs.web.authentication.kerberos.principal</name>
  <value>HTTP/_HOST@YONG.COM</value>
</property>
 
<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>

问题5：

018-11-23 19:11:04,759 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is cdh-server1
2018-11-23 19:11:04,763 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP.  Using privileged resources in combination with SASL RPC data transfer protection is not supported.
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1201)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1101)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:429)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2406)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2293)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2340)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2522)

原因：
配置Kerberos启动HDFS报错，需对DataNode中SASL进行配置，参考官网已经解释了很清楚了：

As of version 2.6.0, SASL can be used to authenticate the data transfer protocol. In this configuration, it is no longer required for secured clusters to start the DataNode as root using jsvc and bind to privileged ports. To enable SASL on data transfer protocol, set dfs.data.transfer.protection in hdfs-site.xml, set a non-privileged port for dfs.datanode.address, set dfs.http.policy to HTTPS_ONLY and make sure the HADOOP_SECURE_DN_USER environment variable is not defined. Note that it is not possible to use SASL on data transfer protocol if dfs.datanode.address is set to a privileged port. This is required for backwards-compatibility reasons.

参考：http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html 里面的Secure DataNode部分

解决：

个人配置：

dfs.data.transfer.protection=integrity
dfs.http.policy=HTTPS_ONLY（原值为HTTP_ONLY）
dfs.datanode.address(原值为0.0.0.0:1019)需改为非特权端口，即大于1024的端口，我这用的是61004
确保HADOOP_SECURE_DN_USER变量为空

参考:https://blog.csdn.net/bugzeroman/article/details/84312302
参考：https://ieevee.com/tech/2016/06/07/kerberos-1.html

参考：https://blog.csdn.net/zhouyuanlinli/article/details/78087395

问题6

java.io.FileNotFoundException: /root/.keystore (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)