main : run as user is xwq main : requested yarn user is xwq User xwq not found

m0_37759590

已于 2023-07-01 10:41:11 修改

阅读量375

点赞数

文章标签： hadoop hdfs mapreduce kerberos

于 2023-06-21 10:30:34 首次发布

本文链接：https://blog.csdn.net/m0_37759590/article/details/131322549

版权

项目场景：

kerberos配置hadoop ha時，重新創建一個新的普通用戶，測試其功能`

问题描述

kerberos配置hadoop ha時，重新創建了一個用戶xwq，讓其執行hadoop jar /opt/ha/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 1 1命令，報錯如下：

[xwq@hadoop102 hive]$ hadoop jar /opt/ha/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 1 1
Number of Maps  = 1
Samples per Map = 1
2023-06-21 10:13:36,763 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #0
Starting Job
2023-06-21 10:13:37,121 INFO hdfs.DFSClient: Created token for xwq: HDFS_DELEGATION_TOKEN owner=xwq@EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1687313617117, maxDate=1687918417117, sequenceNumber=14, masterKeyId=12 on ha-hdfs:mycluster
2023-06-21 10:13:37,122 INFO security.TokenCache: Got dt for hdfs://mycluster; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:mycluster, Ident: (token for xwq: HDFS_DELEGATION_TOKEN owner=xwq@EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1687313617117, maxDate=1687918417117, sequenceNumber=14, masterKeyId=12)
2023-06-21 10:13:37,214 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/xwq/.staging/job_1687309231357_0007
2023-06-21 10:13:37,241 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:13:37,306 INFO input.FileInputFormat: Total input files to process : 1
2023-06-21 10:13:37,319 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:13:37,342 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:13:37,353 INFO mapreduce.JobSubmitter: number of splits:1
2023-06-21 10:13:37,448 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:13:37,462 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1687309231357_0007
2023-06-21 10:13:37,462 INFO mapreduce.JobSubmitter: Executing with tokens: [Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:mycluster, Ident: (token for xwq: HDFS_DELEGATION_TOKEN owner=xwq@EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1687313617117, maxDate=1687918417117, sequenceNumber=14, masterKeyId=12)]
2023-06-21 10:13:37,615 INFO conf.Configuration: resource-types.xml not found
2023-06-21 10:13:37,615 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-06-21 10:13:37,867 INFO impl.YarnClientImpl: Submitted application application_1687309231357_0007
2023-06-21 10:13:37,901 INFO mapreduce.Job: The url to track the job: http://hadoop102:8088/proxy/application_1687309231357_0007/
2023-06-21 10:13:37,902 INFO mapreduce.Job: Running job: job_1687309231357_0007
2023-06-21 10:13:44,994 INFO mapreduce.Job: Job job_1687309231357_0007 running in uber mode : false
2023-06-21 10:13:44,995 INFO mapreduce.Job:  map 0% reduce 0%
2023-06-21 10:13:50,042 INFO mapreduce.Job:  map 100% reduce 0%
2023-06-21 10:13:53,058 INFO mapreduce.Job: Task Id : attempt_1687309231357_0007_r_000000_1000, Status : FAILED
[2023-06-21 10:13:51.502]Application application_1687309231357_0007 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is xwq
main : requested yarn user is xwq
User xwq not found

解决方案：

在每hadoop集群的每台節點上都需要創建xwq用戶，執行如下命令：

 useradd xwq
echo xwq | passwd --stdin xwq
usermod -a -G hadoop xwq
kadmin -p admin/admin -wNTVfPQY9kNs6 -q"addprinc -pw xwq xwq"

然後再執行hadoop jar /opt/ha/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 1 1命令，成功：

[xwq@hadoop102 ~]$ hadoop jar /opt/ha/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 1 1
Number of Maps  = 1
Samples per Map = 1
2023-06-21 10:27:07,443 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #0
Starting Job
2023-06-21 10:27:07,754 INFO hdfs.DFSClient: Created token for xwq: HDFS_DELEGATION_TOKEN owner=xwq@EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1687314427749, maxDate=1687919227749, sequenceNumber=16, masterKeyId=12 on ha-hdfs:mycluster
2023-06-21 10:27:07,755 INFO security.TokenCache: Got dt for hdfs://mycluster; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:mycluster, Ident: (token for xwq: HDFS_DELEGATION_TOKEN owner=xwq@EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1687314427749, maxDate=1687919227749, sequenceNumber=16, masterKeyId=12)
2023-06-21 10:27:07,838 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/xwq/.staging/job_1687309231357_0009
2023-06-21 10:27:07,859 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:27:07,925 INFO input.FileInputFormat: Total input files to process : 1
2023-06-21 10:27:07,939 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:27:07,962 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:27:07,974 INFO mapreduce.JobSubmitter: number of splits:1
2023-06-21 10:27:08,064 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-21 10:27:08,079 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1687309231357_0009
2023-06-21 10:27:08,080 INFO mapreduce.JobSubmitter: Executing with tokens: [Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:mycluster, Ident: (token for xwq: HDFS_DELEGATION_TOKEN owner=xwq@EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1687314427749, maxDate=1687919227749, sequenceNumber=16, masterKeyId=12)]
2023-06-21 10:27:08,222 INFO conf.Configuration: resource-types.xml not found
2023-06-21 10:27:08,222 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-06-21 10:27:08,474 INFO impl.YarnClientImpl: Submitted application application_1687309231357_0009
2023-06-21 10:27:08,508 INFO mapreduce.Job: The url to track the job: http://hadoop102:8088/proxy/application_1687309231357_0009/
2023-06-21 10:27:08,508 INFO mapreduce.Job: Running job: job_1687309231357_0009
2023-06-21 10:27:14,599 INFO mapreduce.Job: Job job_1687309231357_0009 running in uber mode : false
2023-06-21 10:27:14,600 INFO mapreduce.Job:  map 0% reduce 0%
2023-06-21 10:27:20,649 INFO mapreduce.Job:  map 100% reduce 0%
2023-06-21 10:27:24,667 INFO mapreduce.Job:  map 100% reduce 100%
2023-06-21 10:27:25,676 INFO mapreduce.Job: Job job_1687309231357_0009 completed successfully
2023-06-21 10:27:25,752 INFO mapreduce.Job: Counters: 53
	File System Counters
		FILE: Number of bytes read=28
		FILE: Number of bytes written=452853
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=258
		HDFS: Number of bytes written=215
		HDFS: Number of read operations=9
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=3
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=3415
		Total time spent by all reduces in occupied slots (ms)=2005
		Total time spent by all map tasks (ms)=3415
		Total time spent by all reduce tasks (ms)=2005
		Total vcore-milliseconds taken by all map tasks=3415
		Total vcore-milliseconds taken by all reduce tasks=2005
		Total megabyte-milliseconds taken by all map tasks=3496960
		Total megabyte-milliseconds taken by all reduce tasks=2053120
	Map-Reduce Framework
		Map input records=1
		Map output records=2
		Map output bytes=18
		Map output materialized bytes=28
		Input split bytes=140
		Combine input records=0
		Combine output records=0
		Reduce input groups=2
		Reduce shuffle bytes=28
		Reduce input records=2
		Reduce output records=0
		Spilled Records=4
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=83
		CPU time spent (ms)=1110
		Physical memory (bytes) snapshot=641515520
		Virtual memory (bytes) snapshot=5220311040
		Total committed heap usage (bytes)=1142947840
		Peak Map Physical memory (bytes)=371208192
		Peak Map Virtual memory (bytes)=2606309376
		Peak Reduce Physical memory (bytes)=270307328
		Peak Reduce Virtual memory (bytes)=2614001664
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=118
	File Output Format Counters 
		Bytes Written=97
Job Finished in 18.208 seconds
2023-06-21 10:27:25,809 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Estimated value of Pi is 4.00000000000000000000

m0_37759590

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
main : run as user is xwq main : requested yarn user is xwq User xwq not found

kerberos配置hadoop main : run as user is xwq main : requested yarn user is xwq User xwq not found
复制链接

扫一扫