Hadoop系列三：运行job过程日志

最新推荐文章于 2024-05-04 16:43:38 发布
最新推荐文章于 2024-05-04 16:43:38 发布
阅读量5.8k 收藏
点赞数
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/jollypigclub/article/details/49929943
版权 

[bruce@iRobot hadoop]$ $HADOOP_PREFIX/bin/hdfs dfs -mkdir /user
[bruce@iRobot hadoop]$ $HADOOP_PREFIX/bin/hdfs dfs -mkdir /user/bruce


[bruce@iRobot hadoop]$ tail -f logs/*-namenode*.log

2015-11-19 15:01:20,442 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 11Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 37 
2015-11-19 15:05:34,036 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 3 Total time for transactions(ms): 11Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 48 

----------------------------------------------------------------------------


[bruce@iRobot hadoop]$ $HADOOP_PREFIX/bin/hdfs dfs -ls .
15/11/19 15:07:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[bruce@iRobot hadoop]$ ls 
1.txt  bin  bin-mapreduce1  cloudera  etc  examples  examples-mapreduce1  hellohadoop-1.0-SNAPSHOT.jar  include  lib  libexec  logs  sbin  share  src
[bruce@iRobot hadoop]$ cat 1.txt
monkey
sea
funny
hello
world
google
cats
worry
over
new
day
[bruce@iRobot hadoop]$ $HADOOP_PREFIX/bin/hdfs dfs -put 1.txt
[bruce@iRobot hadoop]$ 

（10分钟后执行展示 hdfs的1.txt文件）
[bruce@iRobot hadoop]$ bin/hdfs dfs -ls .
Found 1 items
-rw-r--r--   1 bruce oinstall         59 2015-11-19 15:07 1.txt
[bruce@iRobot hadoop]$ bin/hdfs dfs -cat 1.txt
monkey
sea
funny
hello
world
google
cats
worry
over
new
day
[bruce@iRobot hadoop]$ 



[bruce@iRobot hadoop]$ tail -f logs/*-namenode*.log

2015-11-19 15:07:57,606 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /user/bruce/1.txt._COPYING_. BP-981411196-192.168.100.200-1447912540337 blk_6533808687986917333_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[192.168.100.200:50010|RBW]]}
2015-11-19 15:07:57,858 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.100.200:50010 is added to blk_6533808687986917333_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[192.168.100.200:50010|RBW]]} size 0
2015-11-19 15:07:57,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/bruce/1.txt._COPYING_ is closed by DFSClient_NONMAPREDUCE_1817581491_1


（执行展示 hdfs的1.txt文件后）
2015-11-19 15:17:17,806 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.100.200
2015-11-19 15:17:17,806 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2015-11-19 15:17:17,806 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 3
2015-11-19 15:17:17,806 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 10 Total time for transactions(ms): 14Number of transactions batched in Syncs: 0 Number of syncs: 7 SyncTimes(ms): 76 
2015-11-19 15:17:17,813 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 10 Total time for transactions(ms): 14Number of transactions batched in Syncs: 0 Number of syncs: 8 SyncTimes(ms): 83 
2015-11-19 15:17:17,814 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /u01/hadoopdata/hadoop-bruce/dfs/name/current/edits_inprogress_0000000000000000003 -> /u01/hadoopdata/hadoop-bruce/dfs/name/current/edits_0000000000000000003-0000000000000000012
2015-11-19 15:17:17,814 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 13
2015-11-19 15:17:17,949 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://iRobot:50090/getimage?getimage=1&txid=12&storageInfo=-40:337807999:0:CID-4fb81ac3-7e61-42c2-a707-46f2af89a6e2
2015-11-19 15:17:17,968 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.02s at 0.00 KB/s
2015-11-19 15:17:17,968 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000012 size 358 bytes.
2015-11-19 15:17:17,981 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 2
2015-11-19 15:17:17,981 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/u01/hadoopdata/hadoop-bruce/dfs/name/current/fsimage_0000000000000000000, cpktTxId=0000000000000000000)



[bruce@iRobot hadoop]$ tail -f logs/*datanode*.log

2015-11-19 15:07:57,791 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-981411196-192.168.100.200-1447912540337:blk_6533808687986917333_1002 src: /192.168.100.200:14244 dest: /192.168.100.200:50010
2015-11-19 15:07:57,858 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.100.200:14244, dest: /192.168.100.200:50010, bytes: 59, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1817581491_1, offset: 0, srvID: DS-1689463280-192.168.100.200-50010-1447913181338, blockid: BP-981411196-192.168.100.200-1447912540337:blk_6533808687986917333_1002, duration: 27240542
2015-11-19 15:07:57,858 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-981411196-192.168.100.200-1447912540337:blk_6533808687986917333_1002, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-11-19 15:08:01,620 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-981411196-192.168.100.200-1447912540337:blk_6533808687986917333_1002

（执行展示 hdfs的1.txt文件后）
2015-11-19 15:24:55,875 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.100.200:50010, dest: /192.168.100.200:14676, bytes: 63, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1463876014_1, offset: 0, srvID: DS-1689463280-192.168.100.200-50010-1447913181338, blockid: BP-981411196-192.168.100.200-1447912540337:blk_6533808687986917333_1002, duration: 1766219


[bruce@iRobot hadoop]$ tail -f logs/*secondarynamenode*.log
（执行展示 hdfs的1.txt文件后）
2015-11-19 15:17:17,868 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Image has not changed. Will not download image.
2015-11-19 15:17:17,869 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://192.168.100.200:50070/getimage?getedit=1&startTxId=3&endTxId=12&storageInfo=-40:337807999:0:CID-4fb81ac3-7e61-42c2-a707-46f2af89a6e2
2015-11-19 15:17:17,889 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.02s at 0.00 KB/s
2015-11-19 15:17:17,889 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file edits_tmp_0000000000000000003-0000000000000000012_0000001447917437869 size 0 bytes.
2015-11-19 15:17:17,890 INFO org.apache.hadoop.hdfs.server.namenode.Checkpointer: Checkpointer about to load edits from 1 stream(s).
2015-11-19 15:17:17,890 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /u01/hadoopdata/hadoop-bruce/dfs/namesecondary/current/edits_0000000000000000003-0000000000000000012 expecting start txid #3
2015-11-19 15:17:17,899 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /u01/hadoopdata/hadoop-bruce/dfs/namesecondary/current/edits_0000000000000000003-0000000000000000012 of size 544 edits # 10 loaded in 0 seconds
2015-11-19 15:17:17,900 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /u01/hadoopdata/hadoop-bruce/dfs/namesecondary/current/fsimage.ckpt_0000000000000000012 using no compression
2015-11-19 15:17:17,919 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file of size 358 saved in 0 seconds.
2015-11-19 15:17:17,941 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 2
2015-11-19 15:17:17,941 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/u01/hadoopdata/hadoop-bruce/dfs/namesecondary/current/fsimage_0000000000000000000, cpktTxId=0000000000000000000)
2015-11-19 15:17:17,946 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://192.168.100.200:50070/getimage?putimage=1&txid=12&port=50090&storageInfo=-40:337807999:0:CID-4fb81ac3-7e61-42c2-a707-46f2af89a6e2
2015-11-19 15:17:17,988 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.04s at 0.00 KB/s
2015-11-19 15:17:17,988 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Uploaded image with txid 12 to namenode at 192.168.100.200:50070
2015-11-19 15:17:17,988 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done. New Image Size: 358





[bruce@iRobot hadoop]$ ls /u01/hadoopdata/
hadoop-bruce
[bruce@iRobot hadoop]$ ls /u01/hadoopdata/hadoop-bruce/
dfs  yarn
[bruce@iRobot hadoop]$ ls /u01/hadoopdata/hadoop-bruce/dfs/
data  name  namesecondary
[bruce@iRobot hadoop]$ ls /u01/hadoopdata/hadoop-bruce/dfs/name
current  in_use.lock                                   
[bruce@iRobot hadoop]$ ls /u01/hadoopdata/hadoop-bruce/dfs/name/current/ -lrt
总用量 1052
-rw-r--r-- 1 bruce oinstall     206 11月 19 13:55 VERSION
-rw-r--r-- 1 bruce oinstall     116 11月 19 13:55 fsimage_0000000000000000000
-rw-r--r-- 1 bruce oinstall      62 11月 19 13:55 fsimage_0000000000000000000.md5
-rw-r--r-- 1 bruce oinstall      30 11月 19 14:17 edits_0000000000000000001-0000000000000000002
-rw-r--r-- 1 bruce oinstall       2 11月 19 14:17 seen_txid
-rw-r--r-- 1 bruce oinstall     116 11月 19 14:17 fsimage_0000000000000000002
-rw-r--r-- 1 bruce oinstall      62 11月 19 14:17 fsimage_0000000000000000002.md5
-rw-r--r-- 1 bruce oinstall 1048576 11月 19 15:07 edits_inprogress_0000000000000000003
[bruce@iRobot hadoop]$ 


	（执行展示 hdfs的1.txt文件后的变化）

	[bruce@iRobot hadoop]$ ls /u01/hadoopdata/hadoop-bruce/dfs/name/current/ -lrt
	总用量 1056
	-rw-r--r-- 1 bruce oinstall     206 11月 19 13:55 VERSION
	-rw-r--r-- 1 bruce oinstall      30 11月 19 14:17 edits_0000000000000000001-0000000000000000002
	-rw-r--r-- 1 bruce oinstall     116 11月 19 14:17 fsimage_0000000000000000002
	-rw-r--r-- 1 bruce oinstall      62 11月 19 14:17 fsimage_0000000000000000002.md5
	-rw-r--r-- 1 bruce oinstall     544 11月 19 15:17 edits_0000000000000000003-0000000000000000012
	-rw-r--r-- 1 bruce oinstall 1048576 11月 19 15:17 edits_inprogress_0000000000000000013
	-rw-r--r-- 1 bruce oinstall       3 11月 19 15:17 seen_txid
	-rw-r--r-- 1 bruce oinstall     358 11月 19 15:17 fsimage_0000000000000000012
	-rw-r--r-- 1 bruce oinstall      62 11月 19 15:17 fsimage_0000000000000000012.md5

[bruce@iRobot hadoop]$ ls /u01/hadoopdata/hadoop-bruce/dfs/data/ -lrt
总用量 8
-rw-r--r-- 1 bruce oinstall   11 11月 19 14:06 in_use.lock
drwxr-xr-x 3 bruce oinstall 4096 11月 19 14:06 current
[bruce@iRobot hadoop]$ ls -lrt /u01/hadoopdata/hadoop-bruce/dfs/data/current/
总用量 8
-rw-r--r-- 1 bruce oinstall  189 11月 19 14:06 VERSION
drwx------ 4 bruce oinstall 4096 11月 19 15:08 BP-981411196-192.168.100.200-1447912540337
[bruce@iRobot hadoop]$ ls -lrt /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/
总用量 12
drwxr-xr-x 2 bruce oinstall 4096 11月 19 14:06 tmp
drwxr-xr-x 4 bruce oinstall 4096 11月 19 14:06 current
-rw-r--r-- 1 bruce oinstall   96 11月 19 15:08 dncp_block_verification.log.prev
-rw-r--r-- 1 bruce oinstall    0 11月 19 15:08 dncp_block_verification.log.curr
[bruce@iRobot hadoop]$ 
[bruce@iRobot hadoop]$ cat /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/dncp_block_verification.log.prev 

date="2015-11-19 15:08:01,620"	 time="1447916881620"	 genstamp="1002"	 id="6533808687986917333"
[bruce@iRobot hadoop]$ 
[bruce@iRobot hadoop]$ cat /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/dncp_block_verification.log.curr 
[bruce@iRobot hadoop]$ ls -lrt /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/tmp/
总用量 0
[bruce@iRobot hadoop]$ ls -lrt /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/current/
总用量 12
-rw-r--r-- 1 bruce oinstall  133 11月 19 14:06 VERSION
drwxr-xr-x 2 bruce oinstall 4096 11月 19 15:07 rbw
drwxr-xr-x 2 bruce oinstall 4096 11月 19 15:07 finalized
[bruce@iRobot hadoop]$ ls -lrt /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/current/rbw/
总用量 0
[bruce@iRobot hadoop]$ ls -lrt /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/current/finalized/
总用量 8
-rw-r--r-- 1 bruce oinstall 11 11月 19 15:07 blk_6533808687986917333_1002.meta
-rw-r--r-- 1 bruce oinstall 59 11月 19 15:07 blk_6533808687986917333
[bruce@iRobot hadoop]$ 
[bruce@iRobot hadoop]$ cat /u01/hadoopdata/hadoop-bruce/dfs/data/current/BP-981411196-192.168.100.200-1447912540337/current/finalized/blk_6533808687986917333
monkey
sea
funny
hello
world
google
cats
worry
over
new
day
[bruce@iRobot hadoop]$ 




-------------------------------------------------------------------------------------------------------------------------

执行wordcount job：  15：40左右开始执行

[bruce@iRobot hadoop]$ $HADOOP_PREFIX/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar   wordcount 1.txt 1.sort
15/11/19 15:40:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/11/19 15:40:45 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
15/11/19 15:40:45 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
15/11/19 15:40:46 INFO input.FileInputFormat: Total input paths to process : 1
15/11/19 15:40:46 INFO mapreduce.JobSubmitter: number of splits:1
15/11/19 15:40:46 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/11/19 15:40:46 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15/11/19 15:40:46 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
15/11/19 15:40:46 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
15/11/19 15:40:46 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
15/11/19 15:40:46 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
15/11/19 15:40:46 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15/11/19 15:40:46 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15/11/19 15:40:46 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/11/19 15:40:46 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15/11/19 15:40:46 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15/11/19 15:40:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1447914055057_0001
15/11/19 15:40:46 INFO client.YarnClientImpl: Submitted application application_1447914055057_0001 to ResourceManager at /192.168.100.200:8032
15/11/19 15:40:46 INFO mapreduce.Job: The url to track the job: http://192.168.100.200:54315/proxy/application_1447914055057_0001/
15/11/19 15:40:46 INFO mapreduce.Job: Running job: job_1447914055057_0001
15/11/19 15:40:55 INFO mapreduce.Job: Job job_1447914055057_0001 running in uber mode : false
15/11/19 15:40:55 INFO mapreduce.Job:  map 0% reduce 0%
15/11/19 15:40:59 INFO mapreduce.Job:  map 100% reduce 0%
15/11/19 15:41:04 INFO mapreduce.Job:  map 100% reduce 100%
15/11/19 15:41:04 INFO mapreduce.Job: Job job_1447914055057_0001 completed successfully
15/11/19 15:41:04 INFO mapreduce.Job: Counters: 43
	File System Counters
		FILE: Number of bytes read=97
		FILE: Number of bytes written=145627
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=166
		HDFS: Number of bytes written=59
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=2610
		Total time spent by all reduces in occupied slots (ms)=3157
	Map-Reduce Framework
		Map input records=11
		Map output records=11
		Map output bytes=103
		Map output materialized bytes=97
		Input split bytes=107
		Combine input records=11
		Combine output records=8
		Reduce input groups=8
		Reduce shuffle bytes=97
		Reduce input records=8
		Reduce output records=8
		Spilled Records=16
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=37
		CPU time spent (ms)=2290
		Physical memory (bytes) snapshot=373596160
		Virtual memory (bytes) snapshot=1467797504
		Total committed heap usage (bytes)=327876608
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=59
	File Output Format Counters 
		Bytes Written=59
[bruce@iRobot hadoop]$ 



[bruce@iRobot hadoop]$ tail -f logs/*-namenode*.log
（15：40左右开始执行后）
2015-11-19 15:40:45,789 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 6 Total time for transactions(ms): 1Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 32 
2015-11-19 15:40:46,012 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.jar. BP-981411196-192.168.100.200-1447912540337 blk_-2593472465878570135_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[192.168.100.200:50010|RBW]]}
2015-11-19 15:40:46,157 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.100.200:50010 is added to blk_-2593472465878570135_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[192.168.100.200:50010|RBW]]} size 0
2015-11-19 15:40:46,161 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.jar is closed by DFSClient_NONMAPREDUCE_1732184435_1
2015-11-19 15:40:46,172 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 1 to 10 for /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.jar
2015-11-19 15:40:46,258 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 1 to 10 for /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.split
2015-11-19 15:40:46,275 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.split. BP-981411196-192.168.100.200-1447912540337 blk_1039308966393142348_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[192.168.100.200:50010|RBW]]}
2015-11-19 15:40:46,283 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.100.200:50010 is added to blk_1039308966393142348_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[192.168.100.200:50010|RBW]]} size 0
2015-11-19 15:40:46,285 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.split is closed by DFSClient_NONMAPREDUCE_1732184435_1
2015-11-19 15:40:46,303 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/bruce/.staging/job_1447914055057_0001/job.splitmetainfo. BP-981411196-192.168.100.200-1447912540337 blk_2736067084384974939_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[
最低0.47元/天 解锁文章
BIT工匠
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Hadoop3.x搭建详细教程 | 历史服务器的配置与日志聚合
大数据小禅的博客
05-18 944
🚀 作者 ：“大数据小禅” 🚀 简介：随着版本的迭代，Hadoop也在不断的进行更新与增加新特性，根据生产与业务的需求，掌握最基础的环境搭建是比较重要的，之前的文章已经进行了Hadoop2的安装，Hadoop3的安装也是基于2的前置环境之上，免密与集群前置准备小伙本们可以看一下之前的文章。 🚀 Hadoop2详细搭建教程： Hadoop2安装教程对应的安装包可以通过最下方公众号联系我分享。 1.Hadoop3.1.x版本集群规划 Hadoop由Apache基金会开源，是一个分布式的储存与计算平台。目前H.
Apache Hadoop MapReduce WordCount案例编程入门
猫君之上的博客
03-13 380
一.MapReduce 简介 MapReduce作为Hadoop的三大组件(功能上分)之一，主要为提供大数据平台的分布式计算，虽然比较臃肿，只适合处理离线处理，但是对于理解spark等框架的原理架构会有很大帮助。 二.WordCount案例编写 为了测试方便，因此直接在windows10本地测试本案例 1.准备阶段 1）数据准备 wordCountdemo.rar 解压到某个文夹下，例如本...
最全大数据hadoop 备忘录实时更新笔记(2)
最新发布
2401_84166258的博客
05-04 1014
配置文件ppt有7.看我自己写的命令node1:9870有个browse Directory，这个网站只有查询权限，没有修改权限，可以配置8.hdfs shell其他命令可以看官方文档9.修改用户所属组修改权限 hdfs dfs -chowd [-R] 777/xxx.txt（加-R对子目录生效）我们为什么要修改权限？因为hadoop只有指定的本地用户才能上传，如果我原来用的不是root，就算是root也上传不了，因为强龙压不过地头蛇fsck可以检查指定路径是否正常。
ConcurrentHashMap源码分析
Hellw0rld的博客
10-14 233
继承关系 继承了AbstractMap抽象类和ConcurrentMap接口 线程安全性 线程安全的 JDK1.7里容器有多把锁，每一把锁锁一段数据（segement数组，加上HashEntry单向队列table） JDK1.8版本： ①取消segments字段，直接采用transient volatile HashEntry<K,V>[] table保存数据，采用table数组元素作为锁，从而实现了对每一行数据进行加锁，并发控制使用Synchronized和CAS来操作 将原先table数组＋
MapReduce 计数器简介（转载自：http://www.tuicool.com/articles/qqyIBr）
weixin_33743880的博客
06-04 134
1、计数器简介 在许多情况下，一个用户需要了解待分析的数据，尽管这并非所要执行的分析任务 的核心内容。以统计数据集中无效记录数目的任务为例，如果发现无效记录的比例 相当高，那么就需要认真思考为何存在如此多无效记录。是所采用的检测程序存在 缺陷，还是数据集质量确实很低，包含大量无效记录？如果确定是数据集的质量问 题，则可能需要扩大数据集的规模，以增大有效...
Hadoop 运行过程深入分析
weixin_34242509的博客
05-25 130
我们照着Hadoop教程简单的写了一个例子，它可以用于分析天气数据然后找到某年的最高气温。 我们是用hadoop 0.20的新的API写的，具体代码如下： Mapper类： /**/packagecom.charles.parseweather;importjava.io.IOException;importorg.apache.hadoop.i...
Hadoop从入门到上手企业开发
07-15
017 查看Hadoop 日志以及日志的格式和命名组成 018 Hadoop 守护进程服务三种启动停止方式 019 测试环境(HDFS Shell基本命令和运行WordCount程序) 020 结合WordCount实例讲解Hadoop的数据存储和数据计算 021 Hadoop ...
hadoop 面试题大全
06-04
首先，我们要理解Hadoop的基本架构和运行模式。 1. Hadoop集群的瓶颈通常在于磁盘I/O，这是大数据处理中的常见问题，因为大量数据的读写操作对硬件性能有很高要求。优化I/O性能可以通过使用更快的硬盘、SSD或者采用...
Hadoop部署实验.docx
06-28
该文档旨在帮助读者了解 Hadoop 的基本架构、HDFS 和 MapReduce 的原理，并掌握 Hadoop 的多节点部署过程和 MapReduce 应用程序的实践。 一、Hadoop 概念 1. Hadoop 是一个能够对海量数据进行分布式处理的系统架构...
常见Hadoop面试题
01-21
1. JobTracker 是一个 master 服务，负责调度 Job 的每一个子任务 task 运行于 TaskTracker 上，并监控它们。 2. TaskTracker 是运行在多个节点上的 slaver 服务，负责执行具体的 MapReduce 任务。 3. MapReduce 中...
Linux下Hadoop的分布式配置和使用.doc
02-23
这些文件中包含了Hadoop运行的各种参数，如Namenode和Datanode的地址、副本数量、内存分配等。配置时需要根据实际集群规模和需求进行调整。 ### 6. Hadoop集群启动 Hadoop集群启动包括初始化NameNode、DataNode、...
Web上传文件到HDFS报错java.io.IOException: File xx could only be written to 0 of the
编程爱好者
06-30 980
问题描述 HDFS部署在阿里云服务器，使用web上传文件到阿里云服务器的HDFS中报错：java.io.IOException: File /17035041/服务器测试/test4.txt could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are excluded in this operation. 报错如下图： 2020-06-30 18:22:
Centos7下Hadoop完全分布式安装
蔡政洁的博客
06-03 557
目录Centos7下Hadoop完全分布式安装第一步：安装文件第二步：免密登录配置（1）vim 的安装（2）host配置（3）关闭防火墙（4）实现免密登陆（5）安装NTP时间同步服务第三步：配置环境变量第四步：设置Hadoop配置文件（1）配置hadoop-env.sh文件（2）配置core-site.xml文件（3）配置hdfs-site.xml文件（4）配置mapred-site.xml文件（5）配置yarn-site.xml文件（6）修改 slaves 文件第五步：分发配置到 node2、node3
(转)hadoop 常规错误问题(一)
weixin_34261739的博客
03-19 2318
转至：http://www.freeoa.net/osuport/db/my-hbase-usage-problem-sets_2979.html 本文是我在使用Hbase的过程碰到的一些问题和相应的解决方法，现将这些经过总结分析，以免日后忘记。hadoop新增节点hadoop集群要增加3个节点。在3台主机分别配置了host、与集群内所有机器的ssh登录互信、jdk1.7 等。所有配置、目录...
namenode gc导致的故障一例
weixin_33869377的博客
11-23 662
周末处理的一次故障，这里简单记录下。 故障现象： 6点1分左右开始，Hadoop集群异常，所有的hdfs操作都出现问题。 几十个job报以下错 1 2 3 4 5 6 7 8 FAILED: RuntimeException org.apache.hadoop.ipc.RemoteException(org.apache.ha...
Mac基于docker的hadoop单机环境搭建
u010979584的博客
01-31 3024
hadoop
IOError: [Errno 2] 错误
热门推荐
wang1144的专栏
06-18 1万+
做python开发，经常遇到IOError: [Errno 13]
Hadoop- Namenode经常挂掉 IPC's epoch 9 is less than the last promised epoch 10
weixin_34242819的博客
01-02 745
如题出现Namenode经常挂掉 IPC's epoch 9 is less than the last promised epoch 10， 2019-01-03 05:36:14,774 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741949_1131{UCState=UNDER_CONS...
Hadoop运行问题：INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node 1/10.39.0.124:8032
06-10
这是一个关于 Hadoop 的信息，意思...这个信息通常出现在启动 Hadoop 作业时的日志中，表明作业已经开始执行。如果你遇到了问题，可以检查 ResourceManager 是否正常运行，以及客户端是否能够连接到 ResourceManager。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值