Hadoop系列四: HDFS相关的checkpoint交互

本文深入探讨了Hadoop的HDFS文件系统中Checkpoint机制的工作原理,包括其重要性、交互过程以及如何确保数据一致性。通过理解这些概念,读者能够更好地了解Hadoop集群的数据管理和容错能力。
摘要由CSDN通过智能技术生成



以下为停止运行任何job后,整个hadoop系统的日志输出:

[bruce@iRobot hadoop]$ tail -f logs/*-namenode*.log

2015-11-19 16:17:18,303 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.100.200
2015-11-19 16:17:18,303 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2015-11-19 16:17:18,303 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 13
2015-11-19 16:17:18,303 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 96 Total time for transactions(ms): 10Number of transactions batched in Syncs: 0 Number of syncs: 53 SyncTimes(ms): 269 
2015-11-19 16:17:18,313 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 96 Total time for transactions(ms): 10Number of transactions batched in Syncs: 0 Number of syncs: 54 SyncTimes(ms): 279 
2015-11-19 16:17:18,314 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /u01/hadoopdata/hadoop-bruce/dfs/name/current/edits_inprogress_0000000000000000013 -> /u01/hadoopdata/hadoop-bruce/dfs/name/current/edits_0000000000000000013-0000000000000000108
2015-11-19 16:17:18,314 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 109
2015-11-19 16:17:18,474 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://iRobot:50090/getimage?getimage=1&txid=108&storageInfo=-40:337807999:0:CID-4fb81ac3-7e61-42c2-a707-46f2af89a6e2
2015-11-19 16:17:18,495 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.02s at 47.62 KB/s
2015-11-19 16:17:18,495 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000108 size 1767 bytes.
2015-11-19 16:17:18,514 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 12
2015-11-19 16:17:18,515 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/u01/hadoopdata/hadoop-bruce/dfs/name/current/fsimage_0000000000000000002, cpktTxId=0000000000000000002)
2015-11-19 17:17:18,825 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.100.200
2015-11-19 17:17:18,825 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2015-11-19 17:17:18,825 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 109
2015-11-19 17:17:18,826 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 31 
2015-11-19 17:17:18,836 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 41 
2015-11-19 17:17:18,837 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /u01/hadoopdata/hadoop-bruce/dfs/name/current/edits_inprogress_0000000000000000109 -> /u01/hadoopdata/hadoop-bruce/dfs/name/current/edits_0000000000000000109-0000000000000000110
2015-11-19 17:17:18,837 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 111
2015-11-19 17:17:18,969 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://iRobot:50090/getimage?getimage=1&txid=110&storageInfo=-40:337807999:0:CID-4fb81ac3-7e61-42c2-a707-46f2af89a6e2
2015-11-19 17:17:18,985 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.02s at 62.50 KB/s
2015-11-19 17:17:18,986 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000110 size 1767 bytes.
2015-11-19 17:17:19,004 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 108
2015-11-19 17:17:19
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值