oiv 查看 Fsimage 文件和oev 查看 Edits 文件

最新推荐文章于 2023-04-28 17:31:14 发布
最新推荐文章于 2023-04-28 17:31:14 发布
阅读量1.2k 收藏 1
点赞数 1
分类专栏： 大数据开发相关技术学习笔记 文章标签： hadoop hdfs 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_43224174/article/details/121794194
版权

oiv 查看 Fsimage 文件

1、进入目的路径

[lqs@bdc112 current]$ cd /home/lqs/module/hadoop-3.1.3/data/dfs/name/current

语法：
	hdfs oiv -p 文件类型 -i 镜像文件 -o 转换后文件输出路径

2、使用命令进行转换

[lqs@bdc112 current]$ hdfs oiv -p XML -i fsimage_0000000000000000330 -o /home/lqs/module/hadoop-3.1.3/fsimage1.xml

3、导入idea进行格式化后查看，内容如下：

<?xml version="1.0"?>
<fsimage>
    <version>
        <layoutVersion>-64</layoutVersion>
        <onDiskVersion>1</onDiskVersion>
        <oivRevision>ba631c436b806728f8ec2f54ab1e289526c90579</oivRevision>
    </version>
    <NameSection>
        <namespaceId>2086111572</namespaceId>
        <genstampV1>1000</genstampV1>
        <genstampV2>1031</genstampV2>
        <genstampV1Limit>0</genstampV1Limit>
        <lastAllocatedBlockId>1073741855</lastAllocatedBlockId>
        <txid>330</txid>
    </NameSection>
    <ErasureCodingSection>
        <erasureCodingPolicy>
            <policyId>1</policyId>
            <policyName>RS-6-3-1024k</policyName>
            <cellSize>1048576</cellSize>
            <policyState>DISABLED</policyState>
            <ecSchema>
                <codecName>rs</codecName>
                <dataUnits>6</dataUnits>
                <parityUnits>3</parityUnits>
            </ecSchema>
        </erasureCodingPolicy>

        <erasureCodingPolicy>
            <policyId>2</policyId>
            <policyName>RS-3-2-1024k</policyName>
            <cellSize>1048576</cellSize>
            <policyState>DISABLED</policyState>
            <ecSchema>
                <codecName>rs</codecName>
                <dataUnits>3</dataUnits>
                <parityUnits>2</parityUnits>
            </ecSchema>
        </erasureCodingPolicy>

        <erasureCodingPolicy>
            <policyId>3</policyId>
            <policyName>RS-LEGACY-6-3-1024k</policyName>
            <cellSize>1048576</cellSize>
            <policyState>DISABLED</policyState>
            <ecSchema>
                <codecName>rs-legacy</codecName>
                <dataUnits>6</dataUnits>
                <parityUnits>3</parityUnits>
            </ecSchema>
        </erasureCodingPolicy>

        <erasureCodingPolicy>
            <policyId>4</policyId>
            <policyName>XOR-2-1-1024k</policyName>
            <cellSize>1048576</cellSize>
            <policyState>DISABLED</policyState>
            <ecSchema>
                <codecName>xor</codecName>
                <dataUnits>2</dataUnits>
                <parityUnits>1</parityUnits>
            </ecSchema>
        </erasureCodingPolicy>

        <erasureCodingPolicy>
            <policyId>5</policyId>
            <policyName>RS-10-4-1024k</policyName>
            <cellSize>1048576</cellSize>
            <policyState>DISABLED</policyState>
            <ecSchema>
                <codecName>rs</codecName>
                <dataUnits>10</dataUnits>
                <parityUnits>4</parityUnits>
            </ecSchema>
        </erasureCodingPolicy>

    </ErasureCodingSection>

    <INodeSection>
        <lastInodeId>16468</lastInodeId>
        <numInodes>39</numInodes>
        <inode>
            <id>16385</id>
            <type>DIRECTORY</type>
            <name></name>
            <mtime>1638944340547</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>9223372036854775807</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16386</id>
            <type>DIRECTORY</type>
            <name>tmp</name>
            <mtime>1638944079685</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16387</id>
            <type>DIRECTORY</type>
            <name>hadoop-yarn</name>
            <mtime>1638708219023</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16388</id>
            <type>DIRECTORY</type>
            <name>staging</name>
            <mtime>1638944026810</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16389</id>
            <type>DIRECTORY</type>
            <name>history</name>
            <mtime>1638708219062</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16390</id>
            <type>DIRECTORY</type>
            <name>done</name>
            <mtime>1638944155423</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16391</id>
            <type>DIRECTORY</type>
            <name>done_intermediate</name>
            <mtime>1638944083921</mtime>
            <permission>lqs:supergroup:1777</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16395</id>
            <type>DIRECTORY</type>
            <name>xiyou</name>
            <mtime>1638874848565</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16396</id>
            <type>DIRECTORY</type>
            <name>huaguoshan</name>
            <mtime>1638940862591</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16397</id>
            <type>FILE</type>
            <name>sunwukong.txt</name>
            <replication>2</replication>
            <mtime>1638891402818</mtime>
            <atime>1638891401920</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741826</id>
                    <genstamp>1002</genstamp>
                    <numBytes>42</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16407</id>
            <type>FILE</type>
            <name>sunwukong.txt</name>
            <replication>2</replication>
            <mtime>1638940521097</mtime>
            <atime>1638940520807</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741829</id>
                    <genstamp>1005</genstamp>
                    <numBytes>42</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16408</id>
            <type>DIRECTORY</type>
            <name>lqs</name>
            <mtime>1638940730514</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16409</id>
            <type>DIRECTORY</type>
            <name>test</name>
            <mtime>1638940862591</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16410</id>
            <type>FILE</type>
            <name>lqs.txt</name>
            <replication>2</replication>
            <mtime>1638940748858</mtime>
            <atime>1638940748204</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741830</id>
                    <genstamp>1006</genstamp>
                    <numBytes>42</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16411</id>
            <type>DIRECTORY</type>
            <name>lqs</name>
            <mtime>1638944026810</mtime>
            <permission>lqs:supergroup:0700</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16412</id>
            <type>DIRECTORY</type>
            <name>.staging</name>
            <mtime>1638944359932</mtime>
            <permission>lqs:supergroup:0700</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16415</id>
            <type>DIRECTORY</type>
            <name>input</name>
            <mtime>1638944264244</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16421</id>
            <type>DIRECTORY</type>
            <name>logs</name>
            <mtime>1638944079724</mtime>
            <permission>lqs:lqs:1777</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16422</id>
            <type>DIRECTORY</type>
            <name>lqs</name>
            <mtime>1638944079726</mtime>
            <permission>lqs:lqs:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16423</id>
            <type>DIRECTORY</type>
            <name>logs-tfile</name>
            <mtime>1638944336519</mtime>
            <permission>lqs:lqs:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16424</id>
            <type>DIRECTORY</type>
            <name>application_1638924755371_0002</name>
            <mtime>1638944104711</mtime>
            <permission>lqs:lqs:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16425</id>
            <type>DIRECTORY</type>
            <name>lqs</name>
            <mtime>1638944392624</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16438</id>
            <type>FILE</type>
            <name>
                job_1638924755371_0002-1638944077724-lqs-word+count-1638944096756-0-1-SUCCEEDED-default-1638944088259.jhist
            </name>
            <replication>3</replication>
            <mtime>1638944096890</mtime>
            <atime>1638944096852</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0770</permission>
            <blocks>
                <block>
                    <id>1073741839</id>
                    <genstamp>1015</genstamp>
                    <numBytes>16277</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16439</id>
            <type>FILE</type>
            <name>job_1638924755371_0002_conf.xml</name>
            <replication>3</replication>
            <mtime>1638944097034</mtime>
            <atime>1638944096901</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0770</permission>
            <blocks>
                <block>
                    <id>1073741840</id>
                    <genstamp>1016</genstamp>
                    <numBytes>214600</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16440</id>
            <type>FILE</type>
            <name>bdc114_38083</name>
            <replication>3</replication>
            <mtime>1638944104704</mtime>
            <atime>1638944104602</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:lqs:0640</permission>
            <blocks>
                <block>
                    <id>1073741841</id>
                    <genstamp>1017</genstamp>
                    <numBytes>95203</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16441</id>
            <type>DIRECTORY</type>
            <name>2021</name>
            <mtime>1638944155423</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16442</id>
            <type>DIRECTORY</type>
            <name>12</name>
            <mtime>1638944155423</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16443</id>
            <type>DIRECTORY</type>
            <name>08</name>
            <mtime>1638944155423</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16444</id>
            <type>DIRECTORY</type>
            <name>000000</name>
            <mtime>1638944392624</mtime>
            <permission>lqs:supergroup:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16445</id>
            <type>FILE</type>
            <name>word.txt</name>
            <replication>3</replication>
            <mtime>1638944264329</mtime>
            <atime>1638944264244</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741842</id>
                    <genstamp>1018</genstamp>
                    <numBytes>45</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16451</id>
            <type>DIRECTORY</type>
            <name>application_1638924755371_0003</name>
            <mtime>1638944366469</mtime>
            <permission>lqs:lqs:0770</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16453</id>
            <type>DIRECTORY</type>
            <name>output</name>
            <mtime>1638944358484</mtime>
            <permission>lqs:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16459</id>
            <type>FILE</type>
            <name>part-r-00000</name>
            <replication>3</replication>
            <mtime>1638944358373</mtime>
            <atime>1638944358229</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741849</id>
                    <genstamp>1025</genstamp>
                    <numBytes>38</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16461</id>
            <type>FILE</type>
            <name>_SUCCESS</name>
            <replication>3</replication>
            <mtime>1638944358490</mtime>
            <atime>1638944358484</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0644</permission>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16464</id>
            <type>FILE</type>
            <name>
                job_1638924755371_0003-1638944336424-lqs-word+count-1638944358494-1-1-SUCCEEDED-default-1638944340533.jhist
            </name>
            <replication>3</replication>
            <mtime>1638944358622</mtime>
            <atime>1638944358589</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0770</permission>
            <blocks>
                <block>
                    <id>1073741851</id>
                    <genstamp>1027</genstamp>
                    <numBytes>22304</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16465</id>
            <type>FILE</type>
            <name>job_1638924755371_0003_conf.xml</name>
            <replication>3</replication>
            <mtime>1638944358764</mtime>
            <atime>1638944358636</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:supergroup:0770</permission>
            <blocks>
                <block>
                    <id>1073741852</id>
                    <genstamp>1028</genstamp>
                    <numBytes>214600</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16466</id>
            <type>FILE</type>
            <name>bdc113_37030</name>
            <replication>3</replication>
            <mtime>1638944365783</mtime>
            <atime>1638944365630</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:lqs:0640</permission>
            <blocks>
                <block>
                    <id>1073741853</id>
                    <genstamp>1029</genstamp>
                    <numBytes>33348</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16467</id>
            <type>FILE</type>
            <name>bdc112_35067</name>
            <replication>3</replication>
            <mtime>1638944366122</mtime>
            <atime>1638944366013</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:lqs:0640</permission>
            <blocks>
                <block>
                    <id>1073741854</id>
                    <genstamp>1030</genstamp>
                    <numBytes>35775</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16468</id>
            <type>FILE</type>
            <name>bdc114_38083</name>
            <replication>3</replication>
            <mtime>1638944366465</mtime>
            <atime>1638944366365</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>lqs:lqs:0640</permission>
            <blocks>
                <block>
                    <id>1073741855</id>
                    <genstamp>1031</genstamp>
                    <numBytes>68619</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
    </INodeSection>
    <INodeReferenceSection></INodeReferenceSection>
    <SnapshotSection>
        <snapshotCounter>0</snapshotCounter>
        <numSnapshots>0</numSnapshots>
    </SnapshotSection>
    <INodeDirectorySection>
        <directory>
            <parent>16385</parent>
            <child>16415</child>
            <child>16408</child>
            <child>16453</child>
            <child>16397</child>
            <child>16386</child>
            <child>16395</child>
        </directory>
        <directory>
            <parent>16386</parent>
            <child>16387</child>
            <child>16421</child>
        </directory>
        <directory>
            <parent>16387</parent>
            <child>16388</child>
        </directory>
        <directory>
            <parent>16388</parent>
            <child>16389</child>
            <child>16411</child>
        </directory>
        <directory>
            <parent>16389</parent>
            <child>16390</child>
            <child>16391</child>
        </directory>
        <directory>
            <parent>16390</parent>
            <child>16441</child>
        </directory>
        <directory>
            <parent>16391</parent>
            <child>16425</child>
        </directory>
        <directory>
            <parent>16395</parent>
            <child>16396</child>
        </directory>
        <directory>
            <parent>16408</parent>
            <child>16409</child>
        </directory>
        <directory>
            <parent>16409</parent>
            <child>16410</child>
            <child>16407</child>
        </directory>
        <directory>
            <parent>16411</parent>
            <child>16412</child>
        </directory>
        <directory>
            <parent>16415</parent>
            <child>16445</child>
        </directory>
        <directory>
            <parent>16421</parent>
            <child>16422</child>
        </directory>
        <directory>
            <parent>16422</parent>
            <child>16423</child>
        </directory>
        <directory>
            <parent>16423</parent>
            <child>16424</child>
            <child>16451</child>
        </directory>
        <directory>
            <parent>16424</parent>
            <child>16440</child>
        </directory>
        <directory>
            <parent>16441</parent>
            <child>16442</child>
        </directory>
        <directory>
            <parent>16442</parent>
            <child>16443</child>
        </directory>
        <directory>
            <parent>16443</parent>
            <child>16444</child>
        </directory>
        <directory>
            <parent>16444</parent>
            <child>16438</child>
            <child>16439</child>
            <child>16464</child>
            <child>16465</child>
        </directory>
        <directory>
            <parent>16451</parent>
            <child>16467</child>
            <child>16466</child>
            <child>16468</child>
        </directory>
        <directory>
            <parent>16453</parent>
            <child>16461</child>
            <child>16459</child>
        </directory>
    </INodeDirectorySection>
    <FileUnderConstructionSection></FileUnderConstructionSection>
    <SecretManagerSection>
        <currentId>0</currentId>
        <tokenSequenceNumber>0</tokenSequenceNumber>
        <numDelegationKeys>0</numDelegationKeys>
        <numTokens>0</numTokens>
    </SecretManagerSection>
    <CacheManagerSection>
        <nextDirectiveId>1</nextDirectiveId>
        <numDirectives>0</numDirectives>
        <numPools>0</numPools>
    </CacheManagerSection>
</fsimage>

oev 查看 Edits 文件

1、进入目的路径

[lqs@bdc112 current]$ cd /home/lqs/module/hadoop-3.1.3/data/dfs/name/current

语法：
	hdfs oev -p 文件类型 -i 编辑日志 -o 转换后文件输出路径

2、使用命令进行转换

[lqs@bdc112 current]$ hdfs oev -p XML -i edits_0000000000000000045-0000000000000000055 -o /home/lqs/module/hadoop-3.1.3/edits.xml

3、导入idea进行格式化后查看，内容如下：

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<EDITS>
    <EDITS_VERSION>-64</EDITS_VERSION>
    <RECORD>
        <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
        <DATA>
            <TXID>45</TXID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ADD</OPCODE>
        <DATA>
            <TXID>46</TXID>
            <LENGTH>0</LENGTH>
            <INODEID>16397</INODEID>
            <PATH>/lqs.txt</PATH>
            <REPLICATION>2</REPLICATION>
            <MTIME>1638891401920</MTIME>
            <ATIME>1638891401920</ATIME>
            <BLOCKSIZE>134217728</BLOCKSIZE>
            <CLIENT_NAME>DFSClient_NONMAPREDUCE_-410590844_1</CLIENT_NAME>
            <CLIENT_MACHINE>192.168.10.15</CLIENT_MACHINE>
            <OVERWRITE>true</OVERWRITE>
            <PERMISSION_STATUS>
                <USERNAME>lqs</USERNAME>
                <GROUPNAME>supergroup</GROUPNAME>
                <MODE>420</MODE>
            </PERMISSION_STATUS>
            <ERASURE_CODING_POLICY_ID>0</ERASURE_CODING_POLICY_ID>
            <RPC_CLIENTID>c5157f32-8ffb-48c3-a1e5-c1f898e156cc</RPC_CLIENTID>
            <RPC_CALLID>2</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE>
        <DATA>
            <TXID>47</TXID>
            <BLOCK_ID>1073741826</BLOCK_ID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_SET_GENSTAMP_V2</OPCODE>
        <DATA>
            <TXID>48</TXID>
            <GENSTAMPV2>1002</GENSTAMPV2>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ADD_BLOCK</OPCODE>
        <DATA>
            <TXID>49</TXID>
            <PATH>/lqs.txt</PATH>
            <BLOCK>
                <BLOCK_ID>1073741826</BLOCK_ID>
                <NUM_BYTES>0</NUM_BYTES>
                <GENSTAMP>1002</GENSTAMP>
            </BLOCK>
            <RPC_CLIENTID/>
            <RPC_CALLID>-2</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_CLOSE</OPCODE>
        <DATA>
            <TXID>50</TXID>
            <LENGTH>0</LENGTH>
            <INODEID>0</INODEID>
            <PATH>/lqs.txt</PATH>
            <REPLICATION>2</REPLICATION>
            <MTIME>1638891402818</MTIME>
            <ATIME>1638891401920</ATIME>
            <BLOCKSIZE>134217728</BLOCKSIZE>
            <CLIENT_NAME/>
            <CLIENT_MACHINE/>
            <OVERWRITE>false</OVERWRITE>
            <BLOCK>
                <BLOCK_ID>1073741826</BLOCK_ID>
                <NUM_BYTES>42</NUM_BYTES>
                <GENSTAMP>1002</GENSTAMP>
            </BLOCK>
            <PERMISSION_STATUS>
                <USERNAME>lqs</USERNAME>
                <GROUPNAME>supergroup</GROUPNAME>
                <MODE>420</MODE>
            </PERMISSION_STATUS>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ADD</OPCODE>
        <DATA>
            <TXID>51</TXID>
            <LENGTH>0</LENGTH>
            <INODEID>16398</INODEID>
            <PATH>/test/test1/lqs.txt</PATH>
            <REPLICATION>2</REPLICATION>
            <MTIME>1638892454273</MTIME>
            <ATIME>1638892454273</ATIME>
            <BLOCKSIZE>134217728</BLOCKSIZE>
            <CLIENT_NAME>DFSClient_NONMAPREDUCE_1886145386_1</CLIENT_NAME>
            <CLIENT_MACHINE>192.168.10.15</CLIENT_MACHINE>
            <OVERWRITE>true</OVERWRITE>
            <PERMISSION_STATUS>
                <USERNAME>lqs</USERNAME>
                <GROUPNAME>supergroup</GROUPNAME>
                <MODE>420</MODE>
            </PERMISSION_STATUS>
            <ERASURE_CODING_POLICY_ID>0</ERASURE_CODING_POLICY_ID>
            <RPC_CLIENTID>de910111-6395-46f0-9fac-ee3e8911fa65</RPC_CLIENTID>
            <RPC_CALLID>2</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE>
        <DATA>
            <TXID>52</TXID>
            <BLOCK_ID>1073741827</BLOCK_ID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_SET_GENSTAMP_V2</OPCODE>
        <DATA>
            <TXID>53</TXID>
            <GENSTAMPV2>1003</GENSTAMPV2>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ADD_BLOCK</OPCODE>
        <DATA>
            <TXID>54</TXID>
            <PATH>/lqs/test/lqs1.txt</PATH>
            <BLOCK>
                <BLOCK_ID>1073741827</BLOCK_ID>
                <NUM_BYTES>0</NUM_BYTES>
                <GENSTAMP>1003</GENSTAMP>
            </BLOCK>
            <RPC_CLIENTID/>
            <RPC_CALLID>-2</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_CLOSE</OPCODE>
        <DATA>
            <TXID>55</TXID>
            <LENGTH>0</LENGTH>
            <INODEID>0</INODEID>
            <PATH>/test/test/lqs.txt</PATH>
            <REPLICATION>2</REPLICATION>
            <MTIME>1638892454551</MTIME>
            <ATIME>1638892454273</ATIME>
            <BLOCKSIZE>134217728</BLOCKSIZE>
            <CLIENT_NAME/>
            <CLIENT_MACHINE/>
            <OVERWRITE>false</OVERWRITE>
            <BLOCK>
                <BLOCK_ID>1073741827</BLOCK_ID>
                <NUM_BYTES>42</NUM_BYTES>
                <GENSTAMP>1003</GENSTAMP>
            </BLOCK>
            <PERMISSION_STATUS>
                <USERNAME>lqs</USERNAME>
                <GROUPNAME>supergroup</GROUPNAME>
                <MODE>420</MODE>
            </PERMISSION_STATUS>
        </DATA>
    </RECORD>
</EDITS>
小雏菊的成长
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
FSImage并行加载以及对HDFS OIV的影响
weixin_42473019的博客
04-02 2946
FSImage并行加载 关于NameNode FSImage并行加载现在有很多文章都详细介绍过，比如：https://blog.csdn.net/Androidlushangderen/article/details/100088073，所以这里就不大篇幅说了 简单总结就是开启该功能后，FSImage文件里的FILESUMMARY将会添加INODE_SUB和INODE_DIR_SUB两类SECTION的索引，将INODE和INODE_DIR两个Section进行逻辑拆分，以实现并行加载。 并行加载FSIma
HDFS fsimage和edits中的文件信息查看
热门推荐
Ele_sorriu的博客
10-14 2万+
FSimage文件当中的文件信息查看 使用命令 hdfs oiv cd /export/servers/Hadoop-2.6.0-cdh5.14.0/HadoopDatas/namenodeDatas/current hdfs oiv -i fsimage_0000000000000000864 -p XML -o hello.xml edits当中的文件信息查看 查看命令 hdfs oev cd /export/servers/hadoop-2.6.0-cdh5.1...
hdfs oiv命令查看FsImage镜像文件内容
学亮编程手记
07-29 568
hdfs oiv -p XML -i /var/bigdata/hadoop/full/dfs/name/current/fsimage_0000000000000023560 -o ~/output.xml
edits、fsimage文件查看器
wang_ying_198的专栏
12-01 513
oev是offline edits viewer（离线edits查看器）的缩写，该工具只操作文件因而并不需要hadoop集群处于运行状态。 目前支持的输出格式有binary（hadoop使用的二进制格式）、xml（在不使用参数p时的默认输出格式）和stats（输出edits文件的统计信息）。 oiv是offline image viewer的缩写，用于将fsimage文件的内容
第五章 NameNode和SecondaryNameNode
qq_38358499的博客
05-25 218
5.1 NN和2NN的工作机制 元数据存储，如果存储在内存中，计算快，但是可靠性差；如果存储在磁盘上，计算慢，可靠性高；如果存储在内存+磁盘上，要去两个地方读取数据，效率低，所以，出现了fsImage存储数据，Edits追加数据。磁盘中备份元数据的是FsImage，Edits文件（只进行追加操作，效率高）。每当元数据有更新或者添加元数据时，修改内存中的元数据并追加到Edits中。一旦NameNode节点断电，可以通过FsImage和Edits的合并，合成元数据。但是，如果长时间添加数据到Edits中，会导
NameNode 和 SecondaryNameNode： Fsimage 和 Edits 解析、ovi命令、oev命令
七七的博客
02-28 547
文章目录NameNode 和 SecondaryNameNode： Fsimage 和 Edits 解析、ovi命令、oev命令一、SecondaryNamenode， 专门用于 FsImage 和 Edits 的合并。二、 Fsimage 和 Edits 解析Fsimage和Edits概念1）oiv 查看 Fsimage 文件2）oev 查看 Edits 文件 NameNode 和 SecondaryNameNode： Fsimage 和 Edits 解析、ovi命令、oev命令 一、Secondary
镜像文件(fsimage)和编辑日志(Edits)介绍
一亩三分地
02-03 9950
一、查看日志镜像文件内容 1）镜像文件的作用 通过查看上面的xml文件,可以知道镜像文件实际是存放的目录结构、文件属性等相关信息。另外也可以看到MD5校验文件，这个文件主要是用来判断镜像文件是否发生修改。fsimage文件是NameNode中关于元数据的镜像，一般称为检查点，它是在NameNode启动时对整个文件系统的快照。 fsimage文件是hadoop文件系统元数据的一个...
Day06-HDFS的体系结构(namenode作用、datanode作用、block)、fsimage文件、edits文件和checkpoing检查点机制
长安一片月的博客
06-07 610
HDFS的体系结构(namenode作用、datanode作用、block)、fsimage文件、edits文件和checkpoing检查点机制和Namenode开启过程HDFS的体系结构-------------------------    1.namenode[守护进程]作用：        管理文件系统名称空间；【fsimage元数据镜像文件】 管理客户端对文件的访问。【edits操作文件...
【Hadoop-HDFS】HDFS中Fsimage与Edits详解
最新发布
weixin_53543905的博客
04-28 2347
在HDFS中，NameNode 保存了整个 HDFS 的元数据信息，而这些数据最终会被持久化到 Fsimage 文件和 EditLog 文件。换而言之，NameNode 的元数据信息由Fsimage和Editlog组成。Fsimage 存放上次 checkpoint 生成的文件系统元数据。EditLog 则存放文件系统的操作日志，也就是用户对目录、文件的每个写操作（包括创建、删除、写入等）都会被记录到 Editlog 文件中。注意。【Hadoop-HDFS】HDFS中Fsimage与Edits详解
fsimage和edits详解
Hxm6440的博客
11-08 730
namenode元数据解析 第一次启动namenode格式化后，创建fsimage和edits文件。如果不是第一次启动，直接加载edits和fsimage文件到内存 客户端对元数据进行增删改的请求 namenode记录操作日志，更新滚动日志 namenode在内存中对数据进行增删改查 fsimage保存了最新的元数据检查点，在HDFS启动时加载fsimage的信息，包含了整个HDFS文件系...
查看fsimage和edits文件
Hello 我的
03-12 538
查看fsimage文件： 启动服务器bin/hdfs oiv -i 某个fsimage文件 查看内容bin/hdfs dfs -ls  -R webhdfs://127.0.0.1:5978/ 导出结果bin/hdfs oiv -p XML -i  tmp/dfs/name/current/fsimage_0000000000000000055  -o fsimage.xml 查看ed
(4-3)查看fsimage和edits
鲍礼彬的CSDN博客 ~_~
04-11 1157
查看fsimage文件内容 启动：bin/hdfs oiv -i /usr/local/mydata/dfs/name/current/fsimage_0000000000000000254 查看：bin/hdfs dfs -ls -R webhdfs://127.0.0.1:5978/ bin/hdfs -ls -R / 不启动的话： [root@i-love-yo
解析fsimage文件（文件与数据块）
小小码农
06-03 2353
fsimage文件与editslog文件存储的路径  查看hdfs-site.xml属性： fsimage、editslog文件存储：（可多目录） dfs.namenode.name.dir /opt/data/name blk数据块文件：（可多目录） dfs.datanode.data.dir /opt/data/dn1,/opt/data/dn2 fsimage和editslog文件存储在dfs.namenode.name.dir的值下面。 /opt/data/name/current/
OIV读取*.iv文件
Augusdi的专栏
06-07 2605
 #include #include #include int main(int , char ** argv){ HWND window=SoWin::init(argv[0]); if(!window)exit(1); SoSeparator *root=new SoSeparator; root->ref(); SoInput myInput;
第十五讲.元数据分析：查看解析fsimage和Edits文件
美妙人生IT
08-13 328
视频：【美妙人生】Hadoop课程系列之HDFS--手把手教你精通HDFS 【美妙人生】Hadoop课程系列之HDFS--手把手教你精通HDFS 【视频笔记】 元数据 ------------------------- 1.查看fsimage文件 $>hdfs oiv -i fsimage_0000000000000000022 -o ~/fs22.xml -p XML 【 <?xml version="1.0"?> ...
spark(scala)获取fsimage,并分析出fsimage中的小文件信息
miracle_8的博客
03-24 735
spark(scala)获取fsimage,并分析出fsimage中的小文件信息 项目要求: 获取fsimage并解析, 做聚合操作 采用spark并发处理 // pom.xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" ...
Hadoop oiv、oev
mayaohao的博客
07-09 501
【oiv】offline image viwer --------------------------------------------------------------1.用于查看Hadoop fsimage 2.语法 $> hdfs oiv -i inputfile -o outputfile -P process 3.inputfile: 要查看的fsimage文件 outputfile: 用于保存格式化之后的文件 process: 使用什么进程解码，XML|We...
hadoop查看fsimage
epitomizelu的专栏
05-21 715
一，使用hdfs命令获取FsImage数据文件 hdfs dfsadmin -fetchImage ./tmp_meta 注意，这是本地文件系统 二，执行命令解析fsimage文件 hdfs oiv -i ../tmp_meta/fsimage_0000000000000023025 -o ../tmp_meta_rec/fs.csv -p Delimited 然后可以下载到windows中，用excel打开。 ...
Hadoop基础（三） --- oiv、oev、通过URL访问hdfs以及FileSyatem-- Hadoop的文件系统的常用API
xcvbxv01的博客
08-12 3060
【oiv】offline image viwer    -------------------------------------------------------------- 1.用于查看Hadoop fsimage  2.语法     $&gt; hdfs oiv -i inputfile -o outputfile -P process 3.inputfile: 要查看的fsimage文...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

小雏菊的成长

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值