HDFS Learning Path II

Metadata Assisted Management of HDFS

The role of SecondaryNameNode: SecondaryNameNode periodically merges fsimage and edits, and controls edits within a range

Configure SecondaryNameNode

  • SecondaryNameNode is specified in conf/masters

  • On the machine specified by masters, modify hdfs-site.xml

  •     <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>node03:50090</value>
        </property>
  •  

     

 

 

  Fsimage & Edits:

In the Hadoop cluster, all the metadata information of the NameNode is stored in the Fsimage and Eidts files

These two files constitute the latest metadata information of all data

Fsimage :

  • The image of metadata in the NameNode is generally called a checkpoint, and fsimage stores a complete metadata information

  • At the beginning, the operations on the NameNode are placed in edits

  • The content of fsimage includes all DataNode files under the management of NameNode and the metadata information of the DataNode where the file block and block are located.

  • As the content of edits increases, it needs to be merged with fsimage at a certain point in time.

Edits :

  • edits stores the operation log of the client in the recent period of time

  • When the client writes files to HDFS, it will first be recorded in the edits file

  • When edits are modified, the metadata is also updated

 

 

Metadata information is stored in the configuration file hdfs-site.xml

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///export/servers/hadoop-2.7.6/hadoopDatas/namenodeDatas,file:///export/servers/hadoop-2.7.6/hadoopDatas/namenodeDatas2</value>
    </property>
    
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///export/servers/hadoop-2.7.6/hadoopDatas/datanodeDatas,file:///export/servers/hadoop-2.7.6/hadoopDatas/datanodeDatas2</value>
    </property>
    
    <property>
        <name>dfs.namenode.edits.dir</name>
        <value>file:///export/servers/hadoop-2.7.6/hadoopDatas/nn/edits</value>
    </property>

 

View files in fismage:

#Use the command hdfs oiv
cd /export/servers/hadoop2.7.6/hadoopDatas
hdfs oiv -i fsimage_000000000000864 -p XML -o hello.xml

View the file in edits:

#Use the command hdfs oev
cd /export/servers/hadoop2.7.6/hadoopDatas
hdfs oev -i fsimage_000000000000864 -p XML -o omyedit.xml

 

 

Detailed explanation of the heartbeat mechanism of Datanode and Namenode:

1. hdfs is node01/slave structure, node01 includes namenode and resourcemanager, slave includes datanode and nodemanager
 
2. When node01 starts, an IPC service will be started, waiting for the slave connection
 
3. After the slave is started, it will actively connect to the IPC service and link every 3 seconds. This time can be adjusted. Set the heartbeat, which is a mechanism for connecting every once in a while, called the heartbeat mechanism. Slave reports its information to node01 through the heart hop, and node01 issues commands through the heart hop.
 
4. The Namenode knows the datanode status through heartbeat. Resourcemanager knows nodemanager status through heartbeat
 
5. When node01 does not receive slave information for a long time, it is considered that the slave has died.
 
Note: Calculation result of overtime: the default is 10 minutes and 30 seconds
 
 

Detailed explanation of the checkpoint mechanism of SecondaryNameNode:

SecondaryNamenode, it can assist Namenode to merge fsimage and editlog, reduce the size of editlog file, so as to shorten the next restart time of Namenode and exit safe mode as soon as possible.
The merging cycle of the two files, called the checkpoint mechanism (checkpoint), can be modified through the hdfs-default.xml configuration file:
 
<property>
 <name>dfs.namenode.checkpoint.period</name>
 <value>3600</value>
 <description>Seconds between checkpoints, default is 1 hour</description>
</property> 
<property>
 <name>dfs.namenode.checkpoint.txns</name>
 <value>1000000</value>
 <description>txid The number of executions reaches 100 w times, also execute checkpoint</description>
</property> 
<property>
 <name>dfs.namenode.checkpoint.check.period</name>
 <value>60</value>
 <description>60 Check in seconds txid execution times</description>
</property>

1. SecondaryNamenode requests Namenode to stop using the editlog file being edited, Namenode will create a new editlog file and update the seed_txid file at the same time.

 

2. SecondaryNamenode obtains fsimage and editlog files on Namenode through HTTP protocol.

 

3. The SecondaryNamenode reads the fsimage into memory, analyzes the data in the editlog file step by step, performs a merge operation, and writes it to a new file fsimage_x.ckpt.

 

4. SecondaryNamenode sends the new file fsimage_x.ckpt back to Namenode via HTTP protocol.

 

5. Rename operation, rename the fsimage file

Posted by SJones on Sun, 08 May 2022 08:59:58 +0300