NameNode執行原理淺析
阿新 • • 發佈:2019-02-07
hadoop版本:Apache Hadoop 2.6.0
instance的role type主要有3個,namenode,secondary namenode和data node;其他的還有balancer,backup node等
1.HDFS儲存在本地作業系統磁碟的目錄結構
NameNode:
/data/dfs/nn/current
-rw-r--r-- 1 hdfs hdfs 322K Apr 17 08:41 edits_0000000000007093753-0000000000007096223
-rw-r--r-- 1 hdfs hdfs 338K Apr 17 09:41 edits_0000000000007096224-0000000000007098803
-rw-r--r-- 1 hdfs hdfs 361K Apr 17 10:41 edits_0000000000007098804-0000000000007101548
-rw-r--r-- 1 hdfs hdfs 1.0M Apr 17 11:13 edits_inprogress_0000000000007101549
-rw-r--r-- 1 hdfs hdfs 53M Apr 17 09:41 fsimage_0000000000007098803
-rw-r--r-- 1 hdfs hdfs 62 Apr 17 09:41 fsimage_0000000000007098803.md5
-rw-r--r-- 1 hdfs hdfs 53M Apr 17 10:41 fsimage_0000000000007101548
-rw-r--r-- 1 hdfs hdfs 62 Apr 17 10:41 fsimage_0000000000007101548.md5
-rw-r--r-- 1 hdfs hdfs 8 Apr 17 10:41 seen_txid
-rw-r--r-- 1 hdfs hdfs 173 Nov 13 12:05 VERSION
Secondary NameNode:
/data1/dfs/snn/current
-rw-r--r-- 1 hdfs hdfs 322K Apr 17 08:41 edits_0000000000007093753-0000000000007096223
-rw-r--r-- 1 hdfs hdfs 338K Apr 17 09:41 edits_0000000000007096224-0000000000007098803
-rw-r--r-- 1 hdfs hdfs 361K Apr 17 10:41 edits_0000000000007098804-0000000000007101548
-rw-r--r-- 1 hdfs hdfs 53M Apr 17 09:41 fsimage_0000000000007098803
-rw-r--r-- 1 hdfs hdfs 62 Apr 17 09:41 fsimage_0000000000007098803.md5
-rw-r--r-- 1 hdfs hdfs 53M Apr 17 10:41 fsimage_0000000000007101548
-rw-r--r-- 1 hdfs hdfs 62 Apr 17 10:41 fsimage_0000000000007101548.md5
-rw-r--r-- 1 hdfs hdfs 173 Apr 17 10:41 VERSION
2.NameNode中一共有4種檔案:
edits_{x}_{y}:和fsimage已經merger過的transaction記錄檔案,x為起始transaction記號,y為截止號
edits_inprogress_{y}:尚未和fsimage merger過的transaction記錄檔案,y為transaction截止號
fsimage_{y}:namenode的namespace存在本地的checkpoint檔案,儲存了y為transaction截止號;一般會有兩個,檔案生成時間間隔為設定的checkpoint生成間隔時間;
*NameNode上只有在啟動時才會合併最新的fsimage_{y}和edits_inprogress_{y}檔案