HDFS文件系統空間使用情況分析
[hduser@master1 bin]$ ./hdfs dfsadmin -report
Configured Capacity: 8453768839168 (7.69 TB)
Present Capacity: 7959274510394 (7.24 TB)
DFS Remaining: 4128882869718 (3.76 TB)
DFS Used: 3830391640676 (3.48 TB)
DFS Used%: 48.12%
Under replicated blocks: 18
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (5):
Name: 192.168.11.137:50010 (slave5)
Hostname: slave5
Decommission Status : Normal
Configured Capacity: 1056617381888 (984.05 GB)
DFS Used: 736041209958 (685.49 GB)
Non DFS Used: 66876043079 (62.28 GB)
DFS Remaining: 253700128851 (236.28 GB)
DFS Used%: 69.66%
DFS Remaining%: 24.01%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 30
Last contact: Tue Jul 03 15:06:51 CST 2018
Name: 192.168.11.134:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 2113511358464 (1.92 TB)
DFS Used: 749551308902 (698.07 GB)
Non DFS Used: 120793546401 (112.50 GB)
DFS Remaining: 1243166503161 (1.13 TB)
DFS Used%: 35.46%
DFS Remaining%: 58.82%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 23
Last contact: Tue Jul 03 15:06:50 CST 2018
Name: 192.168.11.133:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 2113511358464 (1.92 TB)
DFS Used: 792363753676 (737.95 GB)
Non DFS Used: 118847225403 (110.69 GB)
DFS Remaining: 1202300379385 (1.09 TB)
DFS Used%: 37.49%
DFS Remaining%: 56.89%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 21
Last contact: Tue Jul 03 15:06:50 CST 2018
Name: 192.168.11.135:50010 (slave3)
Hostname: slave3
Decommission Status : Normal
Configured Capacity: 2113511358464 (1.92 TB)
DFS Used: 752023101542 (700.38 GB)
Non DFS Used: 121481747963 (113.14 GB)
DFS Remaining: 1240006508959 (1.13 TB)
DFS Used%: 35.58%
DFS Remaining%: 58.67%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 26
Last contact: Tue Jul 03 15:06:50 CST 2018
Name: 192.168.11.136:50010 (slave4)
Hostname: slave4
Decommission Status : Normal
Configured Capacity: 1056617381888 (984.05 GB)
DFS Used: 800412266598 (745.44 GB)
Non DFS Used: 66495765928 (61.93 GB)
DFS Remaining: 189709349362 (176.68 GB)
DFS Used%: 75.75%
DFS Remaining%: 17.95%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 30
2、分析192.168.11.133
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LVroot
17G 12G 4.9G 70% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 190M 125M 56M 70% /boot
/dev/mapper/VolGroup00-LVhome
477M 2.4M 449M 1% /home
/dev/mapper/VolGroup00-LVcloud
2.9G 247M 2.5G 9% /opt/cloud
/dev/mapper/VolGroup00-LVtmp
7.8G 19M 7.4G 1% /tmp
/dev/mapper/VolGroup00-LVvar
25G 786M 23G 4% /var
/dev/mapper/VolGroup1-LVdata1
985G 384G 551G 42% /data
/dev/mapper/VolGroup2-LVdata2
985G 365G 570G 40% /data2
數據統計：
/data和/data2為數據存放目錄
總的容量 = 985G + 985G = 1970G = 1.92T
已使用容量 = 384G + 365G = 749G
未使用容量 = 551G + 570G = 1121G
未顯示容量 = 1970G - （749G + 1121G）= 100G
/data目錄中的hadoop和hbase的解壓文件占用10G左右
dfs文件系統分析：
Name: 192.168.11.133:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 2113511358464 (1.92 TB)
DFS Used: 792363753676 (737.95 GB)
Non DFS Used: 118847225403 (110.69 GB)
DFS Remaining: 1202300379385 (1.09 TB)
DFS Used%: 37.49%
DFS Remaining%: 56.89%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 21
Last contact: Tue Jul 03 15:06:50 CST 2018
說明解析：
Configured Capacity:配置容量
DFS Used:DFS已使用的容量
Non DFS Used:非DFS使用的容量
DFS Remaining:DFS剩余的容量
數據統計：
Configured Capacity = 1.92 TB = 1970 GB
DFS Used = 737.95 GB
Non DFS Used = 110.69 GB
DFS Remaining = 1.09 TB = 1116.16 GB
DFS Used + Non DFS Used + DFS Remaining = 737.95 GB + 110.69 GB + 1116.16 GB = 1964.8 GB = Configured Capacity
所以得出結果：
du -h和./hdfs dfsadmin -report顯示的容量對應關系為：
總的容量 ---> Configured Capacity
已使用容量 ---> DFS Used
未使用容量 ---> DFS Remaining
未顯示容量 + /data下非dfs文件 ---> Non DFS Used
非dfs文件指的是不是同過dfs文件系統上傳的文件，通過mv、cp等方式傳輸的文件。
統計192.168.11.136和192.168.11.133類似。
3、尋找消失的100G
Non DFS Used占用了110.69GB，/data下非dfs文件有10GB左右，那麽剩下的100GB去哪了？
查看保留空間
# tune2fs -l /dev/mapper/VolGroup1-LVdata1
Reserved block count: 13107148
13107148 * 4 / 1024 /1024 = 49.99
# tune2fs -l /dev/mapper/VolGroup2-LVdata2
Reserved block count: 13107148
13107148 * 4 / 1024 /1024 = 49.99
消失的100G就是兩個分區的保留空間之和
參考：
https://community.hortonworks.com/questions/42122/hdfs-non-dfs-used.html
https://stackoverflow.com/questions/18477983/what-exactly-non-dfs-used-means
HDFS文件系統空間使用情況分析