Hadoop 學習筆記（五）HDFS 常用 Shell 命令
阿新 • • 發佈：2020-10-11
下面我們一起學習 HDFS常用 Shell 命令，開始之前，我們啟動 HDFS 和 Yarn;
[hui@hadoop103 ~]$ /opt/module/hadoop-2.7.2/sbin/start-dfs.sh
Starting namenodes on [hadoop103]
hadoop103: starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hui-namenode-hadoop103.out
hadoop103: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hui-datanode-hadoop103.out
hadoop105: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hui-datanode-hadoop105.out
hadoop104: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hui-datanode-hadoop104.out
Starting secondary namenodes [hadoop105]
hadoop105: starting secondarynamenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hui-secondarynamenode-hadoop105.out
[hui@hadoop104 ~]$ /opt/module/hadoop-2.7.2/sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hui-resourcemanager-hadoop104.out
hadoop103: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hui-nodemanager-hadoop103.out
hadoop105: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hui-nodemanager-hadoop105.out
hadoop104: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hui-nodemanager-hadoop104.out
1、命令大全
通過 hadoop fs 可以檢視所有可用的 HDFS 操作命令：
[hui@hadoop104 ~]$ hadoop fs Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-d] [-h] [-R] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-truncate [-w] <length> <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions]
2、檢視 hdfs 根路徑
[hui@hadoop103 ~]$ hadoop fs -ls / Found 1 items drwxr-xr-x - hui supergroup 0 2020-08-12 11:15 /user ^[[A[hui@hadoop103 ~]$ hadoop fs -ls -R / drwxr-xr-x - hui supergroup 0 2020-08-12 11:15 /user drwxr-xr-x - hui supergroup 0 2020-08-12 11:17 /user/xajh -rw-r--r-- 3 hui supergroup 20 2020-08-12 11:17 /user/xajh/huasan
3、建立目錄
[hui@hadoop103 ~]$ hadoop fs -mkdir -p /h1/h2/h3 [hui@hadoop103 ~]$ hadoop fs -ls -R /h1 drwxr-xr-x - hui supergroup 0 2020-09-05 09:48 /h1/h2 drwxr-xr-x - hui supergroup 0 2020-09-05 09:48 /h1/h2/h3
4、上傳檔案
[hui@hadoop103 data]$ hadoop fs -put dept.txt /h1/h2/ [hui@hadoop103 data]$ hadoop fs -copyFromLocal dept.txt /h1/h2/h3 [hui@hadoop103 data]$ hadoop fs -ls -R /h1 drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2/h3 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/h3/dept.txt
5、下載檔案/合併下載
單個下載檔案
[hui@hadoop103 data]$ hadoop fs -get /h1/h2/dept.txt ./dept.txt [hui@hadoop103 data]$ hadoop fs -copyToLocal /h1/h2/dept.txt ./dept1.txt [hui@hadoop103 data]$ ll 總用量 48 -rw-r--r-- 1 hui wd 266 2月 11 2020 business.txt -rw-r--r-- 1 hui wd 71 9月 5 09:55 dept1.txt -rw-r--r-- 1 hui wd 71 12月 10 2018 dept_bak.txt -rw-r--r-- 1 hui wd 71 9月 5 09:55 dept.txt -rw-r--r-- 1 hui wd 669 12月 10 2018 emp.txt -rw-r--r-- 1 hui wd 20 8月 12 11:16 huasan -rw-r--r-- 1 hui wd 134 5月 4 01:13 movie.txt -rw-r--r-- 1 hui wd 11 6月 4 21:29 renyy.txt -rw-r--r-- 1 hui wd 64 5月 2 05:24 stu_bak.txt -rw-r--r-- 1 hui wd 144 2月 10 2020 test0.txt -rw-r--r-- 1 hui wd 128 2月 11 2020 visit.txt -rw-r--r-- 1 hui wd 309 5月 3 20:24 xajh.txt
合併下載檔案
[hui@hadoop103 data]$ hadoop fs -getmerge /h1/h2/dept.txt /h1/h2/h3/dept.txt ./dept_2.txt
6、複製
[hui@hadoop103 data]$ hadoop fs -cp /h1/h2/dept.txt /h1 [hui@hadoop103 data]$ hadoop fs -ls -R / drwxr-xr-x - hui supergroup 0 2020-09-05 10:09 /h1 -rw-r--r-- 3 hui supergroup 71 2020-09-05 10:09 /h1/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2/h3 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/h3/dept.txt drwxr-xr-x - hui supergroup 0 2020-08-12 11:15 /user drwxr-xr-x - hui supergroup 0 2020-08-12 11:17 /user/xajh -rw-r--r-- 3 hui supergroup 20 2020-08-12 11:17 /user/xajh/huasan
7、移動
[hui@hadoop103 ~]$ hadoop fs -mv /h1/dept.txt /user/dept.txt [hui@hadoop103 ~]$ hadoop fs -ls -R / drwxr-xr-x - hui supergroup 0 2020-09-09 12:15 /h1 drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2/h3 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/h3/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-09 12:15 /user -rw-r--r-- 3 hui supergroup 71 2020-09-05 10:09 /user/dept.txt drwxr-xr-x - hui supergroup 0 2020-08-12 11:17 /user/xajh -rw-r--r-- 3 hui supergroup 20 2020-08-12 11:17 /user/xajh/huasan
8、刪除
[hui@hadoop103 ~]$ hadoop fs -rm /user/dept.txt 20/09/09 12:16:51 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /user/dept.txt
刪除空資料夾
[hui@hadoop103 ~]$ hadoop fs -ls -R / drwxr-xr-x - hui supergroup 0 2020-09-09 12:15 /h1 drwxr-xr-x - hui supergroup 0 2020-09-05 09:50 /h1/h2 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-09 12:19 /h1/h2/h3 drwxr-xr-x - hui supergroup 0 2020-09-09 12:16 /user drwxr-xr-x - hui supergroup 0 2020-08-12 11:17 /user/xajh -rw-r--r-- 3 hui supergroup 20 2020-08-12 11:17 /user/xajh/huasan [hui@hadoop103 ~]$ hadoop fs -rmdir /h1/h2/h3 [hui@hadoop103 ~]$ hadoop fs -ls -R / drwxr-xr-x - hui supergroup 0 2020-09-09 12:15 /h1 drwxr-xr-x - hui supergroup 0 2020-09-09 12:20 /h1/h2 -rw-r--r-- 3 hui supergroup 71 2020-09-05 09:50 /h1/h2/dept.txt drwxr-xr-x - hui supergroup 0 2020-09-09 12:16 /user drwxr-xr-x - hui supergroup 0 2020-08-12 11:17 /user/xajh -rw-r--r-- 3 hui supergroup 20 2020-08-12 11:17 /user/xajh/huasan
強制刪除
[hui@hadoop103 ~]$ hadoop fs -rm -r /h1/h2 20/09/09 12:22:26 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /h1/h2 [hui@hadoop103 ~]$ hadoop fs -ls -R / drwxr-xr-x - hui supergroup 0 2020-09-09 12:22 /h1 drwxr-xr-x - hui supergroup 0 2020-09-09 12:16 /user drwxr-xr-x - hui supergroup 0 2020-08-12 11:17 /user/xajh -rw-r--r-- 3 hui supergroup 20 2020-08-12 11:17 /user/xajh/huasan
9、檢視叢集工作狀態
[hui@hadoop103 ~]$ hdfs dfsadmin -report Configured Capacity: 56047030272 (52.20 GB) Present Capacity: 39654842368 (36.93 GB) DFS Remaining: 39654694912 (36.93 GB) DFS Used: 147456 (144 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 192.168.134.130:50010 (hadoop103) Hostname: hadoop103 Decommission Status : Normal Configured Capacity: 18682343424 (17.40 GB) DFS Used: 49152 (48 KB) Non DFS Used: 5873045504 (5.47 GB) DFS Remaining: 12809248768 (11.93 GB) DFS Used%: 0.00% DFS Remaining%: 68.56% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Wed Sep 09 12:27:26 CST 2020 Name: 192.168.134.133:50010 (hadoop105) Hostname: hadoop105 Decommission Status : Normal Configured Capacity: 18682343424 (17.40 GB) DFS Used: 49152 (48 KB) Non DFS Used: 5282631680 (4.92 GB) DFS Remaining: 13399662592 (12.48 GB) DFS Used%: 0.00% DFS Remaining%: 71.72% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Wed Sep 09 12:27:26 CST 2020 Name: 192.168.134.131:50010 (hadoop104) Hostname: hadoop104 Decommission Status : Normal Configured Capacity: 18682343424 (17.40 GB) DFS Used: 49152 (48 KB) Non DFS Used: 5236510720 (4.88 GB) DFS Remaining: 13445783552 (12.52 GB) DFS Used%: 0.00% DFS Remaining%: 71.97% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Wed Sep 09 12:27:27 CST 2020