有關HDFS程式設計：HDFS Shell命令列常見操作
阿新 • • 發佈：2019-01-05
在/usr/local/hadoop/etc/hadoop 目錄下：
幫助相關命令
1.hdfs dfs 可以顯示hdfs常用命令
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>] [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] [-e] <path> ...] [-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] [-v] [-x] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] [-skip-empty-file] <src> <localdst>] [-help [cmd ...]] [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-truncate [-w] <length> <path> ...] [-usage [cmd ...]] Generic options supported are: -conf <configuration file> specify an application configuration file -D <property=value> define a value for a given property -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations. -jt <local|resourcemanager:port> specify a ResourceManager -files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster -libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath -archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machines The general command line syntax is: command [genericOptions] [commandOptions]
2.usage 檢視命令的用法，例如，檢視ls的用法
hdfs dfs -usage ls
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -usage ls
Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]3.help 檢視命名的詳細幫助，例如檢視ls命令的詳細幫助
hdfs dfs -help ls
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -help ls -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...] : List the contents that match the specified file pattern. If path is not specified, the contents of /user/<currentUser> will be listed. For a directory a list of its direct children is returned (unless -d option is specified). Directory entries are of the form: permissions - userId groupId sizeOfDirectory(in bytes) modificationDate(yyyy-MM-dd HH:mm) directoryName and file entries are of the form: permissions numberOfReplicas userId groupId sizeOfFile(in bytes) modificationDate(yyyy-MM-dd HH:mm) fileName -C Display the paths of files and directories only. -d Directories are listed as plain files. -h Formats the sizes of files in a human-readable fashion rather than a number of bytes. -q Print ? instead of non-printable characters. -R Recursively list the contents of directories. -t Sort files by modification time (most recent first). -S Sort files by size. -r Reverse the order of the sort. -u Use time of last access instead of modification for display and sorting. -e Display the erasure coding policy of files and directories.
檢視相關命令
4.ls 檢視檔案或者目錄
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -ls / Found 21 items -rw-r--r-- 2 hadoop supergroup 39 2018-03-21 17:17 /aaaa drwxr-xr-x - hadoop supergroup 0 2018-03-11 17:24 /ffk -rw-r--r-- 2 hadoop supergroup 312 2018-03-21 17:10 /hhh drwxr-xr-x - hadoop supergroup 0 2018-03-11 09:31 /hltest drwxr-xr-x - hadoop supergroup 0 2018-03-11 10:09 /hltestout drwxr-xr-x - hadoop supergroup 0 2018-03-21 17:15 /home drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:01 /input drwxr-xr-x - hadoop supergroup 0 2018-03-08 22:26 /ning drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:17 /output drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:22 /output2 drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:24 /output3 drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:26 /output4 drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:27 /output5 drwxr-xr-x - hadoop supergroup 0 2018-03-11 18:34 /result drwxr-xr-x - hadoop supergroup 0 2018-03-07 21:51 /test drwxr-xr-x - hadoop supergroup 0 2018-03-07 20:48 /testxuning drwx------ - hadoop supergroup 0 2018-03-07 19:00 /tmp drwxr-xr-x - hadoop supergroup 0 2018-03-09 21:04 /xu drwxr-xr-x - hadoop supergroup 0 2018-03-10 21:45 /xun drwxr-xr-x - hadoop supergroup 0 2018-03-10 21:50 /xunin drwxr-xr-x - hadoop supergroup 0 2018-03-11 18:33 /xuning
選項-r可連同其子檔案目錄一起列出：hdfs dfs -ls -r/
5.cat 檢視檔案內容
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -cat /aaaa
/home/hadoop/eclipse/committers-oxygen檔案及目錄相關命令
6.touchz建立一個空資料夾，如果存在指定名稱的非空資料夾，則返回錯誤
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -touchz /mal
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -ls /
Found 22 items
-rw-r--r-- 2 hadoop supergroup 39 2018-03-21 17:17 /aaaa
drwxr-xr-x - hadoop supergroup 0 2018-03-11 17:24 /ffk
-rw-r--r-- 2 hadoop supergroup 312 2018-03-21 17:10 /hhh
drwxr-xr-x - hadoop supergroup 0 2018-03-11 09:31 /hltest
drwxr-xr-x - hadoop supergroup 0 2018-03-11 10:09 /hltestout
drwxr-xr-x - hadoop supergroup 0 2018-03-21 17:15 /home
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:01 /input
-rw-r--r-- 2 hadoop supergroup 0 2018-03-21 19:38 /mal7.appendtofile 向現有檔案中追加內容
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -cat /mal
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -appendToFile /home/hadoop/桌面/aaaa /mal
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -cat /mal
/home/hadoop/eclipse/committers-oxygen8.put從本地檔案系統上傳檔案到HDFS
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -put /home/hadoop/桌面/325 /shiyan
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -ls /
Found 23 items
-rw-r--r-- 2 hadoop supergroup 39 2018-03-21 17:17 /aaaa
drwxr-xr-x - hadoop supergroup 0 2018-03-11 17:24 /ffk
-rw-r--r-- 2 hadoop supergroup 312 2018-03-21 17:10 /hhh
drwxr-xr-x - hadoop supergroup 0 2018-03-11 09:31 /hltest
drwxr-xr-x - hadoop supergroup 0 2018-03-11 10:09 /hltestout
drwxr-xr-x - hadoop supergroup 0 2018-03-21 17:15 /home
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:01 /input
-rw-r--r-- 2 hadoop supergroup 39 2018-03-25 15:38 /mal
drwxr-xr-x - hadoop supergroup 0 2018-03-08 22:26 /ning
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:17 /output
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:22 /output2
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:24 /output3
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:26 /output4
drwxr-xr-x - hadoop supergroup 0 2018-03-07 15:27 /output5
drwxr-xr-x - hadoop supergroup 0 2018-03-11 18:34 /result
-rw-r--r-- 2 hadoop supergroup 39 2018-03-25 15:51 /shiyan
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -cat /shiyan
/home/hadoop/eclipse/committers-oxygen選項-f：如果檔案已經存在，則覆蓋已有檔案
選項-p：保留原檔案的訪問和修改時間，使用者和組，許可權屬性
9.get從HDFS上下載檔案到本地，與put不同，沒有覆蓋本地已有檔案的選項
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -get /shiyan ~
[email protected]:/usr/local/hadoop/etc/hadoop$ cd
[email protected]:~$ ls
cqq examples.desktop hl wordcount 模板 未命名資料夾 音樂
eclipse fk2 mapred-site.xml xuning 視訊 文件 桌面
eclipse-workspace hadoop shiyan 公共的 圖片 下載10.getmerge:將指定的HDFS目錄下的檔案合併成一個檔案並下載到本地，原始檔保留
選項-nl：在每個檔案的最後一行增加一新行
11.copyFromLocal
從本地檔案系統上傳到HDFS，與put命令相同
12.copyToLocal
從HDFS下載檔案到本地系統檔案，與get命令相同
13.moveFromLocal
與put命令相同，只是上傳成功後本地檔案會被刪除
14.cp複製檔案
選項-f如果檔案已存在，覆蓋已有檔案
15.mkdir建立資料夾
選項-p如果上層目錄不存在則遞迴建立所需目錄
16.rm刪除檔案
選項-r，遞迴刪除，可以刪除非空目錄
17.rmdir刪除空目錄
選項--ignore-fail-on-non-empety：忽略非空刪除失敗時的提示
（此時，用此命令來刪除非空目錄，沒有錯誤資訊提示，但是檔案未被刪除）
18.setrep改變一個檔案的副本數
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -setrep 2 /aaaa
Replication 2 set: /aaaa選項-w：命令等待副本數調整完成
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -setrep -w 2 /aaaa
Replication 2 set: /aaaa
Waiting for /aaaa ... done19.expunge清空回收站
20.chgrp修改檔案使用者組
[[email protected] hadoop-2.5.2]$ hdfs dfs -chgrp test /output
[[email protected] hadoop-2.5.2]$ hdfs dfs -ls -R /
drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input
-rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt
-rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt
-rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip
-rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip
drwxr-xr-x - hadoop test 0 2015-04-02 08:34 /output --修改後的使用者組（未建立test組，仍可成功）
-rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /output/input1.txt --目錄下檔案的使用者組未修改選項-R：遞迴修改，如果是目錄，則遞迴的修改其下的檔案和目錄
21.chmod修改檔案許可權
補充檔案許可權知識：
d：表示一個目錄（目錄也是一個特殊的檔案）
-：表示這個是普通檔案
r（read，讀取）：讀取檔案內容；瀏覽目錄
w（write，寫入）:新增、修改檔案內容；刪除、移動目錄內檔案
x（execute，執行）:執行檔案；進入目錄
-：表示不具備該項許可權
將rwx看成二進位制，如果有則用1表示，沒有則用0表示，如：rwx r-x r--即可以表示為111 101 100將其每三位轉換成754
-rwx------:等於數字表示700-rwxr―r--:等於數字表示744-rw-rw-r-x:等於數字表示665drwx―x―x:等於數字表示711
drwx------:等於數字表示700
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -ls /
Found 23 items
-rw-r--r-- 2 hadoop supergroup 39 2018-03-21 17:17 /aaaa
drwxr-xr-x - hadoop supergroup 0 2018-03-11 17:24 /ffk
-rw-r--r-- 2 hadoop supergroup 312 2018-03-21 17:10 /hhh
drwxr-xr-x - hadoop supergroup 0 2018-03-11 09:31 /hltest
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -chmod 754 /hltest
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -ls /
Found 23 items
-rw-r--r-- 2 hadoop supergroup 39 2018-03-21 17:17 /aaaa
drwxr-xr-x - hadoop supergroup 0 2018-03-11 17:24 /ffk
-rw-r--r-- 2 hadoop supergroup 312 2018-03-21 17:10 /hhh
drwxr-xr-- - hadoop supergroup 0 2018-03-11 09:31 /hltest22.getfacl顯示訪問控制列表（ACL）
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -getfacl /hltest
# file: /hltest
# owner: hadoop
# group: supergroup
user::rwx
group::r-x
other::r--選項-R：遞迴顯示
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -getfacl -R /hltest
# file: /hltest
# owner: hadoop
# group: supergroup
user::rwx
group::r-x
other::r--
# file: /hltest/file1
# owner: hadoop
# group: supergroup
user::rw-
group::r--
other::r--
# file: /hltest/file2
# owner: hadoop
# group: supergroup
user::rw-
group::r--
other::r--23.setfacl設定訪問控制列表
統計相關命令
24.count：顯示指定檔案或目錄的：DIR_COUNT、FILE_COUNT、CONTENT_SIZE、FILE_NAME,分別表示子目錄個數（如果指定路徑是目錄，則包含目錄本身）、檔案個數、使用位元組數，以及檔案或目錄名。
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -count /
75 2008 37952956 /25.du顯示檔案大小，如果指定目錄，則會顯示該目錄中每個檔案的大小
39 78 /aaaa
0 0 /ffk
312 624 /hhh
140 280 /hltest
0 0 /hltestout
34929793 69859586 /home
87 174 /input
39 78 /mal
207 414 /ning
93 186 /output
0 0 /output2
93 186 /output3
0 0 /output4
93 186 /output5
2004 4008 /result
39 78 /shiyan
2314 4628 /test
96 192 /testxuning
3017342 6244108 /tmp
92 184 /xu
85 170 /xun
0 0 /xunin
88 176 /xuning26.df檢查檔案系統的磁碟空間佔用情況
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -df /
Filesystem Size Used Available Use%
hdfs://master:9000 820654354432 103129088 753473880064 0%27.stat顯示檔案統計資訊
格式：%b-檔案所佔塊數；%g-檔案所屬的使用者組；%n-檔名；%o-檔案塊大小；%r-備份數；%u-檔案所屬使用者；%y-檔案修改時間
[email protected]:/usr/local/hadoop/etc/hadoop$ hdfs dfs -stat %b,%g,%n,%o,%r,,%u,%y /hltest
0,supergroup,hltest,0,0,,hadoop,2018-03-11 01:31:47以上命令基本全部進行實現，少數不常用命令未曾列舉，僅供個人以後學習參考。