[TOC] ## 1、硬體配置 採用3臺虛擬機器 節點名稱 | IP地址 | 記憶體 | 硬碟 | 節點角色 ---|:---|:---|:---|:--- node1 | 192.168.1.6 | 2GB | 10GB | NameNode、ResoucerManager node2 | 192.168.1.7 | 2GB | 10GB | DataNode、NodeManager、SecondaryNameNode node3 | 192.168.1.8 | 2GB | 10GB | DataNode、NodeManager ## 2、軟體版本 軟體 | 版本 ---|:--- JDK | jdk-8u271 HADOOP | hadoop-3.2.1 ## 3、準備工作 ### 3.1、建立虛擬機器，網路設定為橋接模式 ### 3.2、更改主機名 ```sh [root@node1 ~]# vi /etc/hostname [root@node1 ~]# reboot [root@node1 ~]# cat /etc/hostname node1 [root@node1 ~]# hostname node1 [root@node2 ~]# hostname # 其他節點同理 node2 [root@node3 ~]# hostname node3 ``` ### 3.3、繫結主機名和IP，建立各主機間的聯絡 在 node1 上執行如下步驟： ```sh [root@node1 ~]# vi /etc/hosts # 新增如下三行內容 IP地址 節點名稱 [root@node1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.6 node1 192.168.1.7 node2 192.168.1.8 node3 # 將hosts檔案複製到node2和node3節點 [root@node1 ~]# scp /etc/hosts node2:/etc/ The authenticity of host 'node2 (192.168.1.7)' can't be established. ECDSA key fingerprint is SHA256:8MU51OTPEjoMAEsg3eOMgAJBy3L4nuSMX1RGWN8ew/w. ECDSA key fingerprint is MD5:00:2a:ce:9a:66:9b:42:af:a6:8e:74:07:a9:01:52:dc. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node2,192.168.1.7' (ECDSA) to the list of known hosts. hosts ``` ```sql [root@node1 ~]# ping node2 PING node2 (192.168.1.7) 56(84) bytes of data. 64 bytes from node2 (192.168.1.7): icmp_seq=1 ttl=64 time=0.404 ms 64 bytes from node2 (192.168.1.7): icmp_seq=2 ttl=64 time=0.617 ms 64 bytes from node2 (192.168.1.7): icmp_seq=3 ttl=64 time=0.828 ms ^C --- node2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2016ms rtt min/avg/max/mdev = 0.404/0.616/0.828/0.174 ms [root@node1 ~]# ping node3 PING node3 (192.168.1.8) 56(84) bytes of data. 64 bytes from node3 (192.168.1.8): icmp_seq=1 ttl=64 time=1.59 ms 64 bytes from node3 (192.168.1.8): icmp_seq=2 ttl=64 time=0.496 ms 64 bytes from node3 (192.168.1.8): icmp_seq=3 ttl=64 time=0.443 ms ^C --- node3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2015ms rtt min/avg/max/mdev = 0.443/0.843/1.592/0.530 ms [root@node2 ~]# ping node1 PING node1 (192.168.1.6) 56(84) bytes of data. 64 bytes from node1 (192.168.1.6): icmp_seq=1 ttl=64 time=0.325 ms 64 bytes from node1 (192.168.1.6): icmp_seq=2 ttl=64 time=0.864 ms ^C --- node1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.325/0.594/0.864/0.270 ms [root@node2 ~]# ping node3 PING node3 (192.168.1.8) 56(84) bytes of data. 64 bytes from node3 (192.168.1.8): icmp_seq=1 ttl=64 time=1.58 ms 64 bytes from node3 (192.168.1.8): icmp_seq=2 ttl=64 time=0.728 ms ^C --- node3 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1012ms rtt min/avg/max/mdev = 0.728/1.158/1.589/0.431 ms [root@node3 ~]# ping node1 PING node1 (192.168.1.6) 56(84) bytes of data. 64 bytes from node1 (192.168.1.6): icmp_seq=1 ttl=64 time=0.372 ms 64 bytes from node1 (192.168.1.6): icmp_seq=2 ttl=64 time=0.395 ms ^C --- node1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1003ms rtt min/avg/max/mdev = 0.372/0.383/0.395/0.022 ms [root@node3 ~]# ping node2 PING node2 (192.168.1.7) 56(84) bytes of data. 64 bytes from node2 (192.168.1.7): icmp_seq=1 ttl=64 time=0.874 ms 64 bytes from node2 (192.168.1.7): icmp_seq=2 ttl=64 time=1.03 ms ^C --- node2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1006ms rtt min/avg/max/mdev = 0.874/0.955/1.036/0.081 ms ``` ### 3.4、關閉防火牆 ```sh [root@node1 ~]# systemctl stop firewalld.service [root@node1 ~]# firewall-cmd --state not running [root@node1 ~]# systemctl disable firewalld.service # 禁止firewall開機啟動 ``` ### 3.5、配置宿主機hosts檔案 使宿主機和虛擬機器系統可以相互ping通 `C:\Windows\System32\drivers\etc\hosts` 目錄下，新增如下內容： 192.168.1.6 node1 192.168.1.7 node2 192.168.1.8 node3 ```sh C:\Users\zgg>ping node1 正在 Ping node1 [192.168.1.6] 具有 32 位元組的資料: 來自 192.168.1.6 的回覆: 位元組=32 時間<1ms TTL=64 來自 192.168.1.6 的回覆: 位元組=32 時間<1ms TTL=64 192.168.1.6 的 Ping 統計資訊: 資料包: 已傳送 = 2，已接收 = 2，丟失 = 0 (0% 丟失)， 往返行程的估計時間(以毫秒為單位): 最短 = 0ms，最長 = 0ms，平均 = 0ms Control-C ^C C:\Users\zgg>ping node2 正在 Ping node2 [192.168.1.7] 具有 32 位元組的資料: 來自 192.168.1.7 的回覆: 位元組=32 時間<1ms TTL=64 來自 192.168.1.7 的回覆: 位元組=32 時間<1ms TTL=64 192.168.1.7 的 Ping 統計資訊: 資料包: 已傳送 = 2，已接收 = 2，丟失 = 0 (0% 丟失)， 往返行程的估計時間(以毫秒為單位): 最短 = 0ms，最長 = 0ms，平均 = 0ms Control-C ^C C:\Users\zgg>ping node3 正在 Ping node3 [192.168.1.8] 具有 32 位元組的資料: 來自 192.168.1.8 的回覆: 位元組=32 時間<1ms TTL=64 來自 192.168.1.8 的回覆: 位元組=32 時間<1ms TTL=64 192.168.1.8 的 Ping 統計資訊: 資料包: 已傳送 = 2，已接收 = 2，丟失 = 0 (0% 丟失)， 往返行程的估計時間(以毫秒為單位): 最短 = 0ms，最長 = 0ms，平均 = 0ms Control-C ^C ``` ### 3.6、配置SSH，實現節點間的無密碼登入 無密碼登陸：在 node1 上，通過 `ssh node2` 或 `ssh node3` 就可以登陸到對方計算機上，而不用輸入密碼。 分別在三臺虛擬機器的 `/root` 目錄下執行： ssh-keygen -t rsa 設定 ssh 的金鑰和金鑰的存放路徑。 路徑為`~/.ssh` 進入到 `.ssh` 目錄，執行如下命令，將公鑰放到 authorized_keys 裡： cp id_rsa.pub authorized_keys 將 node1 上的 authorized_keys 放入其他虛擬機器的 `~/.ssh` 目錄下: scp authorized_keys test2:~/.ssh/ scp authorized_keys test3:~/.ssh/ ```sh [root@node1 ~]# ssh node2 Last login: Thu Nov 12 15:38:28 2020 from node1 [root@node2 ~]# exit 登出 Connection to node2 closed. ``` ## 4、安裝JDK 在 node1 上，下載，解壓，並配置環境變數： ```sh [root@node1 opt]# tar -zxvf jdk-8u271-linux-x64.tar.gz ... [root@node1 opt]# vi /etc/profile [root@node1 opt]# source /etc/profile [root@node1 opt]# java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode) [root@node1 opt]# cat /etc/profile # /etc/profile ... export JAVA_HOME=/opt/jdk1.8.0_271 export PATH=.:$JAVA_HOME/bin:$PATH ... ``` 將 jdk1.8.0_271 複製到 node2 和 node3 ```sh [root@node1 opt]# scp -r jdk1.8.0_271/ node2:/opt/ [root@node1 opt]# scp -r jdk1.8.0_271/ node3:/opt/ ``` 將 /etc/profile 複製到 node2 和 node3 ```sh [root@node1 opt]# scp /etc/profile node2:/etc/ profile 100% 1890 1.4MB/s 00:00 [root@node1 opt]# scp /etc/profile node3:/etc/ profile 100% 1890 1.7MB/s 00:00 [root@node2 opt]# source /etc/profile [root@node3 opt]# source /etc/profile ``` ## 5、安裝Hadoop 在 node1 上，下載，解壓，並配置環境變數： ```sh [root@node1 opt]# tar -zxvf hadoop-3.2.1.tar.gz ... [root@node1 opt]# vi /etc/profile [root@node1 opt]# source /etc/profile [root@node1 opt]# cat /etc/profile # /etc/profile ... export JAVA_HOME=/opt/jdk1.8.0_271 export HADOOP_HOME=/opt/hadoop-3.2.1 export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH ``` 將 /etc/profile 複製到 node2 和 node3 ```sh [root@node1 opt]# scp /etc/profile node2:/etc/ profile 100% 1945 1.6MB/s 00:00 [root@node1 opt]# scp /etc/profile node3:/etc/ profile 100% 1945 1.5MB/s 00:00 [root@node2 opt]# source /etc/profile [root@node3 opt]# source /etc/profile ``` 配置配置檔案後，將 hadoop-3.2.1 複製到 node2 和 node3 ```sh [root@node1 opt]# scp -r hadoop-3.2.1/ node2:/opt/ [root@node1 opt]# scp -r hadoop-3.2.1/ node3:/opt/ ``` ## 6、格式化 對 node1 : ```sh [root@node1 hadoop-3.2.1]# hdfs namenode -format # 格式化 2020-11-12 21:43:16,999 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = node1/192.168.1.6 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 3.2.1 STARTUP_MSG: classpath = /opt/hadoop-3.2.1/etc/ ... 2020-11-12 21:43:20,696 INFO common.Storage: Storage directory /opt/hadoop-3.2.1/dfs/namenode has been successfully formatted. 2020-11-12 21:43:20,762 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/hadoop-3.2.1/dfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression 2020-11-12 21:43:20,859 INFO namenode.FSImageFormatProtobuf: Image file /opt/hadoop-3.2.1/dfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds . 2020-11-12 21:43:20,866 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 2020-11-12 21:43:20,874 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown. 2020-11-12 21:43:20,874 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.1.6 ************************************************************/ ``` 如果再次格式化，需要先刪除 namenode 和 datanode 上的 `dfs/namenode` 和 `dfs/datanode`目錄。 ## 7、啟動 可以全部啟動，也可以分別啟動。 ```sh [root@node1 -3.2.1]# sbin/start-all.sh Starting namenodes on [node1] 上一次登入：六 11月 14 20:28:04 CST 2020pts/0 上 Starting datanodes 上一次登入：六 11月 14 20:30:51 CST 2020pts/0 上 Starting secondary namenodes [node2] 上一次登入：六 11月 14 20:30:54 CST 2020pts/0 上 Starting resourcemanager 上一次登入：六 11月 14 20:30:59 CST 2020pts/0 上 Starting nodemanagers 上一次登入：六 11月 14 20:31:05 CST 2020pts/0 上 [root@node1 hadoop-3.2.1]# mapred --daemon start historyserver [root@node1 hadoop]# jps 11524 ResourceManager 11927 Jps 11899 JobHistoryServer 11100 NameNode [root@node2 hadoop-3.2.1]# jps 8210 DataNode 8312 SecondaryNameNode 8393 NodeManager 8507 Jps [root@node3 hadoop-3.2.1]# jps 17760 DataNode 17981 Jps 17870 NodeManager ```    ## 8、測試wordcount ```sh [root@node1 hadoop-3.2.1]# hadoop jar /opt/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcount /in/wc.txt /out 2020-11-15 00:43:37,940 INFO client.RMProxy: Connecting to ResourceManager at node1/192.168.1.6:8032 2020-11-15 00:43:38,763 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1605372113315_0001 2020-11-15 00:43:38,945 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-11-15 00:43:39,647 INFO input.FileInputFormat: Total input files to process : 1 2020-11-15 00:43:39,695 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-11-15 00:43:39,731 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-11-15 00:43:39,770 INFO mapreduce.JobSubmitter: number of splits:1 2020-11-15 00:43:39,960 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-11-15 00:43:39,999 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1605372113315_0001 2020-11-15 00:43:39,999 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2020-11-15 00:43:40,196 INFO conf.Configuration: resource-types.xml not found 2020-11-15 00:43:40,196 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2020-11-15 00:43:40,664 INFO impl.YarnClientImpl: Submitted application application_1605372113315_0001 2020-11-15 00:43:40,808 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1605372113315_0001/ 2020-11-15 00:43:40,809 INFO mapreduce.Job: Running job: job_1605372113315_0001 2020-11-15 00:43:52,004 INFO mapreduce.Job: Job job_1605372113315_0001 running in uber mode : false 2020-11-15 00:43:52,005 INFO mapreduce.Job: map 0% reduce 0% 2020-11-15 00:43:59,092 INFO mapreduce.Job: map 100% reduce 0% 2020-11-15 00:44:05,137 INFO mapreduce.Job: map 100% reduce 100% 2020-11-15 00:44:06,168 INFO mapreduce.Job: Job job_1605372113315_0001 completed successfully 2020-11-15 00:44:06,284 INFO mapreduce.Job: Counters: 54 File System Counters FILE: Number of bytes read=67 FILE: Number of bytes written=452639 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=149 HDFS: Number of bytes written=41 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=4936 Total time spent by all reduces in occupied slots (ms)=3495 Total time spent by all map tasks (ms)=4936 Total time spent by all reduce tasks (ms)=3495 Total vcore-milliseconds taken by all map tasks=4936 Total vcore-milliseconds taken by all reduce tasks=3495 Total megabyte-milliseconds taken by all map tasks=5054464 Total megabyte-milliseconds taken by all reduce tasks=3578880 Map-Reduce Framework Map input records=3 Map output records=9 Map output bytes=93 Map output materialized bytes=67 Input split bytes=92 Combine input records=9 Combine output records=5 Reduce input groups=5 Reduce shuffle bytes=67 Reduce input records=5 Reduce output records=5 Spilled Records=10 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=162 CPU time spent (ms)=1180 Physical memory (bytes) snapshot=322658304 Virtual memory (bytes) snapshot=5471531008 Total committed heap usage (bytes)=170004480 Peak Map Physical memory (bytes)=210452480 Peak Map Virtual memory (bytes)=2732470272 Peak Reduce Physical memory (bytes)=112205824 Peak Reduce Virtual memory (bytes)=2739060736 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=57 File Output Format Counters Bytes Written=41 ``` 出現瞭如下問題： (1)通過 yarn 提交任務出現 `Failed while trying to construct the redirect url to the log server. Log Server url may not be configured` 原因是未配置 historyserver 服務。配置如下屬性： ```xml ``` (2)執行作業時，出現了 `錯誤: 找不到或無法載入主類 org.apache.hadoop.mapreduce.v2.app.MRAppMaster` ```sh [root@node1 hadoop-3.2.1]# hadoop classpath /opt/hadoop-3.2.1/etc/hadoop:/opt/hadoop-3.2.1/share/hadoop/common/lib/*:/opt/hadoop-3.2.1/share/hadoop/common/*:/opt/hadoop-3.2.1/share/hadoop/hdfs:/opt/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.1/share/hadoop/hdfs/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/*:/opt/hadoop-3.2.1/share/hadoop/yarn:/opt/hadoop-3.2.1/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.1/share/hadoop/yarn/* ``` 將上述值新增到 `yarn-site.xml` 檔案如下屬性中： ```xml ``` (3)執行作業時，出現了 `The auxService:mapreduce_shuffle does not exist` 錯誤。 因為在複製 yarn-site.xml 時漏掉了 `yarn.nodemanager.aux-services` 屬性。 (4)第一次執行作業的時候，輸出日誌一直卡在 `INFO mapreduce.Job: Running job: job_1605371813670_0001` 。這個問題首先要考慮配置檔案是否正確，其次考慮yarn的資源分配。 ## 9、注意點 (1)如果某個程序啟動失敗了，考慮配置檔案是不是配置錯誤了，或者格式化的時候未清理上次叢集的id。 (2)如果啟動，出現了 `ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.` 錯誤，說明在 `hadoop-env.sh` 中未配置此項。具體配置內容見下面的配置檔案。 (3)在 Hadoop3.x 中，NameNode 的 web 埠改成了 9870 (4)配置檔案的配置可以同時參考 [官網叢集搭建](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html)、[官網core-site.xml](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml)、[官網hdfs-site.xml](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)、[官網yarn-site.xml](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)、[官網mapred-site.xml](https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml) (5)在跑任務時，注意資源的分配。 ## 10、配置檔案 ```sh [root@node1 hadoop]# pwd /opt/hadoop-3.2.1/etc/hadoop [root@node1 hadoop]# ls capacity-scheduler.xml hadoop-user-functions.sh.example kms-log4j.properties ssl-client.xml.example configuration.xsl hdfs-site.xml kms-site.xml ssl-server.xml.example container-executor.cfg httpfs-env.sh log4j.properties user_ec_policies.xml.template core-site.xml httpfs-log4j.properties mapred-env.cmd workers hadoop-env.cmd httpfs-signature.secret mapred-env.sh yarn-env.cmd hadoop-env.sh httpfs-site.xml mapred-queues.xml.template yarn-env.sh hadoop-metrics2.properties kms-acls.xml mapred-site.xml yarnservice-log4j.properties hadoop-policy.xml kms-env.sh shellprofile.d yarn-site.xml ``` 管理員應該通過設定 `etc/hadoop/hadoop-env.sh`，和可選的 `etc/hadoop/mapred-env.sh`、`etc/hadoop/yarn-env.sh` 指令碼來對 Hadoop 守護程序環境進行個性化設定，比如，設定 namenode 使用多少堆記憶體。 至少，你需要在每個遠端結點上指定 JAVA_HOME 。 ```sh # 在 node1、node2、node3 節點： [root@node1 hadoop]# vi hadoop-env.sh ... ### # Generic settings for HADOOP ### # Technically, the only required environment variable is JAVA_HOME. # All others are optional. However, the defaults are probably not # preferred. Many sites configure these options outside of Hadoop, # such as in /etc/profile.d # The java implementation to use. ... # export JAVA_HOME= export JAVA_HOME=/opt/jdk1.8.0_271 export HDFS_NAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root ``` ```sh # 在 node1、node2、node3 節點： [root@node1 hadoop]# cat core-site.xml ... ``` ```sh # 在 node1、node2、node3 節點： [root@node1 hadoop]# cat hdfs-site.xml ... ``` ```sh # 在 node1、node2、node3 節點： [root@node1 hadoop]# cat mapred-site.xml ... ``` ```sh # 在 node1、node2、node3 節點： [root@node1 hadoop]# cat yarn-site.xml ... ``` ```sh # 在 node1、node2、node3 節點： [root@node1 hadoop]# cat workers node2 node3 ``` ### 10.1、一些配置項解釋 dfs.datanode.data.dir 預設是 `file://${hadoop.tmp.dir}/dfs/data` Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. The directories should be tagged with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS storage policies. The default storage type will be DISK if the directory does not have a storage type tagged explicitly. Directories that do not exist will be created if local filesystem permission