龍芯3A2000移植Hadoop指南
阿新 • • 發佈：2019-01-03
http://www.loongson.cn/news/company/468.html
一、hadoop 簡介
hadoop是一個由Apache基金會所開發的分散式系統基礎架構。使用者可以在不瞭解分散式底層細節的情況下，開發分散式程式。充分利用叢集的威力進行高速運算和儲存。
龍芯3A2000上執行Hadoop
hadoop實現了一個分散式檔案系統（hadoop Distributed File System），簡稱HDFS。HDFS有高容錯性
hadoop的框架最核心的設計就是：HDFS和MapReduce。HDFS為海量的資料提供了儲存，則MapReduce為海量的資料提供了計算。
hadoop是一個能夠對大量資料進行
hadoop是一個能夠讓使用者輕鬆架構和使用的分散式計算平臺。使用者可以輕鬆地在hadoop上開發和執行處理海量資料的應用程式。它主要有以下幾個優點：
高可靠性: hadoop按位儲存和處理資料的能力值得人們信賴。
高擴充套件性: hadoop是在可用的計算機集簇間分配資料並完成計算任務的，這些集簇可以方便地擴充套件到數以千計的節點中。
高效性: hadoop能夠在節點之間動態地移動資料，並保證各個節點的
高容錯性:hadoop能夠自動儲存資料的多個副本，並且能夠自動將失敗的任務重新分配。
低成本: 與一體機、商用資料倉庫以及QlikView、Yonghong Z-Suite等資料集市相比，hadoop是開源的，專案的軟體成本因此會大大降低。
本文主要涉及以下內容：hadoop原始碼編譯，hadoop在分散式計算雲端儲存系統中的部署和應用,同時也將記錄hadoop搭建過程的FAQ和相對解決方案。
hadoop 叢集(cluster) 支援如下3種操作模式：
1. Local/Standalone Mode
完成下載後，預設情況下hadoop 被配置為Standalone 模式，作為單個Java程序執行。
2. Pseudo Distributed Mode
此種模式下,每個hadoop 守護程序,如hdfs,yarn,MapReduce 等分散式部署在不同的機器上，分別作為獨立的Java 程序，這種模式有助於開發。
3. Fully Distributed Mode
完全分散式部署，需要至少2臺機器，作為一個叢集，稍後進行詳解。
二、移植環境
首先給出本機的軟硬體資訊，
軟體環境：
(1)loongnix1.0 系統（2016.8.10版本）。下載地址 www.loongnix.org
(2)核心版本:3.10.84-all
(3)JDK版本:1.8.0_25-rc16-b17 or later
(4)MAVEN:3.2.2 or later
硬體環境:
(1)開發板型別: Loongson-3B-780E-2w-V0.2-demo
(2)韌體版本: loongson-PMON-V3.3.0
本例中使用的hadoop的版本為2.7.2, hadoop 原始碼下載地址，參見附錄中的”hadoop downloads” 連結。hadoop 編譯依賴findbugs和cmake軟體包，建議在編譯前通過yum 命令進行自動安裝,安裝方式如下:
[[email protected] log]$ sudo yum -y install java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless \ java-1.8.0-openjdk findbugs cmake protobuf-compiler
完成安裝後，需要設定如下環境變數，建議將以下內容追加到/et c/profile檔案,並用source 命令使其生效。
export FINDBUGS_HOME=/usr/share/findbugs
export MAVEN_HOME=/usr/share/maven
export MAVEN_OPTS="-Xms256m -Xmx512m"
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.25-5.rc16.fc21.loongson.m
PATH=/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/h
export PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin
Build From Scratch：首先解壓原始碼到自定義目錄(本例採用/usr/local)利用mvn clean package -Pdist,native,src -DskipTests -Dtar 命令進行編譯。
tar xvf hadoop-2.7.2.src.gz -C mkdir /usr/local/
cd /usr/local/hadoop-2.7.2
mvn clean package -Pdist,native,src -DskipTests -Dtar
三、注意事項
(1)本例中採用/usr/local 作為工作目錄需要root許可權
(2)編譯過程報錯,可參見對應FAQ,問題解決後，通過mvn package -Pdist,native,src -DskipTests -Dtar 命令再次啟動編譯。
(3)FAQ的標識由序號(從001開始)和模組名組成,其中者通過冒號間隔。模組名源自maven Reactor 涉及的modules名稱。
四、FAQ
001:Apache hadoop Common
終端報錯:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000000ffe18f46fc, pid=5300, tid=1099154321904
#
# JRE version: OpenJDK Runtime Environment (8.0_25-b17) (build 1.8.0_25-rc16-b17)
# Java VM: OpenJDK 64-Bit Server VM (25.25-b02 mixed mode linux- compressed oops)
# Problematic frame:
# J 62748 C2 scala.tools.asm.ClassWriter.get(Lscala/tools/asm/Item;)Lscala/tools/asm/Item; (49 bytes) @ 0x000000ffe18f46fc [0x000000ffe18f46a0+0x5c]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
解決方法:
此問題與JDK的並行GC相關,編譯hadoop和spark均有遇到,目前的解決方法:調整/etc/profile 檔案MAVEN_OPTS 環境變數為如下內容:
export MAVEN_OPTS="-Xms3560m -Xmx3560m -XX:-UseParallelGC -XX:-UseParallelOldGC"
002: any-modules
終端現象: maven 編譯過程中構件(xxx.jar和xxx.pom) 無法下載。
解決方法: 開啟maven 配置檔案的代理設定選項,並重新安裝ca-certificates
#為maven 設定代理
<proxies>
<!-- proxy
| Specification for one proxy, to be used in connecting to the network.
|-->
<proxy>
<id>proxy01</id>
<active>true</active>
<protocol>http</protocol>
<host>ip_address</host>
<port>port</port>
<nonProxyHosts>localhost</nonProxyHosts>
</proxy>
<proxy>
<id>proxy02</id>
<active>true</active>
<protocol>https</protocol>
<host>ip_address</host>
<port>port</port>
<nonProxyHosts>localhost</nonProxyHosts>
</proxy>
</proxies>
#重新安裝ca-certificates
Sudo yum -y install ca-certificates
注意事項: 凡出現Maven 編譯過程構件無法下載，均可參考本FAQ內容進行適當修改。
五、編譯結果
Maven編譯通過後，將在終端顯示hadoop 的maven Reactor(本次編譯的所有maven 模組)和編譯時間資訊。下面給出的時耗資訊，進攻參考不同軟硬體平臺將會產生差異。
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache hadoop Main ................................. SUCCESS [ 10.769 s]
[INFO] Apache hadoop Project POM .......................... SUCCESS [ 8.793 s]
[INFO] Apache hadoop Annotations .......................... SUCCESS [ 18.834 s]
[INFO] Apache hadoop Assemblies ........................... SUCCESS [ 2.414 s]
[INFO] Apache hadoop Project Dist POM ..................... SUCCESS [ 9.653 s]
[INFO] Apache hadoop Maven Plugins ........................ SUCCESS [ 25.215 s]
[INFO] Apache hadoop MiniKDC .............................. SUCCESS [ 20.682 s]
[INFO] Apache hadoop Auth ................................. SUCCESS [ 26.240 s]
[INFO] Apache hadoop Auth Examples ........................ SUCCESS [ 23.112 s]
[INFO] Apache hadoop Common ............................... SUCCESS [45:23 min]
[INFO] Apache hadoop NFS .................................. SUCCESS [ 45.079 s]
[INFO] Apache hadoop KMS .................................. SUCCESS [01:27 min]
[INFO] Apache hadoop Common Project ....................... SUCCESS [ 1.104 s]
[INFO] Apache hadoop HDFS ................................. SUCCESS [21:45 min]
[INFO] Apache hadoop HttpFS ............................... SUCCESS [02:13 min]
[INFO] Apache hadoop HDFS BookKeeper Journal .............. SUCCESS [ 47.832 s]
[INFO] Apache hadoop HDFS-NFS ............................. SUCCESS [ 34.029 s]
[INFO] Apache hadoop HDFS Project ......................... SUCCESS [ 1.075 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 1.354 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [07:20 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [35:51 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 1.020 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [01:42 min]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [01:58 min]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 25.288 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [01:05 min]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [02:52 min]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 40.356 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 54.780 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 24.110 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 1.017 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 21.223 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 17.608 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 1.145 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 42.659 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 34.614 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 1.905 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [33:18 min]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [32:57 min]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 28.868 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [01:00 min]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 46.223 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 29.643 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 15.580 s]
[INFO] Apache hadoop MapReduce Examples ................... SUCCESS [ 40.229 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 24.719 s]
[INFO] Apache hadoop MapReduce Streaming .................. SUCCESS [ 33.669 s]
[INFO] Apache hadoop Distributed Copy ..................... SUCCESS [ 59.792 s]
[INFO] Apache hadoop Archives ............................. SUCCESS [ 19.986 s]
[INFO] Apache hadoop Rumen ................................ SUCCESS [ 47.303 s]
[INFO] Apache hadoop Gridmix .............................. SUCCESS [ 30.258 s]
[INFO] Apache hadoop Data Join ............................ SUCCESS [ 22.306 s]
[INFO] Apache hadoop Ant Tasks ............................ SUCCESS [ 19.212 s]
[INFO] Apache hadoop Extras ............................... SUCCESS [ 27.362 s]
[INFO] Apache hadoop Pipes ................................ SUCCESS [ 6.723 s]
[INFO] Apache hadoop OpenStack support .................... SUCCESS [ 34.857 s]
[INFO] Apache hadoop Amazon Web Services support .......... SUCCESS [ 37.631 s]
[INFO] Apache hadoop Azure support ........................ SUCCESS [ 30.848 s]
[INFO] Apache hadoop Client ............................... SUCCESS [01:02 min]
[INFO] Apache hadoop Mini-Cluster ......................... SUCCESS [ 3.409 s]
[INFO] Apache hadoop Scheduler Load Simulator ............. SUCCESS [ 33.821 s]
[INFO] Apache hadoop Tools Dist ........................... SUCCESS [ 55.501 s]
[INFO] Apache hadoop Tools ................................ SUCCESS [ 0.768 s]
[INFO] Apache hadoop Distribution ......................... SUCCESS [03:44 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:33 h
[INFO] Finished at: 2016-08-01T14:22:17+08:00
[INFO] Final Memory: 125M/3096M
[INFO] ------------------------------------------------------------------------
本例的編譯結果位於/usr/local/hadoop-2.7.2/hadoop-dist/target/目錄，原始碼包和二進位制包分別為hadoop-2.7.2-src.tar.gz和hadoop-2.7.2.tar.gz。至此hadoop編譯結束。
六、Hadoop 叢集搭建測試
本節採用hadoop ”Fully Distributed Mode” 工作模式,在IP地址分別為10.20.42.22(slave1),10.20.42.22(slave2),10.20.42.199(master)的機器上部署3節點的hadoop叢集。
1. 設定SSH免密碼登入
SSH免密碼登入，假設使用root使用者，在每臺伺服器都生成公鑰，再合併到authorized_keys，具體操作如下：
(1)fadora21預設沒有啟動ssh無密登入，修改/etc/ssh/sshd_config註釋掉以下2行。(每臺機器都要設定)
#RSAAuthentication yes
#PubkeyAuthentication yes
(2)在叢集中的每臺機器上，開啟shell終端輸入命令，ssh-keygen -t rsa，生成key，不要輸入密碼，一直回車，/root就會生成.ssh資料夾,這個檔案一般是隱藏的。(每臺伺服器都要設定)
(3)合併slave節點的公鑰到authorized_keys檔案。在Master伺服器，進入/root/.ssh目錄，使用如下命令:
cat id_rsa.pub>> authorized_keys
ssh [email protected] cat ~/.ssh/id_rsa.pub>> authorized_keys
ssh [email protected] cat ~/.ssh/id_rsa.pub>> authorized_keys
(4)把Master伺服器的authorized_keys、known_hosts複製到Slave伺服器的/root/.ssh目錄
(5)終端輸入ssh [email protected]和ssh [email protected]進行驗證是否免密登陸配置成功
2. 搭建hadoop 3節點叢集
搭建思路:準備1臺主伺服器和2臺從伺服器，從主伺服器可以ssh免密登入從伺服器器。hadoop壓縮包採用上節編譯結果:hadoop-2.7.2.tar.gz。 3臺伺服器的概要資訊如下:
Master 10.20.42.199
Slave1 10.20.42.22
Slave2 10.20.42.10
搭建前提: 伺服器需要安裝JDK並設定好JAVA_HOM等環境變數。可參考下面的例子:
#編輯/etc/profile 檔案並設定JAVA_HOME等環境變數
vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.25-6.b17.rc16.fc21.loongson.mips64el
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
#使環境變數生效 並且驗證jdk 是否生效
source /etc/profile && java -version
開始搭建
解壓hadoop-2.7.2.tar.gz 軟體包,筆者的工作目錄為/home/loongson/,沒有特殊說明下面的配置檔案均來自master伺服器。
(1)解壓hadoop軟體包: tar -xvf hadoop-2.7.2.tar.gz -C /home/loongson
(2)在/home/loongson/hadoop-2.7.2目錄下手動建立tmp、hdfs、hdfs/data、hdfs/name資料夾。
(3)配置/home/hadoop/hadoop-2.7.2/etc/hadoop目錄下的core-site.xml(ip設定成master的地址)
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://10.20.42.199:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/loongson/hadoop/tmp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131702</value>
</property>
</configuration>
(4)配置/home/loongson/hadoop-2.7.2/etc/hadoop目錄下的hdfs-site.xml(ip設定成master的地址)
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/loongson/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/loongson/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>10.20.42.199:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
(5)配置/home/loongson/hadoop-2.7.2/etc/hadoop目錄下的mapred-site.xml.template(ip設定成master的地址)
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>10.20.42.199:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>10.20.42.199:19888</value>
</property>
</configuration>
(6)配置/home/loongson/hadoop-2.7.2/etc/hadoop目錄下的yarn-site.xml(ip設定成master的地址)
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>10.20.42.199:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>10.20.42.199:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>10.20.42.199:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>10.20.42.199:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>10.20.42.199:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>768</value>
</property>
</configuration>
(7)修改位於/home/loongson/hadoop-2.7.2/etc/hadoop目錄hadoop-env.sh,yarn-env.sh中的JAVA_HOME等環境變數。
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.25-6.b17.rc16.fc21.loongson.mips64el
(8)配置/home/loongson/hadoop-2.7.2/etc/hadoop目錄下的slaves檔案，增加2個從slave節點，
10.20.42.10
10.20.42.22
(9)將上述配置好的Hadoop-2.7.2(位於master機器上)使用scp複製到各個slave節點對應位置上
scp -r /home/loongson/hadoop-2.7.2 10.20.42.10:/home/loongson
scp -r /home/loongson/hadoop-2.7.2 10.20.42.22:/home/loongson
(10)在Master伺服器啟動hadoop，從節點會自動啟動，進入/home/loongson/hadoop-2.7.2目錄
(1)關閉機器防火牆:service iptables stop (主從都設定)
(2)初始化node節點:bin/hdfs namenode -format
(3)啟動全部node: sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
16/09/02 08:49:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable
Starting namenodes on [hadoop-master-001]
hadoop-master-001: starting namenode, logging to /home/loongson/hadoop-2.7.2/logs/hadoop-root-namenode-
localhost.localdomain.out
10.20.42.22: starting datanode, logging to /home/loongson/hadoop-2.7.2/logs/hadoop-root-datanode-localhost.localdomain.out
10.20.42.22: /home/loongson/hadoop-2.7.2/bin/hdfs: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.25-6.b17.rc16.fc21.loongson.mips64el
10.20.42.22: /home/loongson/hadoop-2.7.2/bin/hdfs: line 304: /usr/lib/jvm/java-1.8.0-
openjdk-1.8.0.25-6.b17.rc16.fc21.loongson.mips64el/bin/java: 成功
10.20.42.10: starting datanode, logging to /home/loongson/hadoop-2.7.2/logs/hadoop-root-datanode-localhost.localdomain.out
Starting secondary namenodes [hadoop-master-001]
hadoop-master-001: secondarynamenode running as process 18418. Stop it first.
16/09/02 08:50:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
starting yarn daemons
resourcemanager running as process 16937. Stop it first.
10.20.42.10: starting nodemanager, logging to /home/loongson/hadoop-2.7.2/logs/yarn-root-nodemanager-localhost.localdomain.out
10.20.42.22: starting nodemanager, logging to /home/loongson/hadoop-2.7.2/logs/yarn-root-nodemanager-localhost.localdomain.out
10.20.42.22: /home/loongson/hadoop-2.7.2/bin/yarn: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.25-6.b17.rc16.fc21.loongson.mips64el
10.20.42.22: /home/loongson/hadoop-2.7.2/bin/yarn: line 333: /usr/lib/jvm/java-1.8.0-
openjdk-1.8.0.25-6.b17.rc16.fc21.loongson.mips64el/bin/java: 成功
(4)暫停全部節點的命令: sbin/stop-all.sh
(5)輸入jps命令: 如果從節點和主節點顯示類似如下,說明節點搭建成功
master:
32497 OServerMain
3506 SecondaryNameNode
3364 DataNode
5654 Jps
2582 OGremlinConsole
16937 ResourceManager
3263 NameNode
slaves:
21580 Jps
20622 DataNode
(11)從瀏覽器訪問: http://10.20.42.199:8088/或http://10.20.42.199:50070/ 檢視hadop執行情況。下面給出從瀏覽器開啟，看到的hadoop的執行情況截圖:
Hadoop執行預覽和概要資訊:
Hadoop執行情況:
七、下載成品
如果覺得上面的移植過程太複雜，筆者已經準備好了移植完的二進位制，可以直接下載執行：
http://www.loongnix.org/index.php/Apache_hadoop-2.7.2
八、總結
hadoop-2.7.2 在loongnix1.0 系統上正確完成原始碼編譯和搭建小叢集測試，可以作為開發者移植hadoop和進行叢集測試的示範過程。
