Spark On Yarn 詳細配置流程
1、系統與軟體準備
- 系統：centos7
- 軟體：
- hadoop-2.7.2.tar.gz，請勿選擇src版本，否則需要自行編譯
- jdk-7u79-linux-x64.tar.gz
- scala-2.10.3.tgz
- spark-1.6.1-bin-hadoop2.6.tgz，選擇hadooppre-built版本，否則需要安裝後編譯
- zookeeper-3.3.6.tar.gz
# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local
# cd /usr/local
# ln -s jdk1.7.0_45 jdk
# vim /etc/profile 加入以下內容
export JAVA_HOME=/usr/local/jdk export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin
# source /etc/profile
2. Scala安裝
# tar xvzf scala-2.10.3.tgz -C/usr/local
# cd /usr/local
# ln -s scala-2.10.3 scala
# vim /etc/profile 加入以下內容
export SCALA_HOME=/usr/local/scala export PATH=$PATH:$SCALA_HOME/bin
3. SSH免登入配置
#ssh-keygen
在node1下生成的金鑰對：id_rsa和id_rsa.pub，預設儲存在"~/.ssh"目錄下，包括兩個檔案，id_rsa和id_rsa.pub，分別為私鑰和公鑰
將公鑰寫入信任檔案中
#cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
然後修改authorized_keys檔案的許可權
#chmod 644
~/.ssh/authorized_keysnode1中的authorized_keys拷貝至其餘節點的~/.ssh目錄下，修改SSH配置檔案"/etc/ssh/sshd_config"的下列內容（這步可以省略）
RSAAuthentication yes # 啟用 RSA 認證
PubkeyAuthentication yes # 啟用公鑰私鑰配對認證方式
AuthorizedKeysFile .ssh/authorized_keys # 公鑰檔案路徑（和上面生成的檔案同）
#service sshd restart
重啟SSH服務
4.主機名設定
# vim /etc/hosts
10.25.248.1 node1 10.25.248.29 node2 10.25.248.17 node3
#vi /etc/sysconfig/network
HOSTNAME=node1/node2/node3
5. Zookeeper安裝
# mkdir –p /root/data/hadoop/zookeeper/{data,logs}
兩個資料夾都需要預先建立好，否則會執行時會報錯
# vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/root/data/hadoop/zookeeper/data clientPort=2181 server.1=10.25.248.1:2888:3888 server.2=10.25.248.29:2888:3888 server.3=10.25.248.17:2888:3888
接著在/root/data/hadoop/zookeeper/data下分別建立名為myid檔案，檔案內容為上述zoo.cfg中IP地址對應server.[number]中的number
node1 : echo 1 > /root/data/hadoop/zookeeper/data/myid
node2 : echo 2 > /root/data/hadoop/zookeeper/data/myid
node3 : echo 3 > /root/data/hadoop/zookeeper/data/myid
執行zkServer.sh start啟動zookeeper，相應zkServer.shstop退出，zkServer.sh status檢視狀態
分別在node1，node2，node3執行zkServer.sh start，然後通過zkServer.sh status檢視狀態，如果發現每個node當前狀態標記為follower或者leader，那麼測試通過
6. Hadoop（HDFS HA）叢集部署
# tar xvzf hadoop-2.2.0.tgz -C/usr/local
# cd /usr/local
# ln -s hadoop-2.2.0 hadoop
#vim /etc/profile
export HADOOP_HOME=/usr/local/hadoop export HADOOP_PID_DIR=/root/data/hadoop/pids export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native" export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
接下來有8個配置檔案需要修改，配置檔案均在$HADOOP_HOME/etc/hadoop/目錄下
- hadoop-env.sh, mapred-env.sh, yarn-env.sh中加入以下內容
export JAVA_HOME=/usr/local/jdk export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib export HADOOP_HOME=/usr/local/hadoop export HADOOP_PID_DIR=/root/data/hadoop/pids export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native" export HADOOP_PREFIX=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin2，core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://lab369</value> <!--此處不能有‘-’符號--> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/root/data/hadoop/storage/tmp</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>node1:2181,node2:2181,node3:2181</value> </property> <property> <name>ha.zookeeper.session-timeout.ms</name> <value>2000</value> </property> <property> <name>fs.trash.interval</name> <value>4320</value> </property> <property> <name>hadoop.http.staticuser.use</name> <value>root</value> </property> <property> <name>hadoop.proxyuser.hadoop.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hadoop.groups</name> <value>*</value> </property> <property> <name>hadoop.native.lib</name> <value>true</value> </property> </configuration>3， hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:/root/data/hadoop/storage/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/root/data/hadoop/storage/hdfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.blocksize</name> <value>67108864</value> </property> <property> <name>dfs.datanode.du.reserved</name> <value>10737418240</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.permissions</name> <value>true</value> </property> <property> <name>dfs.permissions.enabled</name> <value>true</value> </property> <property> <name>dfs.nameservices</name> <value>lab369</value> </property> <property> <name>dfs.ha.namenodes.lab369</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.lab369.nn1</name> <value>node1:8020</value> </property> <property> <name>dfs.namenode.rpc-address.lab369.nn2</name> <value>node2:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.lab369.nn1</name> <value>node1:53310</value> </property> <property> <name>dfs.namenode.servicerpc-address.lab369.nn2</name> <value>node2:53310</value> </property> <property> <name>dfs.namenode.http-address.lab369.nn1</name> <value><span style="color:#ff0000;">node1:<span style="font-family:Arial, Helvetica, sans-serif;">50070</span></span></value> <!-- 該處不建議佔掉8080埠，很多教程上都直接8080--> </property> <property> <name>dfs.namenode.http-address.lab369.nn2</name> <value><span style="color:#ff0000;">node2:50070</span></value> </property> <property> <name>dfs.datanode.http.address</name> <value><span style="color:#ff0000;">0.0.0.0:50070</span></value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://node1:8485;node2:8485;node3:8485/lab369</value> </property> <property> <name>dfs.client.failover.proxy.provider.lab369</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/root/data/hadoop/storage/hdfs/journal</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>ha.failover-controller.cli-check.rpc-timeout.ms</name> <value>60000</value> </property> <property> <name>ipc.client.connect.timeout</name> <value>60000</value> </property> <property> <name>dfs.image.transfer.bandwidthPerSec</name> <value>41943040</value> </property> <property> <name>dfs.namenode.accesstime.precision</name> <value>3600000</value> </property> <property> <name>dfs.datanode.max.transfer.threads</name> <value>4096</value> </property> </configuration>
4，mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>node1:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>node1:19888</value> </property> <property> <name>mapreduce.application.classpath</name> <value> /usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common/*, /usr/local/hadoop/share/hadoop/common/lib/*, /usr/local/hadoop/share/hadoop/hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, /usr/local/hadoop/share/hadoop/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, /usr/local/hadoop/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/* </value> </property> </configuration>
5，yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>node1:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>node1:8031</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>node1:8032</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>node1:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>node1:80</value> </property> <property> <name>yarn.nodemanager.hostname</name> <value>node3</value> <!-- 每個slave應該對應自己的hostName--> <description>the nodemanagers bind to this port</description> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${yarn.nodemanager.hostname}:80</value> </property> <property> <name>yarn.nodemanager.address</name> <value>${yarn.nodemanager.hostname}:8034</value> <description>the nodemanagers bind to this port</description> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>${hadoop.tmp.dir}/nodemanager/local</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>${hadoop.tmp.dir}/nodemanager/remote</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>${hadoop.tmp.dir}/nodemanager/logs</value> </property> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>604800</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>2</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>10240</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>256</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>40960</value> </property> <property> <name>yarn.scheduler.minimum-allocation-vcores</name> <value>1</value> </property> <property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>8</value> </property> </configuration>
6，slaves（此處如果超過一個節點，不要填主機名，要填IP）
10.25.248.17
接下來依次執行以下命令：
a) 在namenode1上執行，建立名稱空間
# hdfs zkfc -formatZK
b) 在對應的節點上啟動日誌程式journalnode
# cd /usr/local/hadoop && ./sbin/hadoop-daemon.sh start journalnode
c) 格式化主NameNode節點（node1）
# hdfs namenode -format
d) 啟動主NameNode節點
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
e) 格式備NameNode節點（node2）
# hdfs namenode -bootstrapStandby
f) 啟動備NameNode節點（node2）
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
g) 在兩個NameNode節點（node1、node2）上執行
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start zkfc
h) 啟動所有的DataNode節點（node3）
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start datanode
i) 啟動Yarn（node1）
# cd /usr/local/hadoop &&sbin/start-yarn.sh
7，Spark安裝與配置
# tar xvzf spark-0.9.0-incubating.tgz-C/usr/local
# cd /usr/local
# ln -s spark-0.9.0-incubating spark
# vim /etc/profile
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
# source /etc/profile
# cd /usr/local/spark/conf
# mkdir -p /data/spark/tmp
接下來，配置兩個配置檔案，檔案在SPARK_HOME/conf/目錄下
1，spark-env.sh
2，slaves（多個節點不能是主機名）export JAVA_HOME=/usr/local/jdk export SCALA_HOME=/usr/local/scala export HADOOP_HOME=/usr/local/hadoop export SPARK_MASTER_PORT=7077 export SPARK_MASTER_WEBUI_PORT=7070 export SPARK_WORKER_CORES=2 export SPARK_WORKER_MEMORY=1024m export SPARK_WORKER_INSTANCES=2 export SPARK_LOCAL_DIR="/root/data/spark/tmp" export SPARK_JAVA_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK_LOCAL_DIR -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:$SPARK_HOME/logs/gc.log -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=60"
<span style="font-size: 13.3333px;">10.25.248.29</span> <span style="font-size: 13.3333px;">10.25.248.17</span>
分別進入每個節點的/usr/local/spark/sbin目錄下，主節點執行#./start-master.sh，子節點執行#./start-slaves.sh，或者在主節點直接執行sbin/start-all.sh
8，測試流程
1，程序執行情況測試
在每個節點執行#jps指令，若輸出結果為以下內容，則測試通過，否則進入/usr/local/hadoop/logs或者/usr/local/spark/logs目錄下檢視log檔案進行檢查
主namenode
從namenode
datanode節點
其中Master和Worker是Spark相應程序
Hmaster和HregionServer為Hbase相應程序(這裡不需要)
DFSZKFailoverController為Zookeeper相應程序
ResourceManager和NodeManager為Yarn相應程序
其餘為Hadoop相應程序
2，HDFS測試
在任意節點下執行
#hadoopfs –mkdir /test
#hadoopfs –ls /
#hadoopfs –put /test/test.txt
若不報錯，則說明測試通過
3，Spark測試
1) Spark本地模式測試（Spark Standalone）
#run-exampleorg.apache.spark.examples.SparkPi 100
#spark-submit--class org.apache.spark.examples.JavaWordCount --master spark://node1:6066 --deploy-modecluster /usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar ./test.txt
#spark-submit--class org.apache.spark.examples.JavaWordCount --master spark://node1:6066 --deploy-modecluster /usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://[hdfsnamespace]/test/test.txt
2) Spark叢集模式測試（Spark on Yarn）
#spark-submit --class org.apache.spark.examples.JavaWordCount--master yarn --deploy-mode client/usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://[hdfsnamespace]/test/test.txt
#spark-submit --classorg.apache.spark.examples.JavaWordCount --master yarn --deploy-mode cluster/usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://[hdfsnamespace]/test/test.txt