1. 程式人生 > >Scala進階之路-Spark獨立模式集群部署

Scala進階之路-Spark獨立模式集群部署

cas pro 多人 tro splay park def 你在 原來

          Scala進階之路-Spark獨立模式集群部署

                             作者:尹正傑

版權聲明:原創作品,謝絕轉載!否則將追究法律責任。

  我們知道Hadoop解決了大數據的存儲和計算,存儲使用HDFS分布式文件系統存儲,而計算采用MapReduce框架進行計算,當你在學習MapReduce的操作時,尤其是Hive的時候(因為Hive底層其實仍然調用的MapReduce)是不是覺得MapReduce運行的特別慢?因此目前很多人都轉型學習Spark,今天我們就一起學習部署Spark集群吧。

一.準備環境

  如果你的服務器還麽沒有部署Hadoop集群,可以參考我之前寫的關於部署Hadoop高可用的筆記:https://www.cnblogs.com/yinzhengjie/p/9154265.html

1>.啟動HDFS分布式文件系統

技術分享圖片
[yinzhengjie@s101 download]$ more `which xzk.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:[email protected]

#判斷用戶是否傳參
if [ $# -ne 1 ];then
    echo "無效參數,用法為: $0  {start|stop|restart|status}"
    exit
fi

#獲取用戶輸入的命令
cmd=$1

#定義函數功能
function zookeeperManger(){
    
case $cmd in start) echo "啟動服務" remoteExecution start ;; stop) echo "停止服務" remoteExecution stop ;; restart) echo "重啟服務" remoteExecution restart ;; status) echo "查看狀態" remoteExecution status ;;
*) echo "無效參數,用法為: $0 {start|stop|restart|status}" ;; esac } #定義執行的命令 function remoteExecution(){ for (( i=102 ; i<=104 ; i++ )) ; do tput setaf 2 echo ========== s$i zkServer.sh $1 ================ tput setaf 9 ssh s$i "source /etc/profile ; zkServer.sh $1" done } #調用函數 zookeeperManger [yinzhengjie@s101 download]$
[yinzhengjie@s101 download]$ more `which xzk.sh` 技術分享圖片
[yinzhengjie@s101 download]$ more `which xcall.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:[email protected]


#判斷用戶是否傳參
if [ $# -lt 1 ];then
        echo "請輸入參數"
        exit
fi

#獲取用戶輸入的命令
cmd=$@

for (( i=101;i<=105;i++ ))
do
        #使終端變綠色 
        tput setaf 2
        echo ============= s$i $cmd ============
        #使終端變回原來的顏色,即白灰色
        tput setaf 7
        #遠程執行命令
        ssh s$i $cmd
        #判斷命令是否執行成功
        if [ $? == 0 ];then
                echo "命令執行成功"
        fi
done
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ more `which xcall.sh` 技術分享圖片
[yinzhengjie@s101 download]$ more `which xrsync.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:[email protected]

#判斷用戶是否傳參
if [ $# -lt 1 ];then
    echo "請輸入參數";
    exit
fi


#獲取文件路徑
file=$@

#獲取子路徑
filename=`basename $file`

#獲取父路徑
dirpath=`dirname $file`

#獲取完整路徑
cd $dirpath
fullpath=`pwd -P`

#同步文件到DataNode
for (( i=102;i<=105;i++ ))
do
    #使終端變綠色 
    tput setaf 2
    echo =========== s$i %file ===========
    #使終端變回原來的顏色,即白灰色
    tput setaf 7
    #遠程執行命令
    rsync -lr $filename `whoami`@s$i:$fullpath
    #判斷命令是否執行成功
    if [ $? == 0 ];then
        echo "命令執行成功"
    fi
done
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ more `which xrsync.sh` 技術分享圖片
[yinzhengjie@s101 download]$ xzk.sh start
啟動服務
========== s102 zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /soft/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== s103 zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /soft/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== s104 zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /soft/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ xcall.sh jps
============= s101 jps ============
2603 Jps
命令執行成功
============= s102 jps ============
2316 Jps
2287 QuorumPeerMain
命令執行成功
============= s103 jps ============
2284 QuorumPeerMain
2319 Jps
命令執行成功
============= s104 jps ============
2305 Jps
2276 QuorumPeerMain
命令執行成功
============= s105 jps ============
2201 Jps
命令執行成功
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ xzk.sh start

技術分享圖片
[yinzhengjie@s101 download]$ start-dfs.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Starting namenodes on [s101 s105]
s101: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s101.out
s105: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s105.out
s103: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s103.out
s104: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s104.out
s102: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s102.out
Starting journal nodes [s102 s103 s104]
s104: starting journalnode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-journalnode-s104.out
s102: starting journalnode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-journalnode-s102.out
s103: starting journalnode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-journalnode-s103.out
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Starting ZK Failover Controllers on NN hosts [s101 s105]
s101: starting zkfc, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-zkfc-s101.out
s105: starting zkfc, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-zkfc-s105.out
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ xcall.sh jps
============= s101 jps ============
7909 Jps
7526 NameNode
7830 DFSZKFailoverController
命令執行成功
============= s102 jps ============
2817 QuorumPeerMain
4340 JournalNode
4412 Jps
4255 DataNode
命令執行成功
============= s103 jps ============
4256 JournalNode
2721 QuorumPeerMain
4328 Jps
4171 DataNode
命令執行成功
============= s104 jps ============
2707 QuorumPeerMain
4308 Jps
4151 DataNode
4236 JournalNode
命令執行成功
============= s105 jps ============
4388 DFSZKFailoverController
4284 NameNode
4446 Jps
命令執行成功
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ start-dfs.sh

2>.上傳測試數據到HDFS集群中

技術分享圖片
[yinzhengjie@s101 download]$ cat temp 
0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991902010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9+00721+99999102001ADDGF104991999999999999999999
0029029070999991903010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9-00941+99999102001ADDGF108991999999999999999999
0029029070999991904010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9-00611+99999101831ADDGF108991999999999999999999
0029029070999991905010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00561+99999101761ADDGF108991999999999999999999
0029029070999991906010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991907010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9-00671+99999101701ADDGF106991999999999999999999
0029029070999991908010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991909010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9-00721+99999102001ADDGF104991999999999999999999
0029029070999991910010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9+00941+99999102001ADDGF108991999999999999999999
0029029070999991911010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9-00611+99999101831ADDGF108991999999999999999999
0029029070999991912010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00561+99999101761ADDGF108991999999999999999999
0029029070999991913010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991914010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991915010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9-00721+99999102001ADDGF104991999999999999999999
0029029070999991916010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9+00941+99999102001ADDGF108991999999999999999999
0029029070999991917010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991918010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00561+99999101761ADDGF108991999999999999999999
0029029070999991919010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00281+99999101751ADDGF108991999999999999999999
0029029070999991920010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991902010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9+00721+99999102001ADDGF104991999999999999999999
0029029070999991903010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9-00941+99999102001ADDGF108991999999999999999999
0029029070999991904010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991905010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00561+99999101761ADDGF108991999999999999999999
0029029070999991906010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991907010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9-00671+99999101701ADDGF106991999999999999999999
0029029070999991908010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9+00781+99999102001ADDGF108991999999999999999999
0029029070999991909010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9-00721+99999102001ADDGF104991999999999999999999
0029029070999991910010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9+00941+99999102001ADDGF108991999999999999999999
0029029070999991911010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991912010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00561+99999101761ADDGF108991999999999999999999
0029029070999991913010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991914010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991915010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9+00721+99999102001ADDGF104991999999999999999999
0029029070999991916010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9-00941+99999102001ADDGF108991999999999999999999
0029029070999991917010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991918010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00561+99999101761ADDGF108991999999999999999999
0029029070999991919010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991920010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ cat temp 技術分享圖片
[yinzhengjie@s101 download]$ hdfs dfs -mkdir -p  /home/yinzhengjie/data
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ hdfs dfs -ls -R /
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x   - yinzhengjie supergroup          0 2018-07-27 03:44 /home
drwxr-xr-x   - yinzhengjie supergroup          0 2018-07-27 03:44 /home/yinzhengjie
drwxr-xr-x   - yinzhengjie supergroup          0 2018-07-27 03:44 /home/yinzhengjie/data
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ hdfs dfs -mkdir -p /home/yinzhengjie/data

3>.

二.部署Spark集群

三.在Spark集群中執行代碼

Scala進階之路-Spark獨立模式集群部署