1. 程式人生 > >Spark job 部署模式

Spark job 部署模式

mys har clu exe 技術 fail emp pre exec


  Spark job 的部署有兩種模式,Client && Cluster
  spark-submit .. --deploy-mode client | cluster

  【上傳 Jar 包】

[centos@s101 ~]$ hdfs dfs -put myspark.jar data

  【Client】
  默認值,Driver 運行在 Client 端主機上。

spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 myspark.jar /user/centos/temptags.txt

  【cluster】
  Driver 運行在某個 Worker 節點上。客戶端值負責提交 job。

spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 --deploy-mode cluster hdfs://mycluster/user/centos/data/myspark.jar /user/centos/temptags.txt

[centos@s101 ~]$ xcall.sh jps
==================== s101 jps ===================
2981
Master 2568 NameNode 2889 DFSZKFailoverController 3915 Jps ==================== s102 jps =================== 2961 CoarseGrainedExecutorBackend 2450 Worker 2325 JournalNode 2246 DataNode 2187 QuorumPeerMain 3005 Jps ==================== s103 jps =================== 2457 Worker 2331 JournalNode 2188 QuorumPeerMain
3292 CoarseGrainedExecutorBackend 2253 DataNode 3310 Jps ==================== s104 jps =================== 2193 QuorumPeerMain 2981 DriverWrapper 3094 Jps 2455 Worker 2328 JournalNode 2252 DataNode 3038 CoarseGrainedExecutorBackend

[centos@s105 /soft/spark-2.1.0-bin-hadoop2.7/bin]$ ./spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 --deploy-mode cluster hdfs://s101/user/centos/data/myspark.jar /user/centos/temptags.txt

技術分享圖片


Spark job 部署模式