Spark job 部署模式
阿新 • • 發佈:2018-10-13
mys har clu exe 技術 fail emp pre exec
Spark job 的部署有兩種模式,Client && Cluster
spark-submit .. --deploy-mode client | cluster
【上傳 Jar 包】
[centos@s101 ~]$ hdfs dfs -put myspark.jar data
【Client】
默認值,Driver 運行在 Client 端主機上。
spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 myspark.jar /user/centos/temptags.txt
【cluster】
Driver 運行在某個 Worker 節點上。客戶端值負責提交 job。
spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 --deploy-mode cluster hdfs://mycluster/user/centos/data/myspark.jar /user/centos/temptags.txt
[centos@s101 ~]$ xcall.sh jps ==================== s101 jps =================== 2981Master 2568 NameNode 2889 DFSZKFailoverController 3915 Jps ==================== s102 jps =================== 2961 CoarseGrainedExecutorBackend 2450 Worker 2325 JournalNode 2246 DataNode 2187 QuorumPeerMain 3005 Jps ==================== s103 jps =================== 2457 Worker 2331 JournalNode 2188 QuorumPeerMain3292 CoarseGrainedExecutorBackend 2253 DataNode 3310 Jps ==================== s104 jps =================== 2193 QuorumPeerMain 2981 DriverWrapper 3094 Jps 2455 Worker 2328 JournalNode 2252 DataNode 3038 CoarseGrainedExecutorBackend
[centos@s105 /soft/spark-2.1.0-bin-hadoop2.7/bin]$ ./spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 --deploy-mode cluster hdfs://s101/user/centos/data/myspark.jar /user/centos/temptags.txt
Spark job 部署模式