一、Spark遠端除錯配置：

#除錯Master，在master節點的spark-env.sh中新增SPARK_MASTER_OPTS變數
export SPARK_MASTER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10000"
#啟動Master
sbin/start-master.sh


#除錯Worker，在worker節點的spark-env.sh中新增SPARK_WORKER_OPTS變數
export SPARK_WORKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10001"
 

#啟動Worker
sbin/start-slave.sh 1 spark://hadoop1:7077

#除錯spark-submit + app
bin/spark-submit --class cn.itcast.spark.WordCount --master spark://hadoop1:7077 --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" /root/wc.jar hdfs://mycluster/wordcount/input/2.txt hdfs://mycluster/out2 

#除錯spark-submit + app + executor
 

bin/spark-submit --class cn.itcast.spark.WordCount --master spark://hadoop1:7077 --conf "spark.executor.extraJavaOptions=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10003" --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" /root/wc.jar hdfs://mycluster/wordcount/input/2.
 
txt hdfs://mycluster/out2   

二、編寫程式進行除錯

接著：