1. 程式人生 > >spark job提交:spark-submit

spark job提交:spark-submit

一般化的引數:

spark-submit \
 --supervise #driver失敗時重啟 \
--name wordcount   \
--master yarn-cluster \

--num-executors 100 \
--executor-memory 6G \
--executor-cores 4 \
--driver-memory 8G \

--conf spark.default.parallelism=1000 \
--conf spark.storage.memoryFraction=0.5 \
--conf spark.shuffle.memoryFraction=0.3  \

--class a.HelloWord \
/home/wang/test/mywork.jar