pyspark on yarn 叢集方式提交計算的驅動問題
阿新 • • 發佈：2020-12-16
技術標籤：spark
spark-submit \
--master yarn \
--verbose \
--deploy-mode cluster \
--num-executors 1 \
--executor-memory 1G \
--executor-cores 1 \
test.py -table 'ods.tabe' -fields 'dt' -prov hl -dt 20201122
在spark-default.conf
配置
spark.pyspark.python python3 spark.driver.extraClassPath /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar spark.driver.extraLibraryPath /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar
或者在提交的時候配置
--conf spark.pyspark.python=python3 \
--driver-class-path /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar \
--driver-library-path /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar \
否則會報驅動找不到問題