1. 程式人生 > >spark 異常解決:A master URL must be set in your configuration

spark 異常解決:A master URL must be set in your configuration

為了省去每次都建立spark物件,寫了一個基礎類SparkSessionBase,沒想到叢集丟擲異常:A master URL must be set in your configuration。原因就在於spark物件在main函式之外建立,driver就不會把這個spark物件分發到其他work節點上。

class SparkSessionBase {
    protected val spark:SparkSession =SparkSession.builder().enableHiveSupport().getOrCreate();
}

object ImportHiveData2Kafka
extends SparkSessionBase {
def main(args: Array[String]): Unit = { val topic = args(0) val dataType = args(1) val dataSourceType = args(2) var infos: Dataset[String] = null dataSourceType match { case "parquet" => {infos = spark.read.load(args(3
)).toJSON} case "hive" => {infos = spark.sql("select * from " + args(3) + " where dt='" + args(4) +"'").toJSON} case _ => null } infos.foreachPartition(info => importToKafka(topic, dataType, info.toList)) spark.close() } }

spark異常日誌