spark 異常解決:A master URL must be set in your configuration
阿新 • • 發佈:2018-12-09
為了省去每次都建立spark物件,寫了一個基礎類SparkSessionBase,沒想到叢集丟擲異常:A master URL must be set in your configuration。原因就在於spark物件在main函式之外建立,driver就不會把這個spark物件分發到其他work節點上。
class SparkSessionBase {
protected val spark:SparkSession =SparkSession.builder().enableHiveSupport().getOrCreate();
}
object ImportHiveData2Kafka extends SparkSessionBase {
def main(args: Array[String]): Unit = {
val topic = args(0)
val dataType = args(1)
val dataSourceType = args(2)
var infos: Dataset[String] = null
dataSourceType match {
case "parquet" => {infos = spark.read.load(args(3 )).toJSON}
case "hive" => {infos = spark.sql("select * from " + args(3) + " where dt='" + args(4) +"'").toJSON}
case _ => null
}
infos.foreachPartition(info => importToKafka(topic, dataType, info.toList))
spark.close()
}
}