坑一：

使用idea使用

setMaster("local[4]")

模式執行寫好的程式碼沒有問題，一旦使用

setMaster("spark://192.168.160.112:8090")

這種方式就會報classnotfound的錯誤,如下圖，下例中使用了elasticsearch的驅動

[[email protected] estest_jar]# spark-submit --class EStest --master spark://192.168.160.135:7077 --name estest estest.jar

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

17/12/15 16:06:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/12/15 16:06:28 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.

17/12/15 16:06:31 INFO Version: Elasticsearch Hadoop v6.0.0 [8b59a8f82d]

17/12/15 16:06:31 INFO ScalaEsRDD: Reading from [bank2]

[Stage 0:>                                                          (0 + 0) / 5]17/12/15 16:06:31 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.160.135, executor 0): java.lang.ClassNotFoundException: org.elasticsearch.spark.rdd.EsPartition

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

解決方案分為2種：

1.在寫程式碼的時候,直接指明使用jar檔案的路徑

var jars=Seq("/home/javajars/elasticsearch-spark-20_2.11-6.0.0.jar","/home/javajars/postgresql-42.1.4.jar")
val conf = new SparkConf().setAppName("estest").setMaster("spark://192.168.160.135:7077").setJars(jars)
conf.set("es.nodes", "192.168.160.135")
conf.set("es.port", "9200")

2.通過分析spark-worker的log發現，spark工作時只使用sparkhome/jars下的jar檔案

log片段如下：紅色部分是spark工作時使用的jar檔案

17/12/15 00:06:27 INFO ExecutorRunner: Launch command: "/root/Downloads/jdk/bin/java" "-cp" "/spark/conf/:/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=33208" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:33208" "--executor-id" "0" "--hostname" "192.168.160.135" "--cores" "2" "--app-id" "app-20171215000627-0012" "--worker-url" "spark://[email protected]:33341"

只要將所需的jar檔案拷貝到sparkhome/jars下即可，叢集模式我也是在每一個slave的sparkhome/jars下添加了jar檔案

坑二：

在提交打好的jar檔案時報錯找不到--class引數指定的類，報錯如下：

[[email protected] estest_jar]# spark-submit --class EStest --master spark://192.168.160.135:7077 --name estest estest.jar 

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

17/12/14 21:28:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/12/14 21:28:46 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.

17/12/14 21:28:47 INFO Version: Elasticsearch Hadoop v6.0.0 [8b59a8f82d]

17/12/14 21:28:47 INFO ScalaEsRDD: Reading from [bank2]

17/12/14 21:28:49 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.160.135, executor 0): java.lang.ClassNotFoundException: EStest$$anonfun$1

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

解決方法和坑一類似，一個是吧你打好的jar檔案也包含在程式碼裡，紅色部分是我自己打的jar檔案，包含在自己的程式碼裡。。。。。（自己要證明自己是自己的感覺）

var jars=Seq("/root/Downloads/jar/elasticsearch-spark-20_2.11-6.0.0.jar","/root/IdeaProjects/EStest/out/artifacts/estest_jar/estest.jar")
val conf = new SparkConf().setAppName("estest").setMaster("spark://192.168.160.135:7077")
conf.set("es.nodes", "192.168.160.135")
conf.set("es.port", "9200")

還有一種 估計直接把jar檔案打到sparkhome/jars下估計也可行。 坑三： idea打jar包 首先打jar包的方式最好是不包含引用的其他jar檔案,選擇copy to output direc......那個選項。