升級版本說明：從spark2.1的maven配置pom.xml中可以看出版本！！

  <dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_2.11</artifactId>
   <version>2.1.0</version>
  </dependency>

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.11</artifactId>
  <version>2.1.0</version>
   </dependency> 

<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-streaming_2.11</artifactId>
   <version>2.1.0</version>
  </dependency>

 <dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-streaming-kafka_2.11</artifactId>
   <version>1.6.3</version>
  </dependency>

<dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-actors</artifactId>
      <version>2.11.8</version>
      <scope>compile</scope>
    </dependency>

問題一：

More than one scala library found in the build path (/cache_V0.1/lib/spark-assembly-1.6.0-cdh5.8.3-hadoop2.6.0-cdh5.8.3.jar,D:/softinstall/eclipse/plugins/org.scala-lang.scala-library_2.11.8.v20160304-115712-1706a37eb8.jar/cache_V0.1/lib/spark-assembly-1.6.0-cdh5.8.3-hadoop2.6.0-cdh5.8.3.jar,).At least one has an incompatible version. Please update the project build path so it contains only one compatible scala library. cache_V0.1   Unknown Scala Classpath Problem

問題一分析原因：因為和spark2.1有衝突了

問題一解決方法：

1）把eclipse中的build ---->add jar的包spark-assembly-1.6.0-cdh5.8.3-hadoop2.6.0-cdh5.8.3.jar刪除掉，即不從本地載入這個jar包

2）eclipse選中scala的maven專案---》右鍵--》scala---->set scala installe..... ----->選擇2.11.8版本的scala

問題二：

[ERROR] scalac error: bad option: '-make:transitive'

[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.0:compile (default) on project cache_chinalife_amis: 
wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]

問題二分析原因：spark2.1不支援'-make:transitive'

問題二解決方法：註釋掉pom.xmi檔案中的這一行即可：<!-- <parameter value="-make:transitive"/>  -->

問題三：打包完成後，放伺服器跑，報錯：main方法找不到：

java.lang.ClassNotFoundException: StartCBPS8
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:340)
at org.apache.spark.util.Utils$.classForName(Utils.scala:176)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

問題三分析原因：jdk版本不一致導致，伺服器的jdk版本為1.8，本地打包的為1.7，

問題三解決方法：修改本地jdk八版為1.8後重新打包即可

問題四：伺服器上執行時，包類org.apache.spark.scheduler.SparkListenerInterface無法找到

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/scheduler/SparkListenerInterface
at StartCBPS8.main(StartCBPS8.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.scheduler.SparkListenerInterface
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more

問題四問題分析：由於服務上提交任務時候，使用的命令為spark-submit，而安裝的spark為2.1，因此是執行的命令不對，需要使用spark2-submit命令提交

問題四解決方法：

 nohup /usr/bin/spark-submit --class StartCBPS8  --master yarn --deploy-mode client  --conf spark.port.maxRetries=50  --jars $BASEDIR/dom4j-1.3.jar,$BASEDIR/scala-actors-2.11.8.jar,$BASEDIR/ojdbc5.jar,$BASEDIR/jedis-2.7.3.jar,$BASEDIR/SDK.jar,$BASEDIR/fastjson-1.1.33.jar  --executor-memory 1g --executor-cores 1 ../bin/cache_chinalife_amis-0.0.1.jar $PROVICE $2 $3 > ../logs/cache-cbps8_$PROVICE-start.out 2>&1 &

改為:

 nohup /usr/bin/spark2-submit --class StartCBPS8  --master yarn --deploy-mode client  --conf spark.port.maxRetries=50  --jars $BASEDIR/dom4j-1.3.jar,$BASEDIR/scala-actors-2.11.8.jar,$BASEDIR/ojdbc5.jar,$BASEDIR/jedis-2.7.3.jar,$BASEDIR/SDK.jar,$BASEDIR/fastjson-1.1.33.jar  --executor-memory 1g --executor-cores 1 ../bin/cache_chinalife_amis-0.0.1.jar $PROVICE $2 $3 > ../logs/cache-cbps8_$PROVICE-start.out 2>&1 &

問題五：spark命令提交引數不正確

問題五分析原因：spark2-submit和spark-submit提交命令引數不相容

問題五解決方法：修改為spark2-submit相容的提交命令方式，去掉不支援的2個命令： --driver-memory 128m  --driver-java-options "-Xms64m -Xmx128m -XX:PermSize=64M -XX:MaxPermSize=128M" 

nohup /opt/cloudera/parcels/CDH/lib/spark/bin/spark-submit --class StartCBPS8  --master yarn --deploy-mode client --driver-memory 128m  --driver-java-options "-Xms64m -Xmx128m -XX:PermSize=64M -XX:MaxPermSize=128M"  --conf spark.port.maxRetries=40   --jars $BASEDIR/dom4j-1.3.jar,$BASEDIR/scala-actors-2.10.6.jar,$BASEDIR/ojdbc5.jar,$BASEDIR/jedis-2.7.3.jar,$BASEDIR/SDK.jar,$BASEDIR/fastjson-1.1.33.jar  --executor-memory 1g --executor-cores 1 ../bin/cache_chinalife_amis-0.0.1.jar $PROVICE $2 $3  > ../logs/cache-cbps8_$PROVICE-start.out 2>&1 &

改為：

nohup /usr/bin/spark2-submit --class StartCBPS8  --master yarn --deploy-mode client  --conf spark.port.maxRetries=50  --jars $BASEDIR/dom4j-1.3.jar,$BASEDIR/scala-actors-2.11.8.jar,$BASEDIR/ojdbc5.jar,$BASEDIR/jedis-2.7.3.jar,$BASEDIR/SDK.jar,$BASEDIR/fastjson-1.1.33.jar  --executor-memory 1g --executor-cores 1 ../bin/cache_chinalife_amis-0.0.1.jar $PROVICE $2 $3 > ../logs/cache-cbps8_$PROVICE-start.out 2>&1 &