1、spark thriftserver報以下錯誤，其他諸如hive/sparksql等方式均正常 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriverActorSystem-akka.actor.default-dispatcher-379] shutting down ActorSystem [sparkDriverActorSystem] java.lang.OutOfMemoryError: Java heap space 原因：thriftserver的堆記憶體不足

解決辦法： 重啟thriftserver，並調大executor-memory記憶體（不能超過spark總剩餘記憶體，如超過，可調大spark-env.sh中的SPARK_WORKER_MEMORY引數，並重啟spark叢集。

start-thriftserver.sh --master spark://masterip:7077 --executor-memory 2g --total-executor-cores 4 --executor-cores 1 --hiveconf hive.server2.thrift.port=10050 --conf spark.dynamicAllocation.enabled=false 如果調大了executor的記憶體,依舊報此錯誤,仔細分析發現應該不是executor記憶體不足,而是driver記憶體不足,在standalone模式下預設給driver 1G記憶體,當我們提交應用啟動driver時,如果讀取資料太大,driver就可能報記憶體不足。
在spark-defaults.conf中調大driver引數
spark.driver.memory    2g
同時在spark-env.sh中同樣設定
export SPARK_DRIVER_MEMORY=2g
2、Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
JDK6新增錯誤型別。當GC為釋放很小空間佔用大量時間時丟擲。
在JVM中增加該選項 -XX:-UseGCOverheadLimit 關閉限制GC的執行時間（預設啟用 ）
在spark-defaults.conf中增加以下引數
spark.executor.extraJavaOptions -XX:-UseGCOverheadLimit
spark.driver.extraJavaOptions -XX:-UseGCOverheadLimit
3、spark 錯誤描述：

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
原因：

NN主從切換時,spark報上述錯誤,經分析spark-defaults.conf配置引數spark.eventLog.dir寫死了其中一個NN的IP,導致主從切換時，無法讀日誌。
另外，跟應用程式使用的checkpoint也有關係,首次啟動應用程式時,建立了一個新的sparkcontext,而該sparkcontext綁定了具體的NN ip,
往後每次程式重啟,由於應用程式碼【StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext _)】將從已有checkpoint目錄匯入checkpoint 資料來重新建立 StreamingContext 例項。
如果 checkpointDirectory 存在，那麼 context 將匯入 checkpoint 資料。如果目錄不存在，函式 functionToCreateContext 將被呼叫並建立新的 context。
故出現上述異常。

解決：

針對測試系統：

1、將某個NN固定IP改成nameservice對應的值
2、清空應用程式的checkpoint日誌
3、重啟應用後,切換NN,spark應用正常

4、獲取每次記憶體GC資訊

spark-defaults.conf中增加：

spark.executor.extraJavaOptions -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc.log

5、timeout,報錯：

17/10/18 17:33:46 WARN TaskSetManager: Lost task 1393.0 in stage 382.0 (TID 223626, test-ssps-s-04): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason:Executor heartbeat timed out after 173568 ms 17/10/18 17:34:02 WARN NettyRpcEndpointRef: Error sending message [message = KillExecutors(app-20171017115441-0012,List(8))] in 2 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout   at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)      網路或者gc引起,worker或executor沒有接收到executor或task的心跳反饋。      提高 spark.network.timeout 的值，根據情況改成300(5min)或更高。      預設為 120(120s),配置所有網路傳輸的延時             spark.network.timeout 300000

6、通過sparkthriftserver讀取lzo檔案報錯：

ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

   在spark-env.sh中增加如下配置：

  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/hadoop/hadoop-2.2.0/lib/native   export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/hadoop/hadoop-2.2.0/lib/native   export SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/*:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/tools/lib/*:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/*

  並分發到各節點

  重啟spark thrift server執行正常 7、spark worker中釋出executor頻繁報錯,陷入死迴圈（新建->失敗->新建->失敗.....） work日誌： Asked to launch executor app-20171024225907-0018/77347 for org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 worker.ExecutorRunner (Logging.scala:logInfo(58)) - Launch command:  "/home/hadoop/jdk1.7.0_09/bin/java" ...... Executor app-20171024225907-0018/77345 finished with state EXITED message Command exited with code 53 exitStatus 53 executor日誌： ERROR [main] storage.DiskBlockManager (Logging.scala:logError(95)) - Failed to create local dir in . Ignoring this directory. java.io.IOException: Failed to create a temp directory (under ) after 10 attempts! 再看配置檔案spark-env.sh： export SPARK_LOCAL_DIRS=/data/spark/data 設定了spark本地目錄，但機器上並沒有建立該目錄，所以引發錯誤。 ./drun "mkdir -p /data/spark/data" ./drun "chown -R hadoop:hadoop /data/spark" 建立後，重啟worker未再出現錯誤。

8、spark worker異常退出：

worker中日誌： 17/10/25 11:59:58 INFO worker.Worker: Master with url spark://10.10.10.82:7077 requested this worker to reconnect. 17/10/25 11:59:58 INFO worker.Worker: Not spawning another attempt to register with the master, since there is an attempt scheduled already. 17/10/25 11:59:59 INFO worker.Worker: Successfully registered with master spark://10.10.10.82:7077 17/10/25 11:59:59 INFO worker.Worker: Worker cleanup enabled; old application directories will be deleted in: /home/hadoop/spark-1.6.1-bin-2.2.0/work 17/10/25 12:00:00 INFO worker.Worker: Retrying connection to master (attempt # 1) 17/10/25 12:00:00 INFO worker.Worker: Master with url spark://10.10.10.82:7077 requested this worker to reconnect. 17/10/25 12:00:00 INFO worker.Worker: Master with url spark://10.10.10.82:7077 requested this worker to reconnect. 17/10/25 12:00:00 INFO worker.Worker: Not spawning another attempt to register with the master, since there is an attempt scheduled already. 17/10/25 12:00:00 INFO worker.Worker: Asked to launch executor app-20171024225907-0018/119773 for org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 17/10/25 12:00:00 INFO worker.Worker: Connecting to master 10.10.10.82:7077... 17/10/25 12:00:00 INFO spark.SecurityManager: Changing view acls to: hadoop 17/10/25 12:00:00 INFO worker.Worker: Connecting to master 10.10.10.82:7077... 17/10/25 12:00:00 INFO spark.SecurityManager: Changing modify acls to: hadoop 17/10/25 12:00:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop) 17/10/25 12:00:01 ERROR worker.Worker: Worker registration failed: Duplicate worker ID 17/10/25 12:00:01 INFO worker.ExecutorRunner: Launch command: "/home/hadoop/jdk1.7.0_09/bin/java" "-cp" "/home/hadoop/spark-1.6.1-bin-2.2.0/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/*:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/tools/lib/*:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/*:/home/hadoop/spark-1.6.1-bin-2.2.0/conf/:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/spark-assembly-1.6.1-hadoop2.2.0.jar:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/datanucleus-core-3.2.10.jar:/home/hadoop/hadoop-2.2.0/etc/hadoop/" "-Xms1024M" "-Xmx1024M" "-Dspark.driver.port=43546" "-XX:MaxPermSize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:43546" "--executor-id" "119773" "--hostname" "10.10.10.190" "--cores" "1" "--app-id" "app-20171024225907-0018" "--worker-url" "spark://[email protected]:55335" 17/10/25 12:00:02 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:02 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:02 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:03 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:05 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:05 INFO worker.ExecutorRunner: Killing process! 17/10/25 12:00:05 INFO util.ShutdownHookManager: Shutdown hook called 17/10/25 12:00:06 INFO util.ShutdownHookManager: Deleting directory /data/spark/data/spark-ab442bb5-62e6-4567-bc7c-8d00534ba1a3

近期，頻繁出現worker異常退出情況，從worker日誌看到，master要求重新連線並註冊，註冊後，worker繼續連線master,並反饋了一個錯誤資訊： ERROR worker.Worker: Worker registration failed: Duplicate worker ID 之後就突然殺掉所有Executor然後退出worker。 出現該問題的機器記憶體都是16g(其他32g節點,沒有出現worker退出問題)。 再檢視該類節點yarn的 配置，發現分配yarn的資源是15g,懷疑問題節點yarn和spark同處高峰期，導致spark分配不到資源退出。 為驗證猜想，檢視NM日誌，其時間發生點部分容器有自殺情況，但這個不能說明什麼，因為其他時間點也存有該問題。 為檢視問題時間點，節點中yarn資源使用情況，只能到active rm中檢視rm日誌，最後看到節點中在11:46:22到11:59:18， 問題節點yarn佔用資源一直在<memory:15000, vCores:6>。

如此，確實是該問題所致，將yarn資源調整至12g/10c,後續再繼續觀察。

8、spark （4個dd，11個nm）沒有利用非datanode節點上的executor問題

問題描述：跑大job時，總任務數上萬個，但是隻利用了其中4個dd上的executor，另外7臺沒有跑或者跑很少。

分析：spark driver進行任務分配時,會希望每個任務正好分配到要計算的資料節點上，避免網路傳輸。但當task因為其所在資料節點資源正好被分配完而沒機會再分配時，spark會等一段時間（由spark.locality.wait引數控制，預設3s）,超過該時間，就會選擇一個本地化差的級別進行計算。

解決辦法： 將spark.locality.wait設定為0，不等待，任務直接分配，需重啟服務生效

9、spark自帶引數不生效問題

spark thrift測試系統經常出問題，調整了driver記憶體引數，依舊報問題。

常見問題狀態：連線spark thrift無響應，一會提示OutOfMemoryError: Java heap space

後來發現設定的driver記憶體引數沒有生效，環境配置檔案spark-env.sh設定了SPARK_DAEMON_MEMORY=1024m，覆蓋了啟動引數設定的

--driver-memory 4G ,導致引數設定沒生效。