spark問題與排查

阿新 • • 發佈：2019-01-12

1、spark thriftserver報以下錯誤，其他諸如hive/sparksql等方式均正常
ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriverActorSystem-akka.actor.default-dispatcher-379] shutting down ActorSystem [sparkDriverActorSystem]
java.lang.OutOfMemoryError: Java heap space 原因：thriftserver的堆記憶體不足

解決辦法：重啟thriftserver，並調大executor-memory記憶體（不能超過spark總剩餘記憶體，如超過，可調大spark-env.sh中的SPARK_WORKER_MEMORY引數，並重啟spark叢集。

start-thriftserver.sh --master spark://masterip:7077 --executor-memory 2g --total-executor-cores 4 --executor-cores 1 --hiveconf hive.server2.thrift.port=10050 --conf spark.dynamicAllocation.enabled=false 如果調大了executor的記憶體,依舊報此錯誤,仔細分析發現應該不是executor記憶體不足,而是driver記憶體不足,在standalone模式下預設給driver 1G記憶體,當我們提交應用啟動driver時,如果讀取資料太大,driver就可能報記憶體不足。
在spark-defaults.conf中調大driver引數
spark.driver.memory 2g
同時在spark-env.sh中同樣設定
export SPARK_DRIVER_MEMORY=2g

2、Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
JDK6新增錯誤型別。當GC為釋放很小空間佔用大量時間時丟擲。
在JVM中增加該選項 -XX:-UseGCOverheadLimit 關閉限制GC的執行時間（預設啟用）
在spark-defaults.conf中增加以下引數
spark.executor.extraJavaOptions -XX:-UseGCOverheadLimit
spark.driver.extraJavaOptions -XX:-UseGCOverheadLimit
3、spark 錯誤描述：

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
原因：

NN主從切換時,spark報上述錯誤,經分析spark-defaults.conf配置引數spark.eventLog.dir寫死了其中一個NN的IP,導致主從切換時，無法讀日誌。
另外，跟應用程式使用的checkpoint也有關係,首次啟動應用程式時,建立了一個新的sparkcontext,而該sparkcontext綁定了具體的NN ip,
往後每次程式重啟,由於應用程式碼【StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext _)】將從已有checkpoint目錄匯入checkpoint 資料來重新建立 StreamingContext 例項。
如果 checkpointDirectory 存在，那麼 context 將匯入 checkpoint 資料。如果目錄不存在，函式 functionToCreateContext 將被呼叫並建立新的 context。
故出現上述異常。

解決：

針對測試系統：
1、將某個NN固定IP改成nameservice對應的值
2、清空應用程式的checkpoint日誌
3、重啟應用後,切換NN,spark應用正常

4、獲取每次記憶體GC資訊

spark-defaults.conf中增加：
spark.executor.extraJavaOptions -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc.log

5、timeout,報錯：

17/10/18 17:33:46 WARN TaskSetManager: Lost task 1393.0 in stage 382.0 (TID 223626, test-ssps-s-04): ExecutorLostFailure (executor 0 exited caused by one of the running tasks)
Reason:Executor heartbeat timed out after 173568 ms
17/10/18 17:34:02 WARN NettyRpcEndpointRef: Error sending message [message = KillExecutors(app-20171017115441-0012,List(8))] in 2 attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)

     網路或者gc引起,worker或executor沒有接收到executor或task的心跳反饋。
     提高 spark.network.timeout 的值，根據情況改成300(5min)或更高。
   預設為 120(120s),配置所有網路傳輸的延時
   spark.network.timeout 300000

6、通過sparkthriftserver讀取lzo檔案報錯：

ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

在spark-env.sh中增加如下配置：

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/hadoop/hadoop-2.2.0/lib/native
export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/hadoop/hadoop-2.2.0/lib/native
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/*:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/tools/lib/*:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/*

並分發到各節點

重啟spark thrift server執行正常

7、spark worker中釋出executor頻繁報錯,陷入死迴圈（新建->失敗->新建->失敗.....）
work日誌：
Asked to launch executor app-20171024225907-0018/77347 for org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
worker.ExecutorRunner (Logging.scala:logInfo(58)) - Launch command: "/home/hadoop/jdk1.7.0_09/bin/java" ......
Executor app-20171024225907-0018/77345 finished with state EXITED message Command exited with code 53 exitStatus 53
executor日誌：
ERROR [main] storage.DiskBlockManager (Logging.scala:logError(95)) - Failed to create local dir in . Ignoring this directory.
java.io.IOException: Failed to create a temp directory (under ) after 10 attempts!
再看配置檔案spark-env.sh：
export SPARK_LOCAL_DIRS=/data/spark/data
設定了spark本地目錄，但機器上並沒有建立該目錄，所以引發錯誤。
./drun "mkdir -p /data/spark/data"
./drun "chown -R hadoop:hadoop /data/spark"
建立後，重啟worker未再出現錯誤。

8、spark worker異常退出：

worker中日誌：
17/10/25 11:59:58 INFO worker.Worker: Master with url spark://10.10.10.82:7077 requested this worker to reconnect.
17/10/25 11:59:58 INFO worker.Worker: Not spawning another attempt to register with the master, since there is an attempt scheduled already.
17/10/25 11:59:59 INFO worker.Worker: Successfully registered with master spark://10.10.10.82:7077
17/10/25 11:59:59 INFO worker.Worker: Worker cleanup enabled; old application directories will be deleted in: /home/hadoop/spark-1.6.1-bin-2.2.0/work
17/10/25 12:00:00 INFO worker.Worker: Retrying connection to master (attempt # 1)
17/10/25 12:00:00 INFO worker.Worker: Master with url spark://10.10.10.82:7077 requested this worker to reconnect.
17/10/25 12:00:00 INFO worker.Worker: Master with url spark://10.10.10.82:7077 requested this worker to reconnect.
17/10/25 12:00:00 INFO worker.Worker: Not spawning another attempt to register with the master, since there is an attempt scheduled already.
17/10/25 12:00:00 INFO worker.Worker: Asked to launch executor app-20171024225907-0018/119773 for org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
17/10/25 12:00:00 INFO worker.Worker: Connecting to master 10.10.10.82:7077...
17/10/25 12:00:00 INFO spark.SecurityManager: Changing view acls to: hadoop
17/10/25 12:00:00 INFO worker.Worker: Connecting to master 10.10.10.82:7077...
17/10/25 12:00:00 INFO spark.SecurityManager: Changing modify acls to: hadoop
17/10/25 12:00:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/10/25 12:00:01 ERROR worker.Worker: Worker registration failed: Duplicate worker ID
17/10/25 12:00:01 INFO worker.ExecutorRunner: Launch command: "/home/hadoop/jdk1.7.0_09/bin/java" "-cp" "/home/hadoop/spark-1.6.1-bin-2.2.0/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/*:/home/hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/*:/home/hadoop/hadoop-2.2.0/share/hadoop/common/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/*:/home/hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/*:/home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-2.2.0/share/hadoop/tools/lib/*:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/*:/home/hadoop/spark-1.6.1-bin-2.2.0/conf/:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/spark-assembly-1.6.1-hadoop2.2.0.jar:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/spark-1.6.1-bin-2.2.0/lib/datanucleus-core-3.2.10.jar:/home/hadoop/hadoop-2.2.0/etc/hadoop/" "-Xms1024M" "-Xmx1024M" "-Dspark.driver.port=43546" "-XX:MaxPermSize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:43546" "--executor-id" "119773" "--hostname" "10.10.10.190" "--cores" "1" "--app-id" "app-20171024225907-0018" "--worker-url" "spark://[email protected]:55335"
17/10/25 12:00:02 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:02 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:02 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:03 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:04 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:05 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:05 INFO worker.ExecutorRunner: Killing process!
17/10/25 12:00:05 INFO util.ShutdownHookManager: Shutdown hook called
17/10/25 12:00:06 INFO util.ShutdownHookManager: Deleting directory /data/spark/data/spark-ab442bb5-62e6-4567-bc7c-8d00534ba1a3

近期，頻繁出現worker異常退出情況，從worker日誌看到，master要求重新連線並註冊，註冊後，worker繼續連線master,並反饋了一個錯誤資訊：
ERROR worker.Worker: Worker registration failed: Duplicate worker ID
之後就突然殺掉所有Executor然後退出worker。
出現該問題的機器記憶體都是16g(其他32g節點,沒有出現worker退出問題)。
再檢視該類節點yarn的配置，發現分配yarn的資源是15g,懷疑問題節點yarn和spark同處高峰期，導致spark分配不到資源退出。
為驗證猜想，檢視NM日誌，其時間發生點部分容器有自殺情況，但這個不能說明什麼，因為其他時間點也存有該問題。
為檢視問題時間點，節點中yarn資源使用情況，只能到active rm中檢視rm日誌，最後看到節點中在11:46:22到11:59:18，
問題節點yarn佔用資源一直在<memory:15000, vCores:6>。
如此，確實是該問題所致，將yarn資源調整至12g/10c,後續再繼續觀察。

8、spark （4個dd，11個nm）沒有利用非datanode節點上的executor問題

問題描述：跑大job時，總任務數上萬個，但是隻利用了其中4個dd上的executor，另外7臺沒有跑或者跑很少。

分析：spark driver進行任務分配時,會希望每個任務正好分配到要計算的資料節點上，避免網路傳輸。但當task因為其所在資料節點資源正好被分配完而沒機會再分配時，spark會等一段時間（由spark.locality.wait引數控制，預設3s）,超過該時間，就會選擇一個本地化差的級別進行計算。

解決辦法：將spark.locality.wait設定為0，不等待，任務直接分配，需重啟服務生效

9、spark自帶引數不生效問題

spark thrift測試系統經常出問題，調整了driver記憶體引數，依舊報問題。

常見問題狀態：連線spark thrift無響應，一會提示OutOfMemoryError: Java heap space

後來發現設定的driver記憶體引數沒有生效，環境配置檔案spark-env.sh設定了SPARK_DAEMON_MEMORY=1024m，覆蓋了啟動引數設定的

--driver-memory 4G ,導致引數設定沒生效。

spark問題與排查

spark問題與排查

linux系統故障分析與排查

Spark與緩存

MySQL復制異常大掃盲：快速溯源與排查錯誤全解

Linux系統故障分析與排查--日誌分析

spark與flume整合

Azure Redis 緩存使用註意事項與排查問題文檔整理

大數據Spark與Storm技術選型

百度快照被劫持怎麼修復與排查

【無人駕駛系列四】基於Spark與ROS分散式無人駕駛模擬平臺

支援kubernetes原生Spark 與其他應用的結合(mysql,postgresql,oracle,hdfs,hbase)

spark原理和spark與mapreduce的最大區別

spark與storm的對比(轉載)

(轉載)一次生產系統 Full GC 問題分析與排查總結

spark與storm功能對比與選型思考

spark與storm的對比

一個MySQL 5.7分割槽表效能下降的案例分析與排查

MySQL複製異常大掃盲：快速溯源與排查錯誤全解

記一次生產DB2資料庫鎖超時問題的分析與排查

一 Spark與kafka基本特點

spark問題與排查

相關推薦