hs_err_prd.log日誌分析
java程式執行的時候,突然JVM發生錯誤導致崩潰時,會生成一個hs_err_pid_xxx.log這樣的檔案,該檔案包含了導致 JVM crash 的重要資訊,我們可以通過分析該檔案定位到導致 JVM Crash 的原因,從而修復保證系統穩定,預設該檔案出生在專案得根目錄。
這個檔案主要包含如下內容:
- 日誌頭資訊
- crash 的執行緒資訊
- 所有執行緒資訊
- 安全點和鎖資訊
- 堆資訊
- 原生代碼快取
- 編譯事件
- gc 相關時間
- 逆優化
- 重新定義類
- 內部異常資訊
- jvm 記憶體對映
- jvm 啟動引數
- 伺服器資訊
一、日誌標頭檔案
翻譯過來的意思大概就是:
沒有足夠的記憶體,Java執行時環境無法繼續。本機記憶體分配(malloc)無法為ChunkPool :: allocate分配32744個位元組可能的原因:系統沒有物理RAM或交換空間該程序正在啟用CompressedOops的情況下執行,並且Java Heap可能阻止了本機堆的增長可能的解決方案:
減少系統上的記憶體負載
增加實體記憶體或交換記憶體
檢查交換後備儲存是否已滿
減小Java堆大小(-Xmx / -Xms)
減小Java執行緒數減小Java執行緒堆疊大小(-Xss)
設定較大的程式碼-XX:ReservedCodeCacheSize =
進行快取記憶體JVM以基於零的壓縮Oops模式執行,在該模式下,Java堆位於前32GB地址空間中。 Java堆基址是本機堆增長的最大限制。請使用-XX:HeapBaseMinAddress
設定Java Heap基礎並將Java Heap放置在32GB以上的虛擬地址上。
該輸出檔案可能被截斷或不完整。記憶體不足錯誤(allocation.cpp:272),pid = 16756,tid = 0x0000000000006b78
JRE版本:Java(TM)SE執行時環境(8.0_231-b11)(內部版本1.8.0_231-b11)
Java VM:Java HotSpot(TM) )64位伺服器VM(25.231-b11混合模式Windows-amd64壓縮的oops)
無法寫入核心轉儲。預設情況下,在Windows客戶端版本上不啟用小型轉儲
這段資訊簡述了發生異常的資訊,是伺服器記憶體不足,除此之外,JVM 自身的 bug,應用程式錯誤,JVM 引數,JNI 呼叫錯誤,版本資訊和配置資訊等也會導致JVM Crash。
Out of Memory Error (allocation.cpp:272), pid=16756, tid=0x0000000000006b78
複製程式碼
- pid=16756 程序號
- tid=0x0000000000006b78 執行緒號
二、crash 的執行緒資訊
--------------- T H R E A D ---------------
Current thread (0x000000001fd37800): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=27512, stack(0x0000000020170000,0x0000000020270000)]
Stack: [0x0000000020170000,0x0000000020270000]
[error occurred during error reporting (printing stack bounds), id 0xc0000005]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
Current CompileTask:
C2: 19813 10229 4 java.net.URLClassLoader$1::run (5 bytes)
複製程式碼
執行緒資訊
Current thread (0x0000000001e94800): JavaThread "pool-1-thread-2" [_thread_in_native, id=30111, stack(0x00007f3d567e5000,0x00007f3d568e6000)]
複製程式碼
- 0x0000000001e94800:出錯的執行緒指標
- JavaThread:執行緒型別,執行緒可能還會有以下:
- JavaThread:Java執行緒
- VMThread : JVM 的內部執行緒
- CompilerThread:用來呼叫JITing,實時編譯裝卸class 。 通常,jvm會啟動多個執行緒來處理這部分工作,執行緒名稱後面的數字也會累加,例如:CompilerThread1
- GCTaskThread:執行gc的執行緒
- WatcherThread:JVM 週期性任務排程的執行緒,是一個單例物件
- ConcurrentMarkSweepThread:jvm在進行CMS GC的時候,會建立一個該執行緒去進行GC,該執行緒被建立的同時會建立一個SurrogateLockerThread(簡稱SLT)執行緒並且啟動它,SLT啟動之後,處於等待階段。CMST開始GC時,會發一個訊息給SLT讓它去獲取Java層Reference物件的全域性鎖:Lock
- pool-1-thread-2:執行緒名稱
- _thread_in_native:當前執行緒狀態
- _thread_in_native:執行緒當前狀態,狀態列舉包括:
- _thread_uninitialized:執行緒還沒有建立,它只在記憶體原因崩潰的時候才出現
- _thread_new:執行緒已經被建立,但是還沒有啟動
- _thread_in_native:執行緒正在執行原生代碼,一般這種情況很可能是原生代碼有問題
- _thread_in_vm:執行緒正在執行虛擬機器程式碼
- _thread_in_Java:執行緒正在執行解釋或者編譯後的Java程式碼
- _thread_blocked:執行緒處於阻塞狀態
- …_trans:以_trans結尾,執行緒正處於要切換到其它狀態的中間狀態
- id=30111 :執行緒id
- stack(0x00007f3d567e5000,0x00007f3d568e6000):棧區間
三、執行緒資訊
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x000000002404f000 JavaThread "com.alibaba.nacos.naming.beat.sender" daemon [_thread_blocked, id=7612, stack(0x0000000021740000,0x0000000021840000)]
0x0000000024047800 JavaThread "DestroyJavaVM" [_thread_blocked, id=25856, stack(0x0000000002fb0000,0x00000000030b0000)]
0x0000000024048800 JavaThread "http-nio-51000-AsyncTimeout" daemon [_thread_blocked, id=12996, stack(0x0000000030020000,0x0000000030120000)]
0x000000002404e800 JavaThread "http-nio-51000-Acceptor-0" daemon [_thread_in_native, id=22020, stack(0x000000002ff20000,0x0000000030020000)]
0x000000002404d800 JavaThread "http-nio-51000-ClientPoller-1" daemon [_thread_in_native, id=11852, stack(0x000000002fe20000,0x000000002ff20000)]
0x0000000024046000 JavaThread "http-nio-51000-ClientPoller-0" daemon [_thread_in_native, id=23656, stack(0x000000002fd20000,0x000000002fe20000)]
0x0000000024047000 JavaThread "http-nio-51000-exec-10" daemon [_thread_blocked, id=11672, stack(0x000000002fc20000,0x000000002fd20000)]
0x000000002404b800 JavaThread "http-nio-51000-exec-9" daemon [_thread_blocked, id=6460, stack(0x000000002fb20000,0x000000002fc20000)]
0x000000002404d000 JavaThread "http-nio-51000-exec-8" daemon [_thread_blocked, id=4524, stack(0x000000002fa20000,0x000000002fb20000)]
0x000000002404c000 JavaThread "http-nio-51000-exec-7" daemon [_thread_blocked, id=7536, stack(0x000000002f920000,0x000000002fa20000)]
0x000000002404a800 JavaThread "http-nio-51000-exec-6" daemon [_thread_blocked, id=15648, stack(0x000000002f820000,0x000000002f920000)]
0x0000000024045800 JavaThread "http-nio-51000-exec-5" daemon [_thread_blocked, id=17580, stack(0x000000002f720000,0x000000002f820000)]
0x0000000022175800 JavaThread "http-nio-51000-exec-4" daemon [_thread_blocked, id=23840, stack(0x000000002f620000,0x000000002f720000)]
0x0000000021bf9000 JavaThread "http-nio-51000-exec-3" daemon [_thread_blocked, id=20868, stack(0x000000002f520000,0x000000002f620000)]
0x0000000021bfc000 JavaThread "http-nio-51000-exec-2" daemon [_thread_blocked, id=9416, stack(0x000000002f420000,0x000000002f520000)]
0x0000000021bff800 JavaThread "http-nio-51000-exec-1" daemon [_thread_blocked, id=7856, stack(0x000000002f320000,0x000000002f420000)]
0x0000000021bfc800 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_in_native, id=17036, stack(0x000000002f060000,0x000000002f160000)]
0x0000000021bf8000 JavaThread "dubbo-remoting-server-heartbeat-thread-1" daemon [_thread_blocked, id=11892, stack(0x000000002edf0000,0x000000002eef0000)]
0x0000000021bf9800 JavaThread "NettyServerBoss-7-1" daemon [_thread_in_native, id=8756, stack(0x000000002e8a0000,0x000000002e9a0000)]
0x0000000021bfe000 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=15228, stack(0x000000002f220000,0x000000002f320000)]
0x0000000021bfd800 JavaThread "xxl-job, executor ExecutorRegistryThread" daemon [_thread_blocked, id=20704, stack(0x000000002e7a0000,0x000000002e8a0000)]
0x0000000021bfb000 JavaThread "nioEventLoopGroup-5-1" [_thread_in_native, id=19896, stack(0x000000002e6a0000,0x000000002e7a0000)]
0x0000000021bfa800 JavaThread "Thread-17" daemon [_thread_blocked, id=23024, stack(0x000000002eaf0000,0x000000002ebf0000)]
0x0000000021bf0800 JavaThread "Thread-16" daemon [_thread_blocked, id=9520, stack(0x000000002e9f0000,0x000000002eaf0000)]
0x0000000021bf6000 JavaThread "xxl-job, executor TriggerCallbackThread" daemon [_thread_blocked, id=15680, stack(0x000000002e2f0000,0x000000002e3f0000)]
0x0000000021bf4800 JavaThread "xxl-job, executor JobLogFileCleanThread" daemon [_thread_blocked, id=18152, stack(0x000000002ada0000,0x000000002aea0000)]
0x0000000021bf6800 JavaThread "com.alibaba.nacos.naming.beat.sender" daemon [_thread_blocked, id=856, stack(0x000000002e400000,0x000000002e500000)]
0x0000000021bf5000 JavaThread "JetCacheDefaultExecutor" daemon [_thread_blocked, id=4936, stack(0x000000002e5a0000,0x000000002e6a0000)]
0x0000000021bf3000 JavaThread "DubboClientReconnectTimer-thread-2" daemon [_thread_blocked, id=20420, stack(0x000000002d9f0000,0x000000002daf0000)]
0x0000000021bf1800 JavaThread "com.alibaba.nacos.client.naming.updater" daemon [_thread_blocked, id=13332, stack(0x000000002d6f0000,0x000000002d7f0000)]
0x0000000021bf2000 JavaThread "com.alibaba.nacos.client.naming.updater" daemon [_thread_blocked, id=12236, stack(0x000000002d5f0000,0x000000002d6f0000)]
0x0000000022179000 JavaThread "com.alibaba.nacos.client.naming.updater" daemon [_thread_blocked, id=12008, stack(0x000000002d4f0000,0x000000002d5f0000)]
0x0000000022178800 JavaThread "com.alibaba.nacos.client.naming.updater" daemon [_thread_blocked, id=26380, stack(0x000000002d3f0000,0x000000002d4f0000)]
0x0000000022177800 JavaThread "com.alibaba.nacos.client.naming.updater" daemon [_thread_blocked, id=5812, stack(0x000000002d2f0000,0x000000002d3f0000)]
0x0000000022176000 JavaThread "dubbo-remoting-client-heartbeat-thread-1" daemon [_thread_blocked, id=27148, stack(0x000000002c020000,0x000000002c120000)]
0x000000002217d000 JavaThread "NettyClientWorker-4-1" daemon [_thread_in_native, id=22332, stack(0x000000002bf20000,0x000000002c020000)]
0x000000002217c000 JavaThread "DubboClientReconnectTimer-thread-1" daemon [_thread_blocked, id=22356, stack(0x000000002be20000,0x000000002bf20000)]
0x000000002217b800 JavaThread "DubboSaveRegistryCache-thread-1" daemon [_thread_blocked, id=23092, stack(0x000000002bd20000,0x000000002be20000)]
0x000000002216e800 JavaThread "com.alibaba.nacos.client.naming.updater" daemon [_thread_blocked, id=23676, stack(0x000000002b680000,0x000000002b780000)]
0x0000000022172800 JavaThread "DubboRegistryFailedRetryTimer-thread-1" daemon [_thread_blocked, id=22928, stack(0x000000002b920000,0x000000002ba20000)]
0x0000000022171000 JavaThread "com.alibaba.nacos.naming.push.receiver" daemon [_thread_in_native, id=11576, stack(0x000000002b820000,0x000000002b920000)]
0x000000002216e000 JavaThread "com.alibaba.nacos.naming.failover" daemon [_thread_blocked, id=16176, stack(0x000000002b230000,0x000000002b330000)]
0x0000000022173000 JavaThread "com.alibaba.nacos.naming.beat.sender" daemon [_thread_blocked, id=22464, stack(0x000000002b580000,0x000000002b680000)]
0x000000002216f800 JavaThread "com.alibaba.nacos.naming.beat.sender" daemon [_thread_blocked, id=17320, stack(0x000000002b480000,0x000000002b580000)]
0x0000000022171800 JavaThread "com.alibaba.nacos.naming.client.listener" daemon [_thread_blocked, id=26024, stack(0x000000002b380000,0x000000002b480000)]
0x0000000022174000 JavaThread "Timer-0" daemon [_thread_blocked, id=23244, stack(0x0000000022f40000,0x0000000023040000)]
0x0000000022174800 JavaThread "Log4j2-TF-9-Scheduled-3" daemon [_thread_blocked, id=11696, stack(0x000000002aee0000,0x000000002afe0000)]
0x000000002659f800 JavaThread "qos-boss-1-1" daemon [_thread_in_native, id=12388, stack(0x000000002aca0000,0x000000002ada0000)]
0x00000000265a1000 JavaThread "Druid-ConnectionPool-Destroy-613765444" daemon [_thread_blocked, id=16984, stack(0x000000002a6a0000,0x000000002a7a0000)]