最近線上的一個tomcat總是無緣無故崩潰，tomcat日誌裡又沒有報任何錯誤，於是調出JVM的崩潰日誌檢視，一般崩潰日誌在啟動目錄下，比如tomcat的bin目錄下，但是如果你用自己寫的指令碼啟動的tomcat，則這個日誌可能就在你放指令碼的目錄下。

#

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfe572980, pid=2383, tid=108 // SIGSEGV (0xb)這種問題一般是呼叫JNI或者使用了JNI的第三方庫造成的，我們可以查'UNIX訊號表"，可以查到這個是因為非法訪問了記憶體或者是訪問了無效指標造成的。
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) Server VM (24.80-b11 mixed mode solaris-sparc )
# Problematic frame:
# V  [libjvm.so+0x572980]  void Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x8  // 這裡V表示一種frame type,這裡看到是垃圾回收時出現錯誤,這裡表示是執行libjvm造成的
#
# Core dump written. Default location: /glodon/deploy/core or core.2383   //表示崩潰的core檔案的位置，實際上沒有找到，由於是用這個位置的指令碼啟動的，所以core檔案生在這裡
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#


---------------  T H R E A D  ---------------


Current thread (0x00501000):  GCTaskThread [stack: 0x2e600000,0x2e680000] [id=108] // 這裡說明是垃圾回收執行緒執行時崩潰


siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000c


Registers:
 G1=0x0000000f G2=0x00000001 G3=0xffffffff G4=0x00000000
 G5=0x00000010 G6=0x00000000 G7=0xfa0b5200 Y=0x00000000
 O0=0x00000022 O1=0x00000379 O2=0x00000000 O3=0x00000022
 O4=0x00000021 O5=0x00000000 O6=0x2e67f880 O7=0xfe572cac
 L0=0x00003ffe L1=0x00003fff L2=0x004e5df0 L3=0x00000001
 L4=0x004e5dac L5=0x0000037a L6=0x00000000 L7=0x00504358
 I0=0x2e67fa84 I1=0x53a5a710 I2=0x00000005 I3=0x006b2d38
 I4=0x006b2d38 I5=0x00000000 I6=0x2e67f928 I7=0xfe572968
 PC=0xfe572980 nPC=0xfe572984

Top of Stack: (sp=0x2e67f880)
0x2e67f880:   00003ffe 00003fff 004e5df0 00000001
0x2e67f890:   004e5dac 0000037a 00000000 00504358
0x2e67f8a0:   2e67fa84 53a5a710 00000005 006b2d38
0x2e67f8b0:   006b2d38 00000000 2e67f928 fe572968
0x2e67f8c0:   fee63d24 0013c800 ff2352ec 00000000
  此處省略N行......................................
I3=0x006b2d38 is an unknown value
I4=0x006b2d38 is an unknown value
I5=0x00000000 is an unknown value
I6=0x2e67f928 is an unknown value
I7=0xfe572968: JVM_GetClassDeclaredMethods+0x2998ec in /usr/jdk/instances/jdk1.7.0/jre/lib/sparc/server/libjvm.so at 0xfe000000




Stack: [0x2e600000,0x2e680000],  sp=0x2e67f880,  free space=510k //當前棧還剩餘510K，夠用
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x572980]  void Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x8
V  [libjvm.so+0x572970]  bool Par_MarkFromRootsClosure::do_bit(unsigned)+0xb0
V  [libjvm.so+0x3ac368]  bool BitMap::iterate(BitMapClosure*,unsigned,unsigned)+0x84
V  [libjvm.so+0x5663f8]  void CMSConcMarkingTask::do_scan_and_mark(int,CompactibleFreeListSpace*)+0x2b0
V  [libjvm.so+0x565e5c]  void CMSConcMarkingTask::work(unsigned)+0xc4
V  [libjvm.so+0xbb7d38]  void YieldingFlexibleGangWorker::loop()+0xd0
V  [libjvm.so+0xa04a00]  java_start+0x338




---------------  P R O C E S S  ---------------


Java Threads: ( => current thread )
  0x02852c00 JavaThread "qtp28852019-947" [_thread_blocked, id=1053, stack(0x24a80000,0x24b00000)]
  0x052a5000 JavaThread "qtp28852019-940" [_thread_blocked, id=1046, stack(0x17c80000,0x17d00000)]
  0x04e23800 JavaThread "qtp28852019-931" [_thread_blocked, id=1037, stack(0x17e80000,0x17f00000)]
  0x02035c00 JavaThread "http-bio-8080-exec-111" daemon [_thread_in_vm, id=1036, stack(0x18180000,0x18200000)]
  0x04e16c00 JavaThread "http-bio-8080-exec-110" daemon [_thread_blocked, id=1035, stack(0x18580000,0x18600000)]
  此處省略N行......................................
  0x009cbc00 JavaThread "Finalizer" daemon [_thread_blocked, id=113, stack(0x2df80000,0x2e000000)]
  0x009ca400 JavaThread "Reference Handler" daemon [_thread_blocked, id=112, stack(0x2e080000,0x2e100000)]
  0x00028c00 JavaThread "main" [_thread_in_native, id=2, stack(0xfde80000,0xfdf00000)]


Other Threads:
  0x009c7400 VMThread [stack: 0x2e180000,0x2e200000] [id=111]
  0x009f3800 WatcherThread [stack: 0x2d980000,0x2da00000] [id=119]


=>0x00501000 (exited) GCTaskThread [stack: 0x2e600000,0x2e680000] [id=108]


VM state:not at safepoint (normal execution) // 不在安全點，說明這個時候JVM執行的程式碼不在for迴圈結束或者方法結束的地方，也恰好說明呼叫的是JNI的東西，因為JVM呼叫JNI的時候不會等待其執行到安全點，關於安全點的問題，這裡不討論，大家可以查其他資料。


VM Mutex/Monitor currently owned by a thread: None


Heap //崩潰瞬間還是回收回來了，因此基本排除是因為JVM記憶體崩潰引起的問題。
 par new generation   total 235968K, used 21333K [0x36400000, 0x46400000, 0x46400000)
  eden space 209792K,   8% used [0x36400000, 0x376654c0, 0x430e0000)
  from space 26176K,   9% used [0x430e0000, 0x433501a0, 0x44a70000)
  to   space 26176K,   0% used [0x44a70000, 0x44a70000, 0x46400000)
 concurrent mark-sweep generation total 474160K, used 436253K [0x46400000, 0x6330c000, 0xb6400000)  // cms垃圾回收的老生代情況
 concurrent-mark-sweep perm gen total 262144K, used 141342K [0xb6400000, 0xc6400000, 0xf6400000) // cms永久代情況

Card table byte_map: [0x35c00000,0x36202000] byte_map_base: 0x35a4e000

Polling page: 0xff3f6000

Code Cache  [0xfbc00000, 0xfd4f8000, 0xfdc00000)
 total_blobs=7341 nmethods=7026 adapters=247 free_code_cache=7377Kb largest_free_block=7400896

Compilation events (10 events):
Event: 14607.071 Thread 0x009ee400 8453             org.apache.coyote.http11.InternalInputBuffer::fill (178 bytes)
Event: 14607.077 Thread 0x009ee400 nmethod 8453 0xfc1ef488 code [0xfc1ef5a0, 0xfc1ef7c4]
Event: 14612.564 Thread 0x009ecc00 8454 %           gboat2.base.core.web.MetadataSupportStrategy::getOperaList @ 146 (230 bytes)
Event: 14612.599 Thread 0x009ecc00 nmethod 8454% 0xfd4e05c8 code [0xfd4e07c0, 0xfd4e11d8]
Event: 14612.745 Thread 0x009ee400 8455             gboat2.base.core.web.MetadataSupportStrategy::getOperaList (230 bytes)
Event: 14612.797 Thread 0x009ee400 nmethod 8455 0xfd4ee208 code [0xfd4ee620, 0xfd4ef700]
Event: 14626.459 Thread 0x009ecc00 8456             com.sun.tools.javac.tree.TreeInfo::hasConstructors (34 bytes)
Event: 14626.462 Thread 0x009ecc00 nmethod 8456 0xfd0fec08 code [0xfd0fed00, 0xfd0fee10]
Event: 14643.412 Thread 0x009ee400 8457             com.itextpdf.text.pdf.PdfReader::readArray (71 bytes)
Event: 14643.424 Thread 0x009ee400 nmethod 8457 0xfd4dde88 code [0xfd4de000, 0xfd4de504]


GC Heap History (10 events): // 最後10次垃圾回收事件
Event: 14626.265 GC heap before // 這個是事件發生的時間戳，虛擬機器（tomcat)啟動以來的秒數
{Heap before GC invocations=803 (full 3): //這裡的full 3表示從tomcat開始到現在一共進行了3次full型別的垃圾回收；“Heap before GC invocations=803”這個表示在當前垃圾回收之前，進行了多少次垃圾回收工作，這裡表示進行了803次的垃圾回收
 par new generation   total 235968K, used 215681K [0x36400000, 0x46400000, 0x46400000) //新生代使用約256M（不到256，因為有一個survivor不能用）
  eden space 209792K, 100% used [0x36400000, 0x430e0000, 0x430e0000)//約209M
  from space 26176K,  22% used [0x44a70000, 0x450305d8, 0x46400000) //約26M，這個地址現在是from，下個階段將變成to
  to   space 26176K,   0% used [0x430e0000, 0x430e0000, 0x44a70000) //約26M
 concurrent mark-sweep generation total 474160K, used 434490K [0x46400000, 0x6330c000, 0xb6400000)  //老生代併發清理，總共約463M，用去424M，剩餘約39M
 concurrent-mark-sweep perm gen total 262144K, used 141295K [0xb6400000, 0xc6400000, 0xf6400000)    //永久代，總共約256M，用掉約141M，剩餘約115M
Event: 14626.282 GC heap after // 和before比較起來，垃圾回收消耗時間27毫秒
Heap after GC invocations=804 (full 3): // 回收
 par new generation   total 235968K, used 3183K [0x36400000, 0x46400000, 0x46400000)
  eden space 209792K,   0% used [0x36400000, 0x36400000, 0x430e0000)
  from space 26176K,  12% used [0x430e0000, 0x433fbd30, 0x44a70000)
  to   space 26176K,   0% used [0x44a70000, 0x44a70000, 0x46400000)
 concurrent mark-sweep generation total 474160K, used 435308K [0x46400000, 0x6330c000, 0xb6400000) // 剩餘38M
 concurrent-mark-sweep perm gen total 262144K, used 141295K [0xb6400000, 0xc6400000, 0xf6400000)   // 不變
}
  此處省略N行......................................
Event: 14692.050 GC heap before //相隔25秒
{Heap before GC invocations=807 (full 3): // 滿了
 par new generation   total 235968K, used 214960K [0x36400000, 0x46400000, 0x46400000)
  eden space 209792K,  99% used [0x36400000, 0x430dfbe8, 0x430e0000)
  from space 26176K,  19% used [0x44a70000, 0x44f7c738, 0x46400000)
  to   space 26176K,   0% used [0x430e0000, 0x430e0000, 0x44a70000)
 concurrent mark-sweep generation total 474160K, used 435958K [0x46400000, 0x6330c000, 0xb6400000)    // 不變
 concurrent-mark-sweep perm gen total 262144K, used 141342K [0xb6400000, 0xc6400000, 0xf6400000)      // 增加27K
Event: 14692.067 GC heap after
Heap after GC invocations=808 (full 3): // 回收
 par new generation   total 235968K, used 2496K [0x36400000, 0x46400000, 0x46400000)
  eden space 209792K,   0% used [0x36400000, 0x36400000, 0x430e0000)
  from space 26176K,   9% used [0x430e0000, 0x433501a0, 0x44a70000)
  to   space 26176K,   0% used [0x44a70000, 0x44a70000, 0x46400000)
 concurrent mark-sweep generation total 474160K, used 436253K [0x46400000, 0x6330c000, 0xb6400000)    // 增加295K，剩餘37M
 concurrent-mark-sweep perm gen total 262144K, used 141342K [0xb6400000, 0xc6400000, 0xf6400000)      // 不變120802K
}          


Deoptimization events (10 events): // 非優化事件

Event: 14023.027 Thread 0x0382c400 Uncommon trap: reason=null_check action=make_not_entrant pc=0xfd3f6188 

  此處省略N行...................................... method=org.apache.struts2.jasper.compiler.TagLibraryInfoImpl.parseTLD(Lorg/apache/struts2/jasper/JspCompilationContext;Ljava/lang/String;Ljava/io/InputStream;Ljava/net/URL;)V @ 573

Internal exceptions (10 events): // 內部異常
Event: 14691.724 Thread 0x02035c00 Threw 0x41055a00 at /HUDSON/workspace/7u-2-build-solaris-sparc/jdk7u80/2329/hotspot/src/share/vm/prims/jvm.cpp:1319
Event: 14691.724 Thread 0x02035c00 Threw 0x410577a0 at /HUDSON/workspace/7u-2-build-solaris-sparc/jdk7u80/2329/hotspot/src/share/vm/prims/jvm.cpp:1319
  此處省略N行......................................


Events (10 events):
Event: 14692.045 Executing VM operation: GenCollectForAllocation
Event: 14692.068 Executing VM operation: GenCollectForAllocation done
Event: 14692.068 Executing VM operation: CMS_Initial_Mark       // 初始標記
Event: 14692.092 Executing VM operation: CMS_Initial_Mark done  // 
Event: 14692.093 Executing VM operation: RevokeBias
Event: 14692.096 Executing VM operation: RevokeBias done
Event: 14692.097 Executing VM operation: RevokeBias
Event: 14692.098 Executing VM operation: RevokeBias done
Event: 14692.099 Executing VM operation: RevokeBias
Event: 14692.101 Executing VM operation: RevokeBias done




Dynamic libraries:
0x00010000 /usr/jdk/instances/jdk1.7.0/jre/bin/java
0xff370000 /usr/lib/libthread.so.1
  此處省略N行......................................
0x36220000 /usr/jdk/instances/jdk1.7.0/jre/lib/sparc/libjpeg.so


VM Arguments:
jvm_args: -Djava.util.logging.config.file=/GBP/apache-tomcat-7.0.57/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx2048m -Xmn256m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -XX:ErrorFile=/var/log/java/java_error%p.log -Djava.endorsed.dirs=/GBP/apache-tomcat-7.0.57/endorsed -Dcatalina.base=/GBP/apache-tomcat-7.0.57 -Dcatalina.home=/GBP/apache-tomcat-7.0.57 -Djava.io.tmpdir=/GBP/apache-tomcat-7.0.57/temp 
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD


Environment Variables:
JAVA_HOME=/usr/java
JRE_HOME=/usr/java/jre
CLASSPATH=:/usr/java/lib:/usr/java/bin:/usr/java/jre/lib
PATH=/usr/sbin:/usr/bin:/usr/java/bin:/usr/java/jre/bin:/GBP/apache-tomcat-7.0.57/bin:/usr/sfw/bin:/opt/csw/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/sun4v/sbin:/opt/sun/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin
LD_LIBRARY_PATH=/usr/lib:/usr/local/lib:/usr/openwin/lib
SHELL=/usr/bin/bash


Signal Handlers:
SIGSEGV: [libjvm.so+0xba1e58], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0xba1e58], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0xa08414], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0xa08414], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0xa0d058], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c




---------------  S Y S T E M  ---------------


OS:                   Oracle Solaris 10 8/11 s10s_u10wos_17b SPARC
  Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
                            Assembled 23 August 2011


uname:SunOS 5.10 Generic_147440-01 sun4v
  (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:1.18 1.11 1.05


CPU:total 256 v9, popc, vis1, vis2, vis3, blk_init, cbcond, sun4v, niagara_plus


Memory: 8k page, physical 267911168k(133769880k free)


vm_info: Java HotSpot(TM) Server VM (24.80-b11) for solaris-sparc JRE (1.7.0_80-b15), built on Apr 10 2015 18:48:54 by "java_re" with Sun Studio 12u1


time: Thu Mar  3 16:18:34 2016
elapsed time: 14704 seconds

        上oracle官方檢視並沒有人提出這個bug，因此判斷有可能是因為java呼叫JNI庫造成的記憶體錯誤，盤了下專案中使用的技術，並且查看了tomcat打出的業務日誌，發現在呼叫jmagick庫的時候程式一直報錯，這是個上傳圖片打水印的元件，呼叫了動態庫造成的問題。剛好這個水印功能不需要，則關閉了這個功能，不再代用打水印的API，觀察了一週，系統再也沒有崩潰過。

        當然了問題的排除過程，並沒有這麼順利，由於之前對JVM的崩潰日誌接觸不多，查了不少資料。這裡強調一句，網上的JVM崩潰場景不同，每個人遇到的問題也不一樣，注重思路就行。比如我這個文章能提醒大家呼叫JNI可能會出現JVM崩潰的問題，大家按照這個思路排查了也許能解決自己的問題，這種問題一般排除的時候需要好多天，但是真正動手結局也許一分鐘都用不到。