性能問題案例分析
阿新 • • 發佈：2019-02-03
standard ioe reg ror proxy object 容器連接 cep inter
準備
本次測試具備的環境很簡單，項目都是 java 項目，jdk 為1.7（1.8 也成），tomcat 為7版本，war 包放進 tomcat 路徑下的 webapps 就成。
然後改數據庫配置文件
sql 文件放到數據庫內執行，生成相應的數據庫以及表
首頁：http://www.ganziwen.cn:8082/dangdang/main/main.jsp
註冊頁面：http://www.ganziwen.cn:8082/dangdang/user/registForm.jsp
線程棧死鎖
問題描述
數據庫保證已經完全連上，但是首頁的圖片無法加載出來
排查方案
出現這種問題嘗試結合之前講的架構從幾個方面去分析，主要是排查法：
一、請求沒法送到服務器
- 負載機 cpu、內存、網絡、磁盤
二、請求發送到服務器但是無法被處理
- 應用服務器/數據庫服務器 cpu、內存、網絡、磁盤
- 容器連接池
- 數據庫連接池
- 代碼邏輯
- sql 語句慢
- JVM 的堆內存/gc 頻繁（jstat -gcutil pid）
- 線程棧問題（jstack pid）
- 磁盤 io(nmon/sar -d/iostat -x)
排查過程
這裏我們的負載機排查很好排查：直接訪問一個服務器上的其他鏈接：
從這張圖片，可以側面論證，我們的負載機是沒有問題的，最起碼可以訪問服務器，同時也說明 web 容器的連接池並沒有滿，因為可以訪問我們的 82 端口，那麽嘗試從其他方面去考慮，看看 jvm 和線程棧
首先我們查一下 java 進程，得到 pid 為 8071：
# ps -ef|grep java|grep -v grep root 8071 1 0 13:10 pts/0 00:00:10 /opt/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dignore.endorsed.dirs= -classpath /opt/tomcat7/bin/bootstrap.jar:/opt/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat7 -Dcatalina.home=/opt/tomcat7 -Djava.io.tmpdir=/opt/tomcat7/temp org.apache.catalina.startup.Bootstrap start
然後我們看一下 jvm 的 gc 情況：
# jstat -gcutil 8071 3000 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 64.96 82.81 96.73 92.25 25 0.189 1 0.051 0.239 0.00 100.00 64.96 82.81 96.73 92.25 25 0.189 1 0.051 0.239 0.00 100.00 65.23 82.81 96.73 92.25 25 0.189 1 0.051 0.239 0.00 100.00 65.23 82.81 96.73 92.25 25 0.189 1 0.051 0.239
可以發現。，gc 的頻率並不是很高，暫時排除是 gc 頻繁導致的
接下來看看線程棧問題：
jstack 8071 > 1.log
打開 1.log：
我們可以發現，在http-nio-8082-exec-XXX這種的線程中狀態幾乎都為 waitting 或者 blocked，而且有一個 deadblock
Found one Java-level deadlock: ============================= "http-bio-8082-exec-10": waiting to lock monitor 0x00007fa8a4005c98 (object 0x00000000e161a920, a java.lang.Object), which is held by "http-bio-8082-exec-8" "http-bio-8082-exec-8": waiting to lock monitor 0x00007fa888005c98 (object 0x00000000e16159d8, a java.lang.Object), which is held by "http-bio-8082-exec-10"
我們看到這裏，大概可以判斷到應該是線程棧死鎖導致的，而且可以看到在鎖住的線程調用的方法的路徑以及方法名是：org.tarena.common.DbUtil.getConnection(DbUtil.java:43)
"http-nio-8082-exec-109" #139 daemon prio=5 os_prio=0 tid=0x00007f915c840800 nid=0x2087 waiting for monitor entry [0x00007f91336b2000] java.lang.Thread.State: BLOCKED (on object monitor) at org.tarena.common.DbUtil.getConnection(DbUtil.java:43) - waiting to lock <0x00000000ec36cbb8> (a java.lang.Object) at org.tarena.dao.impl.DangDAOImpl.getConnection(DangDAOImpl.java:18) at org.tarena.dao.impl.BookDAOImpl.findHotBoardBooks(BookDAOImpl.java:159) at org.tarena.action.main.HotBoardBook.execute(HotBoardBook.java:20) at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:441) at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:280) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:243) at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:165) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:252) at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:68) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:122) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:179) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:75) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:94) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:235) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:89) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:130) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:267) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:126) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:138) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:165) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:179) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:52) at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:488) at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77) at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734) - locked <0x00000000e0f1c108> (a org.apache.tomcat.util.net.NioChannel) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
性能問題案例分析