NameNode是如何儲存元資料的？
阿新 • • 發佈：2020-07-09
# 問題描述
公司做的是一個支付系統，會對接很多第三方公司。
突然有一天,有一家第三方（簡稱金花平臺）反應收不到我們的通知訊息。
# 排查過程
我們登陸自己的伺服器，檢查程式日誌，是有給金花平臺傳送通知的。而且大多訂單都是通知成功而且金花平臺也成功返回了。
仔細檢查日誌後，發現金花平臺說的沒有收到通知的訂單其實在我們伺服器的日誌裡面是有傳送的，只是沒有返回
與金花平臺溝通後，手動模擬程式給金花平臺傳送通知後，金花平臺反饋之前沒有收到通知的訂單經過手動模擬傳送通知成功了
# 定位問題
與研發溝通後，緊急更新加了一些通知模組更加詳細的日誌{具體是一些內部互動以及通知第三方時候的每一步日誌}
和公司負責研發的負責人商量後，決定從頭開始檢查一邊，檢查伺服器的資源，還有服務自身的執行緒池
等檢查到執行緒池的時候發現 負責通知的模組有死鎖的執行緒池，釋放不了，卡在了dns解析
具體檢視死鎖執行緒池的方法：
ps -ef|grep 專案名稱 # 1.找到對應專案的程式號
jstack 程式號 # 實時獲取正在執行的執行緒池
下面是獲取到的執行緒池其中的一個帶鎖的
"Thread-2199" #2595 prio=5 os_prio=0 tid=0x00007f20fc0a5800 nid=0x10b0d in Object.wait() [0x00007f20a940d000]java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.net.InetAddress.checkLookupTable(InetAddress.java:1393)
- locked <0x00000000804a4e78> (a java.util.HashMap)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:)at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at vip.dcpay.util.http.HttpHelper.execute(HttpHelper.java:246)
at vip.dcpay.util.http.HttpHelper.post(HttpHelper.java:113)
at vip.dcpay.order.notify.domain.util.CallbackInterfaceUtil.callbackNotify(CallbackInterfaceUtil.java:84)
at vip.dcpay.order.notify.domain.service.TaskRetryService.notify(TaskRetryService.java:85)
at vip.dcpay.order.notify.domain.service.TaskRetryService$$FastClassBySpringCGLIB$$3b7fcca6.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:180)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at vip.dcpay.order.notify.domain.service.TaskRetryService$$EnhancerBySpringCGLIB$$f6eef3da.notify(<generated>)
at vip.dcpay.order.notify.domain.service.CallBackReceiveService$1.run(CallBackReceiveService.java:56)
at java.lang.Thread.run(Thread.java:748)
# 問題處理方案