一次永久代洩漏（perm洩漏）排查
阿新 • • 發佈：2019-07-14
寫在前面
網上關於perm區洩露的文章比較少，特別是對於動態類載入方面問題的分析比較少，在此記錄下。
perm區問題一般兩種解決方案：
- 啟動時perm區問題，一般修改PermSize,MaxPermSize引數即可
- 執行時動態生成類載入問題，這種問題比較難搞，需要關心動態載入了哪些類
週末早晨收到幾臺機器告警(fullgc告警)(perm大於90%告警)，為快速解決問題，先把幾臺機器重啟解決，留下了一臺禁用埠保留現場進行問題分析。平時上線發版比較頻繁，發版後jvm回收，如果較長時間沒有發版可能會造成洩露，收到告警。
吃完早飯後登上機器進行排查。
排查過程
登入機器，檢視記憶體使用高的程序：
top
pid:15298
既然是perm區問題，檢視永久代情況：
jmap -permstat pid > 15298dump.permstat
class_loader classes bytes parent_loader alive? type <bootstrap> 3630 21866152 null live <internal> 0x0000000705e4df00 1 2008 0x00000006c002e450 dead sun/reflect/DelegatingClassLoader@0x00000007c0050c30 0x00000007282e1040 4 23480 0x00000006c002e450 dead com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040
通過awk統計type型別，檢視載入了哪種型別的類：
awk '{ arr[$6]+=$3 } END { for (key in arr) printf("%s\t%s\n", key, arr[key]) }' 15298dump.permstat | sort -k2,2
ava/net/URLClassLoader@0x00000007c01c1598 0 java/util/ResourceBundle$RBClassLoader@0x00000007c0337380 0 type 0 sun/reflect/DelegatingClassLoader@0x00000007c0050c30 11361872 sun/reflect/misc/MethodUtil@0x00000007c777bf60 134528 <internal> 21866152 com/alibaba/fastjson/util/ASMClassLoader@0x00000007c44de0e0 4785232 N/A 5146 sun/misc/Launcher$AppClassLoader@0x00000007c021c2d8 5423216 com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040 5488288 sun/misc/Launcher$ExtClassLoader@0x00000007c01c1978 615216 org/eclipse/jetty/webapp/WebAppClassLoader@0x00000007c06e0da0 738107040
WebAppClassLoader載入最多，達到了738107040，近738m。
知道了載入的類最多，怎麼分析具體載入了哪種類呢？
之前一般通過Java -verbose檢視啟動類載入過程。
對於執行時貌似有兩種方式：
- 基於ClassFilter寫一個攔截器，dump特定的類，使用SA的jar($JAVA_HOME/lib/sa-jdi.jar)編譯好類，在編譯好的類目錄下呼叫下面的命令進行dump
- 問了(政威老師)，準備試試arthas，https://alibaba.github.io/arthas/classloader.html
機器上安裝好arthas-boot.jar，並啟動：
java -jar arthas-boot.jar
選擇程序，執行類後臺匯出，便後續分析：
classloader -a >> &
開啟匯出檔案：
hash:null, BootstrapClassLoader
[B
[C
[D
[F
[I
[J
[Lcom.sun.activation.registries.MimeTypeFile;
[Lcom.sun.imageio.plugins.jpeg.DHTMarkerSegment$Htable;
[Lcom.sun.imageio.plugins.jpeg.ImageTypeProducer;
[Lcom.sun.imageio.plugins.jpeg.JPEGImageReader$CallBackLock$State;
[Lcom.sun.imageio.plugins.jpeg.JPEGImageWriter$CallBackLock$State;
[Lcom.sun.imageio.plugins.jpeg.SOFMarkerSegment$ComponentSpec;
[Lcom.sun.imageio.plugins.jpeg.SOSMarkerSegment$ScanComponentSpec;
[Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry;
[Lcom.sun.jmx.mbeanserver.MXBeanMapping;
[Lcom.sun.org.apache.xalan.internal.utils.FeatureManager$Feature;
[Lcom.sun.org.apache.xalan.internal.utils.FeaturePropertyBase$State;
[Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$Limit;
[Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$NameMap;
[Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$State;
[Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityPropertyManager$Property;
[Lcom.sun.org.apache.xerces.internal.impl.XMLEntityManager$CharacterBuffer;
[Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMLeaf;
[Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMNode;
[Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMStateSet;
[Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator;
[Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator;
[Lcom.sun.org.apache.xerces.internal.impl.dv.XSSimpleType;
[Lcom.sun.org.apache.xerces.internal.impl.dv.xs.AbstractDateTimeDV$DateTimeData;
[Lcom.sun.org.apache.xerces.internal.impl.dv.xs.TypeValidator;
[Lcom.sun.org.apache.xerces.internal.impl.dv.xs.XSSimpleTypeDecl;
[Lcom.sun.org.apache.xerces.internal.impl.xpath.regex.RegularExpression;
[Lcom.sun.org.apache.xerces.internal.impl.xs.SchemaGrammar;
[Lcom.sun.org.apache.xerces.internal.impl.xs.SubstitutionGroupHandler$OneSubGroup;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSAnnotationImpl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSAttributeUseImpl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSComplexTypeDecl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSElementDecl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSGroupDecl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSNotationDecl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.XSParticleDecl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.identity.IdentityConstraint;
[Lcom.sun.org.apache.xerces.internal.impl.xs.identity.XPathMatcher;
[Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMLeaf;
[Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMValidator;
[Lcom.sun.org.apache.xerces.internal.impl.xs.opti.DefaultNode;
[Lcom.sun.org.apache.xerces.internal.impl.xs.opti.NodeImpl;
[Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.OneAttr;
[Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.XSDocumentInfo;
[Lcom.sun.org.apache.xerces.internal.impl.xs.util.SimpleLocator;
[Lcom.sun.org.apache.xerces.internal.impl.xs.util.XInt;
[Lcom.sun.org.apache.xerces.internal.util.Status;
[Lcom.sun.org.apache.xerces.internal.util.SymbolHash$Entry;
[Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry;
[Lcom.sun.org.apache.xerces.internal.util.XMLAttributesImpl$Attribute;
[Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit;
[Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap;
[Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State;
[Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$Property;
[Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State;
[Lcom.sun.org.apache.xerces.internal.xni.QName;
[Lcom.sun.org.apache.xerces.internal.xni.XMLLocator;
[Lcom.sun.org.apache.xerces.internal.xni.grammars.Grammar;
[Lcom.sun.org.apache.xerces.internal.xni.grammars.XSGrammar;
[Lcom.sun.org.apache.xerces.internal.xs.ShortList;
[Lcom.sun.org.apache.xerces.internal.xs.XSAnnotation;
[Lcom.sun.org.apache.xerces.internal.xs.XSAttributeUse;
[Lcom.sun.org.apache.xerces.internal.xs.XSComplexTypeDefinition;
[Lcom.sun.org.apache.xerces.internal.xs.XSElementDeclaration;
[Lcom.sun.org.apache.xerces.internal.xs.XSIDCDefinition;
[Lcom.sun.org.apache.xerces.internal.xs.XSModelGroupDefinition;
[Lcom.sun.org.apache.xerces.internal.xs.XSNamespaceItem;
[Lcom.sun.org.apache.xerces.internal.xs.XSNotationDeclaration;
[Lcom.sun.org.apache.xerces.internal.xs.XSObject;
[Lcom.sun.org.apache.xerces.internal.xs.XSParticle;
[Lcom.sun.org.apache.xerces.internal.xs.XSSimpleTypeDefinition;
[Lcom.sun.org.apache.xerces.internal.xs.XSTerm;
[Lcom.sun.org.apache.xerces.internal.xs.XSTypeDefinition;
[Lcom.sun.org.apache.xerces.internal.xs.datatypes.XSDateTime;
[Lcom.sun.org.apache.xml.internal.dtm.DTM;
[Lcom.sun.org.apache.xml.internal.dtm.DTMAxisTraverser;
[Lcom.sun.org.apache.xml.internal.dtm.DTMIterator;
[Lcom.sun.org.apache.xml.internal.dtm.ref.ExpandedNameTable$HashEntry;
[Lcom.sun.org.apache.xml.internal.dtm.ref.ExtendedType;
[Lcom.sun.org.apache.xpath.internal.Expression;
[Lcom.sun.org.apache.xpath.internal.ExpressionNode;
[Lcom.sun.org.apache.xpath.internal.XPathVisitable;
[Lcom.sun.org.apache.xpath.internal.axes.LocPathIterator;
[Lcom.sun.org.apache.xpath.internal.axes.PathComponent;
[Lcom.sun.org.apache.xpath.internal.axes.PredicatedNodeTest;
[Lcom.sun.org.apache.xpath.internal.axes.SubContextList;
[Lcom.sun.org.apache.xpath.internal.objects.XObject;
[Lcom.sun.org.apache.xpath.internal.patterns.NodeTest;
[Lcom.sun.xml.internal.ws.org.objectweb.asm.Item;
[Lcom.sun.xml.internal.ws.org.objectweb.asm.Type;
[Ljava.awt.AWTKeyStroke;
[Ljava.awt.Dimension;
[Ljava.awt.Queue;
[Ljava.awt.event.ActionListener;
[Ljava.awt.event.ComponentListener;
[Ljava.awt.event.FocusListener;
[Ljava.awt.event.HierarchyBoundsListener;
......
統計最多的類：
#!/usr/bin/python
from collections import Counter
package_name_count_dic = {}
for s in open('14.txt'):
full_qualified_name = s.strip()
if not full_qualified_name:
continue
class_name_index = full_qualified_name.rfind('.')
if class_name_index >= 0:
package_name, class_name = full_qualified_name[:class_name_index], full_qualified_name[class_name_index + 1:]
else:
package_name = class_name = full_qualified_name
if package_name in package_name_count_dic:
package_name_count_dic[package_name] += 1
else:
package_name_count_dic[package_name] = 1
k = Counter(package_name_count_dic)
high = k.most_common(5)
for p, c in high:
print p, c
最多的幾個類：
真凶：ma.glasnost.orika.generated。
檢視程式碼中誰使用了orika類庫。
發現是閃購同學，由於我們的系統目前對接多方，閃購同學還在我們系統做程式碼開發，在codereview上存在一些問題。
@Component
public class SGMapperFactory implements FactoryBean<MapperFactory> {
@Override
public MapperFactory getObject() {
return new DefaultMapperFactory.Builder().build();
}
@Override
public Class<?> getObjectType() {
return MapperFactory.class;
}
@Override
public boolean isSingleton() {
return true;
}
}
每次呼叫getObject都會新建立DefaultMapperFactory物件。MapperGenerator 每次會動態產生類。
至此問題排查，推動閃購同學改動。
後記
看程式碼是3月份程式碼寫的，為何到現在才發現問題呢？
首先是平時發版比較頻繁，jvm發版後問題解決。 最近發版比較少，同時這些是擴容機器，在最近的幾次發版中並沒有發版，所以造成類載入持續一段時間，最後造成永久代洩露。 為儘早發現問題需要在流程上進行控制，比如增加codereview細緻程度，在灰度發版後對發版機器進行引流壓測，儘早發現問