使用sun.misc.Unsafe及反射對記憶體進行內省(introspection)
對於一個有經驗的JAVA程式設計師來說,瞭解一個或者其它的JAVA物件佔用了多少記憶體,這將會非常有用。你可能已經聽說過我們所生活的世界,儲存容量將不再是一個問題,這個對於你的文字編輯器來說可能是對的(不過,開啟一個包含大量的圖片以及圖表的文件,看看你的編輯器會消耗多少記憶體),對於一個專用伺服器軟體來說也可能是對的(至少在你的企業成長到足夠大或者是在同一臺伺服器執行其它的軟體之前),對於基於雲的軟體來說也可能是對的,如果你足夠的富有可以花足夠的錢可以買頂級的伺服器硬體。
然而,現實是你的軟體如果是受到了記憶體限制,需要做的是花錢優化它而不是嘗試獲取更好的硬體(原文:in the real world your software will once reach a point where it makes sense to spend money in its optimization rather than trying to obtain an even better hardware)(目前你可以獲取到的最好的商業伺服器是64G記憶體),此時你不得不分析你的應用程式來找出是哪個資料結構消耗了大部份的記憶體。對於這種分析任務,最好的工具就是一個好的效能分析工具,但是你可以在剛開始的時候,使用分析你程式碼中的物件這種用廉價的方式。這篇文章描述了使用基於Oracle JDK的ClassIntrospector類,來分析你的應用程式記憶體消耗。
我曾經在文章字串包裝第1部分:將字元轉換為位元組中提到了JAVA物件記憶體結構,例如我曾經寫過,在JAVA1.7.0_06以前,一個具有28個字元的字串會佔用104個位元組,事實上,我在寫這篇文章的時候,通過自己的效能分析器證實了我的計算結果。現在我們使用Oracle JDK中特殊類sun.misc.Unsafe,通過純JAVA來實現一個JAVA物件內省器(introspector)。
我們使用sun.misc.Unsafe的以下方法:
在sun.misc.Unsafe中有兩個額外的內省方法:staticFieldBase及staticFieldOffset,但是在這篇文章中不會使用到。這兩個方法對於非安全的讀寫靜態方法會有用。//獲取位元組物件中非靜態方法的偏移量(get offset of a non-static field in the object in bytes public native long objectFieldOffset(java.lang.reflect.Field field); //獲取陣列中第一個元素的偏移量(get offset of a first element in the array) public native int arrayBaseOffset(java.lang.Class aClass); //獲取陣列中一個元素的大小(get size of an element in the array) public native int arrayIndexScale(java.lang.Class aClass); //獲取JVM中的地址值(get address size for your JVM) public native int addressSize();
我們應如何找到一個物件的記憶體佈局?
1、迴圈的在分析類及父類上呼叫Class.getDeclaredFields
,獲取所有物件的欄位,包括其父類中的欄位;
2、針對非靜態欄位(通過Field.getModifiers() & Modifiers.STATIC
判斷靜態欄位),通過使用Unsafe.objectFieldOffset
在其父類中獲取一個欄位的偏移量以及該欄位的shallow(注:shallow指的是當前物件本身的大小)大小:基礎型別的預設值及4個或8個位元組的物件引用(更多看下面);
3、對陣列來說,呼叫Unsafe.arrayBaseOffset及Unsafe.arrayIndexScale
,陣列的整個shallow大小將會是 當前陣列的偏移量+每個陣列的大小*陣列的長度(原文是:offset + scale * Array.getLength(array)
),當然了也包括對陣列本身引用的大小(看前面提到的);
4、別忘了物件圖的迴圈引用,因而就需要對前面已經分析過的物件進行跟蹤記錄(針對這些情況,推薦使用IdentityHashMap
)
Java物件引用大小是一個非常不確定的值(原文:Java Object
reference size is quite a virtual value),它可能是4個位元組或者是8個位元組,這個取決於你的JVM設定以及給了多少記憶體給JVM,針對32G以上的堆,它就總是8個位元組,但是針對小一點的堆就是4個位元組除非你在JVM設定裡關掉設定-XX:-UseCompressedOops
(我不確定這個功能是在JVM的哪個版本加進來的,或者是預設是開啟的)。結果就是,安全的方式獲取對像引用的大小就是找到Object[]陣列中一個元素的大小:unsafe.arrayIndexScale(
Object[].class )
,針對這種情況,Unsafe.addressSize
倒不實用了。
針對32G以下堆記憶體中例用4位元組引用的一點小小注意。一個正常的4個位元組的指標可以定位到4G地址空間任何地址。如果我們假設所有已分配的物件將通過8位元組邊界對齊,在我們的32位指標中我們將不再需要最低3位(這些位將總是等於零)。這意味著我們可以儲存35位地址在32位中。(這一節附上原文如下:
A small implementation note on 4 byte references on under 32G heaps. A normal 4 byte pointer could addressany byte in 4G address space. If we will assume that all allocated objects will be aligned by 8 bytes boundary, we won’t need 3 lowest bits in our 32 bit pointers anymore (these bits will always be equal to zeroes). This means that we can store 35 bit addresses in 32 bit value:)
32_bit_reference = ( int ) ( actual_64_bit_pointer >> 3 )
35位允許定址 32位*8=4G*8=32G地址空間。
寫這個工具時發現的其它的一些有趣的事情
1、要列印陣列的內容,必須使用Arrays.toString(包括基本型別及物件陣列);
2、你必須要小心 - 內省方法(introspection method)只接受物件作為欄位值,因此你最終可能處在無限迴圈中:整型打包成整數,以便傳遞到內省的方法。裡面你會發現一個Integer.value欄位,並嘗試再次內省了 - 瞧,你又回到了開始的地方!
3、要內省(introspect)物件陣列中所有非空的值 - 這僅僅是間接的物件圖中的外部level(原文:this is just an extra level of indirection in the object graph)
如何使用ClassIntrospector
類?僅需要例項化它並且在你的任意的物件中呼叫它的例項內省(introspect
)方法,它會返回一個ObjectInfo物件,這個物件與你的“根‘物件有關,這個物件將指向它的所有子項,我想這可能是足夠的列印其toString方法的結果和/或呼叫ObjectInfo.getDeepSize方法(原文:I think it may be sufficient to print its
toString
method result and/or to call ObjectInfo.getDeepSize
method),它將通過你的”根“物件引用,返回你的所有物件的總記憶體消耗。
ClassIntrospector
不是執行緒安全的,但是你可以在同一個執行緒中任意多次呼叫內省(introspect
)方法。
總結:
1、你可以使用sun.misc.Unsafe
的這些方法獲取Java物件的佈局資訊:objectFieldOffset
,
arrayBaseOffset
and arrayIndexScale
;
2、Java物件引用的大小取決於你當前的環境,根據不同JVM的設定以及分配給JVM的記憶體大小,它可能是4個或者8個位元組。在大於32G的堆中,對像引用的大小總會是8個位元組,但是在一個比較小的堆中它就會是4個位元組,除非關閉JVM設定:-XX:-UseCompressedOops
。
原始碼
ClassIntrospector:
import sun.misc.Unsafe;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.math.BigDecimal;
import java.util.*;
/**
* This class could be used for any object contents/memory layout printing.
*/
public class ClassIntrospector
{
public static void main(String[] args) throws IllegalAccessException {
final ClassIntrospector ci = new ClassIntrospector();
final Map<String, BigDecimal> map = new HashMap<String, BigDecimal>( 10);
map.put( "one", BigDecimal.ONE );
map.put( "zero", BigDecimal.ZERO );
map.put( "ten", BigDecimal.TEN );
final ObjectInfo res;
res = ci.introspect( "0123456789012345678901234567" );
//res = ci.introspect( new TestObjChild() );
//res = ci.introspect(map);
//res = ci.introspect( new String[] { "str1", "str2" } );
//res = ci.introspect(ObjectInfo.class);
//res = ci.introspect( new TestObj() );
System.out.println( res.getDeepSize() );
System.out.println( res );
}
/** First test object - testing various arrays and complex objects */
private static class TestObj
{
protected final String[] strings = { "str1", "str2" };
protected final int[] ints = { 14, 16 };
private final Integer i = 28;
protected final BigDecimal bigDecimal = BigDecimal.ONE;
@Override
public String toString() {
return "TestObj{" +
"strings=" + (strings == null ? null : Arrays.asList(strings)) +
", ints=" + Arrays.toString( ints ) +
", i=" + i +
", bigDecimal=" + bigDecimal +
'}';
}
}
/** Test class 2 - testing inheritance */
private static class TestObjChild extends TestObj
{
private final boolean[] flags = { true, true, false };
private final boolean flag = false;
@Override
public String toString() {
return "TestObjChild{" +
"flags=" + Arrays.toString( flags ) +
", flag=" + flag +
'}';
}
}
private static final Unsafe unsafe;
/** Size of any Object reference */
private static final int objectRefSize;
static
{
try
{
Field field = Unsafe.class.getDeclaredField("theUnsafe");
field.setAccessible(true);
unsafe = (Unsafe)field.get(null);
objectRefSize = unsafe.arrayIndexScale( Object[].class );
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}
/** Sizes of all primitive values */
private static final Map<Class, Integer> primitiveSizes;
static
{
primitiveSizes = new HashMap<Class, Integer>( 10 );
primitiveSizes.put( byte.class, 1 );
primitiveSizes.put( char.class, 2 );
primitiveSizes.put( int.class, 4 );
primitiveSizes.put( long.class, 8 );
primitiveSizes.put( float.class, 4 );
primitiveSizes.put( double.class, 8 );
primitiveSizes.put( boolean.class, 1 );
}
/**
* Get object information for any Java object. Do not pass primitives to this method because they
* will boxed and the information you will get will be related to a boxed version of your value.
* @param obj Object to introspect
* @return Object info
* @throws IllegalAccessException
*/
public ObjectInfo introspect( final Object obj ) throws IllegalAccessException
{
try
{
return introspect( obj, null );
}
finally { //clean visited cache before returning in order to make this object reusable
m_visited.clear();
}
}
//we need to keep track of already visited objects in order to support cycles in the object graphs
private IdentityHashMap<Object, Boolean> m_visited = new IdentityHashMap<Object, Boolean>( 100 );
private ObjectInfo introspect( final Object obj, final Field fld ) throws IllegalAccessException
{
//use Field type only if the field contains null. In this case we will at least know what's expected to be
//stored in this field. Otherwise, if a field has interface type, we won't see what's really stored in it.
//Besides, we should be careful about primitives, because they are passed as boxed values in this method
//(first arg is object) - for them we should still rely on the field type.
boolean isPrimitive = fld != null && fld.getType().isPrimitive();
boolean isRecursive = false; //will be set to true if we have already seen this object
if ( !isPrimitive )
{
if ( m_visited.containsKey( obj ) )
isRecursive = true;
m_visited.put( obj, true );
}
final Class type = ( fld == null || ( obj != null && !isPrimitive) ) ?
obj.getClass() : fld.getType();
int arraySize = 0;
int baseOffset = 0;
int indexScale = 0;
if ( type.isArray() && obj != null )
{
baseOffset = unsafe.arrayBaseOffset( type );
indexScale = unsafe.arrayIndexScale( type );
arraySize = baseOffset + indexScale * Array.getLength( obj );
}
final ObjectInfo root;
if ( fld == null )
{
root = new ObjectInfo( "", type.getCanonicalName(), getContents( obj, type ), 0, getShallowSize( type ),
arraySize, baseOffset, indexScale );
}
else
{
final int offset = ( int ) unsafe.objectFieldOffset( fld );
root = new ObjectInfo( fld.getName(), type.getCanonicalName(), getContents( obj, type ), offset,
getShallowSize( type ), arraySize, baseOffset, indexScale );
}
if ( !isRecursive && obj != null )
{
if ( isObjectArray( type ) )
{
//introspect object arrays
final Object[] ar = ( Object[] ) obj;
for ( final Object item : ar )
if ( item != null )
root.addChild( introspect( item, null ) );
}
else
{
for ( final Field field : getAllFields( type ) )
{
if ( ( field.getModifiers() & Modifier.STATIC ) != 0 )
{
continue;
}
field.setAccessible( true );
root.addChild( introspect( field.get( obj ), field ) );
}
}
}
root.sort(); //sort by offset
return root;
}
//get all fields for this class, including all superclasses fields
private static List<Field> getAllFields( final Class type )
{
if ( type.isPrimitive() )
return Collections.emptyList();
Class cur = type;
final List<Field> res = new ArrayList<Field>( 10 );
while ( true )
{
Collections.addAll( res, cur.getDeclaredFields() );
if ( cur == Object.class )
break;
cur = cur.getSuperclass();
}
return res;
}
//check if it is an array of objects. I suspect there must be a more API-friendly way to make this check.
private static boolean isObjectArray( final Class type )
{
if ( !type.isArray() )
return false;
if ( type == byte[].class || type == boolean[].class || type == char[].class || type == short[].class ||
type == int[].class || type == long[].class || type == float[].class || type == double[].class )
return false;
return true;
}
//advanced toString logic
private static String getContents( final Object val, final Class type )
{
if ( val == null )
return "null";
if ( type.isArray() )
{
if ( type == byte[].class )
return Arrays.toString( ( byte[] ) val );
else if ( type == boolean[].class )
return Arrays.toString( ( boolean[] ) val );
else if ( type == char[].class )
return Arrays.toString( ( char[] ) val );
else if ( type == short[].class )
return Arrays.toString( ( short[] ) val );
else if ( type == int[].class )
return Arrays.toString( ( int[] ) val );
else if ( type == long[].class )
return Arrays.toString( ( long[] ) val );
else if ( type == float[].class )
return Arrays.toString( ( float[] ) val );
else if ( type == double[].class )
return Arrays.toString( ( double[] ) val );
else
return Arrays.toString( ( Object[] ) val );
}
return val.toString();
}
//obtain a shallow size of a field of given class (primitive or object reference size)
private static int getShallowSize( final Class type )
{
if ( type.isPrimitive() )
{
final Integer res = primitiveSizes.get( type );
return res != null ? res : 0;
}
else
return objectRefSize;
}
}
ObjectInfo:
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
/**
* This class contains object info generated by ClassIntrospector tool
*/
public class ObjectInfo {
/** Field name */
public final String name;
/** Field type name */
public final String type;
/** Field data formatted as string */
public final String contents;
/** Field offset from the start of parent object */
public final int offset;
/** Memory occupied by this field */
public final int length;
/** Offset of the first cell in the array */
public final int arrayBase;
/** Size of a cell in the array */
public final int arrayElementSize;
/** Memory occupied by underlying array (shallow), if this is array type */
public final int arraySize;
/** This object fields */
public final List<ObjectInfo> children;
public ObjectInfo(String name, String type, String contents, int offset, int length, int arraySize,
int arrayBase, int arrayElementSize)
{
this.name = name;
this.type = type;
this.contents = contents;
this.offset = offset;
this.length = length;
this.arraySize = arraySize;
this.arrayBase = arrayBase;
this.arrayElementSize = arrayElementSize;
children = new ArrayList<ObjectInfo>( 1 );
}
public void addChild( final ObjectInfo info )
{
if ( info != null )
children.add( info );
}
/**
* Get the full amount of memory occupied by a given object. This value may be slightly less than
* an actual value because we don't worry about memory alignment - possible padding after the last object field.
*
* The result is equal to the last field offset + last field length + all array sizes + all child objects deep sizes
* @return Deep object size
*/
public long getDeepSize()
{
return length + arraySize + getUnderlyingSize( arraySize != 0 );
}
private long getUnderlyingSize( final boolean isArray )
{
long size = 0;
for ( final ObjectInfo child : children )
size += child.arraySize + child.getUnderlyingSize( child.arraySize != 0 );
if ( !isArray && !children.isEmpty() )
size += children.get( children.size() - 1 ).offset + children.get( children.size() - 1 ).length;
return size;
}
private static final class OffsetComparator implements Comparator<ObjectInfo>
{
@Override
public int compare( final ObjectInfo o1, final ObjectInfo o2 )
{
return o1.offset - o2.offset; //safe because offsets are small non-negative numbers
}
}
//sort all children by their offset
public void sort()
{
Collections.sort( children, new OffsetComparator() );
}
@Override
public String toString() {
final StringBuilder sb = new StringBuilder();
toStringHelper( sb, 0 );
return sb.toString();
}
private void toStringHelper( final StringBuilder sb, final int depth )
{
depth( sb, depth ).append("name=").append( name ).append(", type=").append( type )
.append( ", contents=").append( contents ).append(", offset=").append( offset )
.append(", length=").append( length );
if ( arraySize > 0 )
{
sb.append(", arrayBase=").append( arrayBase );
sb.append(", arrayElemSize=").append( arrayElementSize );
sb.append( ", arraySize=").append( arraySize );
}
for ( final ObjectInfo child : children )
{
sb.append( '\n' );
child.toStringHelper(sb, depth + 1);
}
}
private StringBuilder depth( final StringBuilder sb, final int depth )
{
for ( int i = 0; i < depth; ++i )
sb.append( '\t' );
return sb;
}
}