Externally Stored Fields in InnoDB

This article discusses the storage (inline and external) of field data in the InnoDB storage engine. All fields of variable length types like VARCHAR, VARBINARY, BLOB and TEXT can be stored inline within the clustered index record, or stored externally in separate BLOB pages outside of the index record (but within the same tablespace). All of these fields can be classified as large objects. These large objects are either binary large objects or character large objects. The binary large objects do not have an associated character set, whereas the character large objects do.    >>本文主要討論mysql innodb儲存引擎下如何儲存大物件。所有的變長型別，如VARCHAR,VARBINARY,BLOG,TEXT即可能被儲存在索引頁中(innodb 表是索引組織表)，也可能被儲存在external pages(溢位頁，溢位頁同索引頁在同一個表空間)。至於這些列是儲存在B-tree pages還是被儲存在external pages，主要取決於下面三個因素：
1).大物件的size；2).整個行的size；3).innodb 的 row format。大物件可能是二進位制大物件也可能是字串大物件，二進位制大物件沒有相關字符集，而字串大物件有字符集。

Within the InnoDB storage engine there is no difference between the way character large objects and binary large objects are handled. Throughout this article we will use the term “BLOB field” to refer to any of the aforementioned field types that can be chosen for external storage.    >>對於innodb儲存引擎來說，對於二進位制和字串大物件的處理都是一樣的。本文下面的描述中 “BLOB field”只是一個統稱，它代表mysql中支援的所有大物件比如BLOB,TEXT,LONG VARCHAR(應該還包括LONG VARBINARY)

This article will provide information about the following: >>下面的文章包含如下幾部分內容

• Explain when a BLOB field will be stored inline and when it will be stored externally, with respect to the clustered index record.    >>本文說明了主鍵索引中的大物件什麼情況下會被儲存在B-tree pages，什麼情況下會被儲存在external pages中(注意只有主鍵索引中可能存在大物件)
• The structure of the BLOB reference.    >>BLOB reference的結構
• The BLOB prefix that is stored in the clustered index record, when the BLOB is stored externally.    >>如果表的row_format為COMPACT或者REDUNDANT時如果大物件被儲存在external pages，BLOG欄位會在索引頁儲存 768 bytes資料加上一個20 bytes的指標(BLOB reference)，其他的儲存在external pages。如果表的row_format為DYNAMIC或者COMPRESSED(innodb_file_format=Barracuda在支援之前的REDUNDANT和COMPACT row_format基礎上新增了Dynamic和Compressed兩種 row_format),時，如果大物件被儲存在external pages，索引頁中不會再儲存BLOB prefix(BLOB 前768 bytes 資料)只儲存BLOB reference。
• Utility gdb functions to examine the BLOB reference and the record offsets array.    >>使用gdb 程式分析BLOB reference和記錄的位置偏移量

The BLOB fields are associated with the clustered index records (the primary key) of a table. Only the clustered index can store a BLOB field externally. A secondary index cannot have externally stored fields. For the purposes of this article, we won’t deal with any secondary indexes.    >>只有聚集索引才能儲存大物件在external pages，所以我們下面說的都是聚集索引中大物件如何儲存。

The Schema

The following example table will be used to present the information:    >>下面是用來說明問題的示例表

A single clustered index record can have 1 or more externally stored BLOBs. So for the given table definition of t1, there are 4 possible ways that the BLOB fields of f2 and f3 can be stored:    >>測試表t1中有f2和f3兩個列儲存的大物件，所以理論上innodb 有如下4中方式儲存它們

1. f2 and f3 are both stored inline within the clustered index page    >>f2和f3都儲存在索引頁中
2. f2 is stored inline, while f3 is stored externally    >>f2儲存在索引頁中，f3儲存在 external pages中
3. f3 is stored inline, while f2 is stored externally    >>f3儲存在索引頁中，f2儲存在 external pages中
4. both f2 and f3 are stored externally    >>f2和f3都儲存在 external pages中

In the following sections, let us see which of the BLOB columns are externally stored and which of them are stored inline, for each sample row we created above. Note that the row format of table t1 is not explicitly specified. In MySQL 5.6, it will default to the COMPACT row format.  Please keep this in mind as we discuss the example.    >>下面我們來看一下，上面表中的每一行記錄中大物件是如何儲存的。注意我們建表時沒有指定row_format ，所以t1表使用 mysql 5.6中的預設值 COMPACT。

Overview of BLOB Storage

The BLOB data can be stored inline in the clustered index record, or it can be stored externally in separate BLOB pages. These external BLOB pages are allocated from the same tablespace in which the clustered index resides. The BLOB data will always be stored inline whenever possible though. If and only if this is not possible because of the record size, then the BLOB field will be stored externally.  This is true for all of the current row formats: REDUNDANT, COMPACT, DYNAMIC, and COMPRESSED. Let’s now take a look at the storage details for the BLOB columns in our example table.    >>innodb引擎下，大物件即可能被儲存在索引頁中也可能被儲存在external pages中(external pages同聚集索引頁在同一個表空間)。在可能的情況下大物件會被儘可能的儲存在索引頁中。當記錄的size 過大，無法保證一個索引頁中儲存最少兩條記錄時，大物件就會被儲存在external pages中。這個原則對當前的row format，REDUNDANT，COMPACT，DYNAMIC，COMPRESSED都是適用的。

In MySQL 5.6, the default row format of a table is COMPACT, so that’s what our t1 table is using. The default page size is 16K, so that’s also what we’re using. The maximum record size that can be stored in a 16K page using the COMPACT row format is 8126 bytes. The function page_get_free_space_of_empty() will tell us the total free space available in a page. The value returned by this function, which is then divided by 2, will give us the maximum allowed record size. The division by 2 is required because an index page must contain a minimum of 2 records. Let’s look at an example (the argument value of “1” tells the function that the row format for the page is COMPACT):    >>Mysql 5.6 下 row format 的預設值是COMPACT，預設的page size是 16k。這時表中單條記錄最大尺寸為 8126 bytes. page_get_free_space_of_empty()能夠計算出page 總得剩餘空間，用該值除以2 即得出記錄的最大尺寸(除以2是因為一個頁中最少要儲存兩個記錄)。在如上條件下，記錄的最大尺寸為8126 bytes

The following table shows the storage details of the BLOB columns for each row in table t1. Keeping in mind the maximum allowed record size, it is clear as to why the 60000 bytes of BLOB data is always stored externally. It simply will not fit within a single clustered index record.    >>下面的表格顯示了測試表中，各行中 f2和f3列如何儲存。正如上面所分析的，單條記錄最大 size 為 8216 bytes，所以size 為60000 bytes的 大物件都被儲存在 external pages中

Inline Storage of BLOBs

As mentioned previously, no BLOB fields will be externally stored if the size of the record is less than the maximum record size allowed in a page. In our example table, for the row with a primary key value of 1, no BLOB fields need to be stored externally because the full record size is less than 8126 bytes. The following tables give the sizing details for each of the rows in our example table:    >>當行的 size小於 8126 bytes時，所有大物件都儲存在索引頁。所以測試表中的第一條記錄中的大物件不需要儲存在 external pages中。下面的表格描述了測試表中各行的具體大小，以及被儲存在external pages中的列

As we can see, the BLOB fields are stored externally until the record size falls below the limit. In the table above, column 2 gives the initial clustered index record size. If this size is greater than the allowed maximum size of the record (shown in column 5), then the function dtuple_convert_big_rec() is invoked to choose the fields destined for external storage. Column 3 lists the fields that have been chosen by this function for external storage. Column 4 shows the clustered index record size after moving the chosen fields to external storage. Again, this value must be less than the maximum record size, shown in column 5 (which is 8126 bytes in our example).    >>上面的表中第二列是每行記錄的 size. 如果行的size 大於 所允許的最大size(本例中是8126 bytes)，那麼會呼叫dtuple_convert_big_rec()函式來選擇哪些列將被儲存在external pages。上面表格的第三列是每行中被儲存在external pages中的列資訊。上面表格的第四列表示除了external pages儲存資料外，主鍵索引的size(這個值肯定是小於 maximum record size的)。上面表格的第五列表示當前條件下maximum record size值

All of the size details provided above are obtained through the debugger from the callers of this function.

For more clarity, let me explain the clustered index record length for the first row with a primary key value of 1  (shown as 6027 bytes). The length of the user fields f1, f2, and f3 are 4 bytes, 3000 bytes, and 3000 bytes respectively. The length of the system fields DB_ROLL_PTR and DB_TRX_ID are 7 bytes and 6 bytes respectively. The record header stores the length of the f2 and f3 fields, taking 4 bytes (2 bytes for each field). The record header also contains a null bit array, which for this record takes up 1 byte. Lastly, the record header contains REC_N_NEW_EXTRA_BYTES—which is 5 bytes for the COMPACT row format—of additional information. The complete storage details for the final record are presented in the following table:    >>下面我們來分析一下測試表中第一條記錄的6027 bytes是如何計算出來的。第一行中，f1，f2，f3的長度分別為4 bytes，3000 bytes，3000 bytes。標識回滾段和事務id的列分別佔7 bytes和6 bytes。record header中還包含 null標識佔1 byte。最後REC_N_NEW_EXTRA_BYTES佔用5 bytes(row format 為 COMPACT時)。這些值加起來 正好為6027 bytes

You can refer to the documentation in  for more details about the COMPACT and REDUNDANT row formats. The REDUNDANT row format is also explained in the blog article InnoDB REDUNDANT Row Format.

Choosing Fields for External Storage

As discussed above, the function dtuple_convert_big_rec() is invoked to decide which parts of the oversized clustered index record will be chosen for external storage. This function makes use of the following rules to decide this:    >>當行的size 大於maximum record size值時，會呼叫dtuple_convert_big_rec()決定哪些大物件被儲存在 external pages。該函式通過如下規則確定哪些大物件會被儲存在external中

• No fixed length fields can be chosen for external storage.    >>固定長度型別的列不會被儲存在 external pages中，如char
• No variable length fields whose size is less than or equal to 40 (2 * BTR_EXTERN_FIELD_REF_SIZE) bytes will be chosen for external storage.  >>可變長度列型別 如果size小於等於40 bytes(2 * BTR_EXTERN_FIELD_REF_SIZE)，不會被儲存在external pages中
• No variable length fields whose size is less than the BLOB prefix size will be chosen for external storage.  This means that in the case of REDUNDANT and COMPACT row formats, if the field data length is less than or equal to 768 bytes (DICT_ANTELOPE_MAX_INDEX_COL_LEN), then it will not be chosen for external storage. This rule is not applicable for DYNAMIC and COMPRESSED row formats, because their BLOB prefix is 0 bytes.    >>可變長度列型別，如果size 小於 BLOB prefix size，不會被儲存在external pages中(這條規則使用於 row_format 為COMPACT或者REDUNDANT 的表，不適於row_format為DYNAMIC 和 COMPRESSED 的表)。

In the function dtuple_convert_big_rec(), we examine one BLOB field at a time for potential external storage, and then move it to external storage if it passes the criteria noted above, until the clustered index record size falls within the maximum allowed. Larger fields will be selected for external storage before smaller fields, to ensure that maximum space savings happens in the clustered index page. This ensures that more records can be stored in each clustered index page.    >>dtuple_convert_big_rec()函式一次檢查一個BLOB field列，通過上面的過濾條件來決定該BLOB field是否被儲存在external pages中，然後再檢查該行中的下一個BLOB field，直到列的size 小於當前條件下的 maximum size。在同一行中，size越大的field 越先被dtuple_convert_big_rec()函式檢查，以確保index pages中能保留更多的資料。

BLOB Reference

When a BLOB field is stored externally, a BLOB reference is stored in the clustered index record. The BLOB reference will be stored after the BLOB prefix, if any. This BLOB reference is 20 bytes, and it contains the following information:    >>>>當一個BLOB field被儲存在external pages時，會在index pages中儲存該BLOB field的BLOB reference(指示該BLOB的size,external page位置等元資料資訊)。如果表的row format為COMPACT或者REDUNDANT，那麼BLOB reference儲存在BLOB prefix之後。BLOB reference 佔用20 bytes空間，包含如下資訊：

1. The space identifier (4 bytes)    >>>>表空間資訊(下圖中的Space ID部分)
2. The page number where the first page of the BLOB is stored (4 bytes)    >>>>儲存BLOB field的起始 page號(下圖中的Page Number部分)
3. The offset of the BLOB header within that page (4 bytes)    >>>>起始page中 BLOB header 的位移量(下圖中的Offset)
4. The total size of the BLOB data (8 bytes)    >>>>該BLOB 的 total size(除了上面三個以外的剩餘部分)

Even though 8 bytes are available to store the total size of the BLOB data, only the last 4 bytes are actually used. This means that within InnoDB, the maximum size of a single BLOB field is currently 4GB.    >>>>儘管上面我們提到有8 bytes的空間可以用來儲存BLOB size資訊，但是實際上僅有 4 bytes空間可以真正用來儲存BLOB size 資訊。這就意味著，innodb中可以儲存的最大 BLOB field size 是4 GB(4 bytes=32 bit ,2^32=4GB)

In the length field, two bits are used to store the ownership and inheritance information, which are not discussed in this article. We will cover that in a subsequent blog article. The most significant bit of the length field is used to store ownership information and the second most significant bit is used to store the inheritance information.    >>>>上圖中最後儲存BLOB size資訊部分，用 2 bits來儲存 ownership和inheritance 資訊，至於ownership和inheritance本文我們不做討論，在下篇的部落格中再做說明。

Here is a gdb function to print the contents of a BLOB reference. This function takes a pointer to the external BLOB reference as an argument. The calls to ntohl() are required because all data on disk is stored in network byte order by InnoDB.   >>>>這裡有一個 gdb 函式，可以用來列印 BLOB reference。

BLOB Prefix

When a BLOB field is stored externally, we may also store a prefix of the value in the clustered index record, depending on the row format used. For the REDUNDANT and COMPACT row formats, a BLOB prefix of 768 bytes is stored in the clustered index record. For the DYNAMIC and COMPRESSED row formats, a BLOB prefix is never stored in the clustered index record. The BLOB prefix would be followed by the BLOB reference.    >>>>如果我們表的row format是COMPACT 或者REDUNDANT，那麼external pages儲存的BLOB field，會在index pages中儲存一個 BLOB prefix(BLOB field最開始的 768 bytes資料)，在BLOB prefix 之後儲存著該BLOB 的BLOB reference。如果表的row format是 DYNAMIC，或者compressed，那麼不會在index pages中儲存BLOB prefix，所有的BLOB資料都被儲存在 external pages中，index page中只保留該BLOB相關的BLOB reference。

The BLOB prefix, when available, helps to calculate the secondary index key without needing to fetch the externally stored BLOB data (which involves at least one extra page load/fetch). This is possible because the maximum length of a secondary index key is 767 bytes. If we attempt to create a secondary index with a bigger length it will be automatically truncated with a warning. For example, consider the following statement:    >>>>