1. 程式人生 > >toast 技術之一

toast 技術之一

os: centos 7.4
db: postgresql 9.6

TOAST的全稱是: 超尺寸屬性儲存技術(The Oversized-Attribute Storage Technique)。
PostgreSQL使用固定的頁面尺寸(通常是8kB),並且不允許元組跨越多個額頁面,因此不可能直接儲存非常大的域值。
為了克服這個限制,大的域值會被壓縮並/或分解成多個物理行。這些處理對使用者都是透明的,只是在大部分的後端程式碼上有一些小的影響。
這個技術的暱稱是TOAST(或者"切片面包之後的最好的東西")。
TOAST 機制也被用來提升記憶體中大型資料值的處理。

線外磁碟上 TOAST 儲存

TOAST程式碼程式碼識別四種不同的在磁碟上儲存可TOAST列的策略:

PLAIN 避免壓縮或者線外儲存;而且它禁用變長型別的單位元組頭部。這是不可TOAST資料型別列的唯一可能的策略。只是對那些不能TOAST的資料型別才有可能。

EXTENDED 允許壓縮和線外儲存。這是大多數可TOAST資料型別的預設策略。 首先將嘗試進行壓縮,如果行仍然太大,那麼則進行線外儲存。

EXTERNAL 允許線外儲存,但是不許壓縮。使用EXTERNAL將令那些在寬text和 bytea列上的子串操作更快(代價是增加了儲存空間), 因此這些操作被優化為只抓取未壓縮線外資料中需要的部分。

MAIN 允許壓縮,但不允許線外儲存(實際上,在這樣的列上仍然會進行線外儲存,但只是作為沒有辦法把行變得足以放入一頁的情況下的最後手段)。

每個可TOAST的資料型別都為該資料型別的列指定了一個預設策略, 但是一個給定表的列的儲存策略可以用ALTER TABLE SET STORAGE修改。

實踐

postgres=# create table tmp_t0(c0 varchar(100),c1 text);
CREATE TABLE
postgres=# insert into tmp_t0 select proname,prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc from pg_proc where char_length(prosrc) > 1000;
INSERT 0 1
postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;

postgres=# 
postgres=# \d+
                    List of relations
 Schema |  Name  | Type  |  Owner   | Size  | Description 
--------+--------+-------+----------+-------+-------------
 public | tmp_t0 | table | postgres | 72 kB | 
(1 row)

postgres=# \d+ tmp_t0
                                Table "public.tmp_t0"
 Column |          Type          | Modifiers | Storage  | Stats target | Description 
--------+------------------------+-----------+----------+--------------+-------------
 c0     | character varying(100) |           | extended |              | 
 c1     | text                   |           | extended |              |
 

接下來檢視 pg_class

postgres=# select pg_relation_filenode(pc.relname::varchar) as fs_filenode, 
                  pg_relation_filepath(pc.relname::varchar) as fs_filepath,
				  pc.oid,pc.relname,pc.relfilenode,
				  pc.relpages,pc.reltuples,
				  pc.reltoastrelid 
		     from pg_class pc 
			where 1=1 
			  and pc.relname='tmp_t0'
			  and pc.reltoastrelid <> 0
			  ;

 fs_filenode |   fs_filepath    |  oid  | relname | relfilenode | relpages | reltuples | reltoastrelid 
-------------+------------------+-------+---------+-------------+----------+-----------+---------------
       25177 | base/13323/25177 | 25177 | tmp_t0  |       25177 |        0 |         0 |         25180
(1 row)


postgres=# select pg_relation_filenode(pc.relname::varchar) as fs_filenode, 
                  pg_relation_filepath(pc.relname::varchar) as fs_filepath,
				  pc.oid,pc.relname,pc.relfilenode,
				  pc.relpages,pc.reltuples,
				  pc.reltoastrelid,
                  '####'::varchar as flag,
                  pc1.oid as toast_oid,
				  pc1.relname as toast_relname,
				  pc1.reltoastrelid as toast_reltoastrelid
		     from pg_class pc,
                  pg_class pc1			 
			where 1=1 
			  and pc.relname='tmp_t0'
			  and pc.reltoastrelid <> 0
              and pc.reltoastrelid=pc1.oid
			  ;

 fs_filenode |   fs_filepath    |  oid  | relname | relfilenode | relpages | reltuples | reltoastrelid | flag | toast_oid | toast_relname  | toast_reltoastrelid 
-------------+------------------+-------+---------+-------------+----------+-----------+---------------+------+-----------+----------------+---------------------
       25177 | base/13323/25177 | 25177 | tmp_t0  |       25177 |        0 |         0 |         25180 | #### |     25180 | pg_toast_25177 |                   0
(1 row)

繼續多插入些資料

postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;

postgres=# \d+
                    List of relations
 Schema |  Name  | Type  |  Owner   | Size  | Description 
--------+--------+-------+----------+-------+-------------
 public | tmp_t0 | table | postgres | 94 MB | 
(1 row)

postgres=# select t0.c0,char_length(t0.c1) from tmp_t0 t0 order by char_length(t0.c1) desc limit 5;
    c0    | char_length 
----------+-------------
 ts_debug |    54558720
 ts_debug |    27279360
 ts_debug |    27279360
 ts_debug |    27279360
 ts_debug |    27279360
(5 rows)

postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;
 insert into tmp_t0 select c0,c1||c1 from tmp_t0;

postgres=# \d+
                     List of relations
 Schema |  Name  | Type  |  Owner   |  Size   | Description 
--------+--------+-------+----------+---------+-------------
 public | tmp_t0 | table | postgres | 2468 MB | 
(1 row) 

查詢更廣泛些

postgres=# select pc.oid,pc.relname,pc.relfilenode,
				  pc.relpages,pc.reltuples,
				  pc.reltoastrelid 
		     from pg_class pc 
			where 1=1 
			  and pc.relname like '%toast%25177%'
			  ;
			  
  oid  |       relname        | relfilenode | relpages | reltuples | reltoastrelid 
-------+----------------------+-------------+----------+-----------+---------------
 25180 | pg_toast_25177       |       25180 |        0 |         0 |             0
 25182 | pg_toast_25177_index |       25182 |        1 |         0 |             0
(2 rows)

$ ls -l |grep -i 25177;ls -l |grep -i 25180 ;ls -l |grep -i 25182;
-rw------- 1 postgres postgres    2883584 Nov 25 01:05 25177
-rw------- 1 postgres postgres      24576 Nov 25 01:01 25177_fsm
-rw------- 1 postgres postgres 1073741824 Nov 25 00:48 25180
-rw------- 1 postgres postgres 1073741824 Nov 25 01:01 25180.1
-rw------- 1 postgres postgres  409100288 Nov 25 01:06 25180.2
-rw------- 1 postgres postgres     647168 Nov 25 01:05 25180_fsm
-rw------- 1 postgres postgres   28196864 Nov 25 01:06 25182

參考:
http://postgres.cn/docs/9.6/storage-toast.html