toast 技術之一
阿新 • • 發佈:2018-11-28
os: centos 7.4
db: postgresql 9.6
TOAST的全稱是: 超尺寸屬性儲存技術(The Oversized-Attribute Storage Technique)。
PostgreSQL使用固定的頁面尺寸(通常是8kB),並且不允許元組跨越多個額頁面,因此不可能直接儲存非常大的域值。
為了克服這個限制,大的域值會被壓縮並/或分解成多個物理行。這些處理對使用者都是透明的,只是在大部分的後端程式碼上有一些小的影響。
這個技術的暱稱是TOAST(或者"切片面包之後的最好的東西")。
TOAST 機制也被用來提升記憶體中大型資料值的處理。
線外磁碟上 TOAST 儲存
TOAST程式碼程式碼識別四種不同的在磁碟上儲存可TOAST列的策略:
PLAIN 避免壓縮或者線外儲存;而且它禁用變長型別的單位元組頭部。這是不可TOAST資料型別列的唯一可能的策略。只是對那些不能TOAST的資料型別才有可能。
EXTENDED 允許壓縮和線外儲存。這是大多數可TOAST資料型別的預設策略。 首先將嘗試進行壓縮,如果行仍然太大,那麼則進行線外儲存。
EXTERNAL 允許線外儲存,但是不許壓縮。使用EXTERNAL將令那些在寬text和 bytea列上的子串操作更快(代價是增加了儲存空間), 因此這些操作被優化為只抓取未壓縮線外資料中需要的部分。
MAIN 允許壓縮,但不允許線外儲存(實際上,在這樣的列上仍然會進行線外儲存,但只是作為沒有辦法把行變得足以放入一頁的情況下的最後手段)。
每個可TOAST的資料型別都為該資料型別的列指定了一個預設策略, 但是一個給定表的列的儲存策略可以用ALTER TABLE SET STORAGE修改。
實踐
postgres=# create table tmp_t0(c0 varchar(100),c1 text); CREATE TABLE postgres=# insert into tmp_t0 select proname,prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc from pg_proc where char_length(prosrc) > 1000; INSERT 0 1 postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0; insert into tmp_t0 select c0,c1||c1 from tmp_t0; insert into tmp_t0 select c0,c1||c1 from tmp_t0; insert into tmp_t0 select c0,c1||c1 from tmp_t0; postgres=# postgres=# \d+ List of relations Schema | Name | Type | Owner | Size | Description --------+--------+-------+----------+-------+------------- public | tmp_t0 | table | postgres | 72 kB | (1 row) postgres=# \d+ tmp_t0 Table "public.tmp_t0" Column | Type | Modifiers | Storage | Stats target | Description --------+------------------------+-----------+----------+--------------+------------- c0 | character varying(100) | | extended | | c1 | text | | extended | |
接下來檢視 pg_class
postgres=# select pg_relation_filenode(pc.relname::varchar) as fs_filenode,
pg_relation_filepath(pc.relname::varchar) as fs_filepath,
pc.oid,pc.relname,pc.relfilenode,
pc.relpages,pc.reltuples,
pc.reltoastrelid
from pg_class pc
where 1=1
and pc.relname='tmp_t0'
and pc.reltoastrelid <> 0
;
fs_filenode | fs_filepath | oid | relname | relfilenode | relpages | reltuples | reltoastrelid
-------------+------------------+-------+---------+-------------+----------+-----------+---------------
25177 | base/13323/25177 | 25177 | tmp_t0 | 25177 | 0 | 0 | 25180
(1 row)
postgres=# select pg_relation_filenode(pc.relname::varchar) as fs_filenode,
pg_relation_filepath(pc.relname::varchar) as fs_filepath,
pc.oid,pc.relname,pc.relfilenode,
pc.relpages,pc.reltuples,
pc.reltoastrelid,
'####'::varchar as flag,
pc1.oid as toast_oid,
pc1.relname as toast_relname,
pc1.reltoastrelid as toast_reltoastrelid
from pg_class pc,
pg_class pc1
where 1=1
and pc.relname='tmp_t0'
and pc.reltoastrelid <> 0
and pc.reltoastrelid=pc1.oid
;
fs_filenode | fs_filepath | oid | relname | relfilenode | relpages | reltuples | reltoastrelid | flag | toast_oid | toast_relname | toast_reltoastrelid
-------------+------------------+-------+---------+-------------+----------+-----------+---------------+------+-----------+----------------+---------------------
25177 | base/13323/25177 | 25177 | tmp_t0 | 25177 | 0 | 0 | 25180 | #### | 25180 | pg_toast_25177 | 0
(1 row)
繼續多插入些資料
postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
postgres=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------+-------+----------+-------+-------------
public | tmp_t0 | table | postgres | 94 MB |
(1 row)
postgres=# select t0.c0,char_length(t0.c1) from tmp_t0 t0 order by char_length(t0.c1) desc limit 5;
c0 | char_length
----------+-------------
ts_debug | 54558720
ts_debug | 27279360
ts_debug | 27279360
ts_debug | 27279360
ts_debug | 27279360
(5 rows)
postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
postgres=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------+-------+----------+---------+-------------
public | tmp_t0 | table | postgres | 2468 MB |
(1 row)
查詢更廣泛些
postgres=# select pc.oid,pc.relname,pc.relfilenode,
pc.relpages,pc.reltuples,
pc.reltoastrelid
from pg_class pc
where 1=1
and pc.relname like '%toast%25177%'
;
oid | relname | relfilenode | relpages | reltuples | reltoastrelid
-------+----------------------+-------------+----------+-----------+---------------
25180 | pg_toast_25177 | 25180 | 0 | 0 | 0
25182 | pg_toast_25177_index | 25182 | 1 | 0 | 0
(2 rows)
$ ls -l |grep -i 25177;ls -l |grep -i 25180 ;ls -l |grep -i 25182;
-rw------- 1 postgres postgres 2883584 Nov 25 01:05 25177
-rw------- 1 postgres postgres 24576 Nov 25 01:01 25177_fsm
-rw------- 1 postgres postgres 1073741824 Nov 25 00:48 25180
-rw------- 1 postgres postgres 1073741824 Nov 25 01:01 25180.1
-rw------- 1 postgres postgres 409100288 Nov 25 01:06 25180.2
-rw------- 1 postgres postgres 647168 Nov 25 01:05 25180_fsm
-rw------- 1 postgres postgres 28196864 Nov 25 01:06 25182