timescaledb和PG寫入效能測試
阿新 • • 發佈：2018-11-28
目錄
結論摘要
小資料量（執行緒10萬行），無論單多執行緒，有無batch，PG和TSDB效能相仿，PG略強。
增加一條索引，兩者插入效能都有所下降
WAL增長約為插入資料的2倍，但是WAL有上限。
大資料量（2000萬、1億行），BATCHSIZE=1000，PG插入效能依舊強於TSDB。
CPU佔用，TSDB佔用略高，以8執行緒batch1000為例（TSDB:80%, PG: %60）
總體上單獨PG 效能和資源消耗都優於TSDB。
下一步需要找到在何種場景下TSDB插入效能會優於PG。
測試環境
Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
32G memory
SSD harddisk.
postgresql.conf
max_connections = 256 shared_buffers = 3GB # min 128kB work_mem = 16MB # min 64kB maintenance_work_mem = 256MB # min 1MB wal_buffers = 4MB checkpoint_timeout = 10min # range 30s-1d max_wal_size = 2GB checkpoint_completion_target = 0.9 --------------------- 作者：Kun_Tsai 來源：CSDN 原文：https://blog.csdn.net/jacicson1987/article/details/82988556?utm_source=copy 版權宣告：本文為博主原創文章，轉載請附上博文連結！
資料構造
timestamp 每行遞增1s
id 每行遞增1
10個 integer 隨機數 + 9個隨機長度字串
CASE 1 單TIME索引
timescaledb 不設定索引（預設time索引）
PG 設定time索引
ts1=# \d+ ckts1 Table "public.ckts1" Column | Type | Collation | Nullable | Default | Storage | Stats target | Description --------+--------------------------+-----------+----------+---------+----------+--------------+------------- time | timestamp with time zone | | not null | | plain | | id | integer | | | | plain | | col2 | integer | | | | plain | | col3 | integer | | | | plain | | col4 | integer | | | | plain | | col5 | integer | | | | plain | | col6 | integer | | | | plain | | col7 | integer | | | | plain | | col8 | integer | | | | plain | | col9 | integer | | | | plain | | col10 | integer | | | | plain | | col11 | integer | | | | plain | | col12 | character varying(30) | | | | extended | | col13 | character varying(30) | | | | extended | | col14 | character varying(30) | | | | extended | | col15 | character varying(30) | | | | extended | | col16 | character varying(30) | | | | extended | | col17 | character varying(30) | | | | extended | | col18 | character varying(30) | | | | extended | | col19 | character varying(30) | | | | extended | | col20 | character varying(30) | | | | extended | | Indexes: "ckts1_time_idx" btree ("time" DESC) Triggers: ts_insert_blocker BEFORE INSERT ON ckts1 FOR EACH ROW EXECUTE PROCEDURE _timescaledb_internal.insert_blocker() postgres=# \d+ cts1 Table "public.cts1" Column | Type | Collation | Nullable | Default | Storage | Stats target | Description --------+--------------------------+-----------+----------+---------+----------+--------------+------------- time | timestamp with time zone | | not null | | plain | | id | integer | | | | plain | | col2 | integer | | | | plain | | col3 | integer | | | | plain | | col4 | integer | | | | plain | | col5 | integer | | | | plain | | col6 | integer | | | | plain | | col7 | integer | | | | plain | | col8 | integer | | | | plain | | col9 | integer | | | | plain | | col10 | integer | | | | plain | | col11 | integer | | | | plain | | col12 | character varying(30) | | | | extended | | col13 | character varying(30) | | | | extended | | col14 | character varying(30) | | | | extended | | col15 | character varying(30) | | | | extended | | col16 | character varying(30) | | | | extended | | col17 | character varying(30) | | | | extended | | col18 | character varying(30) | | | | extended | | col19 | character varying(30) | | | | extended | | col20 | character varying(30) | | | | extended | | Indexes: "cts1_time_index" btree ("time")
單行寫入
每執行緒10W行資料。每行約180bytes. 一個執行緒寫入約18MB。
看起來兩者效能相差不大，PG稍強5%~10%， 之前此場景下influxdb的測試資料也放在一起做個對比
| 執行緒平均速率（行/s） | 單執行緒 | 2執行緒 | 4執行緒 | 8執行緒 | 16執行緒 | 32執行緒 |
| timescaledb | 1075 | 1008 | 1026 | 1045 | 923 | 654 |
| PG | 1092 | 1020 | 1070 | 1120 | 982 | 737 |
| infuxdb | 422 | 378 | 398 | 370 | 341 | 305 |
| 總速率（行/s） | 單執行緒 | 2執行緒 | 4執行緒 | 8執行緒 | 16執行緒 | 32執行緒 |
| timescaledb | 1075 | 2011 | 4000 | 8166 | 14358 | 20337 |
| PG | 1092 | 2037 | 4037 | 8734 | 15087 | 22445 |
| infuxdb | 422 | 752 | 1588 | 2947 | 5444 | 9732 |
WAL檔案大小兩者基本一致，都是大約資料量的 2倍。WAL檔案增加
| 執行緒數 | 位元組數 | PG_WAL_SIZE | TS_WAL_SIZE |
| 1 | 18697562 | 37428272 | 37464824 |
| 2 | 37379223 | 74547144 | 75026672 |
| 4 | 74785548 | 149803536 | 149525488 |
| 8 | 149548561 | 298455240 | 298400040 |
| 16 | 299134911 | 596227840 | 596302792 |
| 32 | 598385115 | 1191610952 | 1194038480 |
BATCH寫入
單執行緒行數設為1000萬， BATCHSIZE 設為1000
PG優於TIMESCALEDB。
| 每執行緒1000W行資料 | batchsize=1000 | |
| 執行緒平均速率（行/s） | 單執行緒 | 8執行緒 |
| timescaledb | 26732 | 11916 |
| PG | 39113 | 14972 |
| 總速率（行/s） | 單執行緒 | 8執行緒 |
| timescaledb | 26732 | 95140 |
| PG | 39113 | 119528 |
資源佔用
TDSB 32執行緒， CPU平均佔用 57%。 比PG高10%。
PG 32執行緒， CPU平均佔用 46%.
CASE 2 增加一個索引
增加id為索引
ts1=# \d+ ckts1
Table "public.ckts1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+--------------------------+-----------+----------+---------+----------+--------------+-------------
time | timestamp with time zone | | not null | | plain | |
id | integer | | | | plain | |
col2 | integer | | | | plain | |
col3 | integer | | | | plain | |
col4 | integer | | | | plain | |
col5 | integer | | | | plain | |
col6 | integer | | | | plain | |
col7 | integer | | | | plain | |
col8 | integer | | | | plain | |
col9 | integer | | | | plain | |
col10 | integer | | | | plain | |
col11 | integer | | | | plain | |
col12 | character varying(30) | | | | extended | |
col13 | character varying(30) | | | | extended | |
col14 | character varying(30) | | | | extended | |
col15 | character varying(30) | | | | extended | |
col16 | character varying(30) | | | | extended | |
col17 | character varying(30) | | | | extended | |
col18 | character varying(30) | | | | extended | |
col19 | character varying(30) | | | | extended | |
col20 | character varying(30) | | | | extended | |
Indexes:
"ckts1_id_idx" btree (id)
"ckts1_time_idx" btree ("time" DESC)
Triggers:
ts_insert_blocker BEFORE INSERT ON ckts1 FOR EACH ROW EXECUTE PROCEDURE _timescaledb_internal.insert_blocker()
ts1=#
postgres=# \d+ cts1
Table "public.cts1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+--------------------------+-----------+----------+---------+----------+--------------+-------------
time | timestamp with time zone | | not null | | plain | |
id | integer | | | | plain | |
col2 | integer | | | | plain | |
col3 | integer | | | | plain | |
col4 | integer | | | | plain | |
col5 | integer | | | | plain | |
col6 | integer | | | | plain | |
col7 | integer | | | | plain | |
col8 | integer | | | | plain | |
col9 | integer | | | | plain | |
col10 | integer | | | | plain | |
col11 | integer | | | | plain | |
col12 | character varying(30) | | | | extended | |
col13 | character varying(30) | | | | extended | |
col14 | character varying(30) | | | | extended | |
col15 | character varying(30) | | | | extended | |
col16 | character varying(30) | | | | extended | |
col17 | character varying(30) | | | | extended | |
col18 | character varying(30) | | | | extended | |
col19 | character varying(30) | | | | extended | |
col20 | character varying(30) | | | | extended | |
Indexes:
"cst1_id_index" btree (id)
"cts1_time_index" btree ("time")
單行寫入
10萬行資料，單條寫入
和沒有新增索引時效能差不多，PG看起來效能下降更多一點。
| 總速率（行/s） | 單執行緒 |
| timescaledb | 1051 |
| PG | 1019 |
BATCH寫入
每執行緒1000萬行資料，batchsize設為1000， 8執行緒處理，13.9G資料
增加一個索引以後，PG和TSDB比之前效能都有所降低，PG寫入仍舊比TSDB快。
| 每執行緒1000W行資料 | batchsize=1000 | |
| 執行緒平均速率（行/s） | 單執行緒 | 8執行緒 |
| timescaledb | 26142 | 10591 |
| PG | 36861 | 13037 |
| 總速率（行/s） | 單執行緒 | 8執行緒 |
| timescaledb | 26142 | 84588 |
| PG | 36861 | 103806 |
資源佔用
8執行緒,batchsize=1000, TSDB CPU佔用80%左右
8執行緒,batchsize=1000, PG CPU佔用60%左右
CASE 3 大量資料
設定PG的表cts1索引time為DESC（倒序)，測試結果與上面並無不同
postgres=# \d+ cts1
Table "public.cts1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+--------------------------+-----------+----------+---------+----------+--------------+-------------
time | timestamp with time zone | | not null | | plain | |
id | integer | | | | plain | |
col2 | integer | | | | plain | |
col3 | integer | | | | plain | |
col4 | integer | | | | plain | |
col5 | integer | | | | plain | |
col6 | integer | | | | plain | |
col7 | integer | | | | plain | |
col8 | integer | | | | plain | |
col9 | integer | | | | plain | |
col10 | integer | | | | plain | |
col11 | integer | | | | plain | |
col12 | character varying(30) | | | | extended | |
col13 | character varying(30) | | | | extended | |
col14 | character varying(30) | | | | extended | |
col15 | character varying(30) | | | | extended | |
col16 | character varying(30) | | | | extended | |
col17 | character varying(30) | | | | extended | |
col18 | character varying(30) | | | | extended | |
col19 | character varying(30) | | | | extended | |
col20 | character varying(30) | | | | extended | |
Indexes:
"cts1_id_idx" btree (id)
"cts1_time_idx" btree ("time" DESC)
ts1=# \d+ ckts1
Table "public.ckts1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+--------------------------+-----------+----------+---------+----------+--------------+-------------
time | timestamp with time zone | | not null | | plain | |
id | integer | | | | plain | |
col2 | integer | | | | plain | |
col3 | integer | | | | plain | |
col4 | integer | | | | plain | |
col5 | integer | | | | plain | |
col6 | integer | | | | plain | |
col7 | integer | | | | plain | |
col8 | integer | | | | plain | |
col9 | integer | | | | plain | |
col10 | integer | | | | plain | |
col11 | integer | | | | plain | |
col12 | character varying(30) | | | | extended | |
col13 | character varying(30) | | | | extended | |
col14 | character varying(30) | | | | extended | |
col15 | character varying(30) | | | | extended | |
col16 | character varying(30) | | | | extended | |
col17 | character varying(30) | | | | extended | |
col18 | character varying(30) | | | | extended | |
col19 | character varying(30) | | | | extended | |
col20 | character varying(30) | | | | extended | |
Indexes:
"ckts1_id_idx" btree (id)
"ckts1_time_idx" btree ("time" DESC)
Triggers:
ts_insert_blocker BEFORE INSERT ON ckts1 FOR EACH ROW EXECUTE PROCEDURE _timescaledb_internal.insert_blocker()懷疑因為資料量不夠大，導致TSDB寫入效能優勢沒能體現
增大資料量到1億行
batchsize = 1000，單執行緒寫入
| 行數 | 位元組數 | PG執行時間 | TSDB執行時間 | PG行速率 | TSDB行速率(行/s) | TSDB TRUNK數量 |
| 2000萬 | 3.45GB | 565 s | 817 s | 35398 (行/s) | 24480 (行/s) | 30 |
| 1億 | 17.4GB | 2782 s | 3992 s | 35945 (行/s) | 25050 (行/s) | 166 |
結果仍然是PG寫入速度更快。
To be continued...