(譯)MySQL中的直方圖統計資訊
MySQL的直方圖是如何影響執行計劃生成的?
建立直方圖有哪些注意事項?
直方圖和索引對優化器的選擇上有什麼差異,又該如何選擇?
如何判斷直方圖對執行計劃的影響?
MySQL官方blog的這篇文章用非常具體的示例回答了這一系列問題,let's go。
原文地址為https://dev.mysql.com/blog-archive/histogram-statistics-in-mysql/,以下為譯文: 從MySQL 8.0.3開始,您可以建立直方圖統計資訊,以便向優化器提供更多的統計資訊。在這篇博文中,我們將看看如何建立直方圖統計資料,並解釋何時使用直方圖統計資料可能有用。
什麼是直方圖
查詢優化器是資料庫中負責將SQL查詢轉換為儘可能高效的執行計劃的部分。有時,查詢優化器無法找到最有效的計劃,並最終花費比所需更多的時間執行查詢。出現這種情況的主要原因通常是優化器對它要查詢的資料分佈沒有足夠的瞭解:- 每個表中有多少行?
- 每一列有多少不同的值?
- 資料如何分佈在每一列中?
CREATE TABLE bedtime ( person_id INT, time_of_day TIME);對於“time_of_day”這個欄位,大多數值很可能是在11:00PM左右,因為大多數人是在這個時間段睡覺的。所以下面第一個查詢返回的資料行數要比第二個查詢返回的資料要多。
1在沒有任何統計資料可用的情況下,優化器將假設“time_of_day”中的值是均勻分佈的(即,一個人在下午3點左右睡覺的可能性與晚上11點左右睡覺的可能性相同)。如何使查詢優化器意識到資料中的這種偏斜度?對此的一個解決方案是為該列建立直方圖統計資訊。) SELECT * FROM bedtime WHERE time_of_day BETWEEN "22:00:00" AND "23:59:00" 2) SELECT * FROM bedtime WHERE time_of_day BETWEEN "12:00:00" AND "14:00:00"
直方圖是一列資料分佈的近似值。它可以相當準確地告訴您,您的資料是否有偏差,這反過來將幫助資料庫伺服器理解它所包含的資料的性質。直方圖有很多不同的風格,在MySQL中我們選擇支援兩種不同的型別:“單例(等寬)”直方圖和“等高”直方圖。所有直方圖型別的共同點是,它們將資料集分割為一組“桶”,MySQL自動將值劃分為桶,並自動決定建立什麼型別的直方圖。
如何建立和刪除直方圖統計
為了管理直方圖統計資料,我們擴充套件了ANALYZE TABLE,增加了兩個新的子句:
ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON col_name [, col_name] WITH N BUCKETS; ANALYZE TABLE tbl_name DROP HISTOGRAM ON col_name [, col_name];第一種語法允許你同時為一個或多個列建立直方圖統計資料:
mysql> ANALYZE TABLE payment UPDATE HISTOGRAM ON amount WITH 32 BUCKETS; +----------------+-----------+----------+---------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+---------------------------------------------------+ | sakila.payment | histogram | status | Histogram statistics created for column 'amount'. | +----------------+-----------+----------+---------------------------------------------------+ 1 row in set (0.27 sec) mysql> ANALYZE TABLE payment UPDATE HISTOGRAM ON amount, payment_date WITH 32 BUCKETS; +----------------+-----------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+---------------------------------------------------------+ | sakila.payment | histogram | status | Histogram statistics created for column 'amount'. | | sakila.payment | histogram | status | Histogram statistics created for column 'payment_date'. | +----------------+-----------+----------+---------------------------------------------------------+請注意,必須指定桶的數量,並且可以在 1 到 1024 的範圍內。您應該為資料集選擇多少個桶取決於幾個因素;您有多少個不同的值,您的資料集有多大偏差,您需要多高的準確性等。
但是,在一定數量的桶之後,(再繼續加大桶的資料量)對準確性的提高效果相當低。所以我們建議從較低的數字開始,例如 32,如果您發現它不符合您的需求,則增加它。 在上面的例子中,我們可以看到我們已經為列“amount”構建了兩次直方圖。在第一個查詢中,建立了一個新的直方圖。在第二個查詢中,“amount”的直方圖會自動覆蓋。
如果你想刪除你建立的任何直方圖統計資料,你只需使用DROP histogram語法:
mysql> ANALYZE TABLE payment DROP HISTOGRAM ON payment_date; +----------------+-----------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+---------------------------------------------------------+ | sakila.payment | histogram | status | Histogram statistics removed for column 'payment_date'. | +----------------+-----------+----------+---------------------------------------------------------+與UPDATE HISTOGRAM一樣,您可以在同一個命令中指定多個列。值得注意的一個特性是,ANALYZE TABLE命令將嘗試執行儘可能多的工作,即使在命令執行過程中出現了錯誤。假設您指定了三列,但是第二列不存在。伺服器仍然會為第一和第三列建立和儲存直方圖:
mysql> ANALYZE TABLE customer UPDATE HISTOGRAM ON c_birth_day, c_foobar, c_birth_month WITH 32 BUCKETS; +----------------+-----------+----------+----------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+----------------------------------------------------------+ | tpcds.customer | histogram | status | Histogram statistics created for column 'c_birth_day'. | | tpcds.customer | histogram | status | Histogram statistics created for column 'c_birth_month'. | | tpcds.customer | histogram | Error | The column 'c_foobar' does not exist. | +----------------+-----------+----------+----------------------------------------------------------+ 3 rows in set (0.15 sec)
直方圖的建立在資料庫內部是如何實現的?
如果您已經閱讀了MySQL手冊,您可能已經看到了新的系統變數histogram_generation_max_mem_size。這個變數將控制伺服器在生成直方圖統計資料時允許使用的記憶體大小(以位元組計)。那你為什麼要控制它呢?
當您指定想要構建一個直方圖時,伺服器將把所有資料讀入記憶體並在記憶體中執行所有工作(包括排序)。如果您想在一個非常大的表上生成一個直方圖,那麼您可能要冒著將數百兆位元組的資料讀入記憶體的風險,這可能是不可取的。因此,為了處理這個問題,MySQL將計算在給定由系統變數histogram_generation_max_mem_size指定的記憶體量的情況下,它可以將多少行資料放入記憶體中。如果它意識到它只能在給定的記憶體限制內裝入行的一個子集,它將求助於抽樣。這可以通過檢視屬性“取樣率”來觀察:
mysql> SET histogram_generation_max_mem_size = 1000000; Query OK, 0 rows affected (0.00 sec) mysql> ANALYZE TABLE customer UPDATE HISTOGRAM ON c_birth_country WITH 16 BUCKETS; +----------------+-----------+----------+------------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+------------------------------------------------------------+ | tpcds.customer | histogram | status | Histogram statistics created for column 'c_birth_country'. | +----------------+-----------+----------+------------------------------------------------------------+ 1 row in set (0.22 sec) mysql> SELECT histogram->>'$."sampling-rate"' -> FROM information_schema.column_statistics -> WHERE table_name = "customer" -> AND column_name = "c_birth_country"; +---------------------------------+ | histogram->>'$."sampling-rate"' | +---------------------------------+ | 0.048743243211626014 | +---------------------------------+在這裡,我們可以看到優化器通過讀取“c_birth_country”列中大約4.8%的資料建立了一個直方圖。值得注意的是,抽樣是不確定的,因此如果使用抽樣,在同一個資料集上的兩次後續呼叫“ANALYZE TABLE tbl UPDATE HISTOGRAM…”可能會給您兩個不同的直方圖。
Query examples
那麼,使用直方圖統計可以得到什麼呢?讓我們看看TPC-DS Benchmark Suite中的幾個查詢,其中新增一個直方圖可以在查詢執行時間上產生很大的差異。下面我們將使用規模係數為1的TPC-DS,這意味著資料庫的大小大約為1GB。這臺機器是英特爾酷睿i7-4770,執行Debian Stretch和MySQL 8.0 RC1。這個配置是相當標準的,除了innodb_buffer_pool_size被增加到2G,以便我們可以將整個資料庫放入緩衝池中。
為了讓優化器實際使用直方圖提供的統計資料,您只需確保優化器開關“condition_fanout_filter”處於開啟狀態。注意,這在預設情況下是開啟的。
Query 90
Benchmark Suite 將此查詢描述為“具有特定家屬人數的客戶早上通過網際網路售出的商品數量與晚上售出的商品數量之間的比率是多少。僅考慮具有大量內容的網站。mysql> SELECT CAST(amc AS DECIMAL(15, 4)) / CAST(pmc AS DECIMAL(15, 4)) am_pm_ratio -> FROM (SELECT COUNT(*) amc -> FROM web_sales, -> household_demographics, -> time_dim, -> web_page -> WHERE ws_sold_time_sk = time_dim.t_time_sk -> AND ws_ship_hdemo_sk = household_demographics.hd_demo_sk -> AND ws_web_page_sk = web_page.wp_web_page_sk -> AND time_dim.t_hour BETWEEN 9 AND 9 + 1 -> AND household_demographics.hd_dep_count = 2 -> AND web_page.wp_char_count BETWEEN 5000 AND 5200) at, -> (SELECT COUNT(*) pmc -> FROM web_sales, -> household_demographics, -> time_dim, -> web_page -> WHERE ws_sold_time_sk = time_dim.t_time_sk -> AND ws_ship_hdemo_sk = household_demographics.hd_demo_sk -> AND ws_web_page_sk = web_page.wp_web_page_sk -> AND time_dim.t_hour BETWEEN 15 AND 15 + 1 -> AND household_demographics.hd_dep_count = 2 -> AND web_page.wp_char_count BETWEEN 5000 AND 5200) pt -> ORDER BY am_pm_ratio -> LIMIT 100; +-------------+ | am_pm_ratio | +-------------+ | 1.27619048 | +-------------+ 1 row in set (1.48 sec)View Code 正如我們所見,執行查詢大約需要 1.5 秒。這看起來並不多,但是通過在單個列上新增直方圖,我們可以使該查詢的執行速度提高三倍(為了便於閱讀,查詢被截斷了);
mysql> ANALYZE TABLE web_page UPDATE HISTOGRAM ON wp_char_count WITH 8 BUCKETS; +----------------+-----------+----------+----------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+----------------------------------------------------------+ | tpcds.web_page | histogram | status | Histogram statistics created for column 'wp_char_count'. | +----------------+-----------+----------+----------------------------------------------------------+ 1 row in set (0.06 sec) mysql> SELECT ... +-------------+ | am_pm_ratio | +-------------+ | 1.27619048 | +-------------+ 1 row in set (0.50 sec)View Code 對於這個直方圖,查詢現在大約需要0.5秒。為什麼呢?主要原因可以通過檢視謂詞“web_page”找到。wp_char_count BETWEEN 5000 AND 5200 "。在沒有任何統計資料可用的情況下,優化器假定表“web_page”中有11.11%的行匹配給定的謂詞。然而,這是錯誤的。通過檢查表,我們可以看到只有1.6%匹配這個謂詞(60行中有一行):
mysql> SELECT -> (SELECT COUNT(*) FROM web_page WHERE web_page.wp_char_count BETWEEN 5000 AND 5200) -> / -> (SELECT COUNT(*) FROM web_page) AS ratio; +--------+ | ratio | +--------+ | 0.0167 | +--------+ 1 row in set (0.00 sec)有了直方圖統計資訊,優化器現在知道了這一點,並在連線順序中提前推入表(譯註:原文是pushes the table earlier in the join order,應該是將相關的表選為驅動表,符合小表驅動大表的原則),從而生成執行計劃,執行速度提高三倍。
Query 61
該查詢描述為“查詢給定月份和年份中有促銷和沒有促銷的商品的銷售比例”。只有出售給生活在特定時區的客戶的特定類別的產品才會被考慮。”這是一個包含多個連線的複雜大查詢:mysql> SELECT promotions, -> total, -> CAST(promotions AS DECIMAL(15, 4)) / CAST(total AS DECIMAL(15, 4)) * 100 -> FROM (SELECT SUM(ss_ext_sales_price) promotions -> FROM store_sales, -> store, -> promotion, -> date_dim, -> customer, -> customer_address, -> item -> WHERE ss_sold_date_sk = d_date_sk -> AND ss_store_sk = s_store_sk -> AND ss_promo_sk = p_promo_sk -> AND ss_customer_sk = c_customer_sk -> AND ca_address_sk = c_current_addr_sk -> AND ss_item_sk = i_item_sk -> AND ca_gmt_offset = -5 -> AND i_category = 'Home' -> AND ( p_channel_dmail = 'Y' -> OR p_channel_email = 'Y' -> OR p_channel_tv = 'Y' ) -> AND s_gmt_offset = -5 -> AND d_year = 2000 -> AND d_moy = 12) promotional_sales, -> (SELECT SUM(ss_ext_sales_price) total -> FROM store_sales, -> store, -> date_dim, -> customer, -> customer_address, -> item -> WHERE ss_sold_date_sk = d_date_sk -> AND ss_store_sk = s_store_sk -> AND ss_customer_sk = c_customer_sk -> AND ca_address_sk = c_current_addr_sk -> AND ss_item_sk = i_item_sk -> AND ca_gmt_offset = -5 -> AND i_category = 'Home' -> AND s_gmt_offset = -5 -> AND d_year = 2000 -> AND d_moy = 12) all_sales -> ORDER BY promotions, -> total -> LIMIT 100; +------------+------------+--------------------------------------------------------------------------+ | promotions | total | CAST(promotions AS DECIMAL(15, 4)) / CAST(total AS DECIMAL(15, 4)) * 100 | +------------+------------+--------------------------------------------------------------------------+ | 3213210.07 | 5966836.78 | 53.85114741 | +------------+------------+--------------------------------------------------------------------------+ 1 row in set (2.78 sec)View Code 從輸出中可以看到,執行查詢大約需要2.8秒。然而,查詢優化器沒有意識到列“s_gmt_offset”中只有一個不同的值。在沒有任何統計資料可用的情況下,優化器使用一些硬編碼的估計,這假設10%的行將匹配謂詞“ca_gmt_offset = -5”。如果我們為這一列新增一個直方圖,優化器現在知道表中的所有行都將滿足條件,從而為我們提供一個更好的執行計劃(為了更好的可讀性,查詢被截斷):
mysql> ANALYZE TABLE store UPDATE HISTOGRAM ON s_gmt_offset WITH 8 BUCKETS; +-------------+-----------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +-------------+-----------+----------+---------------------------------------------------------+ | tpcds.store | histogram | status | Histogram statistics created for column 's_gmt_offset'. | +-------------+-----------+----------+---------------------------------------------------------+ 1 row in set (0.06 sec) mysql> SELECT ... +------------+------------+--------------------------------------------------------------------------+ | promotions | total | CAST(promotions AS DECIMAL(15, 4)) / CAST(total AS DECIMAL(15, 4)) * 100 | +------------+------------+--------------------------------------------------------------------------+ | 3213210.07 | 5966836.78 | 53.85114741 | +------------+------------+--------------------------------------------------------------------------+ 1 row in set (1.37 sec)View Code 有了這個直方圖,查詢執行時間下降到不到1.4秒,提高了2倍。原因是在第一個計劃中,優化器選擇第一個派生表在表儲存上執行全表掃描,然後分別在<item、store_sales、date_dim、customer和customer_address中執行主鍵查詢。但是,當它意識到表儲存將返回比它預期的更多的行,而沒有可用的直方圖統計資訊時,優化器選擇對錶項執行全表掃描,並分別在store_sales、store、date_dim、customer和最後的customer_address中執行主鍵查詢。
But, why not an index?
你們中的一些人現在可能認為一個索引也能做得同樣好,事實也的確如此:mysql> CREATE INDEX s_gmt_offset_idx ON store (s_gmt_offset); Query OK, 0 rows affected (0.53 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> SELECT ... +------------+------------+--------------------------------------------------------------------------+ | promotions | total | CAST(promotions AS DECIMAL(15, 4)) / CAST(total AS DECIMAL(15, 4)) * 100 | +------------+------------+--------------------------------------------------------------------------+ | 3213210.07 | 5966836.78 | 53.85114741 | +------------+------------+--------------------------------------------------------------------------+ 1 row in set (1.41 sec)
然而,有兩個原因可以解釋為什麼你可能會考慮直方圖而不是索引:
1. 維護索引是有成本的。如果您有一個索引,那麼每次INSERT/UPDATE/DELETE都會導致索引被更新。這不是免費的,而且會影響您的效能。另一方面,直方圖只建立一次,並且永遠不會更新,除非您明確地要求它。因此,它不會損害您的插入/更新/刪除效能。
2. 如果您有一個索引,優化器將執行我們稱為“index dives” 的操作,以估計給定範圍內的記錄數量。這也有一定的成本,如果查詢中有非常長的in -list,那麼成本可能會太高。在這種情況下,直方圖統計要便宜得多,因此可能更合適。
譯者注:簡單地理解,index dives就是MySQL在對where id in (***,***,……)這種語句生成執行計劃的時候,通過掃描索引頁的方式來估算符合條件的資料行數,這種方式潛在的問題就是,如果符合條件的資料頁面很多,那麼久僅在執行計劃評估截斷,就需要掃描大量的資料頁面,可能會造成一定的效能損耗,如果換一種評估方式,也就是基於統計資訊做評估,就可以避免潛在的掃描大量的索引頁的情況(但是基於統計資訊的預估也不是完美的,最大的問題是不夠精準)。index dives的引數為eq_range_index_dive_limit,預設為200。
檢查直方圖統計資訊
直方圖統計資料作為JSON物件儲存在資料字典中,這使得它們既靈活又可讀。例如,您可以使用內建的JSON函式從直方圖中提取資訊。假設您想知道您的柱狀圖是何時為“payment”表中的“amount”列建立/更新的。你可以很容易地使用JSON反引用提取操作符來查詢這些資訊:mysql> SELECT -> HISTOGRAM->>'$."last-updated"' AS last_updated -> FROM INFORMATION_SCHEMA.COLUMN_STATISTICS -> WHERE -> SCHEMA_NAME = "sakila" -> AND TABLE_NAME = "payment" -> AND COLUMN_NAME = "amount"; +----------------------------+ | last_updated | +----------------------------+ | 2017-09-15 11:54:25.000000 | +----------------------------+或者假設你想找出直方圖中有多少個桶與你在ANALYZE TABLE語句中指定的桶的數量進行比較:
mysql> SELECT -> TABLE_NAME, -> COLUMN_NAME, -> HISTOGRAM->>'$."number-of-buckets-specified"' AS num_buckets_specified, -> JSON_LENGTH(HISTOGRAM, '$.buckets') AS num_buckets_created -> FROM INFORMATION_SCHEMA.COLUMN_STATISTICS -> WHERE -> SCHEMA_NAME = "sakila"; +------------+--------------+-----------------------+---------------------+ | TABLE_NAME | COLUMN_NAME | num_buckets_specified | num_buckets_created | +------------+--------------+-----------------------+---------------------+ | payment | amount | 32 | 19 | | payment | payment_date | 32 | 32 | +------------+--------------+-----------------------+---------------------+
關於可以從直方圖中提取什麼樣的資訊,我們參考了手冊中的更多資訊。
優化器跟蹤
如果你想知道直方圖所做的估計,最簡單的方法是檢視EXPLAIN輸出:mysql> EXPLAIN SELECT * FROM customer WHERE c_birth_day BETWEEN 1 AND 10; +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ | 1 | SIMPLE | customer | NULL | ALL | NULL | NULL | NULL | NULL | 98633 | 11.11 | Using where | +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ 1 row in set, 1 warning (0.00 sec) mysql> ANALYZE TABLE customer UPDATE HISTOGRAM ON c_birth_day WITH 32 BUCKETS; +----------------+-----------+----------+--------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------+-----------+----------+--------------------------------------------------------+ | tpcds.customer | histogram | status | Histogram statistics created for column 'c_birth_day'. | +----------------+-----------+----------+--------------------------------------------------------+ 1 row in set (0.10 sec) mysql> EXPLAIN SELECT * FROM customer WHERE c_birth_day BETWEEN 1 AND 10; +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ | 1 | SIMPLE | customer | NULL | ALL | NULL | NULL | NULL | NULL | 98633 | 32.12 | Using where | +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ 1 row in set, 1 warning (0.00 sec)如果你檢視“過濾”列,你會發現它從預設的11.11%變成了更精確的32.12%。然而,如果你有多個條件,其中一些列有直方圖統計資料,而另一些沒有,這將很難知道優化器已經估計了什麼:
mysql> EXPLAIN SELECT * FROM customer WHERE c_birth_day <= 20 AND c_birth_year = 1967; +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ | 1 | SIMPLE | customer | NULL | ALL | NULL | NULL | NULL | NULL | 98633 | 6.38 | Using where | +----+-------------+----------+------------+------+---------------+------+---------+------+-------+----------+-------------+ 1 row in set, 1 warning (0.00 sec)如果你想更詳細地瞭解直方圖所做的估計,你可以檢視查詢的跟蹤:
mysql> SET OPTIMIZER_TRACE = "enabled=on"; Query OK, 0 rows affected (0.00 sec) mysql> SET OPTIMIZER_TRACE_MAX_MEM_SIZE = 1000000; Query OK, 0 rows affected (0.00 sec) mysql> EXPLAIN SELECT * FROM customer WHERE c_birth_day <= 20 AND c_birth_year = 1967; mysql> SELECT JSON_EXTRACT(TRACE, "$**.filtering_effect") FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE; +----------------------------------------------------------------------------------------+ | JSON_EXTRACT(TRACE, "$**.filtering_effect") | +----------------------------------------------------------------------------------------+ | [[{"condition": "(`customer`.`c_birth_day` <= 20)", "histogram_selectivity": 0.6376}]] | +----------------------------------------------------------------------------------------+ 1 row in set (0.00 sec)這裡我們使用JSON_EXTRACT函式從跟蹤輸出中提取相關部分。在這裡我們可以看到,對於使用直方圖的每個條件,我們可以看到估計的選擇性。在本例中,只對其中一個條件(c_birth_day <= 20)的選擇性進行了估計,並且直方圖估計列中63.76%的行將匹配此條件。事實上,這與列中的實際資料相符的:
mysql> SELECT -> (SELECT count(*) FROM customer WHERE c_birth_day <= 20) -> / -> (SELECT COUNT(*) FROM customer) AS ratio; +--------+ | ratio | +--------+ | 0.6376 | +--------+ 1 row in set (0.03 sec)