給 Magento 2 新增快取層的分析與嘗試
雖然黑色星期五有驚無險的過去了, 但是 Magento 2 社群版無法讀寫分離這個限制, 始終是懸在整個網站上的一把利劍。
我之前嘗試過給 Magento 2 寫一個 MySQL 讀寫分離的外掛, 在深入研究了 Magento 2 的資料庫訪問層後, 發現通過一個簡單的外掛, 想做到讀寫分離基本上是不可能的。Magento 2 社群版讀寫資料庫的邏輯裡, 混雜著大量的 Magento 1的程式碼和邏輯, 無法在修改少量程式碼的前提下做到讀寫分離, 後來忙著做網站上的各種需求， 於是讀寫分離就擱置了。
這次黑五, 整個專案的效能瓶頸就是 MySQL, 流量上來之後, 應用伺服器負載基本保持不變, 而資料庫伺服器負載卻翻了3倍多, 而且是在資料庫伺服器提前升級了硬體配置的基礎上。所以我覺得 Magento 2 的資料庫層必須要優化一下, 既然沒法做讀寫分離, 那能不能加個快取層呢？將絕大多數讀取操作轉移到快取層去, 理論上資料庫的負載會相應下降。
要想改的程式碼最少, 就得找對地方。 Magento 2 的資料庫 Adapter 是 Magento\Framework\DB\Adapter\Pdo\Mysql 類, 該類繼承自 Zend_Db_Adapter_Abstract
所有獲取資料的方法如下:
Zend_Db_Adapter_Abstract::fetchAll($sql, $bind = array(), $fetchMode = null) Zend_Db_Adapter_Abstract::fetchAssoc($sql, $bind = array()) Zend_Db_Adapter_Abstract::fetchCol($sql, $bind = array()) Zend_Db_Adapter_Abstract::fetchPairs($sql, $bind = array()) Zend_Db_Adapter_Abstract::fetchOne($sql, $bind = array()) Zend_Db_Adapter_Abstract::fetchRow($sql, $bind = array(), $fetchMode = null)
其中, fetchAll() 和 fetchRow() 是用的最多的兩個。
下面以 fetchRow() 為例, 分析該方案的可行性以及實現方法。
/** * Fetches the first row of the SQL result. * Uses the current fetchMode for the adapter. * * @param string|Zend_Db_Select $sql An SQL SELECT statement. * @param mixed $bind Data to bind into SELECT placeholders. * @param mixed $fetchMode Override current fetch mode. * @return mixed Array, object, or scalar depending on fetch mode.*/ public function fetchRow($sql, $bind = array(), $fetchMode = null)
通過解析 $sql 物件和 $bind 陣列， 可以得到精確的、格式化的資料， 包含
1. 資料庫表名
2. 欄位鍵值對
通過這些資料，可以構建快取的鍵(key)和標籤(tag)， 例如：
$cacheKey = table_name::主鍵鍵值對
或者
$cacheKey = table_name::唯一鍵索引鍵值對
$cacheTags = [
table_name,
table_name::主鍵鍵值對
table_name::唯一鍵索引鍵值對組1,
table_name::唯一鍵索引鍵值對組2,
…
]
cacheTags 的作用是給快取分類, 方便後續清理。
有了 $cacheKey, $cacheTags 之後, 就可以將資料庫查詢的結果儲存到快取中去;
下次再有查詢過來, 先在快取中查詢有無對應的資料, 如果有就直接返回給資料呼叫方了;
那麼如果資料更新了呢？
資料更新分為三種: 1. UPDATE, 2. INSERT, 3 DELETE
對於 UPDATE:
/** * Updates table rows with specified data based on a WHERE clause. * * @param mixed $table The table to update. * @param array $bind Column-value pairs. * @param mixed $where UPDATE WHERE clause(s). * @return int The number of affected rows. * @throws Zend_Db_Adapter_Exception */ public function update($table, array $bind, $where = '')
update() 方法接收 3 個引數， 分別是 table_name, 待更新資料鍵值對, where 條件子句。
剛才我們在構建 $cacheTags 時， 分別有 table_name、table_name::主鍵鍵值對、table_name::唯一鍵索引鍵值對, table_name 是現成的， 其餘兩種tag 需要從 where 子句中解析。 通過解析，最壞情況是 where 子句未解析到任何鍵值對, 最好情況是解析到了所有 filed 鍵值對。最壞情況下， 需要清除 table_name 下的所有快取資料， 而最好情況下, 只需要清除一條快取資料。
對於 INSERT:
/** * Inserts a table row with specified data. * * @param mixed $table The table to insert data into. * @param array $bind Column-value pairs. * @return int The number of affected rows. * @throws Zend_Db_Adapter_Exception */ public function insert($table, array $bind)
insert() 方法接收 2 個引數， 分別是 table_name， 待插入資料鍵值對。 由於新插入的資料根本不存在與快取中， 所以不需要對快取進行操作
對於 DELETE:
/** * Deletes table rows based on a WHERE clause. * * @param mixed $table The table to update. * @param mixed $where DELETE WHERE clause(s). * @return int The number of affected rows. */ public function delete($table, $where = '')
delete() 方法接收 2 個引數, table_name 和 where 子句， 假如能從 where 子句中解析到主鍵鍵值對 或 唯一鍵索引鍵值對, 就只需要清除一條快取記錄， 否則需要清除該 table_name 下的所有快取記錄。
優化效果:
我暫時只是用 ab 測試了 Magento 2 的購物車:
ab -C PHPSESSID=acmsj8q8ld1tvdo77lm5t0dr9b -n 40 -c 5 http://localhost/checkout/cart/
沒有快取的時候：
test-No-Cache-1:
Requests per second: 1.79 [#/sec] (mean) Time per request: 2786.478 [ms] (mean) Time per request: 557.296 [ms] (mean, across all concurrent requests) Percentage of the requests served within a certain time (ms) 50% 756 66% 2064 75% 5635 80% 6150 90% 7632 95% 8530 98% 8563 99% 8563 100% 8563 (longest request) MySQL 程序的 CPU 佔用率保持在 20% ~ 24%
test-No-Cache-2:
Requests per second: 1.84 [#/sec] (mean) Time per request: 2720.852 [ms] (mean) Time per request: 544.170 [ms] (mean, across all concurrent requests) Percentage of the requests served within a certain time (ms) 50% 586 66% 1523 75% 4036 80% 5667 90% 10228 95% 11621 98% 12098 99% 12098 100% 12098 (longest request) MySQL 程序的 CPU 佔用率保持在 20% ~ 24%
有快取的時候：
test-With-Cache-1:
Requests per second: 1.99 [#/sec] (mean) Time per request: 2509.273 [ms] (mean) Time per request: 501.854 [ms] (mean, across all concurrent requests) Percentage of the requests served within a certain time (ms) 50% 489 66% 511 75% 574 80% 637 90% 19073 95% 19553 98% 20063 99% 20063 100% 20063 (longest request) MySQL 程序的 CPU 佔用率保持在 5% 左右
test-With-Cache-2:
Requests per second: 2.10 [#/sec] (mean) Time per request: 2384.145 [ms] (mean) Time per request: 476.829 [ms] (mean, across all concurrent requests) Percentage of the requests served within a certain time (ms) 50% 465 66% 472 75% 565 80% 620 90% 9509 95% 18374 98% 18588 99% 18588 100% 18588 (longest request) MySQL 程序的 CPU 佔用率保持在 5% ~ 7 %
通過上面兩組資料的對比， 很明顯 MySQL 的 CPU 佔用率有了大幅度下降(從 20% 下降到 5%), 可見增加一個快取層對降低 MySQL 負載是有效果的。
但是有一個小問題, 在不使用快取的情況下, Percentage of the requests served within a certain time 這個值，在 90% 這個點之後, 表現要比有快取的情況好, 我猜是大量 unserialize() 操作造成 CPU 資源不夠導致響應緩慢。
經過修改後的 vendor/magento/framework/DB/Adapter/Pdo/Mysql.php:
class Mysql extends \Zend_Db_Adapter_Pdo_Mysql implements AdapterInterface { protected $_cache; public function fetchAll($sql, $bind = array(), $fetchMode = null) { if ($sql instanceof \Zend_Db_Select) { /** @var array $from */ $from = $sql->getPart('from'); $tableName = current($from)['tableName']; $cacheKey = 'FETCH_ALL::' . $tableName . '::' . md5((string)$sql); $cache = $this->getCache(); $data = $cache->load($cacheKey); if ($data === false) { $data = parent::fetchAll($sql, $bind, $fetchMode); $cache->save(serialize($data), $cacheKey, ['FETCH_ALL::' . $tableName], 3600); } else { $data = @unserialize($data); } } else { $data = parent::fetchAll($sql, $bind, $fetchMode); } return $data; } public function fetchRow($sql, $bind = [], $fetchMode = null) { $cacheIdentifiers = $this->resolveSql($sql, $bind); if ($cacheIdentifiers !== false) { $cache = $this->getCache()->getFrontend(); $data = $cache->load($cacheIdentifiers['cacheKey']); if ($data === false) { $data = parent::fetchRow($sql, $bind, $fetchMode); if ($data) { $cache->save(serialize($data), $cacheIdentifiers['cacheKey'], $cacheIdentifiers['cacheTags'], 3600); } } else { $data = @unserialize($data); } } else { $data = parent::fetchRow($sql, $bind, $fetchMode); } return $data; } public function update($table, array $bind, $where = '') { parent::update($table, $bind, $where); $cacheKey = $this->resolveUpdate($table, $bind, $where); if ($cacheKey === false) { $cacheKey = $table; } $this->getCache()->clean([$cacheKey, 'FETCH_ALL::' . $table]); } /** * @return \Magento\Framework\App\CacheInterface */ private function getCache() { if ($this->_cache === null) { $objectManager = \Magento\Framework\App\ObjectManager::getInstance(); $this->_cache = $objectManager->get(\Magento\Framework\App\CacheInterface::class); } return $this->_cache; } /** * @param string|\Zend_Db_Select $sql An SQL SELECT statement. * @param mixed $bind Data to bind into SELECT placeholders. * @return array */ protected function resolveSql($sql, $bind = array()) { $result = false; if ($sql instanceof \Zend_Db_Select) { try { /** @var array $from */ $from = $sql->getPart('from'); $tableName = current($from)['tableName']; $where = $sql->getPart('where'); foreach ($this->getIndexFields($tableName) as $indexFields) { $kv = $this->getKv($indexFields, $where, $bind); if ($kv !== false) { $cacheKey = $tableName . '::' . implode('|', $kv); $cacheTags = [ $tableName, $cacheKey ]; $result = ['cacheKey' => $cacheKey, 'cacheTags' => $cacheTags]; } } }catch (\Zend_Db_Select_Exception $e) { } } return $result; } protected function resolveUpdate($tableName, array $bind, $where = '') { $cacheKey = false; if (is_string($where)) { $where = [$where]; } foreach ($this->getIndexFields($tableName) as $indexFields) { $kv = $this->getKv($indexFields, $where, $bind); if ($kv !== false) { $cacheKey = $tableName . '::' . implode('|', $kv); } } return $cacheKey; } protected function getIndexFields($tableName) { $indexes = $this->getIndexList($tableName); $indexFields = []; foreach ($indexes as $data) { if ($data['INDEX_TYPE'] == 'primary') { $indexFields[] = $data['COLUMNS_LIST']; } elseif ($data['INDEX_TYPE'] == 'unique') { $indexFields[] = $data['COLUMNS_LIST']; } } return $indexFields; } protected function getKv($fields, $where, $bind) { $found = true; $kv = []; foreach ($fields as $field) { $_found = false; if (isset($bind[':' . $field])) { // 在 bind 陣列中查詢 filed value $kv[$field] = $field . '=' .$bind[':' . $field]; $_found = true; } elseif (is_array($where)) { foreach ($where as $case) { // 遍歷 where 條件子句， 查詢 filed value $matches = []; $preg = sprintf('#%s.*=(.*)#', $field); $_result = preg_match($preg, $case, $matches); if ($_result) { $kv[$field] = $field . '=' .trim($matches[1], ' \')'); $_found = true; } } } if (!$_found) { // 其中任一 field 沒找到， $found = false; break; } } return $found ? $kv : false; } }