技術標籤：clickhouse mysql引擎

原文出處：https://bohutang.me/2020/06/08/clickhouse-and-friends-mysql-protocol-write-stack/

上篇的MySQL Protocol和Read呼叫裡介紹了 ClickHouse 一條查詢語句的呼叫棧，本文繼續介紹寫的呼叫棧，開整。

Write請求

1. 建表:

mysql>CREATETABLEtest(aUInt8,bUInt8,cUInt8)ENGINE=MergeTree()PARTITIONBY(a,b)ORDERBYc;
QueryOK,0rowsaffected(0.03sec)
 

2. 寫入資料：

INSERTINTOtestVALUES(1,1,1),(2,2,2);

呼叫棧分析

1. 獲取儲存引擎 OutputStream

DB::StorageMergeTree::write(std::__1::shared_ptr<:iast>const&,DB::Contextconst&)StorageMergeTree.cpp:174
DB::PushingToViewsBlockOutputStream::PushingToViewsBlockOutputStream(std::__1::shared_ptr<:istorage>const&,DB::Contextconst&,std::__1::shared_ptr<:iast>const&,bool)PushingToViewsBlockOutputStream.cpp:110
 
DB::InterpreterInsertQuery::execute()InterpreterInsertQuery.cpp:229
DB::executeQueryImpl(constchar*,constchar*,DB::Context&,bool,DB::QueryProcessingStage::Enum,bool,DB::ReadBuffer*)executeQuery.cpp:364
DB::executeQuery(DB::ReadBuffer&,DB::WriteBuffer&,bool,DB::Context&,std::__1::function,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&)>)executeQuery.cpp:696
 
DB::MySQLHandler::comQuery(DB::ReadBuffer&)MySQLHandler.cpp:311
DB::MySQLHandler::run()MySQLHandler.cpp:141

2. 從 SQL 組裝 InputStream

(1,1,1), (2,2,2)如何組裝成 inputstream 結構呢？

DB::InputStreamFromASTInsertQuery::InputStreamFromASTInsertQuery(std::__1::shared_ptr<:iast>const&,DB::ReadBuffer*,
DB::InterpreterInsertQuery::execute()InterpreterInsertQuery.cpp:300
DB::executeQueryImpl(charconst*,charconst*,DB::Context&,bool,DB::QueryProcessingStage::Enum,bool,DB::ReadBuffer*)executeQuery.cpp:386
DB::MySQLHandler::comQuery(DB::ReadBuffer&)MySQLHandler.cpp:313
DB::MySQLHandler::run()MySQLHandler.cpp:150

然後

res.in=std::make_shared(query_ptr,nullptr,query_sample_block,context,nullptr);
res.in=std::make_shared(res.in,out_streams.at(0));

通過 NullAndDoCopyBlockInputStream的 copyData 方法構造出 Block：

DB::ValuesBlockInputFormat::readRow(std::__1::vector::mutable_ptr<:icolumn>,std::__1::allocator::mutable_ptr<:icolumn>>>&,unsignedlong)ValuesBlockInputFormat.cpp:93
DB::ValuesBlockInputFormat::generate()ValuesBlockInputFormat.cpp:55
DB::ISource::work()ISource.cpp:48
DB::InputStreamFromInputFormat::readImpl()InputStreamFromInputFormat.h:48
DB::IBlockInputStream::read()IBlockInputStream.cpp:57
DB::InputStreamFromASTInsertQuery::readImpl()InputStreamFromASTInsertQuery.h:31
DB::IBlockInputStream::read()IBlockInputStream.cpp:57
voidDB::copyDataImpl<:copydata>*)::$_0&,void(&)(DB::Blockconst&)>(DB::IBlockInputStream&,DB::IBlockOutputStream&,DB::copyData(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::atomic*)::$_0&,void(&)(DB::Blockconst&))copyData.cpp:26
DB::copyData(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::atomic*)copyData.cpp:62
DB::NullAndDoCopyBlockInputStream::readImpl()NullAndDoCopyBlockInputStream.h:47
DB::IBlockInputStream::read()IBlockInputStream.cpp:57
voidDB::copyDataImpl<:__1::>functionconst&,std::__1::functionconst&>(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::functionconst&,std::__1::functionconst&)copyData.cpp:26
DB::copyData(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::functionconst&,std::__1::functionconst&)copyData.cpp:73
DB::executeQuery(DB::ReadBuffer&,DB::WriteBuffer&,bool,DB::Context&,std::__1::function,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&)>)executeQuery.cpp:785
DB::MySQLHandler::comQuery(DB::ReadBuffer&)MySQLHandler.cpp:313
DB::MySQLHandler::run()MySQLHandler.cpp:150

3. 組裝 OutputStream

DB::InterpreterInsertQuery::execute()InterpreterInsertQuery.cpp:107
DB::executeQueryImpl(constchar*,constchar*,DB::Context&,bool,DB::QueryProcessingStage::Enum,bool,DB::ReadBuffer*)executeQuery.cpp:364
DB::executeQuery(DB::ReadBuffer&,DB::WriteBuffer&,bool,DB::Context&,std::__1::function,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&)>)executeQuery.cpp:696
DB::MySQLHandler::comQuery(DB::ReadBuffer&)MySQLHandler.cpp:311
DB::MySQLHandler::run()MySQLHandler.cpp:141

組裝順序:

1. NullAndDoCopyBlockInputStream
2. CountingBlockOutputStream
3. AddingDefaultBlockOutputStream
4. SquashingBlockOutputStream
5. PushingToViewsBlockOutputStream
6. MergeTreeBlockOutputStream

4. 寫入OutputStream

DB::MergeTreeBlockOutputStream::write(DB::Blockconst&)MergeTreeBlockOutputStream.cpp:17
DB::PushingToViewsBlockOutputStream::write(DB::Blockconst&)PushingToViewsBlockOutputStream.cpp:145
DB::SquashingBlockOutputStream::finalize()SquashingBlockOutputStream.cpp:30
DB::SquashingBlockOutputStream::writeSuffix()SquashingBlockOutputStream.cpp:50
DB::AddingDefaultBlockOutputStream::writeSuffix()AddingDefaultBlockOutputStream.cpp:25
DB::CountingBlockOutputStream::writeSuffix()CountingBlockOutputStream.h:37
DB::copyDataImpl<:copydata>*)::&,void(&)(constDB::Block&)>(DB::IBlockInputStream&,DB::IBlockOutputStream&,&,void(&)(constDB::Block&))copyData.cpp:52
DB::copyData(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::atomic*)copyData.cpp:138
DB::NullAndDoCopyBlockInputStream::readImpl()NullAndDoCopyBlockInputStream.h:57
DB::IBlockInputStream::read()IBlockInputStream.cpp:60
voidDB::copyDataImpl<:__1::>functionconst&,std::__1::functionconst&>(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::functionconst&,std::__1::functionconst&)copyData.cpp:29
DB::copyData(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::functionconst&,std::__1::functionconst&)copyData.cpp:154
DB::executeQuery(DB::ReadBuffer&,DB::WriteBuffer&,bool,DB::Context&,std::__1::function,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&)>)executeQuery.cpp:748
DB::MySQLHandler::comQuery(DB::ReadBuffer&)MySQLHandler.cpp:311
DB::MySQLHandler::run()MySQLHandler.cpp:141

通過 copyData 方法，讓資料在 OutputStream 間層層透傳，一直到 MergeTreeBlockOutputStream。

5. 返回 Client

DB::MySQLOutputFormat::finalize()MySQLOutputFormat.cpp:62
DB::IOutputFormat::doWriteSuffix()IOutputFormat.h:78
DB::OutputStreamToOutputFormat::writeSuffix()OutputStreamToOutputFormat.cpp:18
DB::MaterializingBlockOutputStream::writeSuffix()MaterializingBlockOutputStream.h:22
voidDB::copyDataImpl<:__1::>functionconst&,std::__1::functionconst&>(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::functionconst&,std::__1::functionconst&)copyData.cpp:52
DB::copyData(DB::IBlockInputStream&,DB::IBlockOutputStream&,std::__1::functionconst&,std::__1::functionconst&)copyData.cpp:154
DB::executeQuery(DB::ReadBuffer&,DB::WriteBuffer&,bool,DB::Context&,std::__1::function,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&,std::__1::basic_string,std::__1::allocator>const&)>)executeQuery.cpp:748
DB::MySQLHandler::comQuery(DB::ReadBuffer&)MySQLHandler.cpp:311
DB::MySQLHandler::run()MySQLHandler.cpp:141

總結

INSERTINTOtestVALUES(1,1,1),(2,2,2);

首先核心解析 SQL 語句生成 AST，根據 AST 獲取 Interpreter：InterpreterInsertQuery。其次 Interpreter 依次新增相應的 OutputStream。然後從 InputStream 讀取資料，寫入到 OutputStream，stream 會層層滲透，一直寫到底層的儲存引擎。最後寫入到 Socket Output，返回結果。

ClickHouse 的 OutputStream 編排還是比較複雜，缺少類似 Pipeline 的排程和編排，但是由於模式比較固化，目前看還算清晰。

文內連結

• ClickHouse和他的朋友們(2)MySQL Protocol和Read呼叫棧

全文完。

Enjoy ClickHouse:)

葉老師的「MySQL核心優化」大課已升級到MySQL 8.0，掃碼開啟MySQL 8.0修行之旅吧