ES 24 - 如何通過Elasticsearch進行聚合檢索 (分組統計)
目錄
- 1 普通聚合分析
- 1.1 直接聚合統計
- 1.2 先檢索, 再聚合
- 1.3 擴充套件: fielddata和keyword的聚合比較
- 2 巢狀聚合
- 2.1 先分組, 再聚合統計
- 2.2 先分組, 再統計, 最後排序
- 2.3 先分組, 組內再分組, 然後統計、排序
1 普通聚合分析
1.1 直接聚合統計
(1) 計算每個tag下的文件數量, 請求語法:
GET book_shop/it_book/_search { "size": 0, // 不顯示命中(hits)的所有文件資訊 "aggs": { "group_by_tags": { // 聚合結果的名稱, 需要自定義(複製時請去掉此註釋) "terms": { "field": "tags" } } } }
(2) 發生錯誤:
說明: 索引book_shop的mapping對映是ES自動建立的, 它把tag解析成了text型別, 在發起對tag的聚合請求後, 將丟擲如下錯誤:
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [tags] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead." } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [......] }, "status": 400 }
(3) 錯誤分析:
錯誤資訊:
Set fielddata=true on [xxxx] ......
錯誤分析: 預設情況下, Elasticsearch 對 text 型別的欄位(field)禁用了 fielddata;
text 型別的欄位在建立索引時會進行分詞處理, 而聚合操作必須基於欄位的原始值進行分析;
所以如果要對 text 型別的欄位進行聚合操作, 就需要儲存其原始值 —— 建立mapping時指定fielddata=true, 以便通過反轉倒排索引(即正排索引)將索引資料載入至記憶體中.
(4) 解決方案一: 對text型別的欄位開啟fielddata屬性:
將要分組統計的text field(即tags)的fielddata設定為true:
PUT book_shop/_mapping/it_book { "properties": { "tags": { "type": "text", "fielddata": true } } }可參考官方文件進行設定:
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/fielddata.html. 成功後的結果如下:{ "acknowledged": true }再次統計, 得到的結果如下:
{ "took": 153, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 4, "max_score": 0.0, "hits": [] }, "aggregations": { "group_by_tags": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 6, "buckets": [ { "key": "java", "doc_count": 3 }, { "key": "程", "doc_count": 2 }, ...... ] } } }
(5) 解決方法二: 使用內建keyword欄位:
開啟fielddata將佔用大量的記憶體.
Elasticsearch 5.x 版本開始支援通過text的內建欄位keyword作精確查詢、聚合分析:
GET shop/it_book/_search { size": 0, "aggs": { "group_by_tags": { "terms": { "field": "tags.keyword" // 使用text型別的內建keyword欄位 } } } }
1.2 先檢索, 再聚合
(1) 統計name中含有“jvm”的圖書中每個tag的文件數量, 請求語法:
GET book_shop/it_book/_search
{
"query": {
"match": { "name": "jvm" }
},
"aggs": {
"group_by_tags": { // 聚合結果的名稱, 需要自定義. 下面使用內建的keyword欄位:
"terms": { "field": "tags.keyword" }
}
}
}(2) 響應結果:
{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.64072424,
"hits" : [
{
"_index" : "book_shop",
"_type" : "it_book",
"_id" : "2",
"_score" : 0.64072424,
"_source" : {
"name" : "深入理解Java虛擬機器：JVM高階特性與最佳實踐",
"author" : "周志明",
"category" : "程式語言",
"desc" : "Java圖書領域公認的經典著作",
"price" : 79.0,
"date" : "2013-10-01",
"publisher" : "機械工業出版社",
"tags" : [
"Java",
"虛擬機器",
"最佳實踐"
]
}
}
]
},
"aggregations" : {
"group_by_tags" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Java",
"doc_count" : 1
},
{
"key" : "最佳實踐",
"doc_count" : 1
},
{
"key" : "虛擬機器",
"doc_count" : 1
}
]
}
}
}1.3 擴充套件: fielddata和keyword的聚合比較
為某個 text 型別的欄位開啟fielddata欄位後, 聚合分析操作會對這個欄位的所有分詞分別進行聚合, 獲得的結果大多數情況下並不符合我們的需求.
使用keyword內建欄位, 不會對相關的分詞進行聚合, 結果可能更有用.
推薦使用text型別欄位的內建keyword進行聚合操作.
2 巢狀聚合
2.1 先分組, 再聚合統計
(1) 先按tags分組, 再計算每個tag下圖書的平均價格, 請求語法:
GET book_shop/it_book/_search
{
"size": 0,
"aggs": {
"group_by_tags": {
"terms": { "field": "tags.keyword" },
"aggs": {
"avg_price": {
"avg": { "field": "price" }
}
}
}
}
}(2) 響應結果:
"hits" : {
"total" : 3,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"group_by_tags" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Java",
"doc_count" : 3,
"avg_price" : {
"value" : 102.33333333333333
}
},
{
"key" : "程式語言",
"doc_count" : 2,
"avg_price" : {
"value" : 114.0
}
},
......
]
}
}2.2 先分組, 再統計, 最後排序
(1) 計算每個tag下圖書的平均價格, 再按平均價格降序排序, 查詢語法:
GET book_shop/it_book/_search
{
"size": 0,
"aggs": {
"all_tags": {
"terms": {
"field": "tags.keyword",
"order": { "avg_price": "desc" } // 根據下述統計的結果排序
},
"aggs": {
"avg_price": {
"avg": { "field": "price" }
}
}
}
}
}(2) 響應結果:
與#2.1節內容相似, 區別在於按照價格排序顯示了.
2.3 先分組, 組內再分組, 然後統計、排序
(1) 先按價格區間分組, 組內再按tags分組, 計算每個tags組的平均價格, 查詢語法:
GET book_shop/it_book/_search
{
"size": 0,
"aggs": {
"group_by_price": {
"range": {
"field": "price",
"ranges": [
{ "from": 00, "to": 100 },
{ "from": 100, "to": 150 }
]
},
"aggs": {
"group_by_tags": {
"terms": { "field": "tags.keyword" },
"aggs": {
"avg_price": {
"avg": { "field": "price" }
}
}
}
}
}
}
}(2) 響應結果:
"hits" : {
"total" : 3,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"group_by_price" : {
"buckets" : [
{
"key" : "0.0-100.0", // 區間0.0-100.0
"from" : 0.0,
"to" : 100.0,
"doc_count" : 1, // 共查詢到了3條文件
"group_by_tags" : { // 對tags分組聚合
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Java",
"doc_count" : 1,
"avg_price" : {
"value" : 79.0
}
},
......
]
}
},
{
"key" : "100.0-150.0",
"from" : 100.0,
"to" : 150.0,
"doc_count" : 2,
"group_by_tags" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Java",
"doc_count" : 2,
"avg_price" : {
"value" : 114.0
}
},
......
}
]
}
}
]
}
}版權宣告
作者: 馬瘦風(https://healchow.com)
出處: 部落格園 馬瘦風的部落格(https://www.cnblogs.com/shoufeng)
感謝閱讀, 如果文章有幫助或啟發到你, 點個[好文要頂