1. 程式人生 > 其它 >ElasticSearch Scroll滾動查詢(查詢資料超過10000)

ElasticSearch Scroll滾動查詢(查詢資料超過10000)

ElasticSearch官方預設查詢資料是10000條,查詢第10001時會報錯:Result window is too large, from + size must be less than or equal to...

以下是From/Size查詢方法的官方文件:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-from-size.html

今天我們重點介紹一下:Scroll滾動查詢

以下是Scroll官方文件:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-scroll.html

程式碼:

@Autowired
private RestHighLevelClient restHighLevelClient;

public void scrollDemo() {

//構造查詢條件
SearchRequest searchRequest = new SearchRequest("索引庫");
SearchSourceBuilder builder = new SearchSourceBuilder();
//設定查詢超時時間
Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
builder.query(QueryBuilders.rangeQuery("欄位").gte("開始時間").lte("結束時間"));
//設定最多一次能夠取出10000筆資料,從第10001筆資料開始,將開啟滾動查詢 PS:滾動查詢也屬於這一次查詢,只不過因為一次查不完,分多次查
builder.size(10000);
searchRequest.source(builder);
//將滾動放入
searchRequest.scroll(scroll);
SearchResponse searchResponse = null;
try {
searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("查詢索引庫失敗", e.getMessage(), e);
}
SearchHits hits = searchResponse.getHits();
//記錄要滾動的ID
String scrollId = searchResponse.getScrollId();

//TODO 對結果集的處理

//滾動查詢部分,將從第10001筆資料開始取
SearchHit[] hitsScroll = hits.getHits();
while (hitsScroll != null && hitsScroll.length > 0 ) {
//構造滾動查詢條件
SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
searchScrollRequest.scroll(scroll);
try {
//響應必須是上面的響應物件,需要對上一層進行覆蓋
searchResponse = restHighLevelClient.scroll(searchScrollRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("滾動查詢失敗",e.getMessage(),e);
}
scrollId = searchResponse.getScrollId();
hits = searchResponse.getHits();
hitsScroll = hits.getHits();

//TODO 同上面完全一致的結果集處理
}

//清除滾動,否則影響下次查詢
ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
ClearScrollResponse clearScrollResponse = null;
try {
clearScrollResponse = restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("滾動查詢清除失敗",e.getMessage(),e);
}
//清除滾動是否成功
boolean succeeded = clearScrollResponse.isSucceeded();
}

當然,除此之外還有一種查詢方法:Search After(它與Scroll非常相似,但是其引數是無狀態的,始終針對最新版本的搜尋器進行解析。如此在後序遍歷的過程中根據索引更新刪除而改變;導致查詢混亂)

詳情可見官方文件:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-search-after.html