elasticsearch--搜尋_Java基礎使用
如轉載請申明來源
一、搜尋示例
a) 測試資料準備
curl -XPUT localhost:9200/my_index/my_type/_bulk -d '
{ "index": { "_id": 1 }}
{ "title": "The quick brown fox" , "age":"18"}
{ "index": { "_id": 2 }}
{ "title": "The quick brown fox jumps over the lazy dog" , "age":"20" }
{ "index": { "_id": 3 }}
{ "title": "The quick brown fox jumps over the quick dog" , " b) 查詢引數說明
請求示例, 查詢index名為my_index、type名為my_type下所有的資料
from、size: 用於分頁，從第0條開始，取10條資料
sort: 排序的條件
aggs: 聚合分析的條件，與aggregations等價
bool: 用於組合多個查詢條件，後面的內容會講解
curl -XPOST localhost:9200/my_index/my_type/_search?pretty=true -d '
{
" 返回結果：
took: 本次請求處理耗費的時間(單位:ms)
time_out: 請求處理是否超時。tip:如果查詢超時，將返回已獲取的結果，而不是終止查詢
_shards:本次請求涉及的分片資訊，共5個分片處理，成功5個，失敗0個
hits:查詢結果資訊
hits.total: 滿足查詢條件總的記錄數
hits.max_score: 最大評分(相關性)，因為本次沒有查詢條件，所以沒有相關性評分，每條記錄的評分均為1分(_score=1)
hits.hits: 本次查詢返回的結果, 即從from到min(from+size,hits.total)的結果集
hits.hits._score: 本條記錄的相關度評分，因為本次沒有查詢條件，所以沒有相關性評分，每條記錄的評分均為1分
hits.hits._source: 每條記錄的原資料
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 1.0,
"hits" : [ {
"_index" : "my_index",
"_type" : "my_type",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"title" : "The quick brown fox jumps over the lazy dog",
"age" : "20"
}
}, {
"_index" : "my_index",
"_type" : "my_type",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"title" : "Brown fox brown dog",
"age" : "18"
}
}, {
"_index" : "my_index",
"_type" : "my_type",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"title" : "The quick brown fox",
"age" : "18"
}
}, {
"_index" : "my_index",
"_type" : "my_type",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"title" : "The quick brown fox jumps over the quick dog",
"age" : "19"
}
} ]
}
}c) java查詢程式碼
Client client = ConnectionUtil.getLocalClient();
SearchRequestBuilder requestBuilder =
client.prepareSearch("my_index").setTypes("my_type")
.setFrom(0).setSize(10);
Log.debug(requestBuilder);
SearchResponse response = requestBuilder.get();
Log.debug(response);二. 不同搜尋/過濾關鍵字介紹
term, terms, range, exists, missing
match, match_all, multi_match
高亮搜尋、scroll、排序
a) term
主要用於精確匹配，如數值、日期、布林值或未經分析的字串(not_analyzed)
{ "term": { "age": 26 }}
{ "term": { "date": "2014-09-01" }}
{ "term": { "public": true }}
{ "term": { "tag": "full_text" }}Java程式碼:
QueryBuilder ageBuilder = QueryBuilders.termQuery("age", "10");b) terms
和term有點類似，可以允許指定多個匹配條件。如果指定了多個條件，文件會去匹配多個條件，多個條件直接用or連線。以下表示查詢title中包含內容dog或jumps的記錄
{
"terms": {
"title": [ "dog", "jumps" ]
}
}等效於：
"bool" : {
"should" : [ {
"term" : {
"title" : "dog"
}
}, {
"term" : {
"title" : "jumps"
}
} ]
}Java程式碼：
QueryBuilder builder = QueryBuilders.termsQuery("title", "dog", "jumps");
// 與termsQuery等效
builder = QueryBuilders.boolQuery().should(QueryBuilders.termQuery("title", "dog")).should(QueryBuilders.termQuery("title", "jumps"));c) range
允許我們按照指定範圍查詢一批資料。數值、字串、日期等
數值：
{
"range": {
"age": {
"gte": 20,
"lt": 30
}
}
}日期：
"range" : {
"timestamp" : {
"gt" : "2014-01-01 00:00:00",
"lt" : "2014-01-07 00:00:00"
}
}當用於日期欄位時，range 過濾器支援日期數學操作。例如，我們想找到所有最近一個小時的文件：
"range" : {
"timestamp" : {
"gt" : "now-1h"
}
}日期計算也能用於實際的日期，而不是僅僅是一個像 now 一樣的佔位符。只要在日期後加上雙豎線 ||，就能使用日期數學表示式了。
"range" : {
"timestamp" : {
"gt" : "2014-01-01 00:00:00",
"lt" : "2014-01-01 00:00:00||+1M" <1>
}
}<1> 早於 2014 年 1 月 1 號加一個月
範圍操作符包含：
gt :: 大於
gte:: 大於等於
lt :: 小於
lte:: 小於等於
Java程式碼：
QueryBuilders.rangeQuery("age").gte(18).lt(20);過濾字串時，字串訪問根據字典或字母順序來計算。例如，這些值按照字典順序排序：
5, 50, 6, B, C, a, ab, abb, abc, b
Tip: 使用range過濾/查詢時，數字和日期欄位的索引方式讓他們在計算範圍時十分高效。但對於字串來說卻不是這樣。為了在字串上執行範圍操作，Elasticsearch 會在這個範圍內的每個短語執行 term 操作。這比日期或數字的範圍操作慢得多。
+
字串範圍適用於一個基數較小的欄位，一個唯一短語個數較少的欄位。你的唯一短語數越多，搜尋就越慢。
d) exists, missing
exists和missing過濾可以用於查詢文件中是否包含指定欄位或沒有某個欄位，類似於SQL語句中的is not null和is null條件
目前es不推薦使用missing過濾, 使用bool.mustNot + exists來替代
{
"exists": {
"field": "title"
}
}
{
"missing": {
"field": "title"
}
}
"bool" : {
"must_not" : {
"exists" : {
"field" : "title"
}
}
} Java程式碼：
// exits
QueryBuilder builder = QueryBuilders.existsQuery("title");
// missing
builder = QueryBuilders.missingQuery("title");
// instead of missing
builder = QueryBuilders.boolQuery().mustNot(QueryBuilders.existsQuery("title"));e) match, match_all, multi_match
match_all用於查詢所有內容，沒有指定查詢條件
{
"match_all": {}
}常用與合併過濾或查詢結果。
match查詢是一個標準查詢，全文查詢或精確查詢都可以用到他
如果你使用 match 查詢一個全文字欄位，它會在真正查詢之前用分析器先分析match一下查詢字元。使用match查詢字串時，查詢關鍵字和查詢目標均會進行分析(和指定的分詞器有關)，指定not_analyzed除外。
{
"match": {
"tweet": "About Search"
}
}如果用match下指定了一個確切值，在遇到數字，日期，布林值或者not_analyzed 的字串時，它將為你搜索你給定的值：
{ "match": { "age": 26 }}
{ "match": { "date": "2014-09-01" }}
{ "match": { "public": true }}
{ "match": { "tag": "full_text" }}match引數type、operator、minimum_should_match壽命
type取值
boolean: 分析後進行查詢
phrase: 確切的匹配若干個單詞或短語, 如title: “brown dog”, 則查詢title中包含brown和dog, 且兩個是連線在一起的
phrase_prefix: 和phrase類似，最後一個搜尋詞(term)會進行前面部分匹配
官網解釋：The match_phrase_prefix is the same as match_phrase, except that it allows for prefix matches on the last term in the text
operator取值
and: “brown dog”, 包含brown且包含dog
or: “brown dog”, 包含brown或dog
minimum_should_match：取值為整數或者百分數，用於精度控制。如取4，表示需要匹配4個關鍵字，50%，需要匹配一半的關鍵字。設定minimum_should_match時operator將失效
"match" : {
"title" : {
"query" : "BROWN DOG",
"type" : "boolean",
"operator" : "OR",
"minimum_should_match" : "50%"
}
} multi_match查詢允許你做match查詢的基礎上同時搜尋多個欄位：
{
"multi_match": {
"query": "full text search",
"fields": [ "title", "body" ]
}
}tip:
1. 查詢字串時，match與term的區別
term查詢時內容精確匹配，match則會進行分
析器處理，分析器中的分詞器會將搜尋關鍵字分割成單獨的詞(terms)或者標記(tokens)
eg. 查詢title包含Jumps的內容, 用示例資料時，term匹配不到結果，但match會轉化成jumps匹配，然後查詢到結果。和使用的分析器有關，筆者使用的是自帶的標準分析器
http://localhost:9200/my_index/_analyze?pretty=true&field=title&text=Jumps
{
"tokens" : [ {
"token" : "jumps",
"start_offset" : 0,
"end_offset" : 5,
"type" : "<ALPHANUM>",
"position" : 0
} ]
}Java程式碼：
QueryBuilder builder = QueryBuilders.matchAllQuery();
builder = QueryBuilders.matchQuery("title", "Jumps");
builder = QueryBuilders.matchQuery("title", "BROWN DOG!").operator(MatchQueryBuilder.Operator.OR).type(MatchQueryBuilder.Type.BOOLEAN);
builder = QueryBuilders.multiMatchQuery("title", "dog", "jump"); f) 高亮搜尋
本篇暫不介紹
g) 排序
和資料庫中order by類似
"sort": { "date": { "order": "desc" }}Java程式碼：
SearchRequestBuilder requestBuilder =
client.prepareSearch("my_index").setTypes("my_type")
.setFrom(0).setSize(10)
.addSort("age", SortOrder.DESC); h) scroll
scroll 類似於資料庫裡面的遊標，用於快取大量結果資料
一個search請求只能返回結果的一個單頁（10條記錄），而scroll API能夠用來從一個單一的search請求中檢索大量的結果（甚至全部）
，這種行為就像你在一個傳統資料庫內使用一個遊標一樣。
scrolling目的不是為了實時的使用者請求，而是為了處理大量資料。
官網解釋(https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-request-scroll.html)：
While a search request returns a single “page” of results, the scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database.
Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration.
通過scroll檢索資料時，每次會返回一個scroll_id，檢索下一批資料時，這個id必需要傳遞到scroll API
Client client = ConnectionUtil.getLocalClient();
SearchRequestBuilder requestBuilder = client.prepareSearch("my_index").setTypes("my_type")
.setScroll(new TimeValue(20000)) // 設定scroll有效時間
.setSize(2);
System.out.println(requestBuilder);
SearchResponse scrollResp = requestBuilder.get();
System.out.println("totalHits:" + scrollResp.getHits().getTotalHits());
while (true) {
String scrollId = scrollResp.getScrollId();
System.out.println("scrollId:" + scrollId);
SearchHits searchHits = scrollResp.getHits();
for (SearchHit hit : searchHits.getHits()) {
System.out.println(hit.getId() + "~" + hit.getSourceAsString());
}
System.out.println("=================");
// 3. 通過scrollId獲取後續資料
scrollResp = client.prepareSearchScroll(scrollId)
.setScroll(new TimeValue(20000)).execute().actionGet();
if (scrollResp.getHits().getHits().length == 0) {
break;
}
} 三. 組合搜尋
bool: 組合查詢, 包含must, must not, should
搜尋關鍵字的權重
a) bool
上面介紹查詢/過濾關鍵子時多次提到bool，我們現在介紹bool
bool 可以用來合併多個條件，bool可以巢狀bool，已用於組成複雜的查詢條件，它包含以下操作符：
must :: 多個查詢條件的完全匹配,相當於 and。
must_not :: 多個查詢條件的相反匹配，相當於 not。
should :: 至少有一個查詢條件匹配, 相當於 or。
這些引數可以分別繼承一個條件或者一個條件的陣列：
{
"bool": {
"must": { "term": { "folder": "inbox" }},
"must_not": { "match": { "tag": "spam" }},
"should": [
{ "term": { "starred": true }},
{ "range": { "date": { "gte": "2014-01-01" }}}
]
}
}tip: bool下面，must、must_not、should至少需存在一個
Java程式碼：
// (price = 20 OR productID = "1234") AND (price != 30)
QueryBuilder queryBuilder = QueryBuilders.boolQuery()
.should(QueryBuilders.termQuery("price", "20"))
.should(QueryBuilders.termQuery("productId", "1234"))
.mustNot(QueryBuilders.termQuery("price", "30")); b) 搜尋關鍵字權重, 提高查詢得分
假設我們想搜尋包含”full-text search”的文件，但想給包含“Elasticsearch”或者“Lucene”的文件更高的權重。即包含“Elasticsearch”或者“Lucene”的相關性評分比不包含的高，這些文件在結果文件中更靠前。
一個簡單的bool查詢允許我們寫出像下面一樣的非常複雜的邏輯：
"bool": {
"must": {
"match": {
"content": { (1)
"query": "full text search",
"operator": "and"
}
}
},
"should": [ (2)
{ "match": { "content": "Elasticsearch" }},
{ "match": { "content": "Lucene" }}
]
}content欄位必須包含full,text,search這三個單詞。
如果content欄位也包含了“Elasticsearch”或者“Lucene”，則文件會有一個更高的得分。
在上例中，如果想給包含”Elasticsearch”一詞的文件得分更高於”Lucene”，則可以指定一個boost值控制權重，該值預設為1。一個大於1的boost值可以提高查詢子句的相對權重。
"bool": {
"must": {
"match": { (1)
"content": {
"query": "full text search",
"operator": "and"
}
}
},
"should": [
{ "match": {
"content": {
"query": "Elasticsearch",
"boost": 3 (2)
}
}},
{ "match": {
"content": {
"query": "Lucene",
"boost": 2 (3)
}
}}
]
}這些查詢子句的boost值為預設值1。
這個子句是最重要的，因為他有最高的boost值。
這個子句比第一個查詢子句的要重要，但是沒有“Elasticsearch”子句重要。
Java程式碼：
QueryBuilders.matchQuery("title", "Dog").boost(3);
附:測試類完整Java程式碼
package cn.com.axin.elasticsearch.qwzn.share;
import java.net.UnknownHostException;
import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.MatchQueryBuilder;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.sort.SortOrder;
import cn.com.axin.elasticsearch.util.ConnectionUtil;
import cn.com.axin.elasticsearch.util.Log;
/**
* @Title
*
* @author
* @date 2016-8-11
*/
public class Search {
public static void main(String[] args) throws Exception {
// searchAll();
// execQuery(termSearch());
// execQuery(termsSearch());
// execQuery(rangeSearch());
// execQuery(existsSearch());
// execQuery(matchSearch());
execQuery(boolSearch());
// highlightedSearch();
// scorll();
//
}
/**
* @return
*/
private static QueryBuilder boolSearch() {
// age > 30 or last_name is Smith
QueryBuilder queryBuilder = QueryBuilders.boolQuery()
.should(QueryBuilders.rangeQuery("age").gt("30"))
.should(QueryBuilders.matchQuery("last_name", "Smith"));
// 挺高查詢權重
// QueryBuilders.matchQuery("title", "Dog").boost(3);
// QueryBuilders.boolQuery().must(null);
// QueryBuilders.boolQuery().mustNot(null);
return queryBuilder;
}
private static void scorll() {
Client client = null;
try {
client = ConnectionUtil.getLocalClient(); // 獲取Client連線物件
SearchRequestBuilder requestBuilder = client.prepareSearch("my_index").setTypes("my_type")
// .setQuery(QueryBuilders.termQuery("age", "20"))
.setScroll(new TimeValue(20000)) // 設定scroll有效時間
.setSize(2);
System.out.println(requestBuilder);
SearchResponse scrollResp = requestBuilder.get();
System.out.println("totalHits:" + scrollResp.getHits().getTotalHits());
while (true) {
String scrollId = scrollResp.getScrollId();
System.out.println("scrollId:" + scrollId);
SearchHits searchHits = scrollResp.getHits();
for (SearchHit hit : searchHits.getHits()) {
System.out.println(hit.getId() + "~" + hit.getSourceAsString());
}
System.out.println("=================");
// 3. 通過scrollId獲取後續資料
scrollResp = client.prepareSearchScroll(scrollId)
.setScroll(new TimeValue(20000)).execute().actionGet();
if (scrollResp.getHits().getHits().length == 0) {
break;
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (null != client) {
client.close();
}
}
}
/**
* @return
*/
private static void highlightedSearch() {
QueryBuilder builder = QueryBuilders.termsQuery("age", "18");
Client client = null;
try {
client = ConnectionUtil.getLocalClient();
SearchRequestBuilder requestBuilder =
client.prepareSearch("my_index").setTypes("my_type")
.setFrom(0).setSize(10)
.addHighlightedField("age");
// .addSort("age", SortOrder.DESC);
Log.debug(requestBuilder);
SearchResponse response = requestBuilder.get();
Log.debug(response);
} catch (UnknownHostException e) {
e.printStackTrace();
} finally {
if (null != client) {
client.close();
}
}
}
/**
* @return
*/
private static QueryBuilder matchSearch() {
QueryBuilder builder = QueryBuilders.matchAllQuery();
builder = QueryBuilders.matchQuery("title", "Jumps");
/*
type: boolean 分析後進行查詢
phrase: 確切的匹配若干個單詞或短語,
phrase_prefix: The match_phrase_prefix is the same as match_phrase,
except that it allows for prefix matches on the last term in the text
*/
builder = QueryBuilders.matchQuery("title", "BROWN DOG!").operator(MatchQueryBuilder.Operator.OR).type(MatchQueryBuilder.Type.BOOLEAN);
builder = QueryBuilders.multiMatchQuery("title", "dog", "jump");
return builder;
}
/**
* @return
*/
private static QueryBuilder existsSearch() {
// exits
QueryBuilder builder = QueryBuilders.existsQuery("title");
// missing
builder = QueryBuilders.missingQuery("title");
// instead of missing
builder = QueryBuilders.boolQuery().mustNot(QueryBuilders.existsQuery("title"));
return builder;
}
/**
*
*/
private static QueryBuilder rangeSearch() {
// age >= 18 && age < 20
return QueryBuilders.rangeQuery("age").gte(18).lt(20);
}
private static QueryBuilder termSearch(){
QueryBuilder builder = QueryBuilders.termsQuery("title", "brown");
return builder;
}
private static QueryBuilder termsSearch(){
QueryBuilder builder = QueryBuilders.termsQuery("title", "dog", "jumps");
// 與termsQuery等效
builder = QueryBuilders.boolQuery().should(QueryBuilders.termQuery("title", "dog")).should(QueryBuilders.termQuery("title", "jumps"));
return builder;
}
private static void searchAll() {
Client client = null;
try {
client = ConnectionUtil.getLocalClient();
SearchRequestBuilder requestBuilder =
client.prepareSearch("my_index").setTypes("my_type")
.setFrom(0).setSize(10)
.addSort("age", SortOrder.DESC);
Log.debug(requestBuilder);
SearchResponse response = requestBuilder.get();
Log.debug(response);
} catch (UnknownHostException e) {
e.printStackTrace();
} finally {
if (null != client) {
client.close();
}
}
}
/**
* @param builder
* @throws UnknownHostException
*/
private static void execQuery(QueryBuilder builder)
throws UnknownHostException {
Client client = ConnectionUtil.getLocalClient();
SearchRequestBuilder requestBuilder =
client.prepareSearch("my_index").setTypes("my_type")
.setExplain(true)
.setQuery(builder);
Log.debug(requestBuilder);
SearchResponse response = requestBuilder.get();
Log.debug(response);
}
}獲取連線物件的程式碼
/**
* 獲取本地的連線物件(127.0.0.1:9300)
* @return
* @throws UnknownHostException
*/
public static Client getLocalClient() throws UnknownHostException {
return getClient("127.0.0.1", 9300, "es-stu");
}
/**
* 獲取連線物件
* @param host 主機IP
* @param port 埠
* @param clusterName TODO
* @return
* @throws UnknownHostException
*/
private static Client getClient(String host, int port, String clusterName) throws UnknownHostException {
// 引數設定
Builder builder = Settings.settingsBuilder();
// 啟用嗅探功能 sniff
builder.put("client.transport.sniff", true);
// 叢集名
builder.put("cluster.name", clusterName);
Settings settings = builder.build();
TransportClient transportClient = TransportClient.builder().settings(settings).build();
Client client = transportClient.addTransportAddress(
new InetSocketTransportAddress(InetAddress.getByName(host), port));
// 連線多個地址
// transportClient.addTransportAddresses(transportAddress);
return client;
}