elasticsearch 常見查詢及聚合的JAVA API

阿新 • • 發佈：2019-01-12

最近在研究ES 有點心得記錄下備忘

ES 常見查詢

根據ID 進行單個查詢

GetResponse response = client.prepareGet("accounts", "person", "1")
                .setOperationThreaded(false)
                .get();

相對於sql 的 select * from accounts.person where id=1 ;

分頁查詢所有記錄

QueryBuilder qb=new MatchAllQueryBuilder();
        SearchResponse response= client.prepareSearch("accounts").setTypes("person").setQuery(qb).setFrom(0)
                .setSize(100).get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

response.getHits()是所有命中記錄  相較於sql select * from accounts.person limit 100;

根據多條件組合與查詢

 QueryBuilder qb=QueryBuilders.boolQuery().must(QueryBuilders.termQuery("title","JAVA開發工程師")).must(QueryBuilders.termQuery("age",30)) ;

        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(qb).setFrom(0)
                .setSize(100);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

must 就像sql裡的and 相較於sql select * from accounts.person where title='JAVA開發工程師' and age=30

多條件或查詢

 QueryBuilder qb=QueryBuilders.termQuery("user","kimchy14");
        QueryBuilder qb1=QueryBuilders.termQuery("user","kimchy15");

        SortBuilder sortBuilder=SortBuilders.fieldSort("age");
        sortBuilder.order(SortOrder.DESC);
        QueryBuilder s=QueryBuilders.boolQuery().should(qb).should(qb1);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).addSort(sortBuilder).setFrom(0)
                .setSize(100);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

should 就像sql裡的or SortBuilder 的作用不言而喻就是用來排序以上程式碼相較於sql select * from accounts.person where user='kimchy14' or user='kimchy15' ;

範圍查詢

// RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(30,true);
       // RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").gt(30 );
        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").gte(30 );
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFrom(0)
                .setSize(100);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

範圍查詢rangeQuery.from（30，true）方法是大於30 後面的引數是是否包含為true的話就是大於等於30 to就相當於小於如果也有包含引數為true的話就是小於等於 gt 是大於 gte是大於等於 lt是小於 lte是小於等於第一句的builder就相當於 select * from accounts.person where age >=30 and age<=30;

包含查詢

 List<String> strs=new ArrayList<>();
        strs.add("kimchy14");
        strs.add("kimchy15");
        strs.add("kimchy16");
        QueryBuilder qb=QueryBuilders.termsQuery("user",strs);

        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(qb).setFetchSource("age",null).setFrom(0)
                .setSize(100);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

包含查詢使用termsQuery 可以傳列表也可以傳多個引數或者陣列 setFetchSource有兩個引數第一個引數是包含哪些引數第二個引數是排除哪些引數以上這段程式碼就相當於sql select age from accounts.person where user in ('kimchy14','kimchy15','kimchy16');

專門按id進行的包含查詢

QueryBuilder qb=QueryBuilders.idsQuery(0+"");

        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(qb).setFetchSource("age",null).setFrom(0)
                .setSize(100);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits(); 
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

按萬用字元查詢

  QueryBuilder qb = QueryBuilders.wildcardQuery("user", "k*hy17*");
         //Fuzziness fuzziness=Fuzziness.fromEdits(2);

      // QueryBuilder qb = QueryBuilders.fuzzyQuery("user","mchy2").fuzziness(fuzziness);
        //QueryBuilder qb = QueryBuilders.prefixQuery("user", "kimchy2");
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(qb).setFetchSource("user",null).setFrom(0)
                .setSize(100);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }

萬用字元查詢像我們sql裡的like 但是還不一樣 like的百分號可以加到前後但是萬用字元只能加在中間或者末尾不能加在開頭

常見統計統計分為指標和桶桶就是我們統計的樣本指標就是我們平時所查的count sum 與sql不一樣的是我們還可以將統計的樣本拿到就是response.getHits

統計count

AggregationBuilder  termsBuilder = AggregationBuilders.count("ageCount").field("age");

        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(30,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFrom(0).setSize(100).addAggregation(termsBuilder);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }
        ValueCount valueCount= response.getAggregations().get("ageCount");
        long value=valueCount.getValue();

這段程式碼就相當於 sql select count(age) ageCount form accounts.person where age >=30 and age<=30

查詢最大值

 AggregationBuilder  termsBuilder = AggregationBuilders.max("max").field("age");

        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(30,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFrom(0).setSize(100).addAggregation(termsBuilder);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }
        Max valueCount= response.getAggregations().get("max");
        double value=valueCount.getValue();

統計總和

 AggregationBuilder  termsBuilder = AggregationBuilders.sum("sum").field("age");

        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(30,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFrom(0).setSize(100).addAggregation(termsBuilder);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }
        Sum valueCount= response.getAggregations().get("sum");
        double value=valueCount.getValue();

平均數

 AggregationBuilder  termsBuilder = AggregationBuilders.avg("avg").field("age");

        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(30,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFrom(0).setSize(100).addAggregation(termsBuilder);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }
        Avg valueCount= response.getAggregations().get("avg");
        double value=valueCount.getValue();

統計樣本基本指標

        AggregationBuilder  termsBuilder = AggregationBuilders.stats("stats").field("age");

        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(30,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFrom(0).setSize(100).addAggregation(termsBuilder);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();
        SearchHits searchHits =  response.getHits();
        for(SearchHit hit:searchHits.getHits()){
            logger.log(Level.INFO , hit.getSourceAsString());
        }
        Stats valueCount= response.getAggregations().get("stats");
        logger.log(Level.INFO,"max"+valueCount.getMaxAsString());
        logger.log(Level.INFO,"avg"+valueCount.getAvgAsString());
        logger.log(Level.INFO,"sum"+valueCount.getSumAsString());
        logger.log(Level.INFO,"min"+valueCount.getMinAsString());
        logger.log(Level.INFO,"count"+valueCount.getCount());

分組統計相當於group by 後拿各組指標進行統計

分組求各組資料

 AggregationBuilder  termsBuilder = AggregationBuilders.terms("by_age").field("age");
        AggregationBuilder  sumBuilder=AggregationBuilders.sum("ageSum").field("age");
        AggregationBuilder  avgBuilder=AggregationBuilders.avg("ageAvg").field("age");
        AggregationBuilder  countBuilder=AggregationBuilders.count("ageCount").field("age");

        termsBuilder.subAggregation(sumBuilder).subAggregation(avgBuilder).subAggregation(countBuilder);
        //TermsAggregationBuilder all = AggregationBuilders.terms("age").field("age");
        //all.subAggregation(termsBuilder);
        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(36,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(s).setFetchSource(null,"gender").setFrom(0).setSize(100).addAggregation(termsBuilder);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();

        Aggregations terms= response.getAggregations();
        for (Aggregation a:terms){
            LongTerms teamSum= (LongTerms)a;
            for(LongTerms.Bucket bucket:teamSum.getBuckets()){
                logger.info(bucket.getKeyAsString()+"   "+bucket.getDocCount()+"    "+((Sum)bucket.getAggregations().asMap().get("ageSum")).getValue()+"    "+((Avg)bucket.getAggregations().asMap().get("ageAvg")).getValue()+"    "+((ValueCount)bucket.getAggregations().asMap().get("ageCount")).getValue());

            }
        }

第一行 termsBuilder 就相當於根據年齡對資料進行分組 group by 後面對sumBuilder avgBuilder countBuilder等就是在組內求和求平均數求數量

多分組求各組資料


        TermsAggregationBuilder all = AggregationBuilders.terms("by_gender").field("gender");
         AggregationBuilder age = AggregationBuilders.terms("by_age").field("age");
        AggregationBuilder  sumBuilder=AggregationBuilders.sum("ageSum").field("age");
        //AggregationBuilder  avgBuilder=AggregationBuilders.avg("ageAvg").field("age");
       // AggregationBuilder  countBuilder=AggregationBuilders.count("ageCount").field("age");
         all.subAggregation(age.subAggregation(sumBuilder));
        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("age").from(30,true).to(32,true);
        QueryBuilder s=QueryBuilders.boolQuery().must(rangeQueryBuilder);//.must(qb5);
        SearchRequestBuilder sv=client.prepareSearch("accounts").setTypes("person").setQuery(rangeQueryBuilder).addAggregation(all);
        logger.log(Level.INFO,sv.toString());
        SearchResponse response=  sv.get();

        Aggregations terms= response.getAggregations();
        for (Aggregation a:terms){
            StringTerms stringTerms= (StringTerms)a;
             for(StringTerms.Bucket bucket:stringTerms.getBuckets()){
              //  logger.info(bucket.getKeyAsString());
                Aggregation aggs=bucket.getAggregations().getAsMap().get("by_age");
                 LongTerms terms1= (LongTerms)aggs;
                for (LongTerms.Bucket bu:terms1.getBuckets()){
                     logger.info(bucket.getKeyAsString()+"  "+bu.getKeyAsString()+" "+bu.getDocCount()+"    "+((Sum)bu.getAggregations().asMap().get("ageSum")).getValue());
                }

            }
        }

每增加一個分組指標就需要多加一個termsBuilder 其他等一切跟普通分組一樣每次拿到

以上就是我總結的基本的查詢聚合等常見功能其他等諸如求各組前多少資料是用topHits 這些基本夠我們日常操作了。

最後我們總結下精確查詢用term 組合查詢用bool 範圍用range and查詢用must or查詢用should not查詢用must not 常見的接收聚合返回結果的型別 ValueCount AVG SUM MAX MIN 按照英文意義就可以理解分組聚合查詢時候還需要根據實際情況看是返回那種terms

elasticsearch 常見查詢及聚合的JAVA API

elasticsearch 常見查詢及聚合的JAVA API

Elasticsearch 之（43） Java API 實現 ES 的增刪改查、聚合分析

Elasticsearch 5.4.3實戰--Java API調用：搜索

ElasticSearch(十二) Bool Query JAVA API

elasticsearch常見異常及解決辦法

elasticsearch常見錯誤及解決方案

Elasticsearch的學習以及其JAVA API的使用

Elasticsearch 常見查詢

elasticsearch（一）：JAVA api操作

[Elasticsearch] 過濾查詢以及聚合(Filtering Queries and Aggregations) 5

ElasticSearch常見問題及解決方法

大資料(hadoop-HDFS的本地開發環境的配置與常見的HDFS的JAVA API)

Elasticsearch（二）：使用JAVA API實現簡單查詢、聚合查詢

elasticsearch elk最全java api 搜尋聚合、巢狀查詢

Elasticsearch java api 常用查詢方法QueryBuilder構造舉例

Elasticsearch JAVA api輕松搞定groupBy聚合

Elasticsearch Java API 查詢

Elasticsearch - JAVA -API 多條件查詢資料

Elasticsearch的Java API/查詢/分頁等

Elasticsearch 聚合查詢及複合查詢

elasticsearch 常見查詢及聚合的JAVA API

相關推薦