SpringBoot 整合Jest

阿新 • • 發佈：2019-02-06

【1】新增Elasticsearch-starter

pom檔案新增starter如下：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>

SpringBoot預設支援兩種技術和Elasticsearch進行互動：Spring Data Elasticsearch和Jest。

Jest預設不生效，需要匯入io.searchbox.client.JestClient。

這裡寫圖片描述
maven依賴如下：

<!--匯入jest依賴-->
<dependency>
    <groupId>io.searchbox</groupId>
    <artifactId>jest</artifactId>
    <version>5.3.3</version>
</dependency>

Spring Data Elasticsearch主要作用如下：

① ElasticsearchAutoConfiguration中註冊了client，屬性有clusterNodes和clusterName。

這裡寫圖片描述

② ElasticsearchDataAutoConfiguration註冊了ElasticsearchTemplate來操作ES

@Configuration
@ConditionalOnClass({ Client.class, ElasticsearchTemplate.class })
@AutoConfigureAfter(ElasticsearchAutoConfiguration.class)
public class ElasticsearchDataAutoConfiguration {

    @Bean
    @ConditionalOnMissingBean
    @ConditionalOnBean(Client.class)
    public ElasticsearchTemplate elasticsearchTemplate(Client client,
            ElasticsearchConverter converter) {
        try {
            return new ElasticsearchTemplate(client, converter);
        }
        catch (Exception ex) {
            throw new IllegalStateException(ex);
        }
    }

    @Bean
    @ConditionalOnMissingBean
    public ElasticsearchConverter elasticsearchConverter(
            SimpleElasticsearchMappingContext mappingContext) {
        return new MappingElasticsearchConverter(mappingContext);
    }

    @Bean
    @ConditionalOnMissingBean
    public SimpleElasticsearchMappingContext mappingContext() {
        return new SimpleElasticsearchMappingContext();
    }

}

③ ElasticsearchRepositoriesAutoConfiguration 啟用了ElasticsearchRepository

@Configuration
@ConditionalOnClass({ Client.class, ElasticsearchRepository.class })
@ConditionalOnProperty(prefix = "spring.data.elasticsearch.repositories", name = "enabled", havingValue = "true", matchIfMissing = true)
@ConditionalOnMissingBean(ElasticsearchRepositoryFactoryBean.class)
@Import(ElasticsearchRepositoriesRegistrar.class)
public class ElasticsearchRepositoriesAutoConfiguration {

}

ElasticsearchRepository介面原始碼如下(類似於JPA中的介面)：

@NoRepositoryBean
public interface ElasticsearchRepository<T, ID extends Serializable> extends ElasticsearchCrudRepository<T, ID> {
    <S extends T> S index(S var1);

    Iterable<T> search(QueryBuilder var1);

    Page<T> search(QueryBuilder var1, Pageable var2);

    Page<T> search(SearchQuery var1);

    Page<T> searchSimilar(T var1, String[] var2, Pageable var3);

    void refresh();

    Class<T> getEntityClass();
}

【2】JestClient操作測試

application.properties配置如下：

# jest url配置
spring.elasticsearch.jest.uris=http://192.168.2.110:9200

測試類如下：

@RunWith(SpringRunner.class)
@SpringBootTest
public class SpringBootJestTest {

    @Autowired
    JestClient jestClient;

    @Test
    public void index(){

        Article article = new Article();
        article.setId(1);
        article.setAuthor("Tom");
        article.setContent("hello world !");
        article.setTitle("今日訊息");
        //構建一個索引功能，型別為news
        Index index = new Index.Builder(article).index("jest").type("news").build();

        try {
            jestClient.execute(index);
            System.out.println("資料索引成功！");
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

    @Test
    public void search(){
        //查詢表示式
        String json = "{\n" +
                "    \"query\" : {\n" +
                "        \"match\" : {\n" +
                "            \"content\" : \"hello\"\n" +
                "        }\n" +
                "    }\n" +
                "}";
        //構建搜尋功能
        Search search = new Search.Builder(json).addIndex("jest").addType("news").build();

        try {
            SearchResult result = jestClient.execute(search);
            System.out.println(result.getJsonString());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

測試儲存資料結果如下：

這裡寫圖片描述

測試查詢資料結果如下：

這裡寫圖片描述

【3】 Elasticsearch版本調整

application.properties進行配置：

# Spring data elasticsearch配置
spring.data.elasticsearch.cluster-name=elasticsearch
spring.data.elasticsearch.cluster-nodes=192.168.2.110:9300

這裡節點名取自如下圖：
這裡寫圖片描述

啟動主程式，可能報錯如下(ES版本不合適)：

這裡寫圖片描述

檢視Spring Data官網，其中spring data elasticsearch與elasticsearch適配表如下：

這裡寫圖片描述

我們在上篇博文中安裝的ES版本為5.6.10，專案中SpringBoot版本為1.5.12，spring-boot-starter-data-elasticsearch為2.1.11，elasticsearch版本為2.4.6。

兩種解決辦法：① 升級SpringBoot版本；② 安裝2.4.6版本的elasticsearch。

這裡修改暴露的埠，重新使用docker安裝2.4.6版本：

# 拉取2.4.6 映象
docker pull registry.docker-cn.com/library/elasticsearch:2.4.6
# 啟動容器
docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d -p 9201:9200 -p 9301:9300 --name ES02 bc337c8e4f

這裡寫圖片描述

application.properties配置檔案同步修改：

# jest url配置
spring.elasticsearch.jest.uris=http://192.168.2.110:9201

# Spring data elasticsearch配置
spring.data.elasticsearch.cluster-name=elasticsearch
spring.data.elasticsearch.cluster-nodes=192.168.2.110:9301

此時再次啟動程式：

這裡寫圖片描述

【4】ElasticsearchRepository使用

類似於JPA，編寫自定義Repository介面，繼承自ElasticsearchRepository：

public interface BookRepository extends ElasticsearchRepository<Book,Integer> {

    public List<Book> findByBookNameLike(String bookName);
}

這裡第一個引數為物件型別，第二個引數為物件的主鍵型別。

BookRepository 所擁有的方法如下圖：

這裡寫圖片描述

Book原始碼如下：

// 這裡注意註解
@Document(indexName = "elastic",type = "book")
public class Book {

    private Integer id;
    private String bookName;
    private String author;

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    public String getBookName() {
        return bookName;
    }

    public void setBookName(String bookName) {
        this.bookName = bookName;
    }

    public String getAuthor() {
        return author;
    }

    public void setAuthor(String author) {
        this.author = author;
    }

    @Override
    public String toString() {
        return "Book{" +
                "id=" + id +
                ", bookName='" + bookName + '\'' +
                ", author='" + author + '\'' +
                '}';
    }
}

測試類如下：

    @Autowired
    BookRepository bookRepository;

    @Test
    public void testRepository(){
        Book book = new Book();
        book.setAuthor("吳承恩");
        book.setBookName("西遊記");
        book.setId(1);
        bookRepository.index(book);
        System.out.println("BookRepository 存入資料成功！");
    }

測試結果如下圖：

這裡寫圖片描述

測試獲取示例如下：

     @Test
    public void testRepository2(){
        for (Book book : bookRepository.findByBookNameLike("遊")) {
            System.out.println("獲取的book ： "+book);
        } ;
        Book book = bookRepository.findOne(1);
        System.out.println("根據id查詢 ： "+book);
    }

測試結果如下圖：

這裡寫圖片描述

Elasticsearch支援方法關鍵字如下圖所示

這裡寫圖片描述

即，在BookRepository中使用上述關鍵字構造方法，即可使用，Elastic自行實現其功能！

支援@Query註解

如下所示，直接在方法上使用註解：

public interface BookRepository extends ElasticsearchRepository<Book, String> {
    @Query("{"bool" : {"must" : {"field" : {"name" : "?0"}}}}")
    Page<Book> findByName(String name,Pageable pageable);
}

【5】ElasticsearchTemplate使用

存入資料來源碼示例如下：

    @Autowired
    ElasticsearchTemplate elasticsearchTemplate;

    @Test
    public void testTemplate01(){

        Book book = new Book();
        book.setAuthor("曹雪芹");
        book.setBookName("紅樓夢");
        book.setId(2);
        IndexQuery indexQuery = new IndexQueryBuilder().withId(String.valueOf(book.getId())).withObject(book).build();
        elasticsearchTemplate.index(indexQuery);
    }

測試結果如下：

這裡寫圖片描述

查詢資料示例如下：

    @Test
    public void testTemplate02(){
        QueryStringQueryBuilder stringQueryBuilder = new QueryStringQueryBuilder("樓");
        stringQueryBuilder.field("bookName");
        SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(stringQueryBuilder).build();
        Page<Book> books = elasticsearchTemplate.queryForPage(searchQuery,Book.class);
        Iterator<Book> iterator = books.iterator();
        while(iterator.hasNext()){
            Book book = iterator.next();
            System.out.println("該次獲取的book:"+book);
        }
    }

測試結果如下：

這裡寫圖片描述

1.說明

本文主要講解如何使用Spring Boot快速搭建Web框架，結合Spring Data 和 Jest 快速實現對阿里雲ElasticSearch的全文檢索功能。
主要使用元件：
Spring Boot Starter：可以幫助我們快速的搭建spring mvc 環境
Jest：一種rest訪問es的客戶端
elasticsearch：全文檢索
spring data elasticsearch：結合spring data
thymeleaf：web前端模版框架
jquery：js框架
bootstrap：前端樣式框架

2.專案Maven配置

以下為專案Maven配置，尤其需要注意各個元件的版本，以及註釋部分。
各個元件的某些版本組合下回出現各種異常，以下maven為測試可通過的一個版本。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.lewis</groupId>
    <artifactId>esweb</artifactId>
    <version>0.1</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <!--必須用2.0+，否則會有一個類
        Caused by: java.lang.NoSuchMethodError: org.elasticsearch.common.settings.Settings.settingsBuilder()Lorg/elasticsearch/common/settings/Settings$Builder;
        -->
        <version>2.0.0.M7</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <!--
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        -->

        <!--不可使用version 5.3.3，會有一個類的方法找不到-->
        <dependency>
            <groupId>io.searchbox</groupId>
            <artifactId>jest</artifactId>
            <version>5.3.2</version>
        </dependency>

        <!--必須用5.0+，否則會有一個類找不到org/elasticsearch/node/NodeValidationException-->
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>5.3.3</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
            <version>3.0.0.RELEASE</version>
        </dependency>

        <dependency>
            <groupId>com.github.vanroy</groupId>
            <artifactId>spring-boot-starter-data-jest</artifactId>
            <version>3.0.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-thymeleaf</artifactId>
        </dependency>

        <!--
        不需要引用
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>
        -->

        <!--spring boot elasticsearch 缺少的jar，需要單獨引入-->
        <dependency>
            <groupId>net.java.dev.jna</groupId>
            <artifactId>jna</artifactId>
            <version>4.5.1</version>
        </dependency>

        <!--webjars 前端框架，整體管理前端js框架-->
        <dependency>
            <groupId>org.webjars</groupId>
            <artifactId>jquery</artifactId>
            <version>3.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.webjars</groupId>
            <artifactId>bootstrap</artifactId>
            <version>4.0.0</version>
        </dependency>

        <!--When using Spring Boot version 1.3 or higher, it will automatically detect the webjars-locator library on the classpath and use it to automatically resolve the version of any WebJar assets for you. In order to enable this feature, you will need to add the webjars-locator library as a dependency of your application in the pom.xml file-->
        <dependency>
            <groupId>org.webjars</groupId>
            <artifactId>webjars-locator</artifactId>
            <version>0.30</version>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <fork>true</fork>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

建立完成後，專案目錄結構如下:

3.Spring Starter配置

需使用SpringBootApplication啟動
需禁用ElasticsearchAutoConfiguration，ElasticsearchDataAutoConfiguration，否則會有異常
HighLightJestSearchResultMapper Bean留待下面解釋，主要為了解決spring data不支援elasticsearch檢索highlight問題，此處為該Bean的註冊

@SpringBootApplication
@EnableAutoConfiguration(exclude = {ElasticsearchAutoConfiguration.class, ElasticsearchDataAutoConfiguration.class})
public class App {

    public static void main(String[] args) throws Exception {
        SpringApplication.run(App.class, args);
    }

    @Bean
    public HighLightJestSearchResultMapper highLightJestSearchResultMapper(){
        return new HighLightJestSearchResultMapper();
    }

}

3.Entity配置

a) 歌曲Entity如下：

通過對Class進行Document註解，實現與ElasticSearch中的Index和Type一一對應。
該類在最終與ES返回結果對映時，僅對映其中_source部分。即如下圖部分(highlight另說，後面單獨處理了)：

@Document(indexName = "songs",type = "sample",shards = 1, replicas = 0, refreshInterval = "-1")
public class Song extends HighLightEntity{

    @Id
    private Long id;

    private String name;
    private String href;
    private String lyric;
    private String singer;
    private String album;

    public Song(Long id, String name, String href, String lyric, String singer, String album, Map<String, List<String>> highlight) {
       //省略
    }

    public Song() {
    }
    //getter setter 省略...
}

b) 為了解決Spring data elasticsearch問題，此處增加一個抽象類：HighLightEntity，其他Entity需要繼承該類。

package org.leiws.esweb.entity;

import java.io.Serializable;
import java.util.List;
import java.util.Map;

public abstract class HighLightEntity implements Serializable{

    private Map<String, List<String>> highlight;

    public Map<String, List<String>> getHighlight() {
        return highlight;
    }

    public void setHighlight(Map<String, List<String>> highlight) {
        this.highlight = highlight;
    }
}

4.Repository配置

package org.leiws.esweb.repository;
import org.leiws.esweb.entity.Song;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
public interface SongRepository extends ElasticsearchRepository<Song,Long> {
}

5.Service配置

a) 介面

package org.leiws.esweb.service;

import org.leiws.esweb.entity.Song;
import org.springframework.data.domain.Page;

import java.util.List;

/**
 * The interface Song service.
 */
public interface SongService {

    /**
     * Search song list.
     *
     * @param pNum     the p num
     * @param pSize    the p size
     * @param keywords the keywords
     * @return the list
     */
    public Page<Song> searchSong(Integer pNum, Integer pSize, String keywords);
}

b) 實現類

該類實現了具體如何分頁，如何查詢等

package org.leiws.esweb.service.impl;

import com.github.vanroy.springdata.jest.JestElasticsearchTemplate;
import org.apache.log4j.Logger;
import org.elasticsearch.common.lucene.search.function.FiltersFunctionScoreQuery;
import org.elasticsearch.index.query.MatchPhraseQueryBuilder;
import org.elasticsearch.index.query.MatchQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.functionscore.FunctionScoreQueryBuilder;
import org.elasticsearch.index.query.functionscore.ScoreFunctionBuilders;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.leiws.esweb.entity.Song;
import org.leiws.esweb.repository.HighLightJestSearchResultMapper;
import org.leiws.esweb.repository.SongRepository;
import org.leiws.esweb.service.SongService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.SearchQuery;
import org.springframework.stereotype.Service;
import static org.elasticsearch.index.query.QueryBuilders.functionScoreQuery;
import static org.elasticsearch.index.query.QueryBuilders.matchPhraseQuery;

import java.util.List;

@Service
public class SongServiceImpl implements SongService{
    private static final Logger LOGGER = Logger.getLogger(SongServiceImpl.class);
    /* 分頁引數 */
    private final static Integer PAGE_SIZE = 12;          // 每頁數量
    private final static Integer DEFAULT_PAGE_NUMBER = 0; // 預設當前頁碼

    /* 搜尋模式 */
    private final static String SCORE_MODE_SUM = "sum"; // 權重分求和模式
    private final static Float  MIN_SCORE = 10.0F;      // 由於無相關性的分值預設為 1 ，設定權重分最小值為 10

    @Autowired
    SongRepository songRepository;

    @Autowired
    JestElasticsearchTemplate jestElasticsearchTemplate;

    @Autowired
    HighLightJestSearchResultMapper jestSearchResultMapper;

    @Override
    public Page<Song> searchSong(Integer pNum, Integer pSize, String keywords) {
        // 校驗分頁引數
        if (pSize == null || pSize <= 0) {
            pSize = PAGE_SIZE;
        }

        if (pNum == null || pNum < DEFAULT_PAGE_NUMBER) {
            pNum = DEFAULT_PAGE_NUMBER;
        }

        LOGGER.info("\n searchCity: searchContent [" + keywords + "] \n ");
        // 構建搜尋查詢
        SearchQuery searchQuery = getCitySearchQuery(pNum,pSize,keywords);
        LOGGER.info("\n searchCity: searchContent [" + keywords + "] \n DSL  = \n " + searchQuery.getQuery().toString());
//        Page<Song> cityPage = songRepository.search(searchQuery);
        Page<Song> cityPage = jestElasticsearchTemplate.queryForPage(searchQuery,Song.class,jestSearchResultMapper);
        return cityPage;
    }
    /**
     * 根據搜尋詞構造搜尋查詢語句
     *
     * 程式碼流程：
     *      - 權重分查詢
     *      - 短語匹配
     *      - 設定權重分最小值
     *      - 設定分頁引數
     *
     * @param pNum 當前頁碼
     * @param pSize 每頁大小
     * @param searchContent 搜尋內容
     * @return
     */
    private SearchQuery getCitySearchQuery(Integer pNum, Integer pSize,String searchContent) {

        /* elasticsearch 2.4.6 版本寫法
        FunctionScoreQueryBuilder functionScoreQueryBuilder = QueryBuilders.functionScoreQuery()
                .add(QueryBuilders.boolQuery().should(QueryBuilders.matchQuery("lyric", searchContent)),
                        ScoreFunctionBuilders.weightFactorFunction(1000))
                .scoreMode(SCORE_MODE_SUM).setMinScore(MIN_SCORE);
        */


        FunctionScoreQueryBuilder.FilterFunctionBuilder[] functions = {
                new FunctionScoreQueryBuilder.FilterFunctionBuilder(
                        matchPhraseQuery("lyric", searchContent),
                        ScoreFunctionBuilders.weightFactorFunction(1000))
        };
        FunctionScoreQueryBuilder functionScoreQueryBuilder =
                functionScoreQuery(functions).scoreMode(FiltersFunctionScoreQuery.ScoreMode.SUM).setMinScore(MIN_SCORE);

        // 分頁引數
//        Pageable pageable = new PageRequest(pNum, pSize);
        Pageable pageable = PageRequest.of(pNum, pSize);

        //高亮提示
        HighlightBuilder.Field highlightField =  new HighlightBuilder.Field("lyric")
                .preTags(new String[]{"<font color='red'>", "<b>", "<em>"})
                .postTags(new String[]{"</font>", "</b>", "</em>"})
                .fragmentSize(15)
                .numOfFragments(5)

                //highlightQuery必須單獨設定，否則在使用FunctionScoreQuery時，highlight配置不生效，返回結果無highlight元素
                //官方解釋：Highlight matches for a query other than the search query. This is especially useful if you use a rescore query because those are not taken into account by highlighting by default.
                .highlightQuery(matchPhraseQuery("lyric", searchContent));

        return new NativeSearchQueryBuilder()
                .withPageable(pageable)
    //            .withSourceFilter(new FetchSourceFilter(new String[]{"name","singer","lyric"},new String[]{}))
                .withHighlightFields(highlightField)
                .withQuery(functionScoreQueryBuilder).build();
    }
}

c) 解決Spring Data ElasticSearch不支援Highlight的問題

通過自定義實現一個如下的JestSearchResultMapper，解決無法Highlight的問題

package org.leiws.esweb.repository;

//import 省略
public class HighLightJestSearchResultMapper extends DefaultJestResultsMapper {

    private EntityMapper entityMapper;
    private MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext;

    public HighLightJestSearchResultMapper() {
        this.entityMapper = new DefaultEntityMapper();
        this.mappingContext = new SimpleElasticsearchMappingContext();
    }

    public HighLightJestSearchResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext, EntityMapper entityMapper) {
        this.entityMapper = entityMapper;
        this.mappingContext = mappingContext;
    }

    public EntityMapper getEntityMapper() {
        return entityMapper;
    }

    public void setEntityMapper(EntityMapper entityMapper) {
        this.entityMapper = entityMapper;
    }

    @Override
    public <T> AggregatedPage<T> mapResults(SearchResult response, Class<T> clazz) {
        return mapResults(response, clazz, null);
    }

    @Override
    public <T> AggregatedPage<T> mapResults(SearchResult response, Class<T> clazz, List<AbstractAggregationBuilder> aggregations) {
        LinkedList<T> results = new LinkedList<>();
        for (SearchResult.Hit<JsonObject, Void> hit : response.getHits(JsonObject.class)) {
            if (hit != null) {
                T result = mapSource(hit.source, clazz);
                HighLightEntity highLightEntity = (HighLightEntity) result;
                highLightEntity.setHighlight(hit.highlight);
                results.add((T) highLightEntity);
            }
        }

        String scrollId = null;
        if (response instanceof ExtendedSearchResult) {
            scrollId = ((ExtendedSearchResult) response).getScrollId();
        }

        return new AggregatedPageImpl<>(results, response.getTotal(), response.getAggregations(), scrollId);
    }

    private  <T> T mapSource(JsonObject source, Class<T> clazz) {
        String sourceString = source.toString();
        T result = null;
        if (!StringUtils.isEmpty(sourceString)) {
            result = mapEntity(sourceString, clazz);
            setPersistentEntityId(result, source.get(JestResult.ES_METADATA_ID).getAsString(), clazz);
        } else {
            //TODO(Fields results) : Map Fields results
            //result = mapEntity(hit.getFields().values(), clazz);
        }
        return result;
    }

    private <T> T mapEntity(String source, Class<T> clazz) {
        if (isBlank(source)) {
            return null;
        }
        try {
            return entityMapper.mapToObject(source, clazz);
        } catch (IOException e) {
            throw new ElasticsearchException("failed to map source [ " + source + "] to class " + clazz.getSimpleName(), e);
        }
    }
    private <T> void setPersistentEntityId(Object entity, String id, Class<T> clazz) {

        ElasticsearchPersistentEntity<?> persistentEntity = mappingContext.getRequiredPersistentEntity(clazz);
        ElasticsearchPersistentProperty idProperty = persistentEntity.getIdProperty();

        // Only deal with text because ES generated Ids are strings !
        if (idProperty != null) {
            if (idProperty.getType().isAssignableFrom(String.class)) {
                persistentEntity.getPropertyAccessor(entity).setProperty(idProperty, id);
            }
        }
    }
}

上面類的大部分程式碼來源於：DefaultJestResultsMapper
重點修改部分為：

@Override
    public <T> AggregatedPage<T> mapResults(SearchResult response, Class<T> clazz, List<AbstractAggregationBuilder> aggregations) {
        LinkedList<T> results = new LinkedList<>();
        for (SearchResult.Hit<JsonObject, Void> hit : response.getHits(JsonObject.class)) {
            if (hit != null) {
                T result = mapSource(hit.source, clazz);
                HighLightEntity highLightEntity = (HighLightEntity) result;
                highLightEntity.setHighlight(hit.highlight);
                results.add((T) highLightEntity);
            }
        }

        String scrollId = null;
        if (response instanceof ExtendedSearchResult) {
            scrollId = ((ExtendedSearchResult) response).getScrollId();
        }

        return new AggregatedPageImpl<>(results, response.getTotal(), response.getAggregations(), scrollId);
    }

6.Controller

相對簡單，如普通的Spring Controller

@Controller
@RequestMapping(value = "/search")
public class SearchController {

    @Autowired
    SongService songService;

    /**
     * Song list string.
     *
     * @param map the map
     * @return the string
     */
    @RequestMapping(method = RequestMethod.GET)
    public String songList(@RequestParam(value = "pNum") Integer pNum,
                           @RequestParam(value = "pSize", required = false) Integer pSize,
                           @RequestParam(value = "keywords") String keywords,ModelMap map){
       map.addAttribute("pageSong",songService.searchSong(pNum,pSize,keywords));
       return "songList";
    }
}

7.前端頁面thymeleaf模版

存放目錄為：resources/templates/songList.html

<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org" lang="en">
<head>
    <meta charset="UTF-8"/>
    <title>Title</title>
    <link rel='stylesheet' href='/webjars/bootstrap/css/bootstrap.min.css'>
    <script src="/webjars/jquery/jquery.min.js"></script>
    <script src="/webjars/bootstrap/js/bootstrap.min.js"></script>
</head>
<body>
<form action="/search" class="px-5 py-3" >
    <div class="input-group">
        <input name="keywords" type="text" class="form-control" placeholder="歌詞搜尋，請輸入歌詞內容" aria-label="歌詞搜尋，請輸入歌詞內容" aria-describedby="basic-addon2">
        <div class="input-group-append">
            <button class="btn btn-outline-secondary" type="button">搜尋</button>
        </div>
        <input type="hidden" name="pNum" value="0"/>
    </div>
</form>
<div class="alert alert-light" role="alert">
    為您找到0個結果：
</div>
<ul class="list-group">
    <li th:each="song : ${pageSong.content}" class="list-group-item">
        <div class="row">
            <a th:href="${song.href}">
            <h4 scope="row" th:text="${song.name}" ></h4>
            </a>
            &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
            <h6 scope="row" th:text="${song.singer}" class="align-bottom" ></h6>
        </div>
        <!--
            <td><a th:href="@{/users/update/{userId}(userId=${user.id})}" th:text="${user.name}"></a></td>
        -->
        <div class="row">
            <span th:each="highlight : ${song.highlight}">
                <span th:each="word : ${highlight.value}">
                    <span th:utext="${word}"></span>...
                </span>
            </span>
        </div>
    </li>
</ul>

<nav aria-label="..." class="">
    <ul class="pagination pagination-sm justify-content-center py-5">
        <li class="page-item ">
            <a class="page-link" href="#">
                <span aria-hidden="true">&laquo;</span>
                <span class="sr-only">Previous</span>
            </a>
        </li>
        <li class="page-item"><a class="page-link" href="#">1</a></li>
        <li class="page-item"><a class="page-link" href="#">2</a></li>
        <li class="page-item"><a class="page-link" href="#">3</a></li>
        <li class="page-item">
            <a class="page-link" href="#">
            <span aria-hidden="true">&raquo;</span>
            <span class="sr-only">Next</span>
            </a>
        </li>
    </ul>
</nav>
</body>
</html>

8.阿里雲ElasticSearch連線配置

在resources/application.properties中配置如下：

spring.data.jest.uri=http://1xx.xxx.xxx.xxx:8080
spring.data.jest.username=username
spring.data.jest.password=password
spring.data.jest.maxTotalConnection=50
spring.data.jest.defaultMaxTotalConnectionPerRoute=50
spring.data.jest.readTimeout=5000

9.其他

a) thymeleaf 熱啟動配置，便於測試

在resources/application.properties中配置如下：

spring.thymeleaf.cache=false

在pom.xml中增加：

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
        </dependency>

<build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <fork>true</fork>
                </configuration>
            </plugin>
        </plugins>
    </build>

3.每次還是需要重新compile後，修改的thymeleaf模版程式碼才會自動生效，因為spring boot啟動是以target目錄為準的

b) 阿里雲elasticsearch在esc上配置ngnix代理，以支援本機可以公網訪問，便於開發

購買一臺esc
在ecs上安裝ngnix，並配置代理資訊server 部分如下：

    server {
        listen       8080;
        #listen       [::]:80 default_server;
        server_name  {本機內網ip} {本機外網ip};
        #root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        #include /etc/nginx/default.d/*.conf;

        location / {
                        proxy_pass http://{elasticsearch 內網 ip}:9200;
        }
    }

10. 最後，查詢效果：

===============================================================================================

Elastic Search是一個開源的，分散式，實時搜尋和分析引擎。Spring Boot為Elasticsearch及Spring Data Elasticsearch提供的基於它的抽象提供了基本的配置。Spring Boot提供了一個用於聚集依賴的spring-boot-starter-data-elasticsearch 'StarterPOM'。

ElasticSearch作為搜尋引擎，我們需要解決2大問題：

1，如何將被搜尋的資料在ES上建立反向索引
2， Java程式碼如何與ES互動

其中第一個大問題又分為兩個小問題

1.1，如何初始化已有的資料
1.2，如何同步增量資料

第二個大問題也有兩種整合方式

2.1 Spring Data 9300埠整合
2.2 Restful API 9200埠整合

本篇先解決第二大問題。

第一種方式，利用RestAPI方式，也叫Jest方式：

Pom.xml：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
 <modelVersion>4.0.0</modelVersion> 
 
 <groupId>yejingtao.demo.springcloud</groupId> 
 <artifactId>demo-jest-elasticsearch</artifactId> 
 <version>0.0.1-SNAPSHOT</version> 
 <packaging>jar</packaging> 
 
 <name>demo-jest-elasticsearch</name> 
 <url>http://maven.apache.org</url> 
 
 <properties> 
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
 </properties> 
  
 <parent> 
    <groupId>org.springframework.boot</groupId> 
    <artifactId>spring-boot-starter-parent</artifactId> 
    <version>1.5.6.RELEASE</version> 
  </parent> 
   
  <dependencies> 
    <dependency> 
      <groupId>org.springframework.boot</groupId> 
      <artifactId>spring-boot-starter-web</artifactId> 
    </dependency> 
    <dependency> 
      <groupId>org.springframework.boot</groupId> 
      <artifactId>spring-boot-starter-data-elasticsearch</artifactId> 
    </dependency> 
    <dependency> 
      <groupId>io.searchbox</groupId> 
      <artifactId>jest</artifactId> 
    </dependency> 
    <dependency> 
      <groupId>net.java.dev.jna</groupId> 
      <artifactId>jna</artifactId> 
    </dependency> 
  </dependencies> 
</project>

Application.yml:

server: 
 port: 7081 
 
spring: 
 elasticsearch: 
  jest: 
   uris: 
   - http://192.168.226.133:9200 
   read-timeout: 5000

注意這裡是9200埠

主程式：最簡單的Spring boot啟動程式：

@SpringBootApplication 
public class ESApplication { 
 
  public static void main(String[] args) { 
    SpringApplication.run(ESApplication.class); 
  } 
}

定義好ES中的實體類和對ES操作的介面：

public class Entity implements Serializable{ 
 
  private static final long serialVersionUID = -763638353551774166L; 
   
  public static final String INDEX_NAME = "index_entity"; 
   
  public static final String TYPE = "tstype"; 
 
  private Long id; 
   
  private String name; 
   
  public Entity() { 
    super(); 
  } 
   
  public Entity(Long id, String name) { 
    this.id = id; 
    this.name = name; 
  } 
 
  public Long getId() { 
    return id; 
  } 
 
  public void setId(Long id) { 
    this.id = id; 
  } 
 
  public String getName() { 
    return name; 
  } 
 
  public void setName(String name) { 
    this.name = name; 
  } 
   
   
}

public interface CityESService { 
   
  void saveEntity(Entity entity); 
   
  void saveEntity(List<Entity> entityList); 
   
  List<Entity> searchEntity(String searchContent); 
}

介面實現：

@Service 
public class CityESServiceImpl implements CityESService{ 
   
  private static final Logger LOGGER = LoggerFactory.getLogger(CityESServiceImpl.class); 
   
  @Autowired 
  private JestClient jestClient; 
   
  @Override 
  public void saveEntity(Entity entity) { 
    Index index = new Index.Builder(entity).index(Entity.INDEX_NAME).type(Entity.TYPE).build(); 
    try { 
      jestClient.execute(index); 
      LOGGER.info("ES 插入完成"); 
    } catch (IOException e) { 
      e.printStackTrace(); 
      LOGGER.error(e.getMessage()); 
    } 
  } 
   
   
  /** 
   * 批量儲存內容到ES 
   */ 
  @Override 
  public void saveEntity(List<Entity> entityList) { 
    Bulk.Builder bulk = new Bulk.Builder(); 
    for(Entity entity : entityList) { 
      Index index = new Index.Builder(entity).index(Entity.INDEX_NAME).type(Entity.TYPE).build(); 
      bulk.addAction(index); 
    }     
    try { 
      jestClient.execute(bulk.build()); 
      LOGGER.info("ES 插入完成"); 
    } catch (IOException e) { 
      e.printStackTrace(); 
      LOGGER.error(e.getMessage()); 
    } 
  } 
   
  /** 
   * 在ES中搜索內容 
   */ 
  @Override 
  public List<Entity> searchEntity(String searchContent){ 
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder(); 
    //searchSourceBuilder.query(QueryBuilders.queryStringQuery(searchContent)); 
    //searchSourceBuilder.field("name"); 
    searchSourceBuilder.query(QueryBuilders.matchQuery("name",searchContent)); 
    Search search = new Search.Builder(searchSourceBuilder.toString()) 
        .addIndex(Entity.INDEX_NAME).addType(Entity.TYPE).build(); 
    try { 
      JestResult result = jestClient.execute(search); 
      return result.getSourceAsObjectList(Entity.class); 
    } catch (IOException e) { 
      LOGGER.error(e.getMessage()); 
      e.printStackTrace(); 
    } 
    return null;     
  } 
}

這裡插入資料的方式給了兩種，一種是單次API直接插入，一種是利用ES的bulk批量插入。

做一個controller方面我們測試：

啟動後在瀏覽器中請求http://localhost:7081/entityController/search?name=%E4%BA%BA%E6%89%8B%E4%BA%95

得到結果：

這裡只返回了9條記錄，而理論上ES預設的size是10，應該不是分頁的問題，而是隻能檢索出9條匹配記錄，用Kibana連上相同的搜尋確認下：

這裡用的是standard分詞方式，將每個中文都作為了一個term，凡是包含“人”“手”“井”的都被搜尋了出來，只是評分不同，如果想支援只能中文索引需要依賴ik外掛

OK，RestFul方式對ElasticSearch的檢索已經搞定了，更多的擴充套件可以慢慢研究下QueryBuilders裡的原始碼和批註。

第二種方式，利用Spring Data客戶端方式：

事先說明此方式有個弊端，讓我掉了坑裡好久才爬上來，Spring Data ElasticSearch必須與ElasticSearch版本相匹配，否則在對接時ES端會報版本不匹配錯誤，例如我ES是5.6.1版本，Spring boot是1.5.6版本，錯誤如下：

為解決這個問題我查找了一些資料，Spring Data與elasticsearch版本對應關係如下：

spring data elasticsearch	elasticsearch
3.0.0.RC2	5.5.0
3.0.0.M4	5.4.0
2.0.4.RELEASE	2.4.0
2.0.0.RELEASE	2.2.0
1.4.0.M1	1.7.3
1.3.0.RELEASE	1.5.2
1.2.0.RELEASE	1.4.4
1.1.0.RELEASE	1.3.2
1.0.0.RELEASE	1.1.1

而我用的Spring Boot 1.5.6版本對應的Spring Data ElasticSearch是2.1.6版本，不支援5.X的ES，所以報錯。到本博文撰寫為止，Spring Boot的RELEASE版本最新的是1.5.8，對應的Spring Data ElasticSearch是2.1.8，仍不支援5.X的ES，所以如果一定要使用Java客戶端方式整合ES只能放棄Spring Boot直接使用Spring Data和Spring MVC，或者降低ES的版本使之與Spring boot匹配。

pom.xml依賴：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
 <modelVersion>4.0.0</modelVersion> 
 
 <groupId>yejingtao.demo.springcloud</groupId> 
 <artifactId>demo-data-elasticsearch</artifactId> 
 <version>0.0.1-SNAPSHOT</version> 
 <packaging>jar</packaging> 
 
 <name>demo-data-elasticsearch</name> 
 <url>http://maven.apache.org</url> 
 
 <properties> 
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
 </properties> 
  
 <parent> 
    <groupId>org.springframework.boot</groupId> 
    <artifactId>spring-boot-starter-parent</artifactId> 
    <version>1.5.8.RELEASE</version> 
  </parent> 
   
  <dependencies> 
    <dependency> 
      <groupId>org.springframework.boot</groupId> 
      <artifactId>spring-boot-starter-web</artifactId> 
    </dependency> 
    <dependency> 
      <groupId>org.springframework.boot</groupId> 
      <artifactId>spring-boot-starter-data-elasticsearch</artifactId> 
    </dependency> 
  </dependencies> 
</project>

不再引用Jest。

application.yml:

server: 
 port: 7081 
 
spring: 
 data: 
  elasticsearch: 
   cluster-nodes: 192.168.226.133:9300 
   cluster-name: my-es 
   repositories: 
    enabled: true

注意這裡是9300埠

Controller、主程式、Service介面同Jest專案不變，不再羅列

實體類稍作變化，指定ES中的index和type：

@Document(indexName="index_entity", type="tstype")

多一個Repository介面，無需實現類，spring data標準用法：

/** 
 * Entity ES操作類 
 * @author yejingtao 
 * 
 */ 
public interface EntityRepository extends ElasticsearchRepository<Entity,Long>{ 
 
}

Service實現類與Jest的天壤之別了，從語法上可以看出更像是對資料庫層的操作：

@Service 
public class CityESServiceImpl implements CityESService{ 
   
  private static final Logger LOGGER = LoggerFactory.getLogger(CityESServiceImpl.class); 
   
  int PAGE_SIZE = 15; //預設分頁大小 
   
  int PAGE_NUMBER = 0; //預設當前分頁 
   
  String SCORE_MODE_SUM = "sum"; //權重分求和模式 
   
  Float MIN_SCORE = 10.0F; //由於無相關性的分值預設為1， 設定權重分最小值為10 
   
  @Autowired 
  EntityRepository entityRepository; 
   
  /** 
   * 儲存內容到ES 
   */ 
  @Override 
  public Long saveEntity(Entity entity) { 
    Entity entityResult = entityRepository.save(entity); 
    return entityResult.getId(); 
  } 
   
  /** 
   * 在ES中搜索內容 
   */ 
  @Override 
  public List<Entity> searchEntity(int pageNumber, int pageSize, String searchContent){ 
    if(pageSize==0) { 
      pageSize = PAGE_SIZE; 
    } 
    if(pageNumber<0) { 
      pageNumber = PAGE_NUMBER; 
    } 
     
    SearchQuery searchQuery = getEntitySearchQuery(pageNumber,pageSize,searchContent); 
     
    LOGGER.info("\n searchCity: searchContent [" + searchContent + "] \n DSL = \n "  
        + searchQuery.getQuery().toString()); 
 
     
    Page<Entity> cityPage = entityRepository.search(searchQuery); 
    return cityPage.getContent(); 
  } 
   
  /** 
   * 組裝搜尋Query物件 
   * @param pageNumber 
   * @param pageSize 
   * @param searchContent 
   * @return 
   */ 
  private SearchQuery getEntitySearchQuery(int pageNumber, int pageSize, String searchContent) { 
    FunctionScoreQueryBuilder functionScoreQueryBuilder = QueryBuilders.functionScoreQuery() 
        .add(QueryBuilders.matchPhraseQuery("name", searchContent), 
            ScoreFunctionBuilders.weightFactorFunction(1000)) 
        //.add(QueryBuilders.matchPhraseQuery("other", searchContent), 
            //ScoreFunctionBuilders.weightFactorFunction(1000)) 
        .scoreMode(SCORE_MODE_SUM).setMinScore(MIN_SCORE); 
    //設定分頁，否則只能按照ES預設的分頁給 
    Pageable pageable = new PageRequest(pageNumber, pageSize); 
    return new NativeSearchQueryBuilder().withPageable(pageable).withQuery(functionScoreQueryBuilder).build(); 
  } 
   
}

測試方式同Jest。

這兩種方式，從設計上來講屬於兩種思路，Spring Data的思路就是將ElasticSearch當自家的資料倉庫來管理，直接通過Java客戶端程式碼操作ES；Jest的思路是將ElasticSearch當為獨立的服務端，自己作為客戶端用相容性最強的RestFul格式來與之互動。
個人比較傾向於Jest方式，第一相容性好，不需要考慮版本的問題。第二，從ElasticSearch本身的設計上來分析，9200是對外服務埠，9300是內部管理和叢集通訊埠，請求9200獲取搜尋服務更符合ES的設計初衷，不會影響叢集內部的通訊。

SpringBoot 整合Jest

【1】新增Elasticsearch-starter

【2】JestClient操作測試

【3】 Elasticsearch版本調整

【4】ElasticsearchRepository使用

【5】ElasticsearchTemplate使用

1.說明

2.專案Maven配置

3.Spring Starter配置

3.Entity配置

a) 歌曲Entity如下：

b) 為了解決Spring data elasticsearch問題，此處增加一個抽象類：HighLightEntity，其他Entity需要繼承該類。

4.Repository配置

5.Service配置

a) 介面

b) 實現類

c) 解決Spring Data ElasticSearch不支援Highlight的問題

6.Controller

7.前端頁面thymeleaf模版

8.阿里雲ElasticSearch連線配置

9.其他

a) thymeleaf 熱啟動配置，便於測試

b) 阿里雲elasticsearch在esc上配置ngnix代理，以支援本機可以公網訪問，便於開發

10. 最後，查詢效果：

相關推薦