安裝elasticsearch中文切詞外掛hanlp
阿新 • • 發佈:2019-01-25
hanlp好處的,就是它的data字典比較齊全.
github上有國人寫hanlp支援es的外掛
https://github.com/pengcong90/elasticsearch-analysis-hanlp
下載發現解壓按它的安裝要求總找不到hanlp.properties檔案
將原始碼git下來,發現路徑有問題.
package org.elasticsearch.index.analysis;
import com.hankcs.hanlp.HanLP;
import com.hankcs.hanlp.utility.Predefine;
import com.hankcs.lucene4.HanLPIndexAnalyzer;
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.inject.assistedinject.Assisted;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.env.Environment;
import org.elasticsearch.index.IndexSettings;
/**
*/
public class HanLPAnalyzerProvider extends AbstractIndexAnalyzerProvider <HanLPIndexAnalyzer> {
private final HanLPIndexAnalyzer analyzer;
private static String sysPath = String.valueOf(System.getProperties().get("user.dir"));
@Inject
public HanLPAnalyzerProvider(IndexSettings indexSettings, Environment env, @Assisted String name, @Assisted Settings settings) {
super (indexSettings, name, settings);
//原來路徑
//Predefine.HANLP_PROPERTIES_PATH = sysPath.substring(0, sysPath.length()-4) + "/plugins/analysis-hanlp/hanlp.properties";
//修改後正確路徑
Predefine.HANLP_PROPERTIES_PATH = sysPath + "/plugins/analysis-hanlp/hanlp.properties";
analyzer = new HanLPIndexAnalyzer(true);
}
public static HanLPAnalyzerProvider getIndexAnalyzerProvider(IndexSettings indexSettings, Environment env, String name, Settings settings) {
return new HanLPAnalyzerProvider(indexSettings, env, name, settings);
}
public static HanLPAnalyzerProvider getSmartAnalyzerProvider(IndexSettings indexSettings, Environment env, String name, Settings settings) {
return new HanLPAnalyzerProvider(indexSettings, env, name, settings);
}
@Override
public HanLPIndexAnalyzer get() {
return this.analyzer;
}
}
因為它的hanlp版本是1.2.8,最新版本是1.5.4
修改pom.xml為
<dependency>
<groupId>com.hankcs</groupId>
<artifactId>hanlp</artifactId>
<version>portable-1.5.4</version>
<!--<systemPath>${pom.basedir}/lib/hanlp-1.2.8.jar</systemPath>-->
<!--<scope>system</scope>-->
</dependency>
打包編譯
在$ES_HOME下/plugins建立analysis-hanlp檔案
目錄下結構為
hanlp.properties屬性(可以直接從https://github.com/hankcs/HanLP 的realease下載修改root路徑就行了)
#本配置檔案中的路徑的根目錄,根目錄+其他路徑=完整路徑(支援相對路徑,請參考:https://github.com/hankcs/HanLP/pull/254)
#Windows使用者請注意,路徑分隔符統一使用/
root=/opt/elasticsearch-5.5.1/plugins/analysis-hanlp
#核心詞典路徑
CoreDictionaryPath=data/dictionary/CoreNatureDictionary.txt
#2元語法詞典路徑
BiGramDictionaryPath=data/dictionary/CoreNatureDictionary.ngram.txt
#停用詞詞典路徑
CoreStopWordDictionaryPath=data/dictionary/stopwords.txt
#同義詞詞典路徑
CoreSynonymDictionaryDictionaryPath=data/dictionary/synonym/CoreSynonym.txt
#人名詞典路徑
PersonDictionaryPath=data/dictionary/person/nr.txt
#人名詞典轉移矩陣路徑
PersonDictionaryTrPath=data/dictionary/person/nr.tr.txt
#繁簡詞典根目錄
tcDictionaryRoot=data/dictionary/tc
#自定義詞典路徑,用;隔開多個自定義詞典,空格開頭表示在同一個目錄,使用“檔名 詞性”形式則表示這個詞典的詞性預設是該詞性。優先順序遞減。
#另外data/dictionary/custom/CustomDictionary.txt是個高質量的詞庫,請不要刪除。所有詞典統一使用UTF-8編碼。
CustomDictionaryPath=data/dictionary/custom/CustomDictionary.txt; 現代漢語補充詞庫.txt; 全國地名大全.txt ns; 人名詞典.txt; 機構名詞典.txt; 上海地名.txt ns;data/dictionary/person/nrf.txt nrf;
#CRF分詞模型路徑
CRFSegmentModelPath=data/model/segment/CRFSegmentModel.txt
#HMM分詞模型
HMMSegmentModelPath=data/model/segment/HMMSegmentModel.bin
#分詞結果是否展示詞性
ShowTermNature=true
#IO介面卡,實現com.hankcs.hanlp.corpus.io.IIOAdapter介面以在不同的平臺(Hadoop、Redis等)上執行HanLP
#預設的IO介面卡如下,該介面卡是基於普通檔案系統的。
#IOAdapter=com.hankcs.hanlp.corpus.io.FileIOAdapter
plugin-descriptor.properties和plugin-security.policy屬性按 elasticsearch-analysis-hanlp的release包屬性修改.
修改ES啟動,並啟動
vim /opt/elasticsearch-5.5.1config/jvm.options
#新增
-Djava.security.policy=/opt/elasticsearch-5.5.1/plugins/analysis-hanlp/plugin-security.policy
測試安裝成功否命令
GET /_analyze?analyzer=hanlp-index&pretty=true
{
"text":"公安部:各地校車將享最高路權"
}