Elasticsearch之近义词/同义词的使用
目录
1、环境:
ES 7.6.1
kinaba 7.6.1
centos 7
2、未配置同义词分析器的情况
2.1、创建索引
PUT /test_001
{
"settings": {
"index": {
"max_result_window": 1000000
},
"analysis": {
"analyzer": {
"ik_max_word": {
"tokenizer": "ik_max_word",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"properties": {
"goodsName": {
"type": "text",
"analyzer": "ik_max_word"
}
}
}
}
2.2、添加数据
向test_001索引中批量添加数据
POST _bulk
{ "index" : { "_index" : "test_001","_id":1} }
{"id" : 1,"goodsName" : "克而瑞"}
{ "index" : { "_index" : "test_001","_id":2} }
{"id" : 2,"goodsName" : "随便"}
{ "index" : { "_index" : "test_001","_id":3} }
{"id" : 3,"goodsName" : "CRAC"}
2.3、检索数据
查看克而瑞相关信息
GET test_001/_search
{
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"match": {
"goodsName": {
"query": "克而瑞",
"boost": 1
}
}
}
]
}
}
]
}
}
}
结果中没有和克而瑞同义的CRAC的信息
3、使用自定义的同义词分析器
3.1、配置同义词文件
进入到es安装目录的config目录下,
mkdir analysis
cd analysis/
vim my-synonym.txt
my-synonym.txt文件中填写近义词同义词,内容如下:
[root@hadoop180 analysis]# cat my-synonym.txt
搜房,房天下
成交均价,成交单价,房价,售价
保障房,经济适用房,配套商品房,动迁房,廉租房
出租,租赁
买卖,销售
克而瑞,CRAC
[root@hadoop180 analysis]#
3.2、创建索引
建立索引test_002 ,与索引 test001的主要区别在与
PUT /test_002
{
"settings": {
"index": {
"max_result_window": 1000000
},
"analysis": {
"analyzer": {
"ik_max_word": {
"tokenizer": "ik_max_word",
"filter": [
"lowercase",
"asciifolding",
"my_synonym_filter"
]
}
},
"filter": {
"my_synonym_filter": {
"type": "synonym",
"synonyms_path": "analysis/my-synonym.txt"
}
}
}
},
"mappings": {
"properties": {
"goodsName": {
"type": "text",
"analyzer": "ik_max_word"
}
}
}
}
3.3、批量插入测试数据
向 test_002 索引中批量插入数据
POST _bulk
{ "index" : { "_index" : "test_002","_id":1} }
{"id" : 1,"goodsName" : "克而瑞"}
{ "index" : { "_index" : "test_002","_id":2} }
{"id" : 2,"goodsName" : "随便"}
{ "index" : { "_index" : "test_002","_id":3} }
{"id" : 3,"goodsName" : "CRAC"}
3.3、检索数据测试
查询克而瑞相关信息,发现
GET test_002/_search
{
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"match": {
"goodsName": {
"query": "克而瑞",
"boost": 1
}
}
}
]
}
}
]
}
}
}
发现和克而瑞 同义的CRAC的信息也被检索出来了
3.4、新增同义词如何处理?
新添加同义词到 my-synonym.txt 文件中
批量插入三条数据
POST _bulk
{ "index" : { "_index" : "test_002","_id":4} }
{"id" : 4,"goodsName" : "西红柿"}
{ "index" : { "_index" : "test_002","_id":5} }
{"id" : 5,"goodsName" : "番茄"}
{ "index" : { "_index" : "test_002","_id":6} }
{"id" : 6,"goodsName" : "洋柿子"}
查询西红柿相关的信息时,发现只检索到西红柿的记录,没有包含 番茄 和 洋柿子,此时,重新启动es后,再次执行查询语句,发现同义词的信息可以正常检索到
GET test_002/_search
{
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"match": {
"goodsName": {
"query": "西红柿",
"boost": 1
}
}
}
]
}
}
]
}
}
}
3.5 如何不重启es的情况下,动态更新同义词词典?
前期准备:
将同义词词典作为Tomcat的静态资源
同义词词典的访问路径为 http://192.168.6.180:8500/upload/text/my-remote-synonym.txt
my-remote-synonym.txt文件的内容如下:
使用插件elasticsearch-analysis-dynamic-synonym
github地址:https://github.com/bells/elasticsearch-analysis-dynamic-synonym
克隆下后,修改pom.xml中的为安装的ES的版本,因为该项目中引入的ES依赖版本引用了项目的版本
pom.xml中部分内容如下
<groupId>com.bellszhu.elasticsearch</groupId>
<artifactId>elasticsearch-analysis-dynamic-synonym</artifactId>
<version>7.6.1</version>
<packaging>jar</packaging>
<name>elasticsearch-dynamic-synonym</name>
<description>Analysis-plugin for synonym</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<elasticsearch.version>${project.version}</elasticsearch.version>
<maven.compiler.target>1.8</maven.compiler.target>
<elasticsearch.plugin.name>analysis-dynamic-synonym</elasticsearch.plugin.name>
<elasticsearch.assembly.descriptor>${project.basedir}/src/main/assemblies/plugin.xml
</elasticsearch.assembly.descriptor>
<elasticsearch.plugin.classname>com.bellszhu.elasticsearch.plugin.DynamicSynonymPlugin
</elasticsearch.plugin.classname>
<elasticsearch.plugin.jvm>true</elasticsearch.plugin.jvm>
</properties>
参考README.md文件
按常理修改了项目的version后编译打包后将target/releases包下的zip包解压到es安装目录下的plugins/dynamic-synonym
克隆下来的代码默认版本时7.13.2,不修改版本打包正常
由于我是用了ES是7.6.1的版本,修改版本后,编译报错,原因:
在导入import org.elasticsearch.common.logging.DeprecationCategory;时报错
DeprecationCategory该类在7.13.2版本中有,在7.6.1版本中没有该类,看了下7.13.2版本中的代码实现,修改了下代码(主要修改了 private static final DeprecationLogger DEPRECATION_LOGGER 属性赋值的方法),修改后的代码如下:
package com.bellszhu.elasticsearch.plugin.synonym.analysis;
import java.io.IOException;
import java.util.List;
import java.util.Map;
import java.util.WeakHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.synonym.SynonymMap;
// import org.elasticsearch.common.logging.DeprecationCategory;
import org.elasticsearch.common.logging.DeprecationLogger;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.env.Environment;
import org.elasticsearch.index.IndexSettings;
import org.elasticsearch.index.analysis.AbstractTokenFilterFactory;
import org.elasticsearch.index.analysis.AnalysisMode;
import org.elasticsearch.index.analysis.CharFilterFactory;
import org.elasticsearch.index.analysis.CustomAnalyzer;
import org.elasticsearch.index.analysis.TokenFilterFactory;
import org.elasticsearch.index.analysis.TokenizerFactory;
/**
* @author bellszhu
*/
public class DynamicSynonymTokenFilterFactory extends
AbstractTokenFilterFactory {
private static final DeprecationLogger DEPRECATION_LOGGER = getLogger(
DynamicSynonymTokenFilterFactory.class);
private static Logger logger = LogManager.getLogger("dynamic-synonym");
// private DeprecationLogger(String parentLoggerName) {
// this.logger = LogManager.getLogger(getLoggerName(parentLoggerName));
// }
public static DeprecationLogger getLogger(Class<?> aClass) {
return getLogger(toLoggerName(aClass));
}
public static DeprecationLogger getLogger(String name) {
return new DeprecationLogger(LogManager.getLogger(name));
}
private static String toLoggerName(Class<?> cls) {
String canonicalName = cls.getCanonicalName();
return canonicalName != null ? canonicalName : cls.getName();
}
private static String getLoggerName(String name) {
if (name.startsWith("org.elasticsearch")) {
name = name.replace("org.elasticsearch.", "org.elasticsearch.deprecation.");
} else {
name = "deprecation." + name;
}
return name;
}
/**
* Static id generator
*/
private static final AtomicInteger id = new AtomicInteger(1);
private static ScheduledExecutorService pool = Executors.newScheduledThreadPool(1, r -> {
Thread thread = new Thread(r);
thread.setName("monitor-synonym-Thread-" + id.getAndAdd(1));
return thread;
});
private volatile ScheduledFuture<?> scheduledFuture;
private final String location;
private final boolean expand;
private final boolean lenient;
private final String format;
private final int interval;
protected SynonymMap synonymMap;
protected Map<AbsSynonymFilter, Integer> dynamicSynonymFilters = new WeakHashMap<>();
protected final Environment environment;
protected final AnalysisMode analysisMode;
public DynamicSynonymTokenFilterFactory(
IndexSettings indexSettings,
Environment env,
String name,
Settings settings
) throws IOException {
super(indexSettings, name, settings);
this.location = settings.get("synonyms_path");
if (this.location == null) {
throw new IllegalArgumentException(
"dynamic synonym requires `synonyms_path` to be configured");
}
if (settings.get("ignore_case") != null) {
DEPRECATION_LOGGER.deprecated(
"ANALYSIS",
"dynamic synonym ignore_case",
"The ignore_case option on the synonym_graph filter is deprecated. " +
"Instead, insert a lowercase filter in the filter chain before the synonym_graph filter."
);
}
this.interval = settings.getAsInt("interval", 60);
this.expand = settings.getAsBoolean("expand", true);
this.lenient = settings.getAsBoolean("lenient", false);
this.format = settings.get("format", "");
boolean updateable = settings.getAsBoolean("updateable", false);
this.analysisMode = updateable ? AnalysisMode.SEARCH_TIME : AnalysisMode.ALL;
this.environment = env;
}
@Override
public AnalysisMode getAnalysisMode() {
return this.analysisMode;
}
@Override
public TokenStream create(TokenStream tokenStream) {
throw new IllegalStateException(
"Call getChainAwareTokenFilterFactory to specialize this factory for an analysis chain first");
}
@Override
public TokenFilterFactory getChainAwareTokenFilterFactory(
TokenizerFactory tokenizer,
List<CharFilterFactory> charFilters,
List<TokenFilterFactory> previousTokenFilters,
Function<String, TokenFilterFactory> allFilters
) {
final Analyzer analyzer = buildSynonymAnalyzer(tokenizer, charFilters, previousTokenFilters, allFilters);
synonymMap = buildSynonyms(analyzer);
final String name = name();
return new TokenFilterFactory() {
@Override
public String name() {
return name;
}
@Override
public TokenStream create(TokenStream tokenStream) {
// fst is null means no synonyms
if (synonymMap.fst == null) {
return tokenStream;
}
DynamicSynonymFilter dynamicSynonymFilter = new DynamicSynonymFilter(tokenStream, synonymMap, false);
dynamicSynonymFilters.put(dynamicSynonymFilter, 1);
return dynamicSynonymFilter;
}
@Override
public TokenFilterFactory getSynonymFilter() {
// In order to allow chained synonym filters, we return IDENTITY here to
// ensure that synonyms don't get applied to the synonym map itself,
// which doesn't support stacked input tokens
return IDENTITY_FILTER;
}
@Override
public AnalysisMode getAnalysisMode() {
return analysisMode;
}
};
}
Analyzer buildSynonymAnalyzer(
TokenizerFactory tokenizer,
List<CharFilterFactory> charFilters,
List<TokenFilterFactory> tokenFilters,
Function<String, TokenFilterFactory> allFilters
) {
return new CustomAnalyzer(
tokenizer,
charFilters.toArray(new CharFilterFactory[0]),
tokenFilters.stream().map(TokenFilterFactory::getSynonymFilter).toArray(TokenFilterFactory[]::new)
);
}
SynonymMap buildSynonyms(Analyzer analyzer) {
try {
return getSynonymFile(analyzer).reloadSynonymMap();
} catch (Exception e) {
logger.error("failed to build synonyms", e);
throw new IllegalArgumentException("failed to build synonyms", e);
}
}
SynonymFile getSynonymFile(Analyzer analyzer) {
try {
SynonymFile synonymFile;
if (location.startsWith("http://") || location.startsWith("https://")) {
synonymFile = new RemoteSynonymFile(
environment, analyzer, expand, lenient, format, location);
} else {
synonymFile = new LocalSynonymFile(
environment, analyzer, expand, lenient, format, location);
}
if (scheduledFuture == null) {
scheduledFuture = pool.scheduleAtFixedRate(new Monitor(synonymFile),
interval, interval, TimeUnit.SECONDS);
}
return synonymFile;
} catch (Exception e) {
logger.error("failed to get synonyms: " + location, e);
throw new IllegalArgumentException("failed to get synonyms : " + location, e);
}
}
public class Monitor implements Runnable {
private SynonymFile synonymFile;
Monitor(SynonymFile synonymFile) {
this.synonymFile = synonymFile;
}
@Override
public void run() {
if (synonymFile.isNeedReloadSynonymMap()) {
synonymMap = synonymFile.reloadSynonymMap();
for (AbsSynonymFilter dynamicSynonymFilter : dynamicSynonymFilters.keySet()) {
dynamicSynonymFilter.update(synonymMap);
logger.info("success reload synonym");
}
}
}
}
}
将打包后的zip解压到 es的plugins/dynamic-synonym目录下后,重启ES
创建es索引:
说明:
synonyms_path 必须,根据它的值是否是以http://开头来判断是本地文件,还是远程文件。
interval 非必须,默认值是60,单位秒,表示间隔多少秒去检查同义词文件是否有更新。
ignore_case 非必须, 默认值是false
expand 非必须, 默认值是true
format 非必须, 默认值是空字符串, 如果为wordnet,则表示WordNet结构的同义词。
热更新
1.对于本地文件:主要通过文件的修改时间戳(Modify time)来判断是否要重新加载。
2.对于远程文件:synonyms_path 是指一个url。 这个http请求需要返回两个头部,一个是Last-Modified,一个是ETag,只要有一个发生变化,该插件就会去获取新的同义词来更新相应的同义词。
3.同义词更新后索引中的字段不会立即生效,需要reindex或者覆盖更新
PUT /test_003
{
"settings": {
"index": {
"max_result_window": 1000000
},
"analysis": {
"analyzer": {
"ik_max_word": {
"tokenizer": "ik_max_word",
"filter": [
"lowercase",
"asciifolding",
"my_synonym_filter"
]
}
},
"filter": {
"my_synonym_filter": {
"type": "dynamic_synonym",
"interval": 30,
"synonyms_path": "http://192.168.6.180:8500/upload/text/my-remote-synonym.txt"
}
}
}
},
"mappings": {
"properties": {
"goodsName": {
"type": "text",
"analyzer": "ik_max_word"
}
}
}
}
插入两条测试数据
POST _bulk
{ "index" : { "_index" : "test_003","_id":1} }
{"id" : 1,"goodsName" : "地瓜"}
{ "index" : { "_index" : "test_003","_id":2} }
{"id" : 2,"goodsName" : "红薯"}
测试查询地瓜的数据,发现地瓜和红薯两条记录都有查询到
GET test_003/_search
{
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"match": {
"goodsName": {
"query": "地瓜",
"boost": 1
}
}
}
]
}
}
]
}
}
}
动态更新同义词库
修改Tomcat中的静态资源文件 my-remote-synonym.txt
新增 汤圆,元宵
不重启es,查看es的运行日志,可以看到日志中reload remote synonym:
es已经重新加载了远程同义词词典
再向es中添加两条测试数据
POST _bulk
{ "index" : { "_index" : "test_003","_id":3} }
{"id" : 3,"goodsName" : "汤圆"}
{ "index" : { "_index" : "test_003","_id":4} }
{"id" : 4,"goodsName" : "元宵"}
测试查询汤圆,发现汤圆和元宵的信息都被检索出来
GET test_003/_search
{
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"match": {
"goodsName": {
"query": "汤圆",
"boost": 1
}
}
}
]
}
}
]
}
}
}