es 分词器（五）之elasticsearch-analysis-jieba 8.7.0

es 分词器（五）之elasticsearch-analysis-jieba 8.7.0

今天咱们就来讲一下es jieba 8.7.0 分词器的实现，以及8.x其它版本的实现方式，如果想直接使用es 结巴8.x版本，请直接修改pom文件的elasticsearch.version版本号即可，然后打包安装就行，不需要做太多的操作。

一、elasticsearch-jieba-plugin

最近更新的版本为8.4.1，最近更新的时间停留在2022年，从这之后便无人维护此开源项目
GitHub地址：​​https://github.com/sing1ee/elasticsearch-jieba-plugin/tree/8.4.1​​

二、elasticsearch-analysis-jieba

最近更新的版本为6.8.17，比上面的插件更惨，已经有三年无人维护了。
Github地址：​​https://github.com/huaban/elasticsearch-analysis-jieba/tree/dependabot/maven/org.elasticsearch-elasticsearch-6.8.17​​

三、决定换壳elasticsearch-jieba-plugin

当前我开发的项目采用的版本为8.7.0，目前在网上无法找到与之匹配的版本。
ik分词器用户比jieba分词器用户多，因为会对应的es版本不断更新，目前ik分词器的版本已经更新至8.12.2，2024年5月14日位置es的最新版本为8.14.x
2024年5月14日es最新版本为8.14.x

四、编译elasticsearch-analysis-jieba分词器

由于原有的插件【elasticsearch-analysis-jieba】已经很久没有人使用，但我又感觉【elasticsearch-analysis-jieba】这个名称比【elasticsearch-jieba-plugin】【https://github.com/sing1ee/elasticsearch-jieba-plugin/tree/8.4.1】这个好听一点，所以我本地新开了一个【elasticsearch-analysis-jieba】项目，将这个【elasticsearch-jieba-plugin】这个项目的代码复制到新建的项目中，因为这个【elasticsearch-jieba-plugin】使用的是gradle管理，我想使用的是maven仓库，所以修改了一下。

4.1 新增pom.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <name>elasticsearch-analysis-jieba</name>
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-analysis-jieba</artifactId>
    <version>${elasticsearch.version}</version>
    <packaging>jar</packaging>
    <description>jieba Analyzer for Elasticsearch</description>
    <inceptionYear>2011</inceptionYear>

    <properties>
        <elasticsearch.version>8.7.0</elasticsearch.version>
        <maven.compiler.target>17</maven.compiler.target>
        <elasticsearch.assembly.descriptor>${project.basedir}/src/main/assemblies/plugin.xml</elasticsearch.assembly.descriptor>
        <elasticsearch.plugin.name>analysis-jieba</elasticsearch.plugin.name>
        <elasticsearch.plugin.classname>org.elasticsearch.plugin.analysis.jieba.AnalysisJiebaPlugin</elasticsearch.plugin.classname>
        <elasticsearch.plugin.jvm>true</elasticsearch.plugin.jvm>
        <tests.rest.load_packaged>false</tests.rest.load_packaged>
        <skip.unit.tests>true</skip.unit.tests>
    </properties>

    <licenses>
        <license>
            <name>The Apache Software License, Version 2.0</name>
            <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <developers>
        <developer>
            <name>INFINI Labs</name>
            <email>hello@infini.ltd</email>
            <organization>INFINI Labs</organization>
            <organizationUrl>https://infinilabs.com</organizationUrl>
        </developer>
    </developers>

    <parent>
        <groupId>org.sonatype.oss</groupId>
        <artifactId>oss-parent</artifactId>
        <version>9</version>
    </parent>

    <distributionManagement>
        <snapshotRepository>
            <id>oss.sonatype.org</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </snapshotRepository>
        <repository>
            <id>oss.sonatype.org</id>
            <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
        </repository>
    </distributionManagement>

    <repositories>
        <repository>
            <id>oss.sonatype.org</id>
            <name>OSS Sonatype</name>
            <releases><enabled>true</enabled></releases>
            <snapshots><enabled>true</enabled></snapshots>
            <url>https://oss.sonatype.org/content/repositories/releases/</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>${elasticsearch.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>com.huaban</groupId>
            <artifactId>jieba-analysis</artifactId>
            <version>1.0.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.19.0</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13.2</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>${maven.compiler.target}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.11</version>
                <configuration>
                    <includes>
                        <include>**/*Tests.java</include>
                    </includes>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>2.1.2</version>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>

                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <outputDirectory>${project.build.directory}/releases/</outputDirectory>
                    <descriptors>
                        <descriptor>${basedir}/src/main/assemblies/plugin.xml</descriptor>
                    </descriptors>
                    <archive>
                        <manifest>
                            <mainClass>fully.qualified.MainClass</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <profiles>
        <profile>
            <id>disable-java8-doclint</id>
            <activation>
                <jdk>[1.8,)</jdk>
            </activation>
            <properties>
                <additionalparam>-Xdoclint:none</additionalparam>
            </properties>
        </profile>
            <profile>
                <id>release</id>
                <build>
                    <plugins>
                        <plugin>
                            <groupId>org.sonatype.plugins</groupId>
                            <artifactId>nexus-staging-maven-plugin</artifactId>
                            <version>1.6.3</version>
                            <extensions>true</extensions>
                            <configuration>
                                <serverId>oss</serverId>
                                <nexusUrl>https://oss.sonatype.org/</nexusUrl>
                                <autoReleaseAfterClose>true</autoReleaseAfterClose>
                            </configuration>
                        </plugin>
                        <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-release-plugin</artifactId>
                            <version>2.1</version>
                            <configuration>
                                <autoVersionSubmodules>true</autoVersionSubmodules>
                                <useReleaseProfile>false</useReleaseProfile>
                                <releaseProfiles>release</releaseProfiles>
                                <goals>deploy</goals>
                            </configuration>
                        </plugin>
                        <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-compiler-plugin</artifactId>
                            <version>3.5.1</version>
                            <configuration>
                                <source>${maven.compiler.target}</source>
                                <target>${maven.compiler.target}</target>
                            </configuration>
                        </plugin>
                        <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-gpg-plugin</artifactId>
                            <version>1.5</version>
                            <executions>
                                <execution>
                                    <id>sign-artifacts</id>
                                    <phase>verify</phase>
                                    <goals>
                                        <goal>sign</goal>
                                    </goals>
                                </execution>
                            </executions>
                        </plugin>
                        <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-source-plugin</artifactId>
                            <version>2.2.1</version>
                            <executions>
                                <execution>
                                    <id>attach-sources</id>
                                    <goals>
                                        <goal>jar-no-fork</goal>
                                    </goals>
                                </execution>
                            </executions>
                        </plugin>
                        <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-javadoc-plugin</artifactId>
                            <version>2.9</version>
                            <executions>
                                <execution>
                                    <id>attach-javadocs</id>
                                    <goals>
                                        <goal>jar</goal>
                                    </goals>
                                </execution>
                            </executions>
                        </plugin>
                    </plugins>
                </build>
            </profile>
    </profiles>
</project>

4.2 修改plugin-descriptor.properties文件

# Elasticsearch plugin descriptor file
# This file must exist as 'plugin-descriptor.properties' at
# the root directory of all plugins.
#
# A plugin can be 'site', 'jvm', or both.
#
### example site plugin for "foo":
#
# foo.zip <-- zip file for the plugin, with this structure:
#   _site/ <-- the contents that will be served
#   plugin-descriptor.properties <-- example contents below:
#
# site=true
# description=My cool plugin
# version=1.0
#
### example jvm plugin for "foo"
#
# foo.zip <-- zip file for the plugin, with this structure:
#   <arbitrary name1>.jar <-- classes, resources, dependencies
#   <arbitrary nameN>.jar <-- any number of jars
#   plugin-descriptor.properties <-- example contents below:
#
# jvm=true
# classname=foo.bar.BazPlugin
# description=My cool plugin
# version=2.0.0-rc1
# elasticsearch.version=2.0
# java.version=1.7
#
### mandatory elements for all plugins:
#
# 'description': simple summary of the plugin
description=${project.description}
#
# 'version': plugin's version
version=${project.version}
#
# 'name': the plugin name
name=${elasticsearch.plugin.name}
#
# 'classname': the name of the class to load, fully-qualified.
classname=${elasticsearch.plugin.classname}
#
# 'java.version' version of java the code is built against
# use the system property java.specification.version
# version string must be a sequence of nonnegative decimal integers
# separated by "."'s and may have leading zeros
java.version=${maven.compiler.target}
#
# 'elasticsearch.version' version of elasticsearch compiled against
# You will have to release a new version of the plugin for each new
# elasticsearch release. This version is checked when the plugin
# is loaded so Elasticsearch will refuse to start in the presence of
# plugins with the incorrect elasticsearch.version.
elasticsearch.version=${elasticsearch.version}

4.3 新增plugin-security.policy文件

grant {
  // needed because of the hot reload functionality
  permission java.net.SocketPermission "*", "connect,resolve";
  permission java.lang.RuntimePermission "setContextClassLoader";
};

4.4 构建插件

打包

找到打包之后的zip包

放到elasticsearch-8.7.0/plugin/analysis-jieba目录下。

现在，再手动重启一下es就将elasticsearch-analysis-jieba分词器安装好啦。

五、测试jieba分词器

在kibana中创建索引

PUT jieba_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_ana": {
          "tokenizer": "jieba_index",
          "filter": [
            "lowercase"
          ]
        }
      }
    }
  }
}

文本分词器

PUT jieba_index/_analyze
{
  "analyzer" : "my_ana",
  "text" : "黄河之水天上来"
}

返回结果

{
    "tokens": [
        {
            "token": "黄河",
            "start_offset": 0,
            "end_offset": 2,
            "type": "word",
            "position": 0
        },
        {
            "token": "黄河之水天上来",
            "start_offset": 0,
            "end_offset": 7,
            "type": "word",
            "position": 0
        },
        {
            "token": "之水",
            "start_offset": 2,
            "end_offset": 4,
            "type": "word",
            "position": 1
        },
        {
            "token": "天上",
            "start_offset": 4,
            "end_offset": 6,
            "type": "word",
            "position": 2
        },
        {
            "token": "上来",
            "start_offset": 5,
            "end_offset": 7,
            "type": "word",
            "position": 2
        }
    ]
}