ElasticSearch IK分词器基于mysql实现热更新词库

文档转载自：

采坑解决方案https://blog.csdn.net/hqd1870299/article/details/106660017/

ik源码修改参考：https://blog.csdn.net/Sterry_Gong/article/details/112347256

安装的是elasticsearch7.1.1

从https://github.com/medcl/elasticsearch-analysis-ik下载ik源码（导入idea）：

切换到tag v7.1.1

1：修改pom.xml 版本号改为自己的ES对应版本

虽然我们切到了tag v7.1.1,但是pom文件中es的版本依然配置的是7.0.0 ，需要改为7.1.1 ,要和本地安装的es版本一致，我本地安装的就是7.1.1的

<elasticsearch.version>7.1.1</elasticsearch.version>

2.添加mysql的jar包（和mysql数据库对应）

 <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.16</version>
        </dependency>

3. 新增数据源
   在conf目录下新增jdbc-reload.properties

jdbc.url=jdbc:mysql://localhost:3306/test?serverTimezone=GMT
jdbc.user=root
jdbc.password=123456
jdbc.reload.sql=select word from hot_words
jdbc.reload.stopword.sql=select stopword as word from hot_stopwords
jdbc.reload.interval=1000

4. 添加热更新线程HotDictReloadThread
   

public class HotDictReloadThread implements Runnable {

	private static final Logger LOGGER = ESPluginLoggerFactory.getLogger(HotDictReloadThread.class.getName());
	@Override
	public void run() {
		while(true) {
			LOGGER.info("[==========]reload hot dict from mysql......");
			Dictionary.getSingleton().reLoadMainDict();
		}
	}

}

5. 修改原文件Dictionary
   

//启动热词字典加载线程
new Thread(new HotDictReloadThread()).start();

private static Properties prop = new Properties();

//	static {
//		try {
//			Class.forName("com.mysql.jdbc.Driver");
//		} catch (ClassNotFoundException e) {
//			logger.error("error", e);
//		}
//	}

	/**
	 * 从mysql加载热更新词典
	 */
	private void loadMySQLExtDict() {
		Connection conn = null;
		Statement stmt = null;
		ResultSet rs = null;

		try {
			Path file = PathUtils.get(getDictRoot(), "jdbc-reload.properties");
			prop.load(new FileInputStream(file.toFile()));

			logger.info("[==========]jdbc-reload.properties");
			for(Object key : prop.keySet()) {
				logger.info("[==========]" + key + "=" + prop.getProperty(String.valueOf(key)));
			}

			logger.info("[==========]query hot dict from mysql, " + prop.getProperty("jdbc.reload.sql") + "......");

			conn = DriverManager.getConnection(
					prop.getProperty("jdbc.url"),
					prop.getProperty("jdbc.user"),
					prop.getProperty("jdbc.password"));
			stmt = conn.createStatement();
			rs = stmt.executeQuery(prop.getProperty("jdbc.reload.sql"));

			while(rs.next()) {
				String theWord = rs.getString("word");
				logger.info("[==========]hot word from mysql: " + theWord);
				_MainDict.fillSegment(theWord.trim().toCharArray());
			}

			Thread.sleep(Integer.valueOf(String.valueOf(prop.get("jdbc.reload.interval"))));
		} catch (Exception e) {
			logger.error("erorr", e);
		} finally {
			if(rs != null) {
				try {
					rs.close();
				} catch (SQLException e) {
					logger.error("error", e);
				}
			}
			if(stmt != null) {
				try {
					stmt.close();
				} catch (SQLException e) {
					logger.error("error", e);
				}
			}
			if(conn != null) {
				try {
					conn.close();
				} catch (SQLException e) {
					logger.error("error", e);
				}
			}
		}
	}

loadStopWordDict() 方法末尾增加
   this.loadMySQLStopwordDict();

/**
	 * 从mysql加载停用词
	 */
	private void loadMySQLStopwordDict() {
		Connection conn = null;
		Statement stmt = null;
		ResultSet rs = null;

		try {
			Path file = PathUtils.get(getDictRoot(), "jdbc-reload.properties");
			prop.load(new FileInputStream(file.toFile()));

			logger.info("[==========]jdbc-reload.properties");
			for(Object key : prop.keySet()) {
				logger.info("[==========]" + key + "=" + prop.getProperty(String.valueOf(key)));
			}

			logger.info("[==========]query hot stopword dict from mysql, " + prop.getProperty("jdbc.reload.stopword.sql") + "......");

			conn = DriverManager.getConnection(
					prop.getProperty("jdbc.url"),
					prop.getProperty("jdbc.user"),
					prop.getProperty("jdbc.password"));
			stmt = conn.createStatement();
			rs = stmt.executeQuery(prop.getProperty("jdbc.reload.stopword.sql"));

			while(rs.next()) {
				String theWord = rs.getString("word");
				logger.info("[==========]hot stopword from mysql: " + theWord);
				_StopWords.fillSegment(theWord.trim().toCharArray());
			}

			Thread.sleep(Integer.valueOf(String.valueOf(prop.get("jdbc.reload.interval"))));
		} catch (Exception e) {
			logger.error("erorr", e);
		} finally {
			if(rs != null) {
				try {
					rs.close();
				} catch (SQLException e) {
					logger.error("error", e);
				}
			}
			if(stmt != null) {
				try {
					stmt.close();
				} catch (SQLException e) {
					logger.error("error", e);
				}
			}
			if(conn != null) {
				try {
					conn.close();
				} catch (SQLException e) {
					logger.error("error", e);
				}
			}
		}
	}

打包

通过mvn package，打包
本地是jdk 1.8，maven 3.3.9
本地安装好es 7.1.1 ，安装目录不能有空格，不能有中文
将elasticsearch-analysis-ik-7.1.1.zip拷贝到 安装目录\elasticsearch-7.1.1\plugins\analysis-ik下，并解压，解压后如图所示：

jdk 环境配置

将mysql驱动包拷贝到如下路径

修改 jdk/jre/lib/security下的 java.policy，在末尾添加

permission java.net.SocketPermission "127.0.0.1:3306","connect,resolve";

这个ip就是jdbc-reload.properties中配置的数据库ip地址

测试

在mysql中添加词库与停用词
添加新词之前

添加新词

常见问题解决

ES启动报错截图（就是加载驱动报错）：

注：在plugins目录下新建ik文件夹，解压打包的ik，启动ES不起作用，改为命令安装ik分词器,cmd执行（改为自己的目录）：

D:\soft\elasticsearch-7.7.1-node1\bin>elasticsearch-plugin install file:///D:\ideaWorkSpace\elasticsearch-analysis-ik-master\target\releases\elasticsearch-analysis-ik-7.7.1.zip

3：Could not create connection to database server

把mysql-connector-java的版本改成你连接的数据库的版本

4：no suitable driver found for jdbc

在下面目录加上mysql的jar包

5. 权限异常
   在jdk下的java.policy加上该权限，操作如下：
   
   
   6：成功启动ES后，窗口打印的分词汉字显示乱码：

jvm.options中添加-Dfile.encoding=GBK