mysql 中文分词 查找_中文分词技术

# MySQL中文分词

全文索引大体分为两个过程：

* 索引创建(indexer)：将现实世界中所有的结构化数据和非结构化数据提取信息，创建索引的过程

* 搜索索引(search)：就是得到用户的查询请求，搜索创建的索引，然后返回结果的过程

## 编译安装 sphinx+mmsg

### 0. 安装编译依赖工具包

```

yum install make gcc gcc-c++ libtool autoconf automake imake mysql-devel libxml2-devel expat-devel

```

### 下载稳定版源码包并解压

```

[root@localhost.localdomain /usr/local/src]

# wget http://www.coreseek.cn/uploads/csft/3.2/coreseek-3.2.14.tar.gz

[root@localhost.localdomain /usr/local/src]

# tar xf coreseek-3.2.14.tar.gz

[root@localhost.localdomain /usr/local/src]

# cd coreseek-3.2.14

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14]

# ls

csft-3.2.14(sphinx) mmseg-3.2.14 README.txt testpack

其中-- csft-4.1是修改适应了中文环境后的sphinx

Mmseg 是中文分词插件

Testpack是测试用的软件包

```

### [安装 mmseg](http://www.coreseek.cn/products/products-install/install_on_bsd_linux/)

#### cd mmseg

```

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14]

# cd mmseg-3.2.14/

```

#### 执行bootstrap脚本

```

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14/mmseg-3.2.14]

# ./bootstrap

```

#### ./configure --prefix=/usr/local/mmseg

```

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14/mmseg-3.2.14]

# ./configure --prefix=/usr/local/mmseg

```

#### make && make install

```

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14/mmseg-3.2.14]

# make && make install

```

### 安装coreseek

```

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14/csft-3.2.14]

# ./buildconf.sh

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14/csft-3.2.14]

# ./configure --prefix=/usr/local/coreseek --without-unixodbc --with-mmseg --with-mmseg-includes=/usr/local/mmseg/include/mmseg/ --with-mmseg-libs=/usr/local/mmseg/lib/ --with-mysql

[root@localhost.localdomain /usr/local/src/coreseek-3.2.14/csft-3.2.14]

# make && make install

```

## Sphinx的使用

> 1. 数据源---要让sphinx知道,查哪些数据,即针对哪些数据做索引(可以定义多个源)

> 2. 索引配置---针对哪个源做索引, 索引文件放在哪个目录?? 等等

> 3. 搜索服务器---sphinx可以在某个端口(默认9312),以其自身的协议,与外部程序做交互.

**配置数据源**

```

[root@localhost.localdomain /usr/local/coreseek/etc]

# cp sphinx.conf.dist sphinx.conf

[root@localhost.localdomain /usr/local/coreseek/etc]

# vim sphinx.conf

```

如下配置:

source src1 {

type = mysql

sql_host = localhost

sql_user = root

sql_pass = aaaaaa

sql_db = test

sql_query_pre = set names utf8

sql_query_pre = set session query_cache_type=off

sql_query = `select a_id as id,cat_id,title,simtitle,seotitle,tags,source,description,content,dateline,editdateline from article`

sql_attr_uint = a_id

sql_attr_uint = cat_id

sql_attr_timestamp = dateline

sql_attr_timestamp = editdateline

sql_query_info = `SELECT * FROM article WHERE a_id=$id`

}

**索引典型配置**

> index test1 {

> source = test

> path = /usr/local/sphinx/var/data/test1 # 生成索引放在哪

> # stopwords = G:\data\stopwords.txt

> # wordforms = G:\data\wordforms.txt

> # exceptions = /data/exceptions.txt

> charset_dictpath = /usr/local/mmseg/etc/

> charset_type = zh_cn.utf-8

> }

**生成索引文件**

```

[root@localhost.localdomain /usr/local/coreseek/etc]

# /usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/sphinx.conf test1 (test1为索引名称)

Coreseek Fulltext 3.2 [ Sphinx 0.9.9-release (r2117)]

Copyright (c) 2007-2011,

Beijing Choice Software Technologies Inc (http://www.coreseek.com)

using config file '/usr/local/coreseek/etc/sphinx.conf'...

indexing index 'test1'...

collected 8122 docs, 47.6 MB

sorted 8.7 Mhits, 100.0% done

total 8122 docs, 47596333 bytes

total 17.782 sec, 2676636 bytes/sec, 456.75 docs/sec

total 5 reads, 0.011 sec, 4559.8 kb/call avg, 2.3 msec/call avg

total 58 writes, 0.429 sec, 903.8 kb/call avg, 7.3 msec/call avg

```

> **Error 注意：**

> /usr/local/coreseek/bin/indexer: error while loading shared libraries: **libmysqlclient.so.18**: cannot open shared object file: No such file or directory

> 发现**sphinx**的`indexer`依赖库`ibmysqlclient.so.18`找不到，通过编辑此文件来修复这个错误 `/etc/ld.so.conf`

> `vi /etc/ld.so.conf `

> 将下面这句加到文件到尾部，并保存文件

> `/usr/local/mysql/lib `

> 然后运行下面这个命令即可

> `ldconfig`

在命令行测试查询

````

[root@localhost.localdomain /usr/local/coreseek]

# ./bin/search -c etc/sphinx.conf 留学

```