ElasticSearch,简称es,是一个开源的高扩展的分布式全文检索引擎;近乎实时的存储、检索数据;扩展性好,可以扩展到上百台服务器,处理PB级别的数据。Es使用java开发基于lucene作为核心实现所有索引和搜索功能。它通过简单的RESTFul API来隐藏lucene的复杂性,让全文检索变得简单。
维基百科,百度,新浪和阿里等大公司都有使用es构建数据分析系统或日志采集分析体系。
Es自身自带分布式协调功能;仅支持json格式;高级功能多由第三方插件提供;处理实时搜索效率高。
- 二,安装与启动
- 从官网下载安装包。解压到任意的目录,在windows下,执行bin目录下的elasticsearch.bat。es自带的web服务端口是9200,但是没有提供图形化界面。
- 需要安装elasticsearch-head插件。下载elasticsearch-head-master插件压缩包。将压缩包解压到区别于es解压包所在目录的任意目录。
- Elasticsearch-head-master运行在node.js上,需要下载安装node.js。
- 安装完毕后,将grunt安装为全局命令,grunt基于node.js的项目构建工具。在cmd中输入:
npm install -g grunt-cli
5. 进入elasticsearch-head-master目录启动head,在命令提示符输入命令:
npm install
和
grunt server
6. 通过locahost:9100打开图形界面。
7. 需要连接localhost:9200,由于访问localhost:9200会跨域,需要修改
elasticserver/config/elasticsearch.yml,添加:
http.cors.enable: true
http.cors.allow-origin: "*"
注意,冒号后面需要空格。
8. 打开localhost:9100,连接elasticsearch。
- 三,Es客户端操作
- 在elasticsearch-head中新建索引
索引》新建索引,填写索引名称,比如index-username,默认分片数5,副本1。点击概览,可以查看创建的索引。
2. 使用postman创建索引。
Es使用RESTFul API,get查put增delete删post改。http://localhost:9200/app,发送请求,创建index为app的索引。返回json
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "app"
}
在elasticsearch-head的概览中查看索引。
可以创建索引时,使用postman设置mapping,使用put方式url:http://localhost:9200/app2
请求体:
{
"mappings":{
"article":{
"properties":{
"id":{
"type":"long",
"store":true
},
"title":{
"type":"text",
"store":true,
"index":true,
"analyzer":"standard"
},
"content":{
"type":"text",
"store":true,
"index":true,
"analyzer":"standard"
}
}
}
}
}
3. 使用postman的post方法创建索引设置mapping的type
url填写:localhost:9200/app2/hello/_mappings
请求体:
{
"hello":{
"properties":{
"id":{
"type":"long",
"store":true
},
"title":{
"type":"text",
"store":true,
"index":true,
"analyzer":"standard"
},
"content":{
"type":"text",
"store":true,
"index":true,
"analyzer":"standard"
}
}
}
}
4. 使用delete命令删除索引
url:localhost:9200/app1
5. 在es-head中可以通过复合查询做增删改查
6. 向索引库添加文档
使用postman,在url中选择post,url:localhost:9200/app2/article/1,请求体:
{
"id":1,
"title":"春晓",
"content":"春眠不觉晓,处处闻啼鸟。夜来风雨声,花落知多少。"
}
然后,成功后,会收到响应体:
{
"_index": "app2",
"_type": "article",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}
在es-head中查看,数据浏览》选择相应的索引,通过字段填写值可以过滤查询。
使用head的复合查询实现
7. 删除文档
使用postman,选择delete方式,url:localhost:9200/_index/_type/1,比如localhost:9200/app2/article/1
不需要请求体。
使用es-head的复合查询也可以实现删除。
8. 修改文档
修改的底层逻辑是先删除再添加
使用postman,post方式url:localhost:9200/app/article/1
请求体:
{
"id":1,
"title":"春晓修改",
"content":"春眠不觉晓,处处闻啼鸟。夜来风雨声,花落知多少。"
}
使用es-head的复合查询同样可以实现修改。
9. 根据id查询文档
使用postman的get方式,url:localhost:9200/app/article/3
得到结果:
{
"_index": "app",
"_type": "article",
"_id": "3",
"_version": 1,
"found": true,
"_source": {
"id": 3,
"title": "春晓修3",
"content": "春眠不觉晓,处处闻啼鸟。夜来风雨声,花落知多少。"
}
}
10. 根据关键词查询
使用postman的post方式,url:localhost:9200/app/article/_search,请求体:
{
"query":{
"term":{
"title":"修"
}
}
}
11. 使用queryString查询
queryString方式,会先对查询的关键词进行分词。
post请求url,localhost:9200/app/article/_search
请求体:
{
"query":{
"query_string":{
"default_field":"content",
"query":"处处"
}
}
}
12. 在es中查看分析器的分词效果
Get方式url:
localhost:9200/_analyze?analyzer=standard&pretty=true&text=处处闻啼鸟
localhost:9200/_analyze?analyzer=standard&pretty=true&text=Get more sleep. Seriously. Just go to bed and get some sleep.
13. 集成IK分词器
下载elasticsearch-analyzer-ik-5.x.x,解压。在es的plugins下新建ik-analyzer目录,复制里面的内容到这个目录。重启es服务。
[2019-11-17T18:03:45,787][INFO ][o.w.a.d.Monitor ] try load config from F:\xxx\xxx\xxx\
elasticSearch\elasticsearch-5.6.8\config\analysis-ik\IKAnalyzer.cfg.xml
可以看到日志中,打印加载ik分词器的配置文件。
请求get发送url:localhost:9200/_analyze?analyzer=ik_smart&pretty=true&text=世界就是大
使用ik_smart的算法分词结果。
使用ik_max_word查询,localhost:9200/_analyze?analyzer=ik_max_word&pretty=true&text=世界就是大大好
14. 使用中文分析器
新建索引库,指定索引器为ik_smart。Put方式url:http://localhost:9200/app/
{
"mappings":{
"article":{
"properties":{
"id":{
"type":"long",
"store":true
},
"title":{
"type":"text",
"store":true,
"index":true,
"analyzer":"ik_smart"
},
"content":{
"type":"text",
"store":true,
"index":true,
"analyzer":"ik_smart"
}
}
}
}
}
使用post方式添加数据。
查询
post请求url,localhost:9200/app/article/_search
请求体:
{
"query":{
"query_string":{
"default_field":"content",
"query":"春节好瀑布"
}
}
}
- 四,ES集群
ES集群是一个p2p类型的分布式系统,除了集群状态管理外,其他所有的请求都可以发送到集群内任意一台节点上,这个节点可以自己找到需要转发给哪些节点,并且直接跟这些节点通信。从网络架构及服务配置上来说,构建集群所需要的配置很简单。
搭建ES集群
在一台服务器上创建3个节点。复制elasticsearch目录到server1,需要删除data目录,注意,新的节点中不能有data目录否则创建失败。修改config下的elasticsearch.yml文件,添加集群配置
#节点1的配置信息:
#集群名称,保证唯一
cluster.name: s-elasticsearch
#节点名称,必须不一样
node.name: node-1
#必须为本机的ip地址
network.host: 127.0.0.1
#服务端口号,在同一机器下必须不一样
http.port: 9201
#集群间通信端口号,在同一机器下必须不一样
transport.tcp.port: 9301
#设置集群自动发现机器ip集合
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
再复制2份分别为server2和server3。
启动3个节点,打开head,连接任意一个节点,查看是否启动集群成功。
使用测试
使用head创建一个索引。查看集群概览。测试添加数据。
- 五,Java客户端管理ES
- Java客户端创建索引
步骤:
a,使用idea创建一个Empty Project,ElasticSearchClient
b,在工程中添加module,创建maven工程es-client-1,不勾选create from archetype,
c,Pom文件添加依赖
<dependencies>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>5.6.8</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.6.8</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.24</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.21</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.12</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
2. 编写java代码创建索引库
public class ElasticSearchClientTest {
@Test
public void createIndex() {
Settings settings = Settings.builder().put("cluster.name", "s-elasticsearch").build();
PreBuiltTransportClient client = null;
try {
client = new PreBuiltTransportClient(settings);
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9301));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9302));
client.admin().indices().prepareCreate("index_app2").get();
} catch (Exception e){
} finally {
client.close();
}
}
}
3. 创建mapping
@Test
public void setSettings() throws Exception{
Settings settings = Settings.builder().put("cluster.name", "s-elasticsearch").build();
TransportClient client = null;
try{
client = new PreBuiltTransportClient(settings)
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9301));
XContentBuilder xcb = XContentFactory.jsonBuilder()
.startObject()
.startObject("article4")
.startObject("properties")
.startObject("id")
.field("type", "integer")
.field("store", true)
.endObject()
.startObject("title")
.field("type", "string")
.field("store", "yes")
.field("analyzer", "ik_smart")
.endObject()
.startObject("content")
.field("type", "string")
.field("store", "yes")
.field("analyzer", "ik_smart")
.endObject()
.endObject()
.endObject()
.endObject();
// client.admin().indices()
// .preparePutMapping("app2")
// .setType("article2")
// .setSource(xcb)
// .get();
// PutMappingRequest mapping = Requests.putMappingRequest("app2").type("article3").source(xcb);
// client.admin().indices().putMapping(mapping).get();
PutMappingRequest mapping = Requests.putMappingRequest("app2")
.type("article4").source(xcb);
client.admin().indices().putMapping(mapping).get();
} catch (Exception e){
e.printStackTrace();
} finally {
client.close();
}
}
4. 向索引库添加文档
使用XContentBuilder创建文档
TransportClient client;
@Before
public void init() throws Exception{
Settings settings = Settings.builder().put("cluster.name", "s-elasticsearch").build();
client = new PreBuiltTransportClient(settings);
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9301));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9302));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9303));
}
@Test
public void testAddDoc() throws Exception{
XContentBuilder builder = XContentFactory.jsonBuilder()
.startObject()
.field("id", 1)
.field("title", "春晓")
.field("content", "出门不觉晓,花落是大大大;出出来了,一月又一月。")
.endObject();
client.prepareIndex()
.setIndex("app2")
.setType("article")
.setId("1")
.setSource(builder)
.get();
client.close();
}
使用fastjson、jackson等jar包转换json对象获取json字符串创建文档
引入fastjson的依赖
TransportClient client;
@Before
public void init() throws Exception{
Settings settings = Settings.builder().put("cluster.name", "s-elasticsearch").build();
client = new PreBuiltTransportClient(settings);
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9301));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9302));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9303));
}
@Test
public void testAddDoc2() throws Exception{
Article a = new Article();
a.setId(2);
a.setTitle("望路上");
a.setContent("千山鸟飞机,分惊人中美,日照销量生子一,指引身在刺蛇。");
ObjectMapper om = new ObjectMapper();
String jsonA = om.writeValueAsString(a);
client.prepareIndex()
.setIndex("app2")
.setType("article")
.setId("2")
.setSource(jsonA, XContentType.JSON)
.get();
client.close();
}
5. 查询文档
a. 根据id搜索
TransportClient client;
@Before
public void init() throws Exception{
Settings settings = Settings.builder().put("cluster.name", "s-elasticsearch").build();
client = new PreBuiltTransportClient(settings);
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9301));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9302));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9303));
}
@Test
public void searchById() throws Exception{
IdsQueryBuilder qb = QueryBuilders.idsQuery().addIds("1", "2");
SearchResponse sr = client.prepareSearch("app2")
.setTypes("article")
.setQuery(qb)
.get();
SearchHits hits = sr.getHits();
System.out.println("查询结果:" + hits.getTotalHits());
Iterator<SearchHit> iterator = hits.iterator();
while(iterator.hasNext()){
SearchHit sh = iterator.next();
System.out.println(sh.getSourceAsString());
Map<String, Object> sourceMap = sh.getSource();
System.out.println(sourceMap);
}
client.close();
}
b. 根据Team搜索
TransportClient client;
@Before
public void init() throws Exception{
Settings settings = Settings.builder().put("cluster.name", "s-elasticsearch").build();
client = new PreBuiltTransportClient(settings);
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9301));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9302));
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9303));
}
private void search(QueryBuilder qb){
SearchResponse sr = client.prepareSearch("app2")
.setTypes("article")
.setQuery(qb)
.get();
SearchHits hits = sr.getHits();
System.out.println("查询结果:" + hits.getTotalHits());
Iterator<SearchHit> iterator = hits.iterator();
while(iterator.hasNext()){
SearchHit sh = iterator.next();
System.out.println(sh.getSourceAsString());
Map<String, Object> sourceMap = sh.getSource();
System.out.println(sourceMap);
}
client.close();
}
@Test
public void searchById() throws Exception{
IdsQueryBuilder qb = QueryBuilders.idsQuery().addIds("1", "2");
search(qb);
}
@Test
public void searchByTerm() throws Exception{
TermQueryBuilder tqb = QueryBuilders.termQuery("content", "飞机");
search(tqb);
}
c. QueryString查询方式(带分析查询)
@Test
public void queryString() throws Exception {
QueryStringQueryBuilder qb = QueryBuilders.queryStringQuery("飞机上又一月").defaultField("content");
search(qb);
}
e. 查询分页设置
private void search(QueryBuilder qb){
SearchResponse sr = client.prepareSearch("app2")
.setTypes("article")
.setQuery(qb)
.setFrom(0)
.setSize(5)
.get();
SearchHits hits = sr.getHits();
System.out.println("查询结果:" + hits.getTotalHits());
Iterator<SearchHit> iterator = hits.iterator();
while(iterator.hasNext()){
SearchHit sh = iterator.next();
System.out.println(sh.getSourceAsString());
Map<String, Object> sourceMap = sh.getSource();
System.out.println(sourceMap);
}
client.close();
}
f. 查询结果高亮显示
private void search(QueryBuilder qb, String highlight){
HighlightBuilder hb = new HighlightBuilder();
hb.field("content").preTags("<em>").postTags("</em>");
SearchResponse sr = client.prepareSearch("app2")
.setTypes("article")
.setQuery(qb)
.setFrom(0)
.setSize(5)
.highlighter(hb)
.get();
SearchHits hits = sr.getHits();
System.out.println("查询结果:" + hits.getTotalHits());
Iterator<SearchHit> iterator = hits.iterator();
while(iterator.hasNext()){
SearchHit sh = iterator.next();
System.out.println(sh.getSourceAsString());
Map<String, Object> sourceMap = sh.getSource();
System.out.println(sourceMap);
System.out.println("***************高亮结果******************");
Map<String, HighlightField> highlightFields = sh.getHighlightFields();
System.out.println(highlightFields);
HighlightField highlightField = highlightFields.get(highlight);
Text[] fragments = highlightField.fragments();
for(Text t : fragments){
System.out.println(t);
}
}
client.close();
}
@Test
public void searchByTerm() throws Exception{
TermQueryBuilder tqb = QueryBuilders.termQuery("content", "飞机");
search(tqb, "content");
}
六,Spring Data Elasticsearch
Spring data是一个用户简化数据库访问,并支持云服务的开源框架。其主要目的是使得对数据的访问变得方便快捷,并支持map-reduce框架和云计算数据服务。Spring data elasticsearch是基于spring data api简化es操作,将原始操作的客户端api进行封装。Spring data为es项目提供集成搜索引擎。
- 创建spring data elasticsearch工程
引入依赖
<dependencies>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>5.6.8</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.6.8</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.24</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.21</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.12</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
<version>3.0.5.RELEASE</version>
<exclusions>
<exclusion>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>transport-netty4-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>5.0.4.RELEASE</version>
</dependency>
</dependencies>
添加applicationContext.xml配置文件
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/data/elasticsearch
http://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch-1.0.xsd
">
<!--elasticsearch客户端配置-->
<elasticsearch:transport-client id="esClient" cluster-name="s-elasticsearch"
cluster-nodes="127.0.0.1:9301;127.0.0.1:9302" />
<!--包扫描器,扫描dao接口-->
<elasticsearch:repositories base-package="com.s.es.repository" />
<!--es模板类-->
<bean id="elasticsearchTemplate" class="org.springframework.data.elasticsearch.core.ElasticsearchTemplate">
<constructor-arg name="client" ref="esClient" />
</bean>
</beans>
2. 创建entity类
@Document(indexName = "app3", type = "article")
public class Article {
@Id
@Field(type = FieldType.Integer, store=true)
private Integer id;
@Field(type = FieldType.text, store = true, analyzer = "ik_smart")
private String title;
@Field(type = FieldType.text, store = true, analyzer = "ik_smart")
private String content;
public Integer getId() {
return id;
}
public void setId(Integer id) {
this.id = id;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public String getContent() {
return content;
}
public void setContent(String content) {
this.content = content;
}
}
3. 创建类型持久化接口
public interface IArticleRepository extends ElasticsearchRepository<Article, Integer> {
}
4. 创建索引
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:applicationContext.xml")
public class SpringDataElasticsearchTest {
@Autowired
private IArticleRepository articleRepository;
@Autowired
private ElasticsearchTemplate elasticsearchTemplate;
@Test
public void createIndex() throws Exception{
elasticsearchTemplate.createIndex(Article.class);
}
}
5. 添加文档
@Test
public void createDocument() throws Exception{
Article a = new Article();
a.setId(1);
a.setTitle("今天小雪节气");
a.setContent("“花雪随风不厌看,一片飞来一片寒。”《中国天文年历》显示,北京时间11月22日22时59分迎来“小雪”节气。此时节,伴随着冷空气入侵,我国多地呈现出“荷尽已无擎雨盖,菊残");
articleRepository.save(a);
}
6. 删除文档
@Test
public void deleteById() throws Exception{
articleRepository.deleteById(1);
}
7. 修改文档
更新文档,实际上是增加一个与需要更新的文档id相同的文档。
8. 简单查询
@Test
public void findAll() throws Exception{
Iterable<Article> articles = articleRepository.findAll();
Iterator<Article> iterator = articles.iterator();
while(iterator.hasNext()){
Article ar = iterator.next();
System.out.println(ar.toString());
}
}
@Test
public void findById() throws Exception{
Optional<Article> ar = articleRepository.findById(1);
System.out.println(ar.get().toString());
}
9. 自定义查询方法查询
在实体持久化接口增加方法
public interface IArticleRepository extends ElasticsearchRepository<Article, Integer> {
List<Article> findByTitle(String title);
List<Article> findByTitleOrContent(String title, String content);
List<Article> findByTitleOrContent(String title, String content, PageRequest page);
}
查询测试
@Test
public void findByTitle() throws Exception{
List<Article> ars = articleRepository.findByTitle("小雪");
ars.stream().forEach(a -> System.out.println(a));
}
@Test
public void findByTitleOrContent() throws Exception{
List<Article> list = articleRepository.findByTitleOrContent("一片", "一片");
list.stream().forEach(a -> System.out.println(a));
System.out.println("*************************************************");
PageRequest page = PageRequest.of(0, 5);
List<Article> list2 = articleRepository.findByTitleOrContent("一片", "一片", page);
list2.stream().forEach(a -> System.out.println(a));
}
10. NativeSearchQuery查询
@Test
public void testNativeSearchQuery() throws Exception{
NativeSearchQuery build = new NativeSearchQueryBuilder()
.withQuery(QueryBuilders.queryStringQuery("小雪一片一口口").defaultField("title"))
.withPageable(PageRequest.of(0, 5))
.build();
List<Article> list = elasticsearchTemplate.queryForList(build, Article.class);
list.stream().forEach(a -> System.out.println(a));
}