elasticsearch初装笔记

在如今这个数据为王的时代，对于数据处理有了越来越多的需求，毫无疑问elasticsearch是OLTP的一个主流方案

关于elasticsearch的介绍会在其它篇章中介绍，这里只做安装部署的介绍

在这个开源的时代，各个组件拼的是生态

这里主要介绍elasticsearch和其中两个主要的组件

elasticsearch

elasticsearch-analysis-ik

elasticsearch-sql

因为好久没有部署了，重新走一遍流程以便记录

先到elasticsearch官网下载elasticsearch的包，我选择的是6.8.0，为什么选择6.8.0，是因为写这篇博客的时候，elasticsearch-sql组件的最高版本是6.8.0

官网地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch找到你要的版本，并下载

首先：elasticsearch启动不允许使用root用户，所以要先简历专属于elasticsearch的用户组和用户

下载完成后，解压tar包：tar -xvf elasticsearch-6.8.0.tar.gz

进入elasticsearch-6.8.0

安装组件

1.分词组件选择了ik，目前主流的还可以选择elasticsearch-analysis-ansj，这两种都支持自定义分词

bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.8.0/elasticsearch-analysis-ik-6.8.0.zip

2.安装sql插件，使得查询时支持类sql的查询

bin/elasticsearch-plugin install https://github.com/NLPchina/elasticsearch-sql/releases/download/6.8.0.0/elasticsearch-sql-6.8.0.0.zip

安装好了以后可以在elasticsearch-6.8.0/plugins目录下找到这么两个文件夹

由于今天要安装的是集群，所以，这边需要准备至少两台机子，都安装完成

接下来介绍一下elasticsearch的配置文件:elasticsearch.yml

集群名称    cluster.name: my-application

节点名称    node.name: node-1

es的数据存储目录  path.data: /path/to/data

es的日志存储目录  path.logs: /path/to/logs

锁定物理内存地址，防止elasticsearch内存被交换出去,也就是避免es使用swap交换分区  bootstrap.memory_lock: true

为es设置ip绑定   network.host: 192.168.0.1

为es设置自定义端口 http.port: 9200

默认节点列表  discovery.zen.ping.unicast.hosts: ["host1", "host2"]

后面安装elasticsearch_head插件用到的配置

http.cors.enabled: true

http.cors.allow-origin: "*"

http.cors.allow-credentials: false

http.cors.allow-headers: WWW-Authenticate,X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization

配置主节点挂了后，选举数量限制 ，防止脑裂（建议先了解什么是脑裂）discovery.zen.minimum_master_nodes: 3

一个集群中的N个节点启动后,才允许进行数据恢复处理，默认是1 gateway.recover_after_nodes: 3

在一台服务器上禁止启动多个es服务  node.max_local_storage_nodes: 1

设置是否可以通过正则或者_all删除或者关闭索引库  action.destructive_requires_name: true

配置文件修改完成后，启动elsticsearch

启动后报错，以下为踩坑：

需要增加几个参数：

1.修改/etc/security/limits.conf文件，增加配置

* soft nofile 65536

* hard nofile 65536

* soft nproc 4096

* hard nproc 4096

2.修改/etc/sysctl.conf文件

vm.max_map_count=262144   并执行：sysctl -p

3.discovery.zen.ping.unicast.hosts: ["192.168.4.84:9300", "192.168.4.151:9300"]

这个配置里面的端口并不是http端口，而是默认的9300，供于集群发现的一个端口，并且配置在这个属性里面的节点属性node.master都必须为true

启动成功展示：

 

新建一个索引

 curl -H "Content-Type: application/json" -XPOST  http://192.168.4.151:9101/blog/article/ -d '{"title": "New version of Elasticsearch released!", "content": "Version 1.0 released today!", "tags": ["announce", "elasticsearch", "release"] }'

返回：

查询：

还有一种方式建立索引

先将mapping映射关系建立到一个文件中

vi ps_sms_product.json

 

{ 
 "mappings": { 
 "ps_sms_product": { 
 "properties": {
  "balePrice": {"type": "float"},
  "brand": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "brandId": {"type": "long"},
  "categoryId": {"type": "long"},
  "costPrice": {"type": "float"},
  "createTime": {"type": "date"},
  "creator": {"type": "text","analyzer":"ik_max_word","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "creatorId": {"type": "long"},
  "deleteTime": {"type": "date"},
  "employeeId": {"type": "long"},
  "employeeName": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "firstCategoryId": {"type": "long"},
  "firstCategoryName": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "id": {"type": "long"},
  "image": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "lastOperateId": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "lastOperateName": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "lastOperateTime": {"type": "date"},
  "manufacturerCode": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "modifyTime": {"type": "date"},
  "picurl": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "price": {"type": "float"},
  "productCode": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "productName": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "saleNum": {"type": "long"},
  "secondCategoryId": {"type": "long"},
  "secondCategoryName": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}},
  "status": {"type": "long"},
  "store": {"type": "long"},
  "supplierId": {"type": "long"},
  "tenantId": {"type": "long"},
  "valid": {"type": "long"},
  "version": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}
  }
 } 
 } 
}

然后执行：curl -H "Content-Type: application/json" -XPUT  'http://192.168.4.151:9101/ps_sms_product' -d @ps_sms_product.json

 

给之前创建的索引添加数据：

新建data.json

{"index": {"_type":"ps_sms_product"}} 
{"balePrice": 1.1,"brand": "品牌","brandId": 1,"categoryId": 1,"costPrice":2.2,"creator":"帅哥美女","employeeName":"你大爷","manufacturerCode":"A123456789","productCode":"123456","productName":"帅哥"}

{"index": {"_type":"ps_sms_product"}} 
{"balePrice": 1.1,"brand": "品牌","brandId": 1,"categoryId": 1,"costPrice":2.2,"creator":"帅哥","employeeName":"你","manufacturerCode":"A123456789","productCode":"123456","productName":"帅"}

{"index": {"_type":"ps_sms_product"}} 
{"balePrice": 1.1,"brand": "品牌","brandId": 1,"categoryId": 1,"costPrice":2.2,"creator":"帅哥美女你好","employeeName":"大爷","manufacturerCode":"A123456789","productCode":"123456","productName":"美女"}

执行：curl -H "Content-Type: application/json" -s -XPOST 'http://192.168.4.151:9101/ps_sms_product/_bulk?pretty' --data-binary @data_sms_product.json