elasticsearch

elasticsearch的应用面很广，比如下面的例子，将MySQL和elasticsearch相结合：

公司播控平台的数据存储包括MySQL和ElasticSearch（ES）两个部分，编辑、运营的数据首先保存在MySQL中，为了实现模糊搜索和产品关联推荐，特别增加了ES，ES中保存的是节目集的基本信息。
本项目是为了防止实时同步数据出现问题或者系统重新初始化时的全量数据同步而做的。项目主要是从MySQL读取所有的节目集数据写入到ES中。

项目特点：
因为节目集数量较大，不能一次性的读入内存，因此每次读出一部分记录写入ES。ORM使用的是beego。为了提高性能使用了协程，其中读MySQL的部分最大开启20个协程，ES写入部分开启了15个协程（因为ES分片设置的问题，5个协程和15个协程性能映像不大）。

项目主要包括三个文件：
一个文档不仅仅包含它的数据 ，也包含 元数据 —— 有关 文档的信息。 三个必须的元数据元素如下：
_index文档在哪存放 。一个 索引 应该是因共同的特性被分组到一起的文档集合。 索引名必须小写，不能以下划线开头，不能包含逗号。
_type 文档表示的对象类别
_id 文档唯一标识

Elasticsearch 支持 RESTFUL 风格 API，其 API 基本格式如下：http://:/<索引>/<类型>/<文档id>

安装JDK，Node.js，在etc/profile进行环境变量的配置

wilson@wilson-virtual-machine:/myapps/elasticsearch-6.5.3/bin$ ./elasticsearch
[2018-12-17T14:50:26,522][INFO ][o.e.e.NodeEnvironment    ] [6l9CHPE] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [15.1gb], net total_space [44gb], types [ext4]
[2018-12-17T14:50:26,525][INFO ][o.e.e.NodeEnvironment    ] [6l9CHPE] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-12-17T14:50:26,528][INFO ][o.e.n.Node               ] [6l9CHPE] node name derived from node ID [6l9CHPEcS1SJf0aTFCpSvw]; set [node.name] to override
[2018-12-17T14:50:26,528][INFO ][o.e.n.Node               ] [6l9CHPE] version[6.5.3], pid[4443], build[default/tar/159a78a/2018-12-06T20:11:28.826501Z], OS[Linux/4.15.0-20-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_73/25.73-b02]
[2018-12-17T14:50:26,529][INFO ][o.e.n.Node               ] [6l9CHPE] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.ydHHiP6q, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/myapps/elasticsearch-6.5.3, -Des.path.conf=/myapps/elasticsearch-6.5.3/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-12-17T14:50:30,387][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [aggs-matrix-stats]
[2018-12-17T14:50:30,387][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [analysis-common]
[2018-12-17T14:50:30,387][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [ingest-common]
[2018-12-17T14:50:30,387][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [lang-expression]
[2018-12-17T14:50:30,388][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [lang-mustache]
[2018-12-17T14:50:30,388][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [lang-painless]
[2018-12-17T14:50:30,389][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [mapper-extras]
[2018-12-17T14:50:30,389][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [parent-join]
[2018-12-17T14:50:30,389][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [percolator]
[2018-12-17T14:50:30,389][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [rank-eval]
[2018-12-17T14:50:30,389][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [reindex]
[2018-12-17T14:50:30,390][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [repository-url]
[2018-12-17T14:50:30,390][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [transport-netty4]
[2018-12-17T14:50:30,390][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [tribe]
[2018-12-17T14:50:30,391][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-ccr]
[2018-12-17T14:50:30,391][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-core]
[2018-12-17T14:50:30,391][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-deprecation]
[2018-12-17T14:50:30,392][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-graph]
[2018-12-17T14:50:30,392][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-logstash]
[2018-12-17T14:50:30,392][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-ml]
[2018-12-17T14:50:30,392][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-monitoring]
[2018-12-17T14:50:30,393][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-rollup]
[2018-12-17T14:50:30,393][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-security]
[2018-12-17T14:50:30,393][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-sql]
[2018-12-17T14:50:30,394][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-upgrade]
[2018-12-17T14:50:30,394][INFO ][o.e.p.PluginsService     ] [6l9CHPE] loaded module [x-pack-watcher]
[2018-12-17T14:50:30,395][INFO ][o.e.p.PluginsService     ] [6l9CHPE] no plugins loaded
[2018-12-17T14:50:35,285][INFO ][o.e.x.s.a.s.FileRolesStore] [6l9CHPE] parsed [0] roles from file [/myapps/elasticsearch-6.5.3/config/roles.yml]
[2018-12-17T14:50:36,385][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [6l9CHPE] [controller/4505] [Main.cc@109] controller (64 bit): Version 6.5.3 (Build f418a701d70c6e) Copyright (c) 2018 Elasticsearch BV
[2018-12-17T14:50:36,941][DEBUG][o.e.a.ActionModule       ] [6l9CHPE] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-12-17T14:50:37,254][INFO ][o.e.d.DiscoveryModule    ] [6l9CHPE] using discovery type [zen] and host providers [settings]
[2018-12-17T14:50:38,282][INFO ][o.e.n.Node               ] [6l9CHPE] initialized
[2018-12-17T14:50:38,282][INFO ][o.e.n.Node               ] [6l9CHPE] starting ...
[2018-12-17T14:50:38,746][INFO ][o.e.t.TransportService   ] [6l9CHPE] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-12-17T14:50:38,769][WARN ][o.e.b.BootstrapChecks    ] [6l9CHPE] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-12-17T14:50:41,845][INFO ][o.e.c.s.MasterService    ] [6l9CHPE] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {6l9CHPE}{6l9CHPEcS1SJf0aTFCpSvw}{kZ61HICTRAGSD7t3Er2C1Q}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4112064512, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-12-17T14:50:41,854][INFO ][o.e.c.s.ClusterApplierService] [6l9CHPE] new_master {6l9CHPE}{6l9CHPEcS1SJf0aTFCpSvw}{kZ61HICTRAGSD7t3Er2C1Q}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4112064512, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {6l9CHPE}{6l9CHPEcS1SJf0aTFCpSvw}{kZ61HICTRAGSD7t3Er2C1Q}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4112064512, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-12-17T14:50:41,975][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [6l9CHPE] Failed to clear cache for realms [[]]
[2018-12-17T14:50:41,976][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [6l9CHPE] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-12-17T14:50:41,977][INFO ][o.e.n.Node               ] [6l9CHPE] started
[2018-12-17T14:50:42,074][INFO ][o.e.g.GatewayService     ] [6l9CHPE] recovered [0] indices into cluster_state
[2018-12-17T14:50:42,336][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2018-12-17T14:50:42,408][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2018-12-17T14:50:42,443][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.watches] for index patterns [.watches*]
[2018-12-17T14:50:42,488][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2018-12-17T14:50:42,613][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2018-12-17T14:50:42,680][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2018-12-17T14:50:42,710][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2018-12-17T14:50:42,741][INFO ][o.e.c.m.MetaDataIndexTemplateService] [6l9CHPE] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2018-12-17T14:50:42,842][INFO ][o.e.l.LicenseService     ] [6l9CHPE] license [e8f67fd5-abcf-40bc-a2b7-5e06ebeff3df] mode [basic] - valid

推荐