Ubuntu小技巧18--Elasticsearch安装方法

Ubuntu小技巧18--Elasticsearch安装方法

 

Elasticsearch 是一个开源的搜索引擎，建立在一个全文搜索引擎库 Apache Lucene™ 基础之上。 Lucene可以说是当下最先进、高性能、全功能的搜索引擎库--无论是开源还是私有。Elasticsearch 也是使用Java编写的，它的内部使用Lucene做索引与搜索，但是它的目的是使全文检索变得简单， 通过隐藏 Lucene 的复杂性，取而代之的提供一套简单一致的 RESTful API。

Elasticsearch 是一个分布式、可扩展、实时的搜索与数据分析引擎。它能从项目一开始就赋予你的数据以搜索、分析和探索的能力，其不仅仅只是全文搜索，还包括结构化搜索、数据分析、复杂的语言处理、地理位置和对象间关联关系等功能模块。

 

1、安装方法

1.1、安装java

安装 Elasticsearch 之前，需要先安装一个较新的版本的 Java，最好的选择是，你可以从 oracle网站得官方提供的最新版本的 Java。笔者使用jdk-8u201-linux-x64.tar.gz：https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

下载java压缩包，解压到Soft文件夹，vi /etc/environment 加入如下内容:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/xg/soft/jdk1.8.0_201/bin"

CLASSPATH=".:/home/xg/soft/jdk1.8.0_201/lib"
JAVA_HOME=/home/xg/soft/jdk1.8.0_201/lib

source /etc/environment 使之生效，通过java -version查看java配置信息，如下：

java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

注意：如果使用debain系统，可以将如下内容分别加到/etc/profile（root用户） 和~/.bashrc（普通用户）中，然后通过source命令立即生效即可完成java配置。也可以直接执行如下命令添加环境变量：export PATH="$PATH:/home/xgcloud/soft/jdk1.8.0_201/bin"

#For java
JAVA_HOME=/home/xgcloud/soft/jdk1.8.0_201
CLASSPATH=$PATH:$JAVA_HOME/lib/
PATH=$PATH:$JAVA_HOME/bin/
export PATH JAVA_HOME CLASSPATH

 

1.2、安装配置elasticsearch

在官网下载安装包，笔者下载 elasticsearch-6.5.4.tar：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.tar.gz，下载后解压到Soft文件夹。

通过以下4个步骤完成其配置，不配置默认会出错：

（1）更改elasticsearch-6.5.4/config/elasticsearch.yml文件
network.host: 0.0.0.0  #0.0.0.0表示允许所有网络访问，也可以改改为特定网段或者特定ip访问，多个ip之间使用逗号隔开
http.port: 9200 #开放http 9200访问端口

（2）添加组权限
可以通过如下方式新建用户elasearch和组elasearch，然后对elasticsearch文件夹设置访问权限：
groupadd elasearch
useradd elasearch -g elasearch -p elasticsearch
chown -R elasearch:elasearch elasticsearch-6.5.4
也可以通过/etc/passwd或者/etc/group来查看当前系统中的用户和组，笔者当前有xg用户xg组，因此通过如下方式处理即可：
chown -R xg:xg elasticsearch-6.5.4 #chown -R 用户:组 文件(夹)
注意：该设置主要为了解决不能使用root执行的问题(org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root)

（3）修改vm.max_map_count值
vi /etc/sysctl.conf 
添加下面配置：vm.max_map_count=655360
保存并执行命令：sysctl -p
注意：该设置主要用于解决虚拟内存大小问题(max virtual memory areas vm.max_map_count [65530] is too low, increase to at least ...)

（4）启动elasticsearch，命令行输入ES_JAVA_OPTS="-Xms1g -Xmx1g" ./elasticsearch 即可
注意：此处需要切换到非root权限，否则会出错（org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root）；此处需要在终端设置ES_JAVA_OPTS="-Xms1g -Xmx1g" （也可以根据需要设置为其它大小的内存），否则执行的时候内存不够用，出现 killed返回信息。

通过(4) 在终端输入ES_JAVA_OPTS="-Xms1g -Xmx1g" ./elasticsearch 启动elasticsearch，启动成功后信息如下：

$ ES_JAVA_OPTS="-Xms1g -Xmx1g" ./elasticsearch
[2019-01-27T00:55:07,852][INFO ][o.e.e.NodeEnvironment    ] [bMXRZO-] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [56.4gb], net total_space [61.9gb], types [ext4]
[2019-01-27T00:55:07,915][INFO ][o.e.e.NodeEnvironment    ] [bMXRZO-] heap size [990.7mb], compressed ordinary object pointers [true]
[2019-01-27T00:55:07,924][INFO ][o.e.n.Node               ] [bMXRZO-] node name derived from node ID [bMXRZO-XRYWyetQKUdd1Fw]; set [node.name] to override
[2019-01-27T00:55:07,944][INFO ][o.e.n.Node               ] [bMXRZO-] version[6.5.4], pid[2015], build[default/tar/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/4.4.0-116-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_201/25.201-b09]
[2019-01-27T00:55:07,945][INFO ][o.e.n.Node               ] [bMXRZO-] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.kppyH6mw, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Xms1g, -Xmx1g, -Des.path.home=/home/xg/Soft/elasticsearch-6.5.4, -Des.path.conf=/home/xg/Soft/elasticsearch-6.5.4/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-01-27T00:55:14,732][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [aggs-matrix-stats]
[2019-01-27T00:55:14,732][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [analysis-common]
[2019-01-27T00:55:14,733][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [ingest-common]
[2019-01-27T00:55:14,734][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [lang-expression]
[2019-01-27T00:55:14,734][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [lang-mustache]
[2019-01-27T00:55:14,736][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [lang-painless]
[2019-01-27T00:55:14,737][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [mapper-extras]
[2019-01-27T00:55:14,739][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [parent-join]
[2019-01-27T00:55:14,739][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [percolator]
[2019-01-27T00:55:14,740][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [rank-eval]
[2019-01-27T00:55:14,740][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [reindex]
[2019-01-27T00:55:14,741][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [repository-url]
[2019-01-27T00:55:14,749][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [transport-netty4]
[2019-01-27T00:55:14,750][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [tribe]
[2019-01-27T00:55:14,751][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-ccr]
[2019-01-27T00:55:14,752][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-core]
[2019-01-27T00:55:14,756][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-deprecation]
[2019-01-27T00:55:14,757][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-graph]
[2019-01-27T00:55:14,758][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-logstash]
[2019-01-27T00:55:14,758][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-ml]
[2019-01-27T00:55:14,759][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-monitoring]
[2019-01-27T00:55:14,760][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-rollup]
[2019-01-27T00:55:14,760][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-security]
[2019-01-27T00:55:14,761][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-sql]
[2019-01-27T00:55:14,766][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-upgrade]
[2019-01-27T00:55:14,768][INFO ][o.e.p.PluginsService     ] [bMXRZO-] loaded module [x-pack-watcher]
[2019-01-27T00:55:14,771][INFO ][o.e.p.PluginsService     ] [bMXRZO-] no plugins loaded
[2019-01-27T00:55:20,371][INFO ][o.e.x.s.a.s.FileRolesStore] [bMXRZO-] parsed [0] roles from file [/home/xg/Soft/elasticsearch-6.5.4/config/roles.yml]
[2019-01-27T00:55:21,445][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [bMXRZO-] [controller/2079] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Elasticsearch BV
[2019-01-27T00:55:22,422][DEBUG][o.e.a.ActionModule       ] [bMXRZO-] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-01-27T00:55:22,775][INFO ][o.e.d.DiscoveryModule    ] [bMXRZO-] using discovery type [zen] and host providers [settings]
[2019-01-27T00:55:24,611][INFO ][o.e.n.Node               ] [bMXRZO-] initialized
[2019-01-27T00:55:24,611][INFO ][o.e.n.Node               ] [bMXRZO-] starting ...
[2019-01-27T00:55:25,012][INFO ][o.e.t.TransportService   ] [bMXRZO-] publish_address {192.168.199.231:9300}, bound_addresses {[::]:9300}
[2019-01-27T00:55:25,057][INFO ][o.e.b.BootstrapChecks    ] [bMXRZO-] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-01-27T00:55:28,343][INFO ][o.e.c.s.MasterService    ] [bMXRZO-] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {bMXRZO-}{bMXRZO-XRYWyetQKUdd1Fw}{3ar5gJ30TwSV-zp8Q2Lvdg}{192.168.199.231}{192.168.199.231:9300}{ml.machine_memory=2078593024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-01-27T00:55:28,354][INFO ][o.e.c.s.ClusterApplierService] [bMXRZO-] new_master {bMXRZO-}{bMXRZO-XRYWyetQKUdd1Fw}{3ar5gJ30TwSV-zp8Q2Lvdg}{192.168.199.231}{192.168.199.231:9300}{ml.machine_memory=2078593024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {bMXRZO-}{bMXRZO-XRYWyetQKUdd1Fw}{3ar5gJ30TwSV-zp8Q2Lvdg}{192.168.199.231}{192.168.199.231:9300}{ml.machine_memory=2078593024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-01-27T00:55:28,390][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bMXRZO-] publish_address {192.168.199.231:9200}, bound_addresses {[::]:9200}
[2019-01-27T00:55:28,391][INFO ][o.e.n.Node               ] [bMXRZO-] started
[2019-01-27T00:55:28,922][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [bMXRZO-] Failed to clear cache for realms [[]]
[2019-01-27T00:55:29,028][INFO ][o.e.l.LicenseService     ] [bMXRZO-] license [d174d3d1-ca4c-43a7-99ed-5336bf9e40eb] mode [basic] - valid
[2019-01-27T00:55:29,081][INFO ][o.e.g.GatewayService     ] [bMXRZO-] recovered [0] indices into cluster_state

在浏览器中输入http://ip:9200 或者终端输入 curl 'http://localhost:9200/?pretty' 即可返回相关信息，以下为浏览器返回信息：

1.3 常见错误

以上1.2中内容必须配置，否则在Ubuntu上会执行出错；实际执行中，可能由于其它配置导致如下错误，以下为笔者碰见的错误及其解决方法，后续遇见的错误都会在此处加以补充：

(1）max file descriptors [65535] for elasticsearch process is too low, increase to at least 65536

笔者在debain下出现文件描述符偏小错误，查看cat /etc/security/limits.conf 发现默认soft nofile和hard nofile的默认值都为65535，将其更改为65536即可，更改后执行便无该错误。笔者更改后的内容如下：

#此处省略部分内容
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4
*               soft    nofile          65536
*               hard    nofile          65536
root            soft    nofile          65535
root            hard    nofile          65535
root            soft    core            unlimited
root            hard    core            unlimited

# End of file

（2）max number of threads 2048 for user [xx] is too low increase to at least 4096

解决方法：修改/etc/security/limits.conf，在末尾添加如下内容
*    soft    nproc    5000 
*    hard    nproc    5000
保存后重启机器生效后，即可解决该问题！

 

2、说明

系统为Ubuntu 16.04.4 Server LTS (64-bit)，分配内存2G，安装在VM虚拟机上
参考文献：
Elasticsearch: 权威指南：https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html

更新说明：
2019/03/24 更新1.3（2）--解决 max number of threads错误