在利用kubernetes安装elasticsearch时遇到如下错误,我在安装之前已经在各个节点上测试镜像能否正常拉取
[root@dmp-dn-001 /]# docker pull docker.elastic.co/elasticsearch/elasticsearch:6.2.4
Trying to pull repository docker.elastic.co/elasticsearch/elasticsearch ...
6.2.4: Pulling from docker.elastic.co/elasticsearch/elasticsearch
469cfcc7a4b3: Downloading 5.36 MB/73.17 MB
8e27facfa9e0: Downloading 5.386 MB/56.33 MB
cdd15392adc7: Download complete
ddcc70fbd933: Downloading 4.315 MB/108.9 MB
3d3fa0383994: Waiting
15d1376ebd55: Waiting
^C^C^]^]^]
[root@dmp-dn-001 /]
可以看到是可以正常拉取的,那这个时候就通过elasticsearch.yaml创建pod
vim elasticsearch.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: elasticsearch
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
name: elasticsearch-logging
template:
metadata:
name: es
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
containers:
- image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
# imagePullPolicy: Never
name: elasticsearch-logging
ports:
- name: es-port
containerPort: 9200
- name: es-tran-port
containerPort: 9300
volumeMounts:
- name: es-persistent-storage
mountPath: /data
volumes:
- name: es-persistent-storage
emptyDir: {}
在节点机上已经创建好/data目录,这时创建该pod
[root@master elasticsearchyaml]# kubectl apply -f elasticsearch.yaml
replicationcontroller "elasticsearch" created
[root@master elasticsearchyaml]#
查看创建的pod
[root@master elasticsearchyaml]# kubectl get pods
NAME READY STATUS RESTARTS AGE
elasticsearch-dmrkr 0/1 ContainerCreating 0 8s
mysql-flf48 1/1 Running 2 1d
[root@master elasticsearchyaml]#
过了一会镜像下载完成后pod的状态就变成`CrashLoopBackOff`了
[root@master elasticsearchyaml]# kubectl get pods
NAME READY STATUS RESTARTS AGE
elasticsearch-wvw3s 0/1 CrashLoopBackOff 4 3m
mysql-flf48 1/1 Running 2 1d
[root@master elasticsearchyaml]#
通过describe查看该pod发现如下错误,大概意思就是es的container启动失败 :
Error syncing pod, skipping: failed to "StartContainer" for "elasticsearch-logging" with CrashLoopBackOff: "Back-off 10s restarting failed container=elasticsearch-logging pod=elasticsearch-dmrkr_default(e986d851-fd40-11e8-be8c-1866dae7200c)"
于是我在该pod所在节点手动启动给镜像,
[root@dmp-dn-001 /]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/mysql 5.7 702fb0b7837f 6 weeks ago 372 MB
docker.io/mysql latest 2dd01afbe8df 6 weeks ago 485 MB
docker.elastic.co/elasticsearch/elasticsearch 6.2.4 7cb69da7148d 8 months ago 515 MB
registry.access.redhat.com/rhel7/pod-infrastructure latest 99965fb98423 14 months ago 209 MB
[root@dmp-dn-001 /]# docker run 7cb69da7148d
[2018-12-11T12:33:50,380][INFO ][o.e.n.Node ] [] initializing ...
[2018-12-11T12:33:50,427][INFO ][o.e.e.NodeEnvironment ] [QwRx60a] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [81.9gb], net total_space [99.9gb], types [rootfs]
[2018-12-11T12:33:50,427][INFO ][o.e.e.NodeEnvironment ] [QwRx60a] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-12-11T12:33:50,430][INFO ][o.e.n.Node ] node name [QwRx60a] derived from node ID [QwRx60alQq6W6JvCifhBoA]; set [node.name] to override
[2018-12-11T12:33:50,430][INFO ][o.e.n.Node ] version[6.2.4], pid[1], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-327.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-12-11T12:33:50,430][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.6qHwOYNF, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [aggs-matrix-stats]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [analysis-common]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [ingest-common]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [lang-expression]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [lang-mustache]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [lang-painless]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [mapper-extras]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [parent-join]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [percolator]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [rank-eval]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [reindex]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [repository-url]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [transport-netty4]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded module [tribe]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [ingest-geoip]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [ingest-user-agent]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-core]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-deprecation]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-graph]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-logstash]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-ml]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-monitoring]
[2018-12-11T12:33:52,276][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-security]
[2018-12-11T12:33:52,276][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-upgrade]
[2018-12-11T12:33:52,276][INFO ][o.e.p.PluginsService ] [QwRx60a] loaded plugin [x-pack-watcher]
[2018-12-11T12:33:54,344][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/281] [Main.cc@128] controller (64 bit): Version 6.2.4 (Build 524e7fe231abc1) Copyright (c) 2018 Elasticsearch BV
[2018-12-11T12:33:55,549][INFO ][o.e.d.DiscoveryModule ] [QwRx60a] using discovery type [zen]
[2018-12-11T12:33:56,066][INFO ][o.e.n.Node ] initialized
[2018-12-11T12:33:56,066][INFO ][o.e.n.Node ] [QwRx60a] starting ...
[2018-12-11T12:33:56,230][INFO ][o.e.t.TransportService ] [QwRx60a] publish_address {10.255.41.3:9300}, bound_addresses {[::]:9300}
[2018-12-11T12:33:56,241][INFO ][o.e.b.BootstrapChecks ] [QwRx60a] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-12-11T12:33:56,250][INFO ][o.e.n.Node ] [QwRx60a] stopping ...
[2018-12-11T12:33:56,315][INFO ][o.e.n.Node ] [QwRx60a] stopped
[2018-12-11T12:33:56,315][INFO ][o.e.n.Node ] [QwRx60a] closing ...
[2018-12-11T12:33:56,325][INFO ][o.e.n.Node ] [QwRx60a] closed
[root@dmp-dn-001 /]#
我看到了错误的原因就是虚拟内存的权限太小,至少需要262144
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
查看虚拟机可分配内存的大小:
[root@dmp-dn-001 /]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 65530
[root@dmp-dn-001 /]#
接着就是修改虚拟机可分配内存的大小
[root@dmp-dn-001 /]# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
[root@dmp-dn-001 /]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 262144
[root@dmp-dn-001 /]#
启动es镜像:
[root@dmp-dn-001 /]# docker run 7cb69da7148d
[2018-12-11T12:37:08,213][INFO ][o.e.n.Node ] [] initializing ...
[2018-12-11T12:37:08,260][INFO ][o.e.e.NodeEnvironment ] [uTUEV7Q] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [82gb], net total_space [99.9gb], types [rootfs]
[2018-12-11T12:37:08,261][INFO ][o.e.e.NodeEnvironment ] [uTUEV7Q] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-12-11T12:37:08,263][INFO ][o.e.n.Node ] node name [uTUEV7Q] derived from node ID [uTUEV7Q6Q8mgVXsdIkMd6Q]; set [node.name] to override
[2018-12-11T12:37:08,263][INFO ][o.e.n.Node ] version[6.2.4], pid[1], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-327.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-12-11T12:37:08,263][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.QFAKB4sE, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [aggs-matrix-stats]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [analysis-common]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [ingest-common]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [lang-expression]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [lang-mustache]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [lang-painless]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [mapper-extras]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [parent-join]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [percolator]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [rank-eval]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [reindex]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [repository-url]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [transport-netty4]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded module [tribe]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [ingest-geoip]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [ingest-user-agent]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-core]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-deprecation]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-graph]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-logstash]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-ml]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-monitoring]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-security]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-upgrade]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService ] [uTUEV7Q] loaded plugin [x-pack-watcher]
[2018-12-11T12:37:12,144][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/281] [Main.cc@128] controller (64 bit): Version 6.2.4 (Build 524e7fe231abc1) Copyright (c) 2018 Elasticsearch BV
[2018-12-11T12:37:13,395][INFO ][o.e.d.DiscoveryModule ] [uTUEV7Q] using discovery type [zen]
[2018-12-11T12:37:14,060][INFO ][o.e.n.Node ] initialized
[2018-12-11T12:37:14,060][INFO ][o.e.n.Node ] [uTUEV7Q] starting ...
[2018-12-11T12:37:14,171][INFO ][o.e.t.TransportService ] [uTUEV7Q] publish_address {10.255.41.3:9300}, bound_addresses {[::]:9300}
[2018-12-11T12:37:14,182][INFO ][o.e.b.BootstrapChecks ] [uTUEV7Q] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-12-11T12:37:17,240][INFO ][o.e.c.s.MasterService ] [uTUEV7Q] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {uTUEV7Q}{uTUEV7Q6Q8mgVXsdIkMd6Q}{uelRGzEaRaSWbOAaQUwBpw}{10.255.41.3}{10.255.41.3:9300}{ml.machine_memory=134906904576, ml.max_open_jobs=20, ml.enabled=true}
[2018-12-11T12:37:17,247][INFO ][o.e.c.s.ClusterApplierService] [uTUEV7Q] new_master {uTUEV7Q}{uTUEV7Q6Q8mgVXsdIkMd6Q}{uelRGzEaRaSWbOAaQUwBpw}{10.255.41.3}{10.255.41.3:9300}{ml.machine_memory=134906904576, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {uTUEV7Q}{uTUEV7Q6Q8mgVXsdIkMd6Q}{uelRGzEaRaSWbOAaQUwBpw}{10.255.41.3}{10.255.41.3:9300}{ml.machine_memory=134906904576, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-12-11T12:37:17,270][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [uTUEV7Q] publish_address {10.255.41.3:9200}, bound_addresses {[::]:9200}
[2018-12-11T12:37:17,271][INFO ][o.e.n.Node ] [uTUEV7Q] started
[2018-12-11T12:37:17,354][INFO ][o.e.g.GatewayService ] [uTUEV7Q] recovered [0] indices into cluster_state
[2018-12-11T12:37:17,688][INFO ][o.e.l.LicenseService ] [uTUEV7Q] license [674c88a0-48f4-43b0-a71a-cda5cc1637cc] mode [basic] - valid
可以看到启动成功了,这是再看看pod的状态,
[root@master elasticsearchyaml]# kubectl get pods
NAME READY STATUS RESTARTS AGE
elasticsearch-dmrkr 1/1 Running 6 9m
mysql-flf48 1/1 Running 2 1d
[root@master elasticsearchyaml]#
可以看到es的pod在正常运行了,那么问题就解决掉了。为了是虚拟机内存的限制设置始终有效,我们将他写入配置文件,否则重启后刚才的设置将失效,
vim /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
vm.max_map_count=262144
好了,问题已经解决,有问题或者想结交技术类朋友,请加QQ群:526855734