kubernetes安装elasticsearch状态显示CrashLoopBackOff解决

 在利用kubernetes安装elasticsearch时遇到如下错误,我在安装之前已经在各个节点上测试镜像能否正常拉取

[root@dmp-dn-001 /]# docker pull docker.elastic.co/elasticsearch/elasticsearch:6.2.4
Trying to pull repository docker.elastic.co/elasticsearch/elasticsearch ... 
6.2.4: Pulling from docker.elastic.co/elasticsearch/elasticsearch
469cfcc7a4b3: Downloading  5.36 MB/73.17 MB
8e27facfa9e0: Downloading 5.386 MB/56.33 MB
cdd15392adc7: Download complete 
ddcc70fbd933: Downloading 4.315 MB/108.9 MB
3d3fa0383994: Waiting 
15d1376ebd55: Waiting 
^C^C^]^]^]
[root@dmp-dn-001 /]

 可以看到是可以正常拉取的,那这个时候就通过elasticsearch.yaml创建pod

vim elasticsearch.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: elasticsearch 
  labels:
    name: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    name: elasticsearch-logging
  template:
    metadata:
      name: es
      labels:
        name: elasticsearch-logging
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
        - image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4 
          # imagePullPolicy: Never
          name: elasticsearch-logging
          ports:
            - name: es-port
              containerPort: 9200
            - name: es-tran-port
              containerPort: 9300
          volumeMounts:
            - name: es-persistent-storage
              mountPath: /data
      volumes:
        - name: es-persistent-storage
          emptyDir: {}

 在节点机上已经创建好/data目录,这时创建该pod

[root@master elasticsearchyaml]# kubectl apply -f elasticsearch.yaml 
replicationcontroller "elasticsearch" created
[root@master elasticsearchyaml]#

查看创建的pod

[root@master elasticsearchyaml]# kubectl get pods
NAME                  READY     STATUS              RESTARTS   AGE
elasticsearch-dmrkr   0/1       ContainerCreating   0          8s
mysql-flf48           1/1       Running             2          1d
[root@master elasticsearchyaml]#

 过了一会镜像下载完成后pod的状态就变成`CrashLoopBackOff`了

[root@master elasticsearchyaml]# kubectl get pods
NAME                  READY     STATUS             RESTARTS   AGE
elasticsearch-wvw3s   0/1       CrashLoopBackOff   4          3m
mysql-flf48           1/1       Running            2          1d
[root@master elasticsearchyaml]#

 通过describe查看该pod发现如下错误,大概意思就是es的container启动失败 :

Error syncing pod, skipping: failed to "StartContainer" for "elasticsearch-logging" with CrashLoopBackOff: "Back-off 10s restarting failed container=elasticsearch-logging pod=elasticsearch-dmrkr_default(e986d851-fd40-11e8-be8c-1866dae7200c)"

 于是我在该pod所在节点手动启动给镜像,

[root@dmp-dn-001 /]# docker images
REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
docker.io/mysql                                       5.7                 702fb0b7837f        6 weeks ago         372 MB
docker.io/mysql                                       latest              2dd01afbe8df        6 weeks ago         485 MB
docker.elastic.co/elasticsearch/elasticsearch         6.2.4               7cb69da7148d        8 months ago        515 MB
registry.access.redhat.com/rhel7/pod-infrastructure   latest              99965fb98423        14 months ago       209 MB
[root@dmp-dn-001 /]# docker run 7cb69da7148d
[2018-12-11T12:33:50,380][INFO ][o.e.n.Node               ] [] initializing ...
[2018-12-11T12:33:50,427][INFO ][o.e.e.NodeEnvironment    ] [QwRx60a] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [81.9gb], net total_space [99.9gb], types [rootfs]
[2018-12-11T12:33:50,427][INFO ][o.e.e.NodeEnvironment    ] [QwRx60a] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-12-11T12:33:50,430][INFO ][o.e.n.Node               ] node name [QwRx60a] derived from node ID [QwRx60alQq6W6JvCifhBoA]; set [node.name] to override
[2018-12-11T12:33:50,430][INFO ][o.e.n.Node               ] version[6.2.4], pid[1], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-327.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-12-11T12:33:50,430][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.6qHwOYNF, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [aggs-matrix-stats]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [analysis-common]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [ingest-common]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [lang-expression]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [lang-mustache]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [lang-painless]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [mapper-extras]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [parent-join]
[2018-12-11T12:33:52,274][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [percolator]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [rank-eval]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [reindex]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [repository-url]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [transport-netty4]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded module [tribe]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [ingest-geoip]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [ingest-user-agent]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-core]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-deprecation]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-graph]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-logstash]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-ml]
[2018-12-11T12:33:52,275][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-monitoring]
[2018-12-11T12:33:52,276][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-security]
[2018-12-11T12:33:52,276][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-upgrade]
[2018-12-11T12:33:52,276][INFO ][o.e.p.PluginsService     ] [QwRx60a] loaded plugin [x-pack-watcher]
[2018-12-11T12:33:54,344][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/281] [Main.cc@128] controller (64 bit): Version 6.2.4 (Build 524e7fe231abc1) Copyright (c) 2018 Elasticsearch BV
[2018-12-11T12:33:55,549][INFO ][o.e.d.DiscoveryModule    ] [QwRx60a] using discovery type [zen]
[2018-12-11T12:33:56,066][INFO ][o.e.n.Node               ] initialized
[2018-12-11T12:33:56,066][INFO ][o.e.n.Node               ] [QwRx60a] starting ...
[2018-12-11T12:33:56,230][INFO ][o.e.t.TransportService   ] [QwRx60a] publish_address {10.255.41.3:9300}, bound_addresses {[::]:9300}
[2018-12-11T12:33:56,241][INFO ][o.e.b.BootstrapChecks    ] [QwRx60a] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-12-11T12:33:56,250][INFO ][o.e.n.Node               ] [QwRx60a] stopping ...
[2018-12-11T12:33:56,315][INFO ][o.e.n.Node               ] [QwRx60a] stopped
[2018-12-11T12:33:56,315][INFO ][o.e.n.Node               ] [QwRx60a] closing ...
[2018-12-11T12:33:56,325][INFO ][o.e.n.Node               ] [QwRx60a] closed
[root@dmp-dn-001 /]#

我看到了错误的原因就是虚拟内存的权限太小,至少需要262144

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

查看虚拟机可分配内存的大小:

[root@dmp-dn-001 /]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 65530
[root@dmp-dn-001 /]# 

接着就是修改虚拟机可分配内存的大小

[root@dmp-dn-001 /]# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
[root@dmp-dn-001 /]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 262144
[root@dmp-dn-001 /]# 

启动es镜像:

[root@dmp-dn-001 /]# docker run 7cb69da7148d
[2018-12-11T12:37:08,213][INFO ][o.e.n.Node               ] [] initializing ...
[2018-12-11T12:37:08,260][INFO ][o.e.e.NodeEnvironment    ] [uTUEV7Q] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [82gb], net total_space [99.9gb], types [rootfs]
[2018-12-11T12:37:08,261][INFO ][o.e.e.NodeEnvironment    ] [uTUEV7Q] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-12-11T12:37:08,263][INFO ][o.e.n.Node               ] node name [uTUEV7Q] derived from node ID [uTUEV7Q6Q8mgVXsdIkMd6Q]; set [node.name] to override
[2018-12-11T12:37:08,263][INFO ][o.e.n.Node               ] version[6.2.4], pid[1], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-327.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-12-11T12:37:08,263][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.QFAKB4sE, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [aggs-matrix-stats]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [analysis-common]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [ingest-common]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [lang-expression]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [lang-mustache]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [lang-painless]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [mapper-extras]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [parent-join]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [percolator]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [rank-eval]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [reindex]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [repository-url]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [transport-netty4]
[2018-12-11T12:37:10,092][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded module [tribe]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [ingest-geoip]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [ingest-user-agent]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-core]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-deprecation]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-graph]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-logstash]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-ml]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-monitoring]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-security]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-upgrade]
[2018-12-11T12:37:10,093][INFO ][o.e.p.PluginsService     ] [uTUEV7Q] loaded plugin [x-pack-watcher]
[2018-12-11T12:37:12,144][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/281] [Main.cc@128] controller (64 bit): Version 6.2.4 (Build 524e7fe231abc1) Copyright (c) 2018 Elasticsearch BV
[2018-12-11T12:37:13,395][INFO ][o.e.d.DiscoveryModule    ] [uTUEV7Q] using discovery type [zen]
[2018-12-11T12:37:14,060][INFO ][o.e.n.Node               ] initialized
[2018-12-11T12:37:14,060][INFO ][o.e.n.Node               ] [uTUEV7Q] starting ...
[2018-12-11T12:37:14,171][INFO ][o.e.t.TransportService   ] [uTUEV7Q] publish_address {10.255.41.3:9300}, bound_addresses {[::]:9300}
[2018-12-11T12:37:14,182][INFO ][o.e.b.BootstrapChecks    ] [uTUEV7Q] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-12-11T12:37:17,240][INFO ][o.e.c.s.MasterService    ] [uTUEV7Q] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {uTUEV7Q}{uTUEV7Q6Q8mgVXsdIkMd6Q}{uelRGzEaRaSWbOAaQUwBpw}{10.255.41.3}{10.255.41.3:9300}{ml.machine_memory=134906904576, ml.max_open_jobs=20, ml.enabled=true}
[2018-12-11T12:37:17,247][INFO ][o.e.c.s.ClusterApplierService] [uTUEV7Q] new_master {uTUEV7Q}{uTUEV7Q6Q8mgVXsdIkMd6Q}{uelRGzEaRaSWbOAaQUwBpw}{10.255.41.3}{10.255.41.3:9300}{ml.machine_memory=134906904576, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {uTUEV7Q}{uTUEV7Q6Q8mgVXsdIkMd6Q}{uelRGzEaRaSWbOAaQUwBpw}{10.255.41.3}{10.255.41.3:9300}{ml.machine_memory=134906904576, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-12-11T12:37:17,270][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [uTUEV7Q] publish_address {10.255.41.3:9200}, bound_addresses {[::]:9200}
[2018-12-11T12:37:17,271][INFO ][o.e.n.Node               ] [uTUEV7Q] started
[2018-12-11T12:37:17,354][INFO ][o.e.g.GatewayService     ] [uTUEV7Q] recovered [0] indices into cluster_state
[2018-12-11T12:37:17,688][INFO ][o.e.l.LicenseService     ] [uTUEV7Q] license [674c88a0-48f4-43b0-a71a-cda5cc1637cc] mode [basic] - valid

可以看到启动成功了,这是再看看pod的状态,

[root@master elasticsearchyaml]# kubectl get pods
NAME                  READY     STATUS    RESTARTS   AGE
elasticsearch-dmrkr   1/1       Running   6          9m
mysql-flf48           1/1       Running   2          1d
[root@master elasticsearchyaml]#

可以看到es的pod在正常运行了,那么问题就解决掉了。为了是虚拟机内存的限制设置始终有效,我们将他写入配置文件,否则重启后刚才的设置将失效,

vim  /etc/sysctl.conf


# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

vm.max_map_count=262144

好了,问题已经解决,有问题或者想结交技术类朋友,请加QQ群:526855734

 

  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 4
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 4
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值