elk flow mysql_[k8s]elk架构设计-k8s集群里搭建

elasticsearch和mysql的对比

Mapping ~ Schema

master: 负责在集群范围内创建/删除索引,将分片分配给这些节点.

data: 用来保存数据和倒排索引,node.data=ture

client: 将节点配置为客户端节点，并充当负载平衡器，将传入的请求路由到集群中的不同节点。node.master和node.data设置为false

Index templates

索引可使用预定义的模板进行创建,这个模板称作Index templates。模板设置包括settings和mappings，通过模式匹配的方式使得多个索引重用一个模板，例如：

curl -XPUT localhost:9200/_template/template_1 -d '

{

"template" : "te*",

"settings" : {

"number_of_shards" : 1

},

"mappings" : {

"type1" : {

"_source" : {"enabled" : false }

}

}

}

elk设计架构

Elasticsearch最佳实践建议将这些节点分成三个角色：

Master 节点 - 仅用于集群管理，没有数据，没有HTTP API

Client 节点 - 用于客户端使用，无数据， 使用HTTP API

Data 节点 - 旨在存储和索引数据， 没有HTTP API

分为 管理区 存储区 读写区

定制镜像

- es

0. 我把镜像推到了dockerhub: lanny/quay.io_pires_docker-elasticsearch-kubernetes:5.6.0

1. 镜像里用到的yaml: https://github.com/lannyMa/docker-elasticsearch-kubernetes

2. 可以安装自定义插件后重新build

- kibana

0. 用到这个人的这个镜像 cfontes/kibana-xpack-less:5.5.0 https://hub.docker.com/r/cfontes/kibana-xpack-less/

这个人的git(包含了elk的k8s yaml): https://github.com/cfontes/kubernetes-elasticsearch-cluster

- k8s的elk集群的yaml

https://github.com/pires/kubernetes-elasticsearch-cluster

镜像到这个地址: https://github.com/lannyMa/kubernetes-elasticsearch-cluster

内含elk 普罗 harbor的yaml,比较实用

https://github.com/cnych/k8s-repo

集群yaml里搜HTTP_ENABLE,统一打开head访问权限

es-master.yaml

- name: HTTP_ENABLE

value: "true"

- name: HTTP_CORS_ALLOW_ORIGIN

value: "*"

创建集群

创建pv

$ cat es-data_claim.yaml

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

name: opspvc

namespace: kube-ops

annotations:

volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"

spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 1Gi

先创建svc类型

后依次创建 data master client kibana

访问es api

访问kibana

打开kibana,插入示例数据

PUT _bulk

{"index":{"_index":"lagou","_type":"job","_id":"1"}}

{"title":"python分布式爬虫开发","salary_min":15000,"city":"深圳","company":{"name":"腾讯","company_addr":"深圳市软件园"},"publish_date":"2017-11-11","comments":15}

{"index":{"_index":"lagou","_type":"job","_id":"2"}}

{"title":"django开发","salary_min":15000,"city":"上海","company":{"name":"阿里","company_addr":"广州市软件园"},"publish_date":"2017-11-12","comments":20}

打开head查看数据

docker run -d -v /etc/localtime:/etc/localtime --restart=always -p 9100:9100 mobz/elasticsearch-head:5

查看数据

Curator的主要用途:

获取索引列表

过滤索引,删除/快照

name

用途

Alias

别号

Allocation

分配

Close

关

Cluster Routing

群集路由

Create Index

创建索引

Delete Indices

删除索引

Delete Snapshots

删除快照

forceMerge

forceMerge

Index Settings

索引设置

Open

打开

Reindex

重新编制

Replicas

副本

Restore

恢复

Rollover

滚下

Shrink

收缩

Snapshot

快照

es版本兼容

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

curl http://10.100.32.137:9200

{

"name" : "es-data-5c5969967-wb2b8",

"cluster_name" : "myesdb",

"cluster_uuid" : "qSps-b9dRI2ngGHBguJ44Q",

"version" : {

"number" : "6.3.0",

"build_flavor" : "default",

"build_type" : "tar",

"build_hash" : "424e937",

"build_date" : "2018-06-11T23:38:03.357887Z",

"build_snapshot" : false,

"lucene_version" : "7.3.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

"tagline" : "You Know, for Search"

}