es部署–生产环境–9.2.4–跨机房灾备–实践–极限网关
备注
这个实验我没有弄,因为下载不了容器
参考内容:https://cloud.tencent.com/developer/article/1928886
1、环境准备
中间件 | 版本 | 备注 |
---|---|---|
Docker | 20.10.12 | 无 |
Docker Compose | 1.29.2 | 无 |
kibana | 7.16.2 | 无 |
elasticsearch | 7.16.2 | 无 |
infinilabs/gateway | latest | 无 |
1.1、创建docker-compose.yml
mkdir -p /tmp/gateway
cd /tmp/gateway
vim docker-compose.yml
内容
version: '3.8'
services:
# 集群 cluster01
# Elasticsearch
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
container_name: es01
environment:
# 节点名
- node.name=es01
# 集群名
- cluster.name=cluster01
# 指定单节点启动
- discovery.type=single-node
# 开启内存锁定
- bootstrap.memory_lock=true
# 设置内存大小
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
# 启用安全
- xpack.security.enabled=true
# 设置 elastic 用户密码
- ELASTIC_PASSWORD=test123
ulimits:
memlock:
soft: -1
hard: -1
# 映射到主机名的端口 宿主机端口:容器端口
ports:
- 9200:9200
volumes:
- data01:/usr/share/elasticsearch/data
networks:
- elastic
# Kibana
kib01:
image: docker.elastic.co/kibana/kibana:7.16.2
container_name: kib01
ports:
- 5601:5601
environment:
# Elasticsearch 连接信息
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: '["http://es01:9200"]'
ELASTICSEARCH_USERNAME: elastic
ELASTICSEARCH_PASSWORD: test123
networks:
- elastic
# 集群 cluster02
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
container_name: es02
environment:
- node.name=es02
- cluster.name=cluster02
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- xpack.security.enabled=true
- ELASTIC_PASSWORD=test123
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9201:9200
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
kib02:
image: docker.elastic.co/kibana/kibana:7.16.2
container_name: kib02
ports:
- 5602:5601
environment:
ELASTICSEARCH_URL: http://es02:9200
ELASTICSEARCH_HOSTS: '["http://es02:9200"]'
ELASTICSEARCH_USERNAME: elastic
ELASTICSEARCH_PASSWORD: test123
networks:
- elastic
# 极限网关
infini-gateway:
image: infinilabs/gateway:latest
container_name: infini-gateway
ports:
- 2900:2900
- 8000:8000
- 8001:8001
- 8002:8002
volumes:
- ./gateway.yml:/gateway.yml
- data03:/software/infini-gateway
networks:
- elastic
# 存储卷
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
# 网络
networks:
elastic:
driver: bridge
极限网关的配置文件 gateway.yml 如下所示。
# 数据路径
path.data: /software/infini-gateway/data
# 日志路径
path.logs: /software/infini-gateway/log
# 定义 Elasticsearch 集群地址
elasticsearch:
# cluster01 集群
- name: cluster01
enabled: true
endpoint: http://es01:9200
basic_auth:
username: elastic
password: test123
# cluster02 集群
- name: cluster02
enabled: true
endpoint: http://es02:9200
basic_auth:
username: elastic
password: test123
# 定义网关入口
entry:
- name: my_es_entry
enabled: true
router: my_router
network:
binding: 0.0.0.0:8000
# 定义工作流
flow:
# 写请求优先发给主集群, 当主集群不可用时发给备集群
# 当主集群数据写入成功时,记录到队列中,异步消费写入备集群
- name: write-flow
filter:
- if:
# 当主集群可用时
cluster_available: ["cluster01"]
then:
# 先将数据写入主集群
- elasticsearch:
elasticsearch: "cluster01"
# 写入消息队列,等待 pipeline 异步消费到备集群
- queue:
queue_name: "cluster02-queue"
else:
- elasticsearch:
elasticsearch: "cluster02"
- queue:
queue_name: "cluster01-queue"
# 读请求优先发给主集群, 当主集群不可用时发给备集群
- name: read-flow
filter:
- if:
cluster_available: ["cluster01"]
then:
- elasticsearch:
elasticsearch: "cluster01"
else:
- elasticsearch:
elasticsearch: "cluster02"
# 路由规则
router:
- name: my_router
# 默认路由
default_flow: write-flow
# 读请求路由
rules:
- method:
- "GET"
- "HEAD"
pattern:
- "/{any:*}"
flow:
- read-flow
- method:
- "POST"
- "GET"
pattern:
- "/_refresh"
- "/_count"
- "/_search"
- "/_msearch"
- "/_mget"
- "/{any_index}/_count"
- "/{any_index}/_search"
- "/{any_index}/_msearch"
- "/{any_index}/_mget"
flow:
- read-flow
# 定义管道, 异步将数据写入备集群
pipeline:
- name: cluster01-consumer
auto_start: true
keep_running: true
processor:
- queue_consumer:
input_queue: "cluster01-queue"
elasticsearch: "cluster01"
when:
cluster_available: ["cluster01"] # 当集群可用时,才消费队列中的数据
- name: cluster02-consumer
auto_start: true
keep_running: true
processor:
- queue_consumer:
input_queue: "cluster02-queue"
elasticsearch: "cluster02"
when:
cluster_available: ["cluster02"]
1.2、启动容器
入极限网关实验目录,启动容器。
cd infini-gateway
docker-compose up -d
1.3、查看容器状态
$ docker-compose ps
Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------
es01 /bin/tini -- /usr/local/bi ... Up 0.0.0.0:9200->9200/tcp,:::9200->9200/tcp, 9300/tcp
es02 /bin/tini -- /usr/local/bi ... Up 0.0.0.0:9201->9200/tcp,:::9201->9200/tcp, 9300/tcp
infini-gateway /gateway Up 0.0.0.0:2900->2900/tcp,:::2900->2900/tcp,
0.0.0.0:8000->8000/tcp,:::8000->8000/tcp,
0.0.0.0:8001->8001/tcp,:::8001->8001/tcp,
0.0.0.0:8002->8002/tcp,:::8002->8002/tcp
kib01 /bin/tini -- /usr/local/bi ... Up 0.0.0.0:5601->5601/tcp,:::5601->5601/tcp
kib02 /bin/tini -- /usr/local/bi ... Up 0.0.0.0:5602->5601/tcp,:::5602->5601/tcp
1.4、查看极限网关日志,可以看到此时极限网关已经成功连接到集群 cluster01 和 cluster02。
$ docker-compose logs -f infini-gateway
1.5、当两个集群都正常时,cluster01 是主集群,通过极限网关查看集群将返回集群 cluster01 的信息。
$ curl -u elastic:test123 http://11.8.36.25:8000
{
"name" : "es01",
"cluster_name" : "cluster01",
"cluster_uuid" : "0Fj2oq0LRxyNfMbTX2TqZA",
"version" : {
"number" : "7.16.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
"build_date" : "2021-12-18T19:42:46.604893745Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
2、验证同步
插入 3 条数据。
for i in {1..3};do
curl -XPUT -u elastic:test123 -H "Content-Type: Application/json" \
http://11.8.36.25:8000/index-1/_doc/$i \
-d "{\"name\":\"tom_$i\",\"age\":\"$i\"}";done
在集群 cluster01 和 cluster02 分别查询索引 index-1,可以查到相同的数据。
GET index-1/_search
# 返回结果
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"name" : "tom_1",
"age" : "1"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "tom_2",
"age" : "2"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
}
]
}
}
修改 id 为 2 的数据。
curl -XPUT -u elastic:test123 -H "Content-Type: Application/json" \
http://11.8.36.25:8000/index-1/_doc/2 \
-d '{"name":"jack_2","age":"100"}'
再次查询,可以看到两个集群上 id 为 2 的文档都已经修改成功。
GET index-1/_doc/2
# 返回结果
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_version" : 2,
"_seq_no" : 3,
"_primary_term" : 1,
"found" : true,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
}
删除 id 为 1 的数据。
curl -XDELETE -u elastic:test123 \
http://11.8.36.25:8000/index-1/_doc/1
查询索引 index-1 ,此时两个集群上 id 为 1 的文档都已经删除。
GET index-1/_search
# 返回结果
{
"took" : 795,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
}
]
}
}
在网关上对数据进行查询,也是可以正确得到结果的。
curl -XGET -u elastic:test123 \
http://11.8.36.25:8000/index-1/_search?pretty
# 返回结果
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
}
]
}
}
3、故障测试
3.1、模拟备集群(cluster02)故障
停止 cluster02 集群。
docker-compose stop es02
此时在极限网关上可以看到报错说集群 cluster02 不可用。
$ docker-compose logs -f infini-gateway
接下来继续往索引 index-1 中插入 2 条数据。
for i in {4..5};do
curl -XPUT -u elastic:test123 -H "Content-Type: Application/json" \
http://11.8.36.25:8000/index-1/_doc/$i \
-d "{\"name\":\"tom_$i\",\"age\":\"$i\"}";done
查看集群 cluster01 发现新的 2 条数据插入成功。
GET index-1/_search
# 返回结果
{
"took" : 399,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
},
# 新插入的 2 条数据
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"name" : "tom_4",
"age" : "4"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"name" : "tom_5",
"age" : "5"
}
}
]
}
}
访问极限网关的 API 接口查看本地队列状态,由于集群 cluster02 当前不可用,因此刚刚插入的 2 条消息保存在队列中,待集群 cluster02 恢复后再尝试消费。
$ curl http://11.8.36.25:2900/queue/stats
# 返回结果
{
"queue": {
"disk": {
"cluster01-queue": {
"depth": 0,
"metadata": {
"source": "dynamic",
"id": "c79eq49r8oo0golornmg",
"name": "cluster01-queue",
"label": null
}
},
"cluster02-queue": {
"depth": 2, # 未成功异步写入集群 cluster02 的 2 条消息
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golornt0",
"name": "cluster02-queue",
"label": null
}
},
"cluster_state_change": {
"depth": 8,
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golorns0",
"name": "cluster_state_change",
"label": null
}
}
}
}
}
恢复 es02
docker-compose start es02
查看网关日志可以看到集群 cluster02 恢复的信息。
$ docker-compose logs -f infini-gateway
查询集群 cluster02 的数据,可以在 es02 停止期间的数据已经成功写入。
GET index-1/_search
# 返回结果
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
# 新插入的 2 条数据
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"name" : "tom_4",
"age" : "4"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"name" : "tom_5",
"age" : "5"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
}
]
}
}
此时查看队列状态,cluster02-queue 中的消息已经消费完毕。
$ curl http://11.8.36.25:2900/queue/stats
# 返回结果
{
"queue": {
"disk": {
"cluster01-queue": {
"depth": 0,
"metadata": {
"source": "dynamic",
"id": "c79eq49r8oo0golornmg",
"name": "cluster01-queue",
"label": null
}
},
"cluster02-queue": {
"depth": 0, # 消费完毕
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golornt0",
"name": "cluster02-queue",
"label": null
}
},
"cluster_state_change": {
"depth": 10,
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golorns0",
"name": "cluster_state_change",
"label": null
}
}
}
}
}
3.2、模拟主集群(cluster01)故障
停止 cluster01 集群。
docker-compose stop es01
此时在极限网关上可以看到报错说 cluster01 集群不可用。
docker-compose logs -f infini-gateway
接下来继续往 index-1 索引中插入 2 条数据。
for i in {6..7};do
curl -XPUT -u elastic:test123 -H "Content-Type: Application/json" \
http://11.8.36.25:8000/index-1/_doc/$i \
-d "{\"name\":\"tom_$i\",\"age\":\"$i\"}";done
查看集群 cluster02 发现新的 2 条数据插入成功。
GET index-1/_search
# 返回结果
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 6,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"name" : "tom_4",
"age" : "4"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"name" : "tom_5",
"age" : "5"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
},
# 新插入的 2 条数据
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "6",
"_score" : 1.0,
"_source" : {
"name" : "tom_6",
"age" : "6"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "7",
"_score" : 1.0,
"_source" : {
"name" : "tom_7",
"age" : "7"
}
}
]
}
}
访问极限网关的 API 接口查看本地队列状态,由于集群 cluster01 当前不可用,因此刚刚插入的 2 条消息保存在队列中,待集群 cluster01 恢复后再尝试消费
$ curl http://11.8.36.25:2900/queue/stats
# 返回结果
{
"queue": {
"disk": {
"cluster01-queue": {
"depth": 2, # 未成功异步写入集群 cluster01 的 2 条消息
"metadata": {
"source": "dynamic",
"id": "c79eq49r8oo0golornmg",
"name": "cluster01-queue",
"label": null
}
},
"cluster02-queue": {
"depth": 0,
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golornt0",
"name": "cluster02-queue",
"label": null
}
},
"cluster_state_change": {
"depth": 10,
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golorns0",
"name": "cluster_state_change",
"label": null
}
}
}
}
}
恢复 es01。
docker-compose start es01
查看网关日志可以看到集群 cluster01 恢复的信息。
docker-compose logs -f infini-gateway
查询集群 cluster01 的数据,可以在 es01 停止期间的数据已经成功写入。
GET index-1/_search
# 返回结果
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 6,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"name" : "tom_3",
"age" : "3"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "jack_2",
"age" : "100"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"name" : "tom_4",
"age" : "4"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"name" : "tom_5",
"age" : "5"
}
},
# 新插入的 2 条数据
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "6",
"_score" : 1.0,
"_source" : {
"name" : "tom_6",
"age" : "6"
}
},
{
"_index" : "index-1",
"_type" : "_doc",
"_id" : "7",
"_score" : 1.0,
"_source" : {
"name" : "tom_7",
"age" : "7"
}
}
]
}
}
此时查看队列状态,cluster01-queue 中的消息已经消费完毕。
$ curl http://11.8.36.25:2900/queue/stats
# 返回结果
{
"queue": {
"disk": {
"cluster01-queue": {
"depth": 0, # 消费完毕
"metadata": {
"source": "dynamic",
"id": "c79eq49r8oo0golornmg",
"name": "cluster01-queue",
"label": null
}
},
"cluster02-queue": {
"depth": 0,
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golornt0",
"name": "cluster02-queue",
"label": null
}
},
"cluster_state_change": {
"depth": 12,
"metadata": {
"source": "dynamic",
"id": "c79eq6pr8oo0golorns0",
"name": "cluster_state_change",
"label": null
}
}
}
}
}
4、清理现场
用于后续的测试
cd /tmp/gateway
# -v 参数表示删除 volume
docker-compose down -v