ElasticSearch -- ES CDC

TheBiiigBlue

已于 2023-11-23 15:59:25 修改

阅读量678

点赞数

分类专栏： ELK 文章标签： elasticsearch java 大数据

于 2023-03-15 19:07:05 首次发布

本文链接：https://blog.csdn.net/Aeve_imp/article/details/129563640

版权

1、ES cdc安装包

Elasticsearch V6.3.2 CDC 组件

2、安装步骤

集群每个节点都执行如下步骤：

1. 移除之前版本(如果之前安装过)
${elasticsearch_home}/bin/elasticsearch-plugin remove es-cdc

2. 安装
${elasticsearch_home}/bin/elasticsearch-plugin install file:///${elasticsearch_cdc_dir}/es-cdc-v6-1.0-SNAPSHOT.zip

3. 配置es的javax相关权限
${elasticsearch_home}/config/jvm.options 文件加入相关java配置
-Djava.security.policy=/your_elasticsearch_home/plugins/es-cdc/plugin-security.policy

4. 重启 elasticsearch

3、使用

3.1 设置集群属性

PUT _cluster/settings
{
   
  "persistent": {
   
    "indices.cdc.request.timeout.ms": 30000,
    "indices.cdc.send.buffer.bytes": 131072,
    "indices.cdc.acks": "all",
    "indices.cdc.compression.type": "none",
    "indices.cdc.receive.buffer.bytes": 32768,
    "indices.cdc.batch.size": 16384,
    "indices.cdc.linger.ms": 1000,
    "indices.cdc.buffer.memory": 33554432,
    "indices.cdc.bootstrap.servers": "xxx:9092,yyy:9092",
    "indices.cdc.max.request.size": 52428800,
    "indices.cdc.max.in.flight.requests.per.connection": 10,
    "indices.cdc.retry.backoff.ms": 100,
    "indices.cdc.retries": 100,
    "indices.cdc.max.block.ms": 86400000
  }
}

3.2 创建索引时，enable cdc并初始化相关属性

PUT /index
{
   
    "settings":{
   
        "number_of_shards":3,
        "number_of_replicas":1,
        "index.cdc.enabled":true,
        "index.cdc.topic":"cdc_test",
        "index.cdc.pk.column":"doc_id",
        "index.refresh_interval":"1s"
    },
    "mappings":{
   
        "test":{
   
            "properties":{
   
                "content":{
   
                    "type":"text"
                },
                "doc_id":{
   
                    "type":"integer"
                },
                "age":{
   
                    "type":"integer"
                },
                "address":{
   
                    "type":"keyword"
                },
                "keywords":{
   
                    "type":"nested",
                    "properties":{
   
                        "keyword":{
   
                            "type":"keyword"
                        },
                        "frequency":{
   
                            "type":"integer"
                        }
                    }
                }
            }
        }
    }
}

3.3 索引级别属性说明

在这里插入图片描述

3.4 集群级别配置属性

在这里插入图片描述

3.5 测试

3.5.1 insert with POST

请求信息：

POST /index/test/1
{
   "doc_id":1,"content":"peace and hope","age":12,"address":"bj","keywords":[{
   "keyword":"peace","frequency":10},{
   "keyword":"hope","frequency":5}]}

cdc结果：

{
   "op":1,"index":"index","content":{
   "address":"bj","keywords":[{
   "keyword":"peace","frequency":10},{
   "keyword":"hope","frequency":5

最低0.47元/天解锁文章