logstash使用自定义模板问题记录

最新推荐文章于 2024-03-19 15:59:41 发布

原创最新推荐文章于 2024-03-19 15:59:41 发布 · 2.5k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#elasticsearch #logstash

工具安装同时被 3 个专栏收录

11 篇文章

订阅专栏

日常记录

10 篇文章

订阅专栏

环境配置

3 篇文章

订阅专栏

本文档记录了使用Logstash将日志发送到Elasticsearch时，如何自定义模板以优化字段类型，节省存储空间。在尝试安装模板过程中遇到了未执行模板安装和400错误的问题。解决方法包括：去除`manage_template=true`选项，以及针对Elasticsearch 6.x版本的模板结构调整，需要在`mappings`下添加`_doc`字段。

起因：

使用ogstash把日志存储到es，由于修改字段类型节省空间的需要，考虑使用自定义模板

经过：

根据网上配置了logstash.conf

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-test-%{+YYYY.MM.dd}"
timeout => 30
template => "/data/conf/template.json"
template_name => "template-test"
template_overwrite => true
}
}

模板：

{
"order": 0,
"index_patterns": [
"logstash-test-*"
],
"mappings": {
           "dynamic_templates": [
               {
                   "string_fields": {
                       "match": "*",
                       "match_mapping_type": "string",
                       "mapping": {
                           "ignore_above": 256,
                           "type": "keyword"
                       }
                   }
               },
               {
                   "long_fields": {
                       "match": "*",
                       "match_mapping_type": "long",
                       "mapping": {
                           "doc_values": true,
                           "type": "long"
                       }
                   }
               },
               {
                   "date_fields": {
                       "match": "*",
                       "match_mapping_type": "date",
                       "mapping": {
                           "doc_values": true,
                           "type": "date"
                       }
                   }
               },,
               {
                   "text_fields": {
                       "match_mapping_type": "string",
                       "mapping": {
                           "type": "text",
                           "fields": {
                               "raw": {
                                   "type": "keyword",
                                   "ignore_above": 256
                               }
                           }
                       }
                   }
               }
           ],
           "properties": {
               "@timestamp": {
                   "type": "date",
                   "format": "yyyy-MM-dd HH:mm:ss||HH:mm:ss||strict_date_optional_time||epoch_millis"
               },
               "info": {
                   "type": "keyword",
                   "ignore_above": 256
               },
               "cost": {
                   "type": "long"
               },
               "description": {
                   "type": "text",
                   "norms": false,
                   "fields": {
                       "raw": {
                           "ignore_above": 256,
                           "type": "keyword"
                       }
                   }
               },
               "userId": {
                   "type": "keyword",
                   "ignore_above": 256
               }
           }
       },
"settings": {
"index": {
"max_ngram_diff": "253",
"refresh_interval": "5s",
"number_of_shards": "5",
"analysis": {
"normalizer": {
"lowercase_normalizer": {
"filter": [
"lowercase"
],
"type": "custom"
}
}
}
}
}
}

问题：

1、没有执行模板安装

发现自己设置么 manage_template=true;

解决：去掉 manage_template=true;

2、安装模板报400 错误，通过http添加模板报

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"
}
],
"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"
},
"status": 400

}

原因：模板是es7的，要安装的服务器es是6+的

解决：

在Mapping结构下增加_doc {

包含 properties， dynamic_templates等（原谅我，那么长的模板不想ctrl+c ctrl+v）

}

es7+的不需要增加_doc

官网关于这个模板结构的6.8的文档：Removal of mapping types | Elasticsearch Guide [6.8] | Elastic