logstash使用自定义模板问题记录

最新推荐文章于 2023-03-01 11:13:16 发布

祎木一世界

最新推荐文章于 2023-03-01 11:13:16 发布

阅读量2.4k

点赞数

分类专栏：环境配置日常记录工具安装文章标签： elasticsearch logstash

本文链接：https://blog.csdn.net/tttmdds/article/details/121477163

版权

工具安装同时被 3 个专栏收录

11 篇文章 0 订阅

订阅专栏

日常记录

10 篇文章 0 订阅

订阅专栏

环境配置

3 篇文章 0 订阅

订阅专栏

起因：

使用ogstash把日志存储到es，由于修改字段类型节省空间的需要，考虑使用自定义模板

经过：

根据网上配置了logstash.conf

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-test-%{+YYYY.MM.dd}"
timeout => 30
template => "/data/conf/template.json"
template_name => "template-test"
template_overwrite => true
}
}

模板：

{
"order": 0,
"index_patterns": [
"logstash-test-*"
],
"mappings": {
           "dynamic_templates": [
               {
                   "string_fields": {
                       "match": "*",
                       "match_mapping_type": "string",
                       "mapping": {
                           "ignore_above": 256,
                           "type": "keyword"
                       }
                   }
               },
               {
                   "long_fields": {
                       "match": "*",
                       "match_mapping_type": "long",
                       "mapping": {
                           "doc_values": true,
                           "type": "long"
                       }
                   }
               },
               {
                   "date_fields": {
                       "match": "*",
                       "match_mapping_type": "date",
                       "mapping": {
                           "doc_values": true,
                           "type": "date"
                       }
                   }
               },,
               {
                   "text_fields": {
                       "match_mapping_type": "string",
                       "mapping": {
                           "type": "text",
                           "fields": {
                               "raw": {
                                   "type": "keyword",
                                   "ignore_above": 256
                               }
                           }
                       }
                   }
               }
           ],
           "properties": {
               "@timestamp": {
                   "type": "date",
                   "format": "yyyy-MM-dd HH:mm:ss||HH:mm:ss||strict_date_optional_time||epoch_millis"
               },
               "info": {
                   "type": "keyword",
                   "ignore_above": 256
               },
               "cost": {
                   "type": "long"
               },
               "description": {
                   "type": "text",
                   "norms": false,
                   "fields": {
                       "raw": {
                           "ignore_above": 256,
                           "type": "keyword"
                       }
                   }
               },
               "userId": {
                   "type": "keyword",
                   "ignore_above": 256
               }
           }
       },
"settings": {
"index": {
"max_ngram_diff": "253",
"refresh_interval": "5s",
"number_of_shards": "5",
"analysis": {
"normalizer": {
"lowercase_normalizer": {
"filter": [
"lowercase"
],
"type": "custom"
}
}
}
}
}
}

问题：

1、没有执行模板安装

发现自己设置么 manage_template=true;

解决：去掉 manage_template=true;

2、安装模板报400 错误，通过http添加模板报

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"
}
],
"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"
},
"status": 400

}

原因：模板是es7的，要安装的服务器es是6+的

解决：

在Mapping结构下增加_doc {

包含 properties， dynamic_templates等（原谅我，那么长的模板不想ctrl+c ctrl+v）

}

es7+的不需要增加_doc

官网关于这个模板结构的6.8的文档：Removal of mapping types | Elasticsearch Guide [6.8] | Elastic