Elasticsearch实战
一个field的设置是不能被修改的,如果要修改一个Field,那么应该重新按照新的mapping,建立一个index,然后将数据批量查询出来,重新用bulk api写入index中
批量查询的时候,建议采用scroll api,并且采用多线程并发的方式来reindex数据,每次scoll就查询指定日期的一段数据,交给一个线程即可
step1
首先我们建两个文档,然后看下mapping
PUT /vehicle/car/1
{
"name":"benz",
"create": "2020-01-01"
}
PUT /vehicle/car/2
{
"name":"bmw",
"create": "2019-01-01"
}
GET /vehicle/_mapping
{
"vehicle" : {
"mappings" : {
"properties" : {
"create" : {
"type" : "date"
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
step2
一开始,依靠dynamic mapping,插入数据,但是不小心有些数据是2020-01-01这种日期格式的,所以title这种field被自动映射为了date类型,实际上它应该是string类型的,当后期向索引中加入string类型的title值的时候,就会报错,如果此时想修改title的类型,是不可能的
PUT /vehicle/car/3
{
"name":"Range Rover",
"create":"china"
}
{
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [create] of type [date] in document with id '3'. Preview of field's value: 'china'"
}
],
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [create] of type [date] in document with id '3'. Preview of field's value: 'china'",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "failed to parse date field [china] with format [strict_date_optional_time||epoch_millis]",
"caused_by" : {
"type" : "date_time_parse_exception",
"reason" : "Failed to parse with all enclosed parsers"
}
}
},
"status" : 400
}
PUT /vehicle/_mapping/car?include_type_name=true
{
"properties":{
"create":{
"type":"text"
}
}
}
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "mapper [create] cannot be changed from type [date] to [text]"
}
],
"type" : "illegal_argument_exception",
"reason" : "mapper [create] cannot be changed from type [date] to [text]"
},
"status" : 400
}
step3
此时,唯一的办法,就是进行reindex,也就是说,重新建立一个索引,将旧索引的数据查询出来,再导入新索引, 给java应用一个别名,这个别名是指向旧索引的,java应用先用着,java应用先用old_vehicle alias来操作,此时实际指向的是旧的vehicle
PUT /vehicle/_alias/alias_vehicle
{
"acknowledged" : true
}
查询使用alias
GET /alias_vehicle/car/_search
#! Deprecation: [types removal] Specifying types in search requests is deprecated.
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "vehicle",
"_type" : "car",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"name" : "benz",
"create" : "2020-01-01"
}
},
{
"_index" : "vehicle",
"_type" : "car",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "bmw",
"create" : "2019-01-01"
}
}
]
}
}
step4
建立新的索引,并通过scrool分页查询原数据
PUT new_vehicle?include_type_name=true
{
"mappings": {
"car":{
"properties":{
"name":{
"type":"text"
},
"create":{
"type":"text"
}
}
}
}
}
#! Deprecation: [types removal] Using include_type_name in create index requests is deprecated. The parameter will be removed in the next major version.
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "new_vehicle"
}
scroll深度分页查询
GET /vehicle/car/_search?scroll=1m
{
"query": {
"match_all": {}
},
"sort": [
{
"_doc": {
"order": "asc"
}
}
],
"size": 1
}
GET _search/scroll
{
"scroll":"1m",
"scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDXF1ZXJ5QW5kRmV0Y2gBFks2Qi1fV0ptUkJtNlhETTVSU2piSlEAAAAAAAAE5BZSemFEbHd6SlRaQ3E0R29YbXgtcVNR"
}
step5
采用bulk api将scoll查出来的一批数据,批量写入新索引
POST /new_vehicle/car/_bulk
{"index":{"_id":1}}
{"name":"benz","create":"2020-01-01"}
{"index":{"_id":2}}
{"name":"bmw11","create":"2019-01-01"}
step6
将alias_vehicle alias切换到new_vehicle上去,java应用会直接通过index别名使用新的索引中的数据,java应用程序不需要停机,零提交,高可用
POST /_aliases
{
"actions": [
{
"remove": {
"index": "vehicle",
"alias": "alias_vehicle"
}
},
{
"add": {
"index": "new_vehicle",
"alias": "alias_vehicle"
}
}
]
}
#! Deprecation: [types removal] Specifying types in search requests is deprecated.
{
"took" : 568,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "new_vehicle",
"_type" : "car",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"name" : "benz",
"create" : "2020-01-01"
}
},
{
"_index" : "new_vehicle",
"_type" : "car",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "bmw11",
"create" : "2019-01-01"
}
}
]
}
}
至此已经显示了索引的重建以及数据迁移
欢迎访问我的个人博客:小马博客