我最终想出了一个解决方案,使用char_filter在换行符和标点符号上引入额外的标记:
PUT /index
{
"settings": {
"analysis": {
"char_filter": {
"my_mapping": {
"type": "mapping",
"mappings": [ ".=>\\n_PERIOD_\\n", "\\n=>\\n_NEWLINE_\\n" ]
}
},
"analyzer": {
"my_analyzer": {
"tokenizer": "standard",
"char_filter": ["my_mapping"],
"filter": ["lowercase"]
}
}
}
}
}
使用示例字符串进行测试
POST /index/_analyze?analyzer=my_analyzer&pretty
The quick brown fox runs after the rabbit.
Then comes the jumpy frog.
结果如下:
{
"tokens" : [ {
"token" : "the",
"start_offset" : 0,
"end_offset" : 3,
"type" : "",
"position" : 1
}, {
... snip ...
"token" : "rabbit",
"start_offset" : 35,
"end_offset" : 41,
"type" : "",
"position" : 8
}, {
"token" : "_period_",
"start_offset" : 41,
"end_offset" : 41,
"type" : "",
"position" : 9
}, {
"token" : "_newline_",
"start_offset" : 42,
"end_offset" : 42,
"type" : "",
"position" : 10
}, {
"token" : "then",
"start_offset" : 43,
"end_offset" : 47,
"type" : "",
"position" : 11
... snip ...
} ]
}