linux行位换行符,换行符或标点符号作为elasticsearch中的位置间隔

我最终想出了一个解决方案，使用char_filter在换行符和标点符号上引入额外的标记：

PUT /index

{

"settings": {

"analysis": {

"char_filter": {

"my_mapping": {

"type": "mapping",

"mappings": [ ".=>\\n_PERIOD_\\n", "\\n=>\\n_NEWLINE_\\n" ]

}

},

"analyzer": {

"my_analyzer": {

"tokenizer": "standard",

"char_filter": ["my_mapping"],

"filter": ["lowercase"]

}

}

}

}

}

使用示例字符串进行测试

POST /index/_analyze?analyzer=my_analyzer&pretty

The quick brown fox runs after the rabbit.

Then comes the jumpy frog.

结果如下：

{

"tokens" : [ {

"token" : "the",

"start_offset" : 0,

"end_offset" : 3,

"type" : "",

"position" : 1

}, {

... snip ...

"token" : "rabbit",

"start_offset" : 35,

"end_offset" : 41,

"type" : "",

"position" : 8

}, {

"token" : "_period_",

"start_offset" : 41,

"end_offset" : 41,

"type" : "",

"position" : 9

}, {

"token" : "_newline_",

"start_offset" : 42,

"end_offset" : 42,

"type" : "",

"position" : 10

}, {

"token" : "then",

"start_offset" : 43,

"end_offset" : 47,

"type" : "",

"position" : 11

... snip ...

} ]

}