Es 会跟踪每个活动分片的索引,分片在五分钟内如果没有收到任何有关索引的操作,则认为分片不活动了;这将给Es机会去减小分片的资源,同时执行一种特殊刷新,成为:同步刷新;同步刷新也是执行普通的刷新,然后给所有的分片添加一个不同的标记;
当没有索引操作时,id 标记会添加到分片上。标记可以作为一个快速的方式来检查两个分片的lucene索引一致是否一致;这种快速的id 比较主要用于 数据恢复或者重启后跳过第一个也是成本最高的阶段;这种情况下,segment 不需要copy,事务日志的重演会立即执行;当id 标记和flush 一起用的时候,事物日志很可能为空,更加加速了数据的恢复;
这对于具有很多很少更新或者不变的索引的情况(如基于时间的数据),将会更有用;查看分片是否被id 标记,用如下命令:
curl -X GET "dmp9:9201/wzp_test/_stats/commit?level=shards"
{"_shards":{"total":10,"successful":10,"failed":0},"_all":{"primaries":{},"total":{}},"indices":{"wzp_test":{"primaries":{},"total":{},"shards":{"0":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQg==","generation":12,"user_data":{"translog_uuid":"OHqJo56NQoqD0vivoRE_-w","sync_id":"AWdPVMGxhBFTwa1Fywjs","translog_generation":"1543159976461"},"num_docs":411901},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQg==","generation":12,"user_data":{"translog_uuid":"OHqJo56NQoqD0vivoRE_-w","sync_id":"AWdPVMGxhBFTwa1Fywjs","translog_generation":"1543159976461"},"num_docs":411901},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"1":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQQ==","generation":13,"user_data":
{"translog_uuid":"r5RMt3ZDS5ip1Il2cldOKA","sync_id":"AWdPVMGxhBFTwa1Fywjr","translog_generation":"1543160040431"},"num_docs":383147},"shard_path":
{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQQ==","generation":13,"user_data":{"translog_uuid":"r5RMt3ZDS5ip1Il2cldOKA","sync_id":"AWdPVMGxhBFTwa1Fywjr","translog_generation":"1543160040431"},"num_docs":383147},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"2":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAPw==","generation":13,"user_data":{"translog_uuid":"EYkaGyrTTPCtgKArFf0Qyw","sync_id":"AWdPVMGxhBFTwa1Fywjp","translog_generation":"1543159976463"},"num_docs":383110},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAPw==","generation":13,"user_data":{"translog_uuid":"EYkaGyrTTPCtgKArFf0Qyw","sync_id":"AWdPVMGxhBFTwa1Fywjp","translog_generation":"1543159976463"},"num_docs":383110},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"3":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQw==","generation":13,"user_data":{"translog_uuid":"livjpLxlRSakLFnCp8n9Jw","sync_id":"AWdPVMGxhBFTwa1Fywjo","translog_generation":"1543160040432"},"num_docs":410859},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQw==","generation":13,"user_data":{"translog_uuid":"livjpLxlRSakLFnCp8n9Jw","sync_id":"AWdPVMGxhBFTwa1Fywjo","translog_generation":"1543160040432"},"num_docs":410859},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"4":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQA==","generation":13,"user_data":{"translog_uuid":"US1IS1LaR5qkwlJwLLs7Vw","sync_id":"AWdPVMGxhBFTwa1Fywjq","translog_generation":"1543159976461"},"num_docs":416518},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQA==","generation":13,"user_data":{"translog_uuid":"US1IS1LaR5qkwlJwLLs7Vw","sync_id":"AWdPVMGxhBFTwa1Fywjq","translog_generation":"1543159976461"},"num_docs":416518},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}]}}}}
官方文档给的结果示例:
curl -X GET "localhost:9200/twitter/_stats/commit?level=shards"
{
...
"indices": {
"twitter": {
"primaries": {},
"total": {},
"shards": {
"0": [
{
"routing": {
...
},
"commit": {
"id": "te7zF7C4UsirqvL6jp/vUg==",
"generation": 2,
"user_data": {
"sync_id": "AU2VU0meX-VX2aNbEUsD" ,
...
},
"num_docs": 0
}
}
...
],
...
}
}
}
}
同步刷新 API
同步刷新API 允许管理员手动刷新。这对于计划的(回滚)集群重启特别有用,您可以停止索引,并且不希望等待默认的5分钟来自动同步刷新空闲索引;
尽管很方便,但是有一些警告:
-
同步刷新是个尽最大努力的操作:任何正在执行索引的操作都会造成同步刷新失败;这就一位置有些分片可能刷新成功,而有些分片则刷新失败;
-
同步id 的标记在再次刷新是会移除掉;
这是因为刷新会替换低级别的储存标记的lucene 提交点;事物日志中未提交的操作不会移除id 标记;现实中,在任何时间,应该把索引操作视触发的移除标记为刷新一样;
note:有索引操作时,请求同步刷新是无害的;当索引空闲时同步刷新成功,否则失败;任何请求成功的同步刷新在数据恢复时更加迅速;
curl -X POST "localhost:9200/twitter/_flush/synced"返回值将显示有多少分片同步刷新成功,及失败原因;
1)、以下是同步刷新成功的情况;
{
"_shards": {
"total": 4,
"successful": 4,
"failed": 0
},
"twitter": {
"total": 4,
"successful": 4,
"failed": 0
}
}
2)、以下是失败情况及其原因;
{
"_shards": {
"total": 4,
"successful": 2,
"failed": 2
},
"twitter": {
"total": 4,
"successful": 2,
"failed": 2,
"failures": [
{
"shard": 1,
"reason": "[2] ongoing operations on primary"
}
]
}
}
note:当并行索引操作时,以上错误会出现,且http 状态码将是 409 ;
3)、有时候分片copy的时候错误很特殊;copy失败对于数据的快速恢复没有用,但是如果成功了依然有用;具体错误如下:
{
"_shards": {
"total": 4,
"successful": 1,
"failed": 1
},
"twitter": {
"total": 4,
"successful": 3,
"failed": 1,
"failures": [
{
"shard": 1,
"reason": "unexpected error",
"routing": {
"state": "STARTED",
"primary": false,
"node": "SZNr2J_ORxKTLUCydGX4zA",
"relocating_node": null,
"shard": 1,
"index": "twitter"
}
}
]
}
}
同步刷新时copy失败,http 返回码将是409 ;
4)、同步刷新可以再单一索引上用,也可以在全部索引上用;
单一索引:
POST /kimchy,elasticsearch/_flush/synced
全部索引:
POST /_flush/synced