mongoshake实现跨机房的副本集同步

        MongoShake是一个以golang语言进行编写的通用的平台型服务，通过读取MongoDB集群的Oplog操作日志，对MongoDB的数据进行复制，后续通过操作日志实现特定需求。日志可以提供很多场景化的应用，为此，在设计时就考虑了把MongoShake做成通用的平台型服务。通过操作日志，提供日志数据订阅消费PUB/SUB功能，可通过SDK、Kafka、MetaQ等方式灵活对接以适应不同场景(如日志订阅、数据中心同步、Cache异步淘汰等)。集群数据同步是其中核心应用场景，通过抓取oplog后进行回放达到同步目的，实现灾备和多活的业务场景。

下面是我的操作步骤：

1.下载mongo shake

https://github.com/alibaba/MongoShake/releases?spm=a2c4g.11186623.2.21.65fc6ed2GjMvka

tar -xvf mongo-shake-v2.4.7_2.tar.gz

2.创建用户

源副本集：

use admin;

db.createUser({user:'shake',pwd:'shake',roles:[{role:'readWrite',db:'mongoshake'}]})

db.grantRolesToUser("shake",[{role:'readAnyDatabase',db:'admin'}])

db.grantRolesToUser("shake",[{role:'read',db:'local'}])

db.grantRolesToUser("shake",[{role:'read',db:'config'}])

目标副本集：

use admin;

db.createUser({user:'shake',pwd:'shake',roles:[{role:'readWriteAnyDatabase',db:'admin'}]})

3.修改配置文件（粗体为修改的，其他为默认的）

# cat collector.conf|grep -v '#'|grep -v '^$' 

conf.version = 5

id = mongoshake

master_quorum = false

full_sync.http_port = 9101

incr_sync.http_port = 9100

system_profile_port = 9200

log.level = info

log.dir =

log.file = collector.log

log.flush = false

sync_mode = all

mongo_urls = mongodb://shake:shake@xxx,xxx,xxx

mongo_cs_url =

mongo_s_url =

tunnel = direct

tunnel.address = mongodb://shake:shake@xxx,xxx,xxx

tunnel.message = raw

mongo_connect_mode = secondaryPreferred

filter.namespace.black =

filter.namespace.white =

filter.pass.special.db =

filter.ddl_enable = true

checkpoint.storage.url =

checkpoint.storage.db = mongoshake

checkpoint.storage.collection = ckpt_default

checkpoint.start_position = 1970-01-01T00:00:00Z

transform.namespace =

full_sync.reader.collection_parallel = 6

full_sync.reader.write_document_parallel = 8

full_sync.reader.document_batch_size = 128

full_sync.reader.read_document_count = 0

full_sync.collection_exist_drop = true

full_sync.create_index = none

full_sync.executor.insert_on_dup_update = false

full_sync.executor.filter.orphan_document = false

full_sync.executor.majority_enable = false

incr_sync.mongo_fetch_method = oplog

incr_sync.oplog.gids =

incr_sync.shard_key = collection

incr_sync.worker = 8

incr_sync.worker.oplog_compressor = none

incr_sync.target_delay = 0

incr_sync.worker.batch_queue_size = 64

incr_sync.adaptive.batching_max_size = 1024

incr_sync.fetcher.buffer_capacity = 256

incr_sync.executor.upsert = false

incr_sync.executor.insert_on_dup_update = false

incr_sync.conflict_write_to = none

incr_sync.executor.majority_enable = false

incr_sync.change_stream.watch_full_document = false

4.开启

# ./start.sh collector.conf

5.监控

刚开始是full，然后变成增量同步后，才能

# ./mongoshake-stat --port=9100