win10 monstache mongdb es7.8同步

win10 使用monstache进行mongdb3.6与es7.8同步与查询

​ 网上有关于使用mongo-connector的，river的，还有使用logstash的。mongo-connector和river都已经年久失修了，es的版本更新太快，他们已经跟不上版本了，且mongo-connector这个方式很不稳定，容易断。logstash的方法网上也有，但是大多都是基于centos的，在win10上我尝试了一些天发现有一些问题没有法解决。需要的可以自行去百度查询。这里是详细的介绍使用monstache对于mongdb3.6与es7.8的同步。

monstache是最近比较新的使用进行mongdb与es的同步方法。github的如下，有详细的说明选择安装的版本以及配置的信息。因为这个是同步OPLOG的方法进行数据库和es的更新，所以需要先进行mongdb的副本集的制作，这点非常的重要，在下面有详细的介绍方式。

https://rwynn.github.io/monstache-site/start/

https://rwynn.github.io/monstache-site/advanced/

1.制作mongdb副本集！！！！！！！！！！！！！！！很重要。

https://blog.csdn.net/jack_brandy/article/details/88887795，这个是windows写的最详细的操作，十分的干货，没看这个之前吃了很大的亏。

大概的意思就是：一个主节点，一个从节点和一个选举节点，三个节点都有着属于自己的访问端口。首先你要运行你的mongdb的主节点，在将从节点全部加上。
你的操作都是在主节点上，但是从节点会进行同步。但是每个节点都需要启动，启动写在了conf文件中。

下面是我根据网站的操作的一些截图和简述。mong有多种启动方式，我是如下的方式：

每次mong启动都需要一个data下的文件 mongod -f D:\mongodb\data\conf\rs1.conf ，重复把所有的节点都加上。再启动 mongo --port 40000（40000是端口号，可以根据conf的文件修改）。因为文件中rs1的端口是40000，所以rs1是主节点。文件命名为什么都无所谓，但是mong启动必须要有个文件夹用于存放数据和log。

我按照网站的要求写成了rs1，rs2，rs3。这三个都可以成为主节点，因为他们的端口不一样，你启动了哪个节点哪个节点就是主节点。接着跟着网站走就好。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-91JvHgfD-1594028842411)(C:\Users\lenovo\AppData\Roaming\Typora\typora-user-images\image-20200703145517788.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-BcsPj0s9-1594028842415)(C:\Users\lenovo\AppData\Roaming\Typora\typora-user-images\image-20200703150403574.png)]

2.根据你的mongdb版本下载相应的es，monstache。

我的mongdb3,6，之前因为同步mysql装的是es7.8，按理说是没有这两个版本的monstache,要不然就进行es的降级，降到es6.x；要不然升级mongdb到4.x。但是我尝试了一下，下载了Monstache4和6两个版本，为什么这样下载可以下图，自己理解。monstache6这个版本运行时因为mongdb版本的原因会一直报错。但是Monstache4却是可以进行同步和查询的，说明es6和7 的版本在这个问题上差距不大。如果你电脑上是es7也是可以用的。下面我用的是Monstache4。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JhJB5eUl-1594028842417)(C:\Users\lenovo\AppData\Roaming\Typora\typora-user-images\image-20200706172630776.png)]

3.根据官网的网站写相应的配置文件。config.toml

# connection settings

# connect to MongoDB using the following URL
mongo-url = "mongodb://root:<your_mongodb_password>@dds-bp1aadcc629******.mongodb.rds.aliyuncs.com:3717"
# connect to the Elasticsearch REST API at the following node URLs
elasticsearch-urls = ["http://es-cn-mp91kzb8m00******.elasticsearch.aliyuncs.com:9200"]

# frequently required settings

# if you need to seed an index from a collection and not just listen and sync changes events
# you can copy entire collections or views from MongoDB to Elasticsearch
direct-read-namespaces = ["mydb.hotmovies","mydb.col"]

# if you want to use MongoDB change streams instead of legacy oplog tailing use change-stream-namespaces
# change streams require at least MongoDB API 3.6+
# if you have MongoDB 4+ you can listen for changes to an entire database or entire deployment
# in this case you usually don't need regexes in your config to filter collections unless you target the deployment.
# to listen to an entire db use only the database name.  For a deployment use an empty string.
#change-stream-namespaces = ["mydb.col"]

# additional settings

# if you don't want to listen for changes to all collections in MongoDB but only a few
# e.g. only listen for inserts, updates, deletes, and drops from mydb.mycollection
# this setting does not initiate a copy, it is only a filter on the change event listener
#namespace-regex = '^mydb\.col$'
# compress requests to Elasticsearch
#gzip = true
# generate indexing statistics
#stats = true
# index statistics into Elasticsearch
#index-stats = true
# use the following PEM file for connections to MongoDB
#mongo-pem-file = "/path/to/mongoCert.pem"
# disable PEM validation
#mongo-validate-pem-file = false
# use the following user name for Elasticsearch basic auth
elasticsearch-user = "elastic"
# use the following password for Elasticsearch basic auth
elasticsearch-password = "<your_es_password>"
# use 4 go routines concurrently pushing documents to Elasticsearch
elasticsearch-max-conns = 4
# use the following PEM file to connections to Elasticsearch
#elasticsearch-pem-file = "/path/to/elasticCert.pem"
# validate connections to Elasticsearch
#elastic-validate-pem-file = true
# propogate dropped collections in MongoDB as index deletes in Elasticsearch
dropped-collections = true
# propogate dropped databases in MongoDB as index deletes in Elasticsearch
dropped-databases = true
# do not start processing at the beginning of the MongoDB oplog
# if you set the replay to true you may see version conflict messages
# in the log if you had synced previously. This just means that you are replaying old docs which are already
# in Elasticsearch with a newer version. Elasticsearch is preventing the old docs from overwriting new ones.
#replay = false
# resume processing from a timestamp saved in a previous run
resume = true
# do not validate that progress timestamps have been saved
#resume-write-unsafe = false
# override the name under which resume state is saved
#resume-name = "default"
# use a custom resume strategy (tokens) instead of the default strategy (timestamps)
# tokens work with MongoDB API 3.6+ while timestamps work only with MongoDB API 4.0+
resume-strategy = 0
# exclude documents whose namespace matches the following pattern
#namespace-exclude-regex = '^mydb\.ignorecollection$'
# turn on indexing of GridFS file content
#index-files = true
# turn on search result highlighting of GridFS content
#file-highlighting = true
# index GridFS files inserted into the following collections
#file-namespaces = ["users.fs.files"]
# print detailed information including request traces
verbose = true
# enable clustering mode
cluster-name = 'es-cn-mp91kzb8m00******'
# do not exit after full-sync, rather continue tailing the oplog
#exit-after-direct-reads = false
[[mapping]]
namespace = "mydb.hotmovies"
index = "hotmovies"


[[mapping]]
namespace = "mydb.col"
index = "mydbcol"

找了很久，https://help.aliyun.com/document_detail/171650.html?spm=a2c4g.11186623.6.801.545a4059tf4Nb8，有着很详细的参数的介绍。我说几个最重要的参数

direct-read-namespaces = ["mydb.hotmovies","mydb.col"] #用于同步mongdb到es，mydb是数据库，hotmovies是集合，也是就是mysql的表。
namespace-regex = '^mydb\.col$'#这个是功能可以看介绍，mydb是数据库、col是集合，注意^,$不要忘记
#mapping用于指定同步到es的index，其他的同理。
[[mapping]]
namespace = "mydb.col"
index = "mydbcol"

把这个文件放在monstache4\build\windows-amd64中

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xFx5nZHd-1594028842430)(C:\Users\lenovo\AppData\Roaming\Typora\typora-user-images\image-20200706174158436.png)]

4.启动cmd，cd到上面文件夹中，运行monstache -f config.toml

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-3NOchDQD-1594028842434)(C:\Users\lenovo\AppData\Roaming\Typora\typora-user-images\image-20200706174001500.png)]

你的toml的配置文件的不同，这个cmd里东西会用轻微的变化，只要不报错就可以。现在就实现了数据库的同步。当你插入删除，修改数据的时候都会同步到es中。

5.查询python

查询的方法使用python就很简单了，可以看我的与mysql相连的方法中有详细的介绍。

注释：

mongdb的可视化工具我选择的是Robo3T。如果追求可视化来查看数据的同步的情况，可以使用kibana，这个方法很好用，非常建议学习，这样可以很大的提高效率。怎么使用，自行百度。怎么启动和下载需要去es的官网去下载，同时要与es的版本保持一致，一定要保持一致。