mongo与elasticsearch数据同步插件

前提条件：

mongo集群必须是副本集，因为同步插件读取的是日志文件

1.需要安装pip插件（如果已经安装不需再安装）

yum install python-setuptools && easy_install pip

2.在mongo端  安装 mongo-connector。
pip install mongo-connector

3.安装 elastic2-doc-manager（注意这个是针对es2.x版本的，如果es安装的是1.x版本，用elastic-doc-manager）
pip install elastic-doc-manager

插件网址：https://github.com/mongodb-labs

4.启动同步插件

1. 不使用配置文件

• mongo-connector -m mongo的ip:27017 -t elasticsearch的IP:9200 -d elastic2_doc_manager
• 可以在命令行里面加入
• --auto-commit-interval=0此命令保证实时同步数据,0代表无延迟
• --continue-on-error 当数据中有错误会跳过

2.使用配置文件

{     "__comment__": "Configuration options starting with '__' are disabled",     "__comment__": "To enable them, remove the preceding '__'",     "mainAddress": "192.168.106.220:27017",     "oplogFile": "/home/mongo/oplog.timestamp",  自定义     "noDump": false,     "batchSize": -1,     "verbosity": 0,     "__continueOnError": true,当数据中有错误会跳过     "logging": {         "type": "file",         "filename": "/home/mongo/mongo-connector.log",  自定义         "__format": "%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s",         "__rotationWhen": "D",         "__rotationInterval": 1,         "__rotationBackups": 10,         "__type": "syslog",         "__host": "localhost:514"     },     "authentication": {         "__adminUsername": "username",         "__password": "password",         "__passwordFile": "mongo-connector.pwd"     },     "__comment__": "For more information about SSL with MongoDB, please see http://docs.mongodb.org/manual/tutorial/configure-ssl-clients/",     "__ssl": {         "__sslCertfile": "Path to certificate to identify the local connection against MongoDB",         "__sslKeyfile": "Path to the private key for sslCertfile. Not necessary if already included in sslCertfile.",         "__sslCACerts": "Path to concatenated set of certificate authority certificates to validate the other side of the connection",         "__sslCertificatePolicy": "Policy for validating SSL certificates provided from the other end of the connection. Possible values are 'required' (require and validate certificates), 'optional' (validate but don't require a certificate), and 'ignored' (ignore certificates)."     },     "__fields": ["field1", "field2", "field3"],     "__namespaces": {         "excluded.collection": false,         "excluded_wildcard.*": false,         "*.exclude_collection_from_every_database": false,         "included.collection1": true,         "included.collection2": {},         "included.collection4": {             "includeFields": ["included_field", "included.nested.field"]         },         "included.collection5": {             "rename": "included.new_collection5_name",             "includeFields": ["included_field", "included.nested.field"]         },         "included.collection6": {             "excludeFields": ["excluded_field", "excluded.nested.field"]         },         "included.collection7": {             "rename": "included.new_collection7_name",             "excludeFields": ["excluded_field", "excluded.nested.field"]         },         "included_wildcard1.*": true,         "included_wildcard2.*": true,         "renamed.collection1": "something.else1",         "renamed.collection2": {             "rename": "something.else2"         },         "renamed_wildcard.*": {             "rename": "new_name.*"         },         "gridfs.collection": {             "gridfs": true         },         "gridfs_wildcard.*": {             "gridfs": true         }     },     "docManagers": [         {             "docManager": "elastic_doc_manager",             "targetURL": ["192.168.106.218:9200","192.168.106.217:9200","192.168.106.219:9200"],             "bulkSize": 1000,             "__uniqueKey": "_id",             "autoCommitInterval": 1         }     ] }

启动命令：mongo-connector -c /etc/mongo-connector-conf.json & （json的目录自定义）

注意：写入mongo的速度比同步速度快，经过测试elasticsearch配置良好的情况下同步速度大概在1000条/s左右

插件安装的目录在：/usr/lib/python2.7/site-packages下面

elasticsearch、mongo-connector（如果想修改源码可以进入此目录）

转载于:https://www.cnblogs.com/yhx-0408/p/7267704.html