【Maxwell】使用maxwell+kafka+python做binlog增量解析消费

1.maxwell基础
参考http://seanlook.com/2018/01/13/maxwell-binlog/

这里写下我的基础配置

1. [root@ip-10-1-2-175 maxwell-1.12.0]# more config.properties
   
2. 
   
3. log_level=info
   
4. kafka.bootstrap.servers=localhost:9092
   
5. producer=kafka
   
6. host=10.1.2.175
   
7. user=dbaadmin
   
8. password=
   
9. schema_database=maxwell
   
10. kafka.compression.type=snappy
    
11. kafka.retries=0
    
12. kafka.acks=1
    
13. kinesis_stream=maxwell
    
14. sqs_queue_uri=aws_sqs_queue_uri
    

1. producer=kafka   ###生产者选择kafka
2. kafka.bootstrap.servers=10.1.2.175:9092
3. kafka_topic=test    ####主题
4. 
   
5. output_ddl = true   ####ddl输出到 maxwell_ddl topic
   ddl_kafka_topic=maxwell_ddl
   
   
6. producer_partition_by=database

2.python 消费者伪代码

1. import time, json
   
2. from pykafka import KafkaClient
   
3. client = KafkaClient(hosts="10.1.2.175:9092")
   
4. topic = client.topics['test']
   
5. balanced_consumer = topic.get_balanced_consumer(consumer_group='goods_group',auto_commit_enable=True,zookeeper_connect='localhost:2181')
   
6. 
   
7. for message in balanced_consumer:
   
8.     print message.offset, message.value

启动
[root@ip-10-1-2-175 sh]# python cus.py 

3.启动maxwel

1. [root@ip-10-1-2-175 maxwell-1.12.0]# ./bin/maxwell

4.模拟数据库操作

1. mysql> insert into testwell(name) values ('fuck');
   
2. Query OK, 1 row affected (0.00 sec)
   
3. 
   
4. mysql> insert into testwell(name) values ('fuck');
   
5. Query OK, 1 row affected (0.00 sec)
   
6. 
   
7. mysql> insert into testwell(name) values ('fuck');
   
8. Query OK, 1 row affected (0.00 sec)
   
9. 
   
10. mysql>

5.查看消费者

1. [root@ip-10-1-2-175 sh]# python cus.py
   
2. 67 {"database":"test","table":"testwell","type":"insert","ts":1526290515,"xid":208818,"commit":true,"data":{"id":3050,"name":"fuck"}}
   
3. 68 {"database":"test","table":"testwell","type":"insert","ts":1526290515,"xid":208819,"commit":true,"data":{"id":3051,"name":"fuck"}}
   
4. 69 {"database":"test","table":"testwell","type":"insert","ts":1526290517,"xid":208820,"commit":true,"data":{"id":3052,"name":"fuck"}}
   
5. 70 {"database":"test","table":"testwell","type":"insert","ts":1526290519,"xid":208822,"commit":true,"data":{"id":3053,"name":"fuck"}}

6.根据上面的思路可以写出增量同步的伪代码

1. [root@ip-10-1-2-175 sh]#
   
2. import time, json
   
3. from pykafka import KafkaClient
   
4. client = KafkaClient(hosts="10.1.2.175:9092")
   
5. topic = client.topics['test']
   
6. balanced_consumer = topic.get_balanced_consumer(consumer_group='goods_group',auto_commit_enable=True,zookeeper_connect='localhost:2181')
   
7. 
   
8. for message in balanced_consumer:
   
9.     print message.offset, message.value
   
10.    # 对数据库进行操作
    
11.     if message.value ['type'] == 'insert':
    
12.       mysql_insert()
    
13.     elif  message.value['type'] == 'update':
    
14.       mysql_update()
    
15.     elif  message.value['type'] == 'delete':
    
16.       mysql_delete()
    
17.     else:
    
18.       order_option()

其它：
https://mp.weixin.qq.com/s/aGqwWNd6Q2Gm2jilxXS0RA

Kafka Connect 如何实现同步RDS binlog数据?

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/29096438/viewspace-2154470/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/29096438/viewspace-2154470/