第3章 业务数据采集模块
3.1 采集通道
3.2 采集工具
3.3 采集通道Maxwell配置
1)修改Maxwell配置文件config.properties
[atguigu@hadoop102 maxwell]$ vim /opt/module/maxwell/config.properties
2)配置参数如下
log_level=info
producer=kafka
kafka.bootstrap.servers=hadoop102:9092,hadoop103:9092
#kafka topic配置
**kafka_topic=topic_db**
# mysql login info
host=hadoop102
user=maxwell
password=maxwell
jdbc_options=useSSL=false&serverTimezone=Asia/Shanghai
3)重新启动Maxwell
[atguigu@hadoop102 bin]$ mxw.sh restart
4)通道测试
(1)启动Zookeeper以及Kafka集群
(2)启动一个Kafka Console Consumer,消费topic_db数据
[atguigu@hadoop103 kafka]$ bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic topic_db
(3)生成模拟数据
[atguigu@hadoop102 bin]$ cd /opt/module/db_log/
[atguigu@hadoop102 db_log]$ java -jar gmall2020-mock-db-2021-11-14.jar
(4)观察Kafka消费者是否能消费到数据
{
"database": "gmall",
"table": "cart_info",
"type": "update",
"ts": 1592270938,
"xid": 13090,
"xoffset": 1573,
"data": {
"id": 100924,
"user_id": "93",
"sku_id": 16,
"cart_price": 4488,
"sku_num": 1,
"img_url": "http://47.93.148.192:8080/group1/M00/00/02/rBHu8l-sklaALrngAAHGDqdpFtU741.jpg",
"sku_name": "华为 HUAWEI P40 麒麟990 5G SoC芯片 5000万超感知徕卡三摄 30倍数字变焦 8GB+128GB亮黑色全网通5G手机",
"is_checked": null,
"create_time": "2020-06-14 09:28:57",
"operate_time": null,
"is_ordered": 1,
"order_time": "2021-10-17 09:28:58",
"source_type": "2401",
"source_id": null
},
"old": {
"is_ordered": 0,
"order_time": null
}
}