mytest_user.yml:
dataSourceKey: defaultDS
destination: DX_tj_SEND_MYSQL_LOG_MSG
groupId: g1
esMapping:
_index: wx_cp_user_contact_rel_all
_id: id
# upsert: true
# pk: id
sql: "select
CONCAT(a.external_userid, '*', a.follow_user_id) as id,
a.external_userid as externalUserid,
a.follow_user_id as followUserId,
a.follow_user_remark as followUserRemark,
a.follow_user_description as followUserDescription,
date_format( a.follow_user_createtime, '%Y-%m-%d %H:%I:%S') as followUserCreatetime,
a.follow_user_tags_group_name as followUserTagsGroupName,
a.follow_user_tags_tag_name as followUserTagsTagName,
a.follow_user_tags_type as followUserTagsType,
a.follow_user_remark_corp_name as followUserRemarkCorpName,
a.address,
a.corp_id as corpId,
a.follow_user_remark_mobiles as followUserRemarkMobiles,
a.follow_user_state as followUserState ,
a.status ,
a. birthday,
date_format( a.link_time, '%Y-%m-%d %H:%I:%S') as linkTime,
date_format( a.create_time, '%Y-%m-%d %H:%I:%S') as createTime,
b.name,
b.avatar,
b.type,
b.gender,
b.unionid,
b.position,
b.corp_name as corpName,
b.corp_full_name as corpFullName,
b.external_profile as externalProfile,
b.phone
from
wx_cp_user_contact_rel a
left join wx_cp_user_contact b on
a.external_userid = b.external_userid"
# objFields:
# _labels: array:;
etlCondition: "where a.corp_id ={}"
commitBatch: 3000
看了源码才知道 where 条件后面的参数是需要在访问 的时候传入例如:
源码:
全量同步:
curl -X POST http://localhost:8585/etl/es7/mytest_user.yml?params=ww685f2b0e44c63b20
增量同步打开开关(DX_tj_SEND_MYSQL_LOG_MSG为kafka主题,或者canal instanse实例名称):
curl http://127.0.0.1:8585/syncSwitch/DX_tj_SEND_MYSQL_LOG_MSG/on -X PUT