flink- mysql同步数据至starrocks-2.5.0之环境搭建

flink- mysql同步数据至starrocks-2.5.0之环境搭建

一般需要以下几个服务:

• mysql
• flink
  • flink-taskmanager
  • flink-jobmanager
• starrocks
  • starrocks-fe
  • starrocks-be

docker-compose.yml 配置文件

version: "3.3"
services:
  starrocks-fe:
    image: starrocks/fe-ubuntu:2.5.0
    #image: starrocks.docker.scarf.sh/starrocks/allin1-ubuntu
    hostname: starrocks-fe
    container_name: starrocks-fe
    command:
      /opt/starrocks/fe/bin/start_fe.sh
    ports:
      - 8030:8030
      - 9020:9020
      - 9030:9030
    volumes:
       - /etc/localtime:/etc/localtime
    #   - ../../conf/fe.conf:/opt/starrocks/fe/conf/fe.conf
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9030"]
      interval: 5s
      timeout: 5s
      retries: 30
    environment:
      - TZ="Asia/Shanghai"

  starrocks-be:
    #image: starrocks/be-ubuntu:main
    image: starrocks/be-ubuntu:2.5.0
    command:
      - /bin/bash
      - -c
      - |
        sleep 15s; mysql --connect-timeout 2 -h starrocks-fe -P9030 -uroot -e "alter system add backend \"starrocks-be:9050\";"
        /opt/starrocks/be/bin/start_be.sh
    ports:
      - 8040:8040
    hostname: starrocks-be
    container_name: starrocks-be
    depends_on:
      - starrocks-fe
    # volumes:
    #   - ../../conf/be.conf:/opt/starrocks/be/conf/be.conf
    volumes:
       - /etc/localtime:/etc/localtime
    environment:
      - TZ="Asia/Shanghai"
  mysql:
    image: mysql:8
    container_name: mysql
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: test
      TZ: Asia/Shanghai
    command:
      --max_connections=1000
      --character-set-server=utf8mb4
      --collation-server=utf8mb4_general_ci
      --default-authentication-plugin=mysql_native_password
    ports:
      - 3306:3306
    volumes:
      - $PWD/data:/var/lib/mysql
      - $PWD/logs:/logs
      - $PWD/conf/my.cnf:/etc/mysql/my.cnf
      - $PWD/mysql-file:/var/lib/mysql-files
  jobmanager:
    #image: flink:1.16.2-scala_2.12
    image: flink:1.15.4-scala_2.12
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager        
    volumes:
      - $PWD/lib:/opt/flink/lib # //在flink部署jar包时，需要将依赖包上传，所以需要挂载出来
  taskmanager:
    #image: flink:1.16.2-scala_2.12
    image: flink:1.15.4-scala_2.12
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2 
    volumes:
      - $PWD/lib:/opt/flink/lib
# 如果不在同一个docker-comopse创建的服务，请务必将这个服务放在一个网络中，不然，mysql同步至starrocks会请求超时！！
# networks:
#   default:
#     external:
#       name: starrocks_default

启动: docker-compose up -d:

[root@blue starrocks]# docker ps|awk '{print $2}'
ID
flink:1.15.4-scala_2.12
flink:1.15.4-scala_2.12
mysql:8
starrocks/be-ubuntu:2.5.0
starrocks/fe-ubuntu:2.5.0

登陆 starrocks

starrocks@starrocks-fe:/opt/starrocks$ mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > "
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 24
Server version: 5.1.0 StarRocks version 2.5.0

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

注意事项

mysql开启 bin log

mysql 基于cdc ，采用 binlog模式，所以要开启binlog, conf/my.cnf:

[mysqld]
log_bin = mysql-bin
server_id = 1

docker 服务放到同一docker网络中

如果不在同一个docker-comopse创建的服务，请务必将这个服务放在一个网络中，不然，mysql同步至starrocks会请求超时