Flink环境部署

一、基础环境搭建

1.创建用户

groupadd hadoop

useradd -g hadoop hadoop

passwd hadoop

XXX

--添加sudo权限

visudo

hadoop ALL=(ALL) NOPASSWD:ALL

2.安装需要的工具

yum install htop java lrzsz sysstat iotop -y

yum install ansible -y

3.hadoop用户下配置互信

ssh-keygen -t rsa

ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub test31

ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@test31

4.ansible创建基础目录

--替换主机名

vim /etc/ansible/hosts

:%s/master1/test31/g

:%s/master2/test32/g

:%s/slave1/test33/g

5.ansible安装大数据组件

ansible-playbook yumtest.yaml

ansible-playbook common.yaml

ansible-playbook zookeeper.yaml

ansible-playbook scala.yaml

二、flink部署

1.上传并解压flink

cd /data/bigdata/app

rz flink-1.13.6-bin-scala_2.11.tgz

tar -zxvf flink-1.13.6-bin-scala_2.11.tgz

ln -s flink-1.13.6-bin-scala_2.11 flink

2.配置flink

vim .flink/conf/flink-conf.yaml

#配置内存大小

jobmanager.memory.process.size: 6000m

taskmanager.memory.process.size: 8000m

#配置最大任务数

taskmanager.numberOfTaskSlots: 50

#配置平行度，没有大数据量 一个就可以

parallelism.default: 1

#配置checkpoint存储位置

state.backend: rocksdb

execution.checkpointing.interval: 3000

state.checkpoints.dir: hdfs://test31:9001/flink/flink-checkpoints

state.savepoints.dir: hdfs://test31:9001/flink/flink-checkpoints

#开启增量同步

state.backend.incremental: true

至此单节点flink安装完成

3.上传需要的工具包

ll flink/lib

4.启动和关闭

./bin/start-cluster.sh

./bin/stop-cluster.sh

5.通过sql-client同步任务

sql-client.sh embedded

CREATE TABLE s_user1 (


id STRING,
order_info_id STRING,
goods_id STRING,
goods_id_h STRING,
goods_name STRING,
goods_no STRING,
goods_type INT,
package_type STRING,

  PRIMARY KEY (id) NOT ENFORCED
) WITH (
  'connector' = 'mysql-cdc',
  'hostname' = '******',
  'port' = '3306',
  'username' = '******',
  'password' = '******',
  'database-name' = 'test',
  'table-name' = 'user1',
  'server-time-zone' = 'Asia/Shanghai',
  'debezium.snapshot.mode' = 'initial'
);

CREATE TABLE user1(


id STRING,
order_info_id STRING,
goods_id STRING,
goods_id_h STRING,
goods_name STRING,
goods_no STRING,
goods_type INT,
package_type STRING,

  PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'connector' = 'jdbc',
    'url' = 'jdbc:mysql://******:3306/test?characterEncoding=UTF-8',
    'table-name' = 'user1',
    'username' = '******',
    'password' = '******'
);
  
  
INSERT INTO order_goods 
SELECT * FROM s_order_goods;

6.通过浏览器查看后台运行任务

添加白名单端口8001，并开启防火墙

http://******:8081/#/job/running

二、安装flink-streaming-platform-web

1.上传工具包

rz flink-streaming-platform-web.tar.gz

tar -zxvf flink-streaming-platform-web.tar.gz

2.配置flink-streaming-platform-web连接

vim vim flink-streaming-platform-web/conf/application.properties

3.创建元数据库并创建连接用户

CREATE USER flink_user@'%' IDENTIFIED BY '****';

GRANT INSERT,SELECT,UPDATE,DELETE,DROP,CREATE TEMPORARY TABLES,LOCK TABLES,EXECUTE,TRIGGER,SHOW VIEW,EVENT,ALTER ROUTINE,

CREATE ROUTINE,CREATE,INDEX,ALTER,CREATE VIEW ON `flink_web`.* TO flink_user@'%';

创建库脚本：

flink-streaming-platform-web/flink_web.sql at master · zhp8341/flink-streaming-platform-web · GitHub

4.启动关闭flink管理平台

./bin/deploy.sh start

./bin/deploy.sh stop

5.通过浏览器访问flink-streaming-platform-web

添加白名单端口9084，并开启防火墙

http://******:9084/static/ui/index.html#/index

admin / 123456