Flink-1.13.0 sql-client yarn-session模式部署

Flink-1.13.0 sql-client yarn-session模式部署

当前flink：sql-client.sh是 flink下类似spark-sql的sql方式处理数据的操作接口,可以像操作普通有界数据表一样去操作无界数据流，支持流实时查询、join、事件时间、水印延迟、窗口等各种操作，不用写java/scala/python代码一样可以操作数据流，当前主sql引擎采用alibaba Blink sql实现，当前flink-1.13.0版本仍是beta版本,有的公司已经在使用，也是未来的一个趋势

1.依赖

• hadoop-3.1.3（必须）

• apache-zookeeper-3.5.9(必须)

• kafka_2.11-2.4.0（必须）

• hbase-2.2.2（可选）

以上软件确定已安装完成

2.下载安装flink-1.13.0

官网下载改版本，解压

配置环境变量

sudo vim /etc/profile ,添加flink的环境变量到path,还有HADOOP_CLASSPATH必须要配置

export JAVA_HOME=/opt/jdk1.8.0_291
export SCALA_HOME=/opt/scala-2.11.12
export TERM=xterm-color
export HADOOP_HOME=/opt/hadoop-3.1.3
export HIVE_HOME=/opt/apache-hive-3.1.2-bin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/spark-2.4.4-bin-hadoop2.7
export HBASE_HOME=/opt/hbase-2.2.2
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export FLINK_HOME=/opt/flink-1.13.0
export HADOOP_CLASSPATH=`hadoop classpath`
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$SPARK_HOME/bin:$SCALA_HOME/bin:$HBASE_HOME/bin:FLINK_HOME/bin

source /etc/profile,生效环境变量

配置flink

vim $FLINK_HOME/conf/flink-conf.yaml

jobmanager.rpc.address: baby
rest.address: baby
jobmanager.memory.process.size: 1024m
taskmanager.memory.process.size: 1024m
jobmanager.archive.fs.dir: hdfs://baby:9000/completed-jobs/
historyserver.archive.fs.dir: hdfs://baby:9000/completed-jobs/

vim $FLINK_HOME/conf/zoo.cfg

修改zookeeper配置成你的本地配置

创建sql-client配置文件

vim $FLINK_HOME/conf/sql-client-defaults.yaml

# Tables
#==============================================================================

# Define tables here such as sources, sinks, views, or temporal tables.

tables: [] # empty list
# A typical table source definition looks like:
# - name: ...
#   type: source-table
#   connector: ...
#   format: ...
#   schema: ...

# A typical view definition looks like:
# - name: ...
#   type: view
#   query: "SELECT ..."

# A typical temporal table definition looks like:
# - name: ...
#   type: temporal-table
#   history-table: ...
#   time-attribute: ...
#   primary-key: ...
#==============================================================================
# User-defined functions
#==============================================================================

# Define scalar, aggregate, or table functions here.