Linux环境下安装FLink1.10.0并启动SQL-client读取Hive1.2.1的数据

Linux环境下安装FLink1.10.0并启动SQL-client读取Hive数据

首先去官网下载Flink1.10.0的tgz的包，教程如上篇文章上半部分流程一样，然后配置一下FLINK_HOME/conf/sql-client-defaults.yaml：

catalogs:
   - name: myhive   #自己定个名字就行
     type: hive
     hive-conf-dir: /etc/hive/conf  # hive-site.xml的路径
     hive-version: 1.2.1    # hive版本
 execution:
  # select the implementation responsible for planning table programs
  # possible values are 'blink' (used by default) or 'old'
  planner: blink
  # 'batch' or 'streaming' execution
  type: batch  #这里streaming和batch都行
  # allow 'event-time' or only 'processing-time' in sources
  time-characteristic: event-time
  # interval in ms for emitting periodic watermarks
  periodic-watermarks-interval: 200
  # 'changelog' or 'table' presentation of results
  result-mode: table
  # maximum number of maintained rows in 'table' presentation of results
  max-table-result-rows: 1000000
  # parallelism of the program
  parallelism: 1
  # maximum parallelism
  max-parallelism: 128
  # minimum idle state retention in ms
  min-idle-state-retention: 0
  # maximum idle state retention in ms
  max-idle-state-retention: 0
  # current catalog ('default_catalog' by default)
  current-catalog: myhive
  # current database of the current catalog (default database of the catalog by default)
  current-database: secoo_tmp
  # controls how table programs are restarted in case of a failures
  restart-strategy:
    # strategy type
    # possible values are "fixed-delay", "failure-rate", "none", or "fallback" (default)
    type: fallback

配置/etc/profile文件：

export HADOOP_HOME=/usr/hdp/2.4.0.0-169/hadoop
export YARN_CONF_DIR=/etc/hadoop/conf
export HADOOP_CLASSPATH=`hadoop classpath` #非常重要，不添加 运行flink命令时会报错

在FLink安装目录启动yarn-session.sh:

./bin/yarn-session.sh -n 5 -tm 4096 -s 4 -nm 应用名称 -q 队列名称 -d(这个参数可以保证在我们退出客户端时，任务不被立即杀死，还在yarn上持续运行着)

yarn-session的参数介绍
  -n ： 指定TaskManager的数量；
  -d: 以分离模式运行；
  -id：指定yarn的任务ID；
  -j:Flink jar文件的路径;
  -jm：JobManager容器的内存（默认值：MB）;
  -nl：为YARN应用程序指定YARN节点标签;
  -nm:在YARN上为应用程序设置自定义名称;
  -q:显示可用的YARN资源（内存，内核）;