文章目录
1. FLINK单机版安装部署
(1) 下载flink部署包并解压
# 下载Flink安装包(笔者这里测试使用Flink12版本,其它版本下载目录 https://flink.apache.org/downloads.html)
wget https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz
# 解压
tar -xzvf flink-1.12.0-bin-scala_2.11.tgz
解压后目录结构:
(2) 进入conf目录下 flink-conf.yaml 修改rest.port
(8081端口过于常用,容易出现端口冲突,笔者这里改为8778)
(3) 启动集群
,进入bin目录执行下面命令
# 启动Flink集群
./start-cluster.sh
# 停止Flink集群
#./stop-cluster.sh
(4) 访问Flink UI
,链接http://host:8778/#/overview,查看dashboard (这里的host为部署flink的机器host)
2. 提交任务
2.1 在flink ui上提交任务
Submit New Job --> +Add New(上传用户jar包)–> 单击xxx.jar填写入口类等 --> 点击Submit
2.2 通过命令行提交Flink任务
./bin/flink run -d -m host:8778 --class xx.xx.xx.Main xx/xx/xx/test.jar
3. 安装部署常见问题
3.1 jdk版本问题
[ERROR] Raw output from BashJavaUtils:
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/flink/runtime/util/bash/BashJavaUtils : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:472)
Flink需要java 1.8版本, 笔者系统环境为1.7,若不希望改动系统环境jdk版本,可在flink bin目录下config.sh添加JAVA_HOME=xx/xx/xx
3.2 flink rest端口问题
Caused by: java.net.BindException: Could not start rest endpoint on any port in port range 8081
at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:222) ~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherRes
ourceManagerComponentFactory.java:162) ~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:220) ~[flink-dist_2.11-1.12.0.jar:
1.12.0]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:173) ~[flink-dist_2.11-
1.12.0.jar:1.12.0]
flink默认8081端口已被占用,服务无法启动,可修改rest.port,见安装部署步骤(2)
3.3 Flink与hadoop冲突问题
提交hadoop相关任务时,hadoop的jar包已经在classpath下,但仍报类找不到,将Flink Shaded Hadoop 2 Uber jar包放入flink lib目录下,重启flink服务即可。