准备工作
1、安装查看 Java 的版本号,推荐使用 Java 8。
2.安装 brew
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
上面方法失败的话请执行如下操作
在浏览器上进入上面的网站地址,将里面的内容复制到自己新建一个 “install.txt”里面,然后将该文件保存到任意目录,进入该目录下执行
ruby install.txt
安装 Flink
2、在 Mac OS X 上安装 Flink 是非常方便的。推荐通过 homebrew 来安装。
brew install apache-flink
3、检查安装:
kingdeMacBook-Pro:bin king$ flink --version
Version: 1.7.2, Commit ID: ceba8af
4、启动 flink
到该目录下执行/usr/local/Cellar/apache-flink/1.7.2/libexec/bin
./start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host kingdeMacBook-Pro.local.
Starting taskexecutor daemon on host kingdeMacBook-Pro.local.
5、web访问
http://localhost:8081
demo
1、新建一个 maven 项目
创建一个 SocketTextStreamWordCount 文件,加入以下代码:
package com.king.learn.Flink.streaming.wordcount
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector
/**
* @Author: king
* @Date: 2019-02-28
* @Desc: TODO
*/
object SocketTextStreamWordCount {
def main(args: Array[String]): Unit = {
//参数检查
if (args.length != 2) {
System.err.println("USAGE:\nSocketTextStreamWordCount <hostname> <port>")
}
val hostname = args(0)
val port = args(1).toInt
// get the execution environment
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
// get input data by connecting to the socket
val stream: DataStream[String] = env.socketTextStream(hostname, port, '\n')
val sum = stream.flatMap(new SocketTextStreamWordCount.LineSplitter).keyBy(0).sum(1)
sum.print
env.execute("Socket Window WordCount")
}
import org.apache.flink.api.common.functions.FlatMapFunction
final class LineSplitter extends FlatMapFunction[String, (String, Integer)] {
def flatMap(s: String, collector: Collector[(String, Integer)]): Unit = {
val tokens = s.toLowerCase.split("\\W+")
for (token <- tokens) {
if (token.length > 0) collector.collect(new Tuple2[String, Integer](token, 1))
}
}
}
}
maven 打包
注意我这是scala文件需要maven scala 打包插件
<build>
<!--scala待编译的文件目录-->
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<!--scala插件-->
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<!--<arg>-make:transitive</arg>--><!--scala2.11 netbean不支持这个参数-->
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<!-- 这是个编译java代码的 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<source>6</source>
<target>6</target>
<encoding>UTF-8</encoding>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- 这是个编译scala代码的 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!--manven打包插件-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>cn.itcast.rpc.Master</mainClass> <!--main方法-->
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
开启监听 9000 端口:
nc -l 9000
进入 flink 安装目录 bin 下执行以下命令跑程序:
./flink run -c com.king.learn.Flink.streaming.wordcount.SocketTextStreamWordCount /Volumes/mydisk/Projects/IdeaProjects/flink-learn/target/original-flink-learn-1.0-SNAPSHOT.jar 127.0.0.1 9000
注意自己的文件路径
我们可以在 webUI 中看到正在运行的程序
进入/usr/local/Cellar/apache-flink/1.7.2/libexec/log
tail -f flink-king-taskexecutor-0-kingdeMacBook-Pro.local.out