Flink安装与编程实践

系列文章目录

Ubuntu常见基本问题
Hadoop3.1.3安装（单机、伪分布）
Hadoop集群搭建
HBase2.2.2安装（单机、伪分布）
Zookeeper集群搭建
HBase集群搭建
Spark安装和编程实践（Spark2.4.0）
Spark集群搭建

一、安装Flink

1. 先把压缩格式的文件flink-1.9.1-bin-scala_2.11.tgz下载到本地电脑，然后保存在“下载”中
2. 解压安装包flink-1.9.1-bin-scala_2.11.tgz至路径 /usr/local，命令如下

sudo tar -zxvf ~/下载/flink-1.9.1-bin-scala_2.11.tgz -C /usr/local/
cd /usr/local
sudo mv ./flink-1.9.1 ./flink
sudo chown -R hadoop:hadoop ./flink

1、修改环境变量

使用如下命令添加环境变量：

vim ~/.bashrc

增加如下内容：

export FLINK_HOME=/usr/local/flink
export PATH=$FLINK_HOME/bin:$PATH

保存并退出.bashrc文件，然后执行如下命令让配置文件生效：

source ~/.bashrc

2、启动

cd /usr/local/flink
./bin/start-cluster.sh

使用jps命令查看进程，成功啦！！！

3、测试

如果能够看到TaskManagerRunner和StandaloneSessionClusterEntrypoint这两个进程，就说明启动成功。
Flink的JobManager同时会在8081端口上启动一个Web前端，可以在浏览器中输入“http://localhost:8081”来访问。
Flink安装包中自带了测试样例，这里可以运行WordCount样例程序来测试Flink的运行效果，具体命令如下：

cd /usr/local/flink/bin
./flink run /usr/local/flink/examples/batch/WordCount.jar

结果如下：

二、编程实现WordCount程序

1、安装Maven

1. 先把压缩格式的文件apache-maven-3.6.3-bin.zip下载到本地电脑，然后保存在“下载”中
2. 解压安装包apache-maven-3.6.3-bin.zip至路径 /usr/local，命令如下

sudo unzip ~/下载/apache-maven-3.6.3-bin.zip -d /usr/local
cd /usr/local
sudo mv apache-maven-3.6.3/ ./maven
sudo chown -R hadoop ./maven

2、编写代码

在用户主文件夹下创建一个文件夹flinkapp作为应用程序根目录：

cd ~ #进入用户主文件夹
mkdir -p ./flinkapp/src/main/java

① WordCountData.java

sudo vim ./flinkapp/src/main/java/WordCountData.java

修改为：

package cn.edu.xmu;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
 
public class WordCountData {
    public static final String[] WORDS=new String[]{"To be, or not to be,--that is the question:--", "Whether \'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune", "Or to take arms against a sea of troubles,", "And by opposing end them?--To die,--to sleep,--", "No more; and by a sleep to say we end", "The heartache, and the thousand natural shocks", "That flesh is heir to,--\'tis a consummation", "Devoutly to be wish\'d. To die,--to sleep;--", "To sleep! perchance to dream:--ay, there\'s the rub;", "For in that sleep of death what dreams may come,", "When we have shuffled off this mortal coil,", "Must give us pause: there\'s the respect", "That makes calamity of so long life;", "For who would bear the whips and scorns of time,", "The oppressor\'s wrong, the proud man\'s contumely,", "The pangs of despis\'d love, the law\'s delay,", "The insolence of office, and the spurns", "That patient merit of the unworthy takes,", "When he himself might his quietus make", "With a bare bodkin? who would these fardels bear,", "To grunt and sweat under a weary life,", "But that the dread of something after death,--", "The undiscover\'d country, from whose bourn", "No traveller returns,--puzzles the will,", "And makes us rather bear those ills we have", "Than fly to others that we know not of?", "Thus conscience does make cowards of us all;", "And thus the native hue of resolution", "Is sicklied o\'er with the pale cast of thought;", "And enterprises of great pith and moment,", "With this regard, their currents turn awry,", "And lose the name of action.--Soft you now!", "The fair Ophelia!--Nymph, in thy orisons", "Be all my sins remember\'d."};
    public WordCountData() {
    }
    public static DataSet<String> getDefaultTextLineDataset(ExecutionEnvironment env){
        return env.fromElements(WORDS);
    }
}

② WordCountTokenizer.java

sudo vim ./flinkapp/src/main/java/WordCountTokenizer.java

修改为：

package cn.edu.xmu;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;
 
 
public class WordCountTokenizer implements FlatMapFunction<String, Tuple2<String,Integer>>{
 
    public WordCountTokenizer(){}
 
 
    public void flatMap(String value, Collector<Tuple2<String, Integer>> out) throws Exception {
        String[] tokens = value.toLowerCase().split("\\W+");
        int len = tokens.length;
 
        for(int i = 0; i<len;i++){
            String tmp = tokens[i];
            if(tmp.length()>0){
                out.collect(new Tuple2<String, Integer>(tmp,Integer.valueOf(1)));
            }
        }
    }
}

③ WordCount.java

sudo vim ./flinkapp/src/main/java/WordCount.java

修改为：

package cn.edu.xmu;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.operators.AggregateOperator;
import org.apache.flink.api.java.utils.ParameterTool;
 
 
public class WordCount {
 
    public WordCount(){}
 
    public static void main(String[] args) throws Exception {
        ParameterTool params = ParameterTool.fromArgs(args);
        ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
        env.getConfig().setGlobalJobParameters(params);
        Object text;
        //如果没有指定输入路径，则默认使用WordCountData中提供的数据
        if(params.has("input")){
            text = env.readTextFile(params.get("input"));
        }else{
            System.out.println("Executing WordCount example with default input data set.");
            System.out.println("Use -- input to specify file input.");
            text = WordCountData.getDefaultTextLineDataset(env);
        }
 
        AggregateOperator counts = ((DataSet)text).flatMap(new WordCountTokenizer()).groupBy(new int[]{0}).sum(1);
        //如果没有指定输出，则默认打印到控制台
        if(params.has("output")){
            counts.writeAsCsv(params.get("output"),"\n", " ");
            env.execute();
        }else{
            System.out.println("Printing result to stdout. Use --output to specify output path.");
            counts.print();
        }
 
    }
}

④ pom.xml

cd ~/flinkapp
vim pom.xml

修改为：

<project>
    <groupId>cn.edu.xmu</groupId>
    <artifactId>simple-project</artifactId>
    <modelVersion>4.0.0</modelVersion>
    <name>Simple Project</name>
    <packaging>jar</packaging>
    <version>1.0</version>
    <repositories>
        <repository>
            <id>jboss</id>
            <name>JBoss Repository</name>
            <url>http://repository.jboss.com/maven2/</url>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.9.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.11</artifactId>
<version>1.9.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.11</artifactId>
            <version>1.9.1</version>
        </dependency>
    </dependencies>
</project>

3、使用Maven打包Java程序

为了保证Maven能够正常运行，先执行如下命令检查整个应用程序的文件结构：

cd ~/flinkapp
find .

文件结构应该是类似如下的内容：

.
./pom.xml
./src
./src/main
./src/main/java
./src/main/java/WordCount.java
./src/main/java/WordCountTokenizer.java
./src/main/java/WordCountData.java

然后进行打包：

cd ~/flinkapp    #一定把这个目录设置为当前目录
/usr/local/maven/bin/mvn package

成功啦！！！

如果打包很慢，可以进行更换为国内源：Ubuntu常见基本问题

4、通过flink run命令运行程序

最后，可以将生成的JAR包通过flink run命令提交到Flink中运行（请确认已经启动Flink），命令如下：

/usr/local/flink/bin/flink run --class cn.edu.xmu.WordCount ~/flinkapp/target/simple-project-1.0.jar

成功啦！！！