flink写hdfs文件-去重+分桶

Flink 按时间分文件写入HDFS案例
https://blog.csdn.net/qq_31866793/article/details/91038531

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-sequence-file</artifactId>
            <version>1.12-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.12-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-hadoop-compatibility_2.11</artifactId>
            <version>1.12-SNAPSHOT</version>
        </dependency>
// An highlighted block
import org.apache.flink.api.common.functions.RichFilterFunction;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.serialization.SimpleStringEncoder;
import org.apache.flink.api.common.state.StateTtlConfig;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.GlobalConfiguration;
import org.apache.flink.core.fs.Path;
import org.apache.flink.core.io.SimpleVersionedSerializer;
import org.apache.flink.formats.sequencefile.SequenceFileWriterFactory;
import org.apache.flink.runtime.util.HadoopUtils;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig;
import org.apache.flink.streaming.api.functions
  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
要将文件HDFS并指定Gzip压缩格式,可以使用Flink提供的`org.apache.flink.core.fs.FileSystem`和`org.apache.flink.core.fs.Path`类来实现。具体步骤如下: 1. 创建一个`org.apache.flink.core.fs.FileSystem`对象,指定HDFS的URI和配置信息。 2. 创建一个`org.apache.flink.core.fs.Path`对象,指定HDFS文件路径。 3. 调用`FileSystem.create()`方法创建一个输出流。 4. 将数据入输出流,这里可以使用`org.apache.flink.api.common.io.FileOutputFormat`类来实现Gzip压缩。 5. 关闭输出流。 下面是一个示例程序,它将数据HDFS并使用Gzip压缩: ```java import org.apache.flink.api.common.io.FileOutputFormat; import org.apache.flink.core.fs.FileSystem; import org.apache.flink.core.fs.Path; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; public class WriteToHdfsExample { public static void main(String[] args) throws Exception { // 创建执行环境 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // 准备要HDFS的数据 DataStream<String> data = ... // HDFS String outputPath = "hdfs://hadoop-master:9000/path/to/output"; FileSystem hdfs = FileSystem.get(new URI(outputPath), new Configuration()); Path path = new Path(outputPath); FSDataOutputStream outputStream = hdfs.create(path); GzipCompressor gzipCompressor = new GzipCompressor(); FileOutputFormat<String> fileOutputFormat = new TextOutputFormat<>(path, gzipCompressor); fileOutputFormat.setOutputFilePath(path); fileOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE); fileOutputFormat.open(outputStream); data.writeUsingOutputFormat(fileOutputFormat); fileOutputFormat.close(); // 启动任务执行 env.execute("Write to HDFS Example"); } } ``` 在以上示例程序中,`hadoop-master:9000`是HDFS的URI,`/path/to/output`是要入的文件路径。`TextOutputFormat`是Flink提供的一个文本输出格式,它支持Gzip压缩。在`FileOutputFormat`的构造函数中,将`TextOutputFormat`作为参数传入,即可实现Gzip压缩。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值