日志收集系统：将应用产生的数据通过flume收集后发送到Kafka，整理后保存至hbase

最新推荐文章于 2024-01-19 13:43:37 发布
最新推荐文章于 2024-01-19 13:43:37 发布
阅读量928 收藏 1
点赞数
分类专栏： # HBase # Kafka 文章标签： hbase 大数据 kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/d905133872/article/details/130488053
版权

目录

前言：功能描述

第一步：flume拉取日志数据，并将源数据保存至Kafka

flume配置文件：

users：

user_friends_raw：

events：

train：

第二步：Kafka源数据处理

方式一：KafkaStream

EventAttendStream

UserFriendsStream

方式二：sparkStreaming

EventAttendeesRaw -> EventAttendees

UserFriendRaw -> UserFriend

 第三步：Kafka topic数据发送到HBase

EventAttendee -> HBase

Events -> HBase

Train -> HBase

UserFriends -> HBase

Users -> HBase

前言：功能描述

 源数据为日志数据，通过flume采集后发送到Kafka，对源数据进行处理后保存至hbase。

第一步：flume拉取日志数据，并将源数据保存至Kafka

flume配置文件：

users：

users.sources=usersSource
users.channels=usersChannel
users.sinks=userSink

users.sources.usersSource.type=spooldir
users.sources.usersSource.spoolDir=/opt/flumelogfile/users
users.sources.usersSource.deserializer=LINE
users.sources.usersSource.deserializer.maxLineLength=320000
users.sources.usersSource.includePattern=user_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
users.sources.usersSource.interceptors=head_filter
users.sources.usersSource.interceptors.head_filter.type=regex_filter
users.sources.usersSource.interceptors.head_filter.regex=^user_id*
users.sources.usersSource.interceptors.head_filter.excludeEvents=true

users.channels.usersChannel.type=file
users.channels.usersChannel.checkpointDir=/opt/flumelogfile/checkpoint/users
users.channels.usersChannel.dataDirs=/opt/flumelogfile/data/users

users.sinks.userSink.type=org.apache.flume.sink.kafka.KafkaSink
users.sinks.userSink.batchSize=640
users.sinks.userSink.brokerList=192.168.136.20:9092
users.sinks.userSink.topic=users

users.sources.usersSource.channels=usersChannel
users.sinks.userSink.channel=usersChannel

user_friends_raw：

userfriends.sources=userfriendsSource
userfriends.channels=userfriendsChannel
userfriends.sinks=userfriendsSink

userfriends.sources.userfriendsSource.type=spooldir
userfriends.sources.userfriendsSource.spoolDir=/opt/flumelogfile/uf
userfriends.sources.userfriendsSource.deserializer=LINE
userfriends.sources.userfriendsSource.deserializer.maxLineLength=320000
userfriends.sources.userfriendsSource.includePattern=uf_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
userfriends.sources.userfriendsSource.interceptors=head_filter
userfriends.sources.userfriendsSource.interceptors.head_filter.type=regex_filter
userfriends.sources.userfriendsSource.interceptors.head_filter.regex=^user*
userfriends.sources.userfriendsSource.interceptors.head_filter.excludeEvents=true

userfriends.channels.userfriendsChannel.type=file
userfriends.channels.userfriendsChannel.checkpointDir=/opt/flumelogfile/checkpoint/uf
userfriends.channels.userfriendsChannel.dataDirs=/opt/flumelogfile/data/uf

userfriends.sinks.userfriendsSink.type=org.apache.flume.sink.kafka.KafkaSink
userfriends.sinks.userfriendsSink.batchSize=640
userfriends.sinks.userfriendsSink.brokerList=192.168.136.20:9092
userfriends.sinks.userfriendsSink.topic=user_friends_raw

userfriends.sources.userfriendsSource.channels=userfriendsChannel
userfriends.sinks.userfriendsSink.channel=userfriendsChannel

events：

events.sources=eventsSource
events.channels=eventsChannel
events.sinks=eventsSink

events.sources.eventsSource.type=spooldir
events.sources.eventsSource.spoolDir=/opt/flumelogfile/events
events.sources.eventsSource.deserializer=LINE
events.sources.eventsSource.deserializer.maxLineLength=320000
events.sources.eventsSource.includePattern=events_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
events.sources.eventsSource.interceptors=head_filter
events.sources.eventsSource.interceptors.head_filter.type=regex_filter
events.sources.eventsSource.interceptors.head_filter.regex=^event_id*
events.sources.eventsSource.interceptors.head_filter.excludeEvents=true

events.channels.eventsChannel.type=file
events.channels.eventsChannel.checkpointDir=/opt/flumelogfile/checkpoint/events
events.channels.eventsChannel.dataDirs=/opt/flumelogfile/data/events

events.sinks.eventsSink.type=org.apache.flume.sink.kafka.KafkaSink
events.sinks.eventsSink.batchSize=640
events.sinks.eventsSink.brokerList=192.168.136.20:9092
events.sinks.eventsSink.topic=events

events.sources.eventsSource.channels=eventsChannel
events.sinks.eventsSink.channel=eventsChannel

event_attendees_raw：

ea.sources=eaSource
ea.channels=eaChannel
ea.sinks=eaSink

ea.sources.eaSource.type=spooldir
ea.sources.eaSource.spoolDir=/opt/flumelogfile/ea
ea.sources.eaSource.deserializer=LINE
ea.sources.eaSource.deserializer.maxLineLength=320000
ea.sources.eaSource.includePattern=ea_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
ea.sources.eaSource.interceptors=head_filter
ea.sources.eaSource.interceptors.head_filter.type=regex_filter
ea.sources.eaSource.interceptors.head_filter.regex=^event*
ea.sources.eaSource.interceptors.head_filter.excludeEvents=true

ea.channels.eaChannel.type=file
ea.channels.eaChannel.checkpointDir=/opt/flumelogfile/checkpoint/ea
ea.channels.eaChannel.dataDirs=/opt/flumelogfile/data/ea

ea.sinks.eaSink.type=org.apache.flume.sink.kafka.KafkaSink
ea.sinks.eaSink.batchSize=640
ea.sinks.eaSink.brokerList=192.168.136.20:9092
ea.sinks.eaSink.topic=event_attendees_raw

ea.sources.eaSource.channels=eaChannel
ea.sinks.eaSink.channel=eaChannel

train：

train.sources=trainSource
train.channels=trainChannel
train.sinks=trainink

train.sources.trainSource.type=spooldir
train.sources.trainSource.spoolDir=/opt/flumelogfile/train
train.sources.trainSource.deserializer=LINE
train.sources.trainSource.deserializer.maxLineLength=320000
train.sources.trainSource.includePattern=train_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
train.sources.trainSource.interceptor=head_filter
train.sources.trainSource.interceptor.head_filter.type=regex_filter
train.sources.trainSource.interceptor.head_filter.regex=^user*
train.sources.trainSource.interceptor.head_filter.excludeEvents=true

train.channels.trainChannel.type=file
train.channels.trainChannel.checkpointDir=/opt/flumelogfile/checkpoint/train
train.channels.trainChannel.dataDirs=/opt/flumelogfile/data/train

train.sinks.trainink.type=org.apache.flume.sink.kafka.KafkaSink
train.sinks.trainink.batchSize=640
train.sinks.trainink.brokerList=192.168.136.20:9092
train.sinks.trainink.topic=train

train.sources.trainSource.channels=trainChannel
train.sinks.trainink.channel=trainChannel

 

第二步：Kafka源数据处理

user_friends和event_attendees需要对元数据进行处理，在这里我们有两种方式。一种是KafkaStream将数据消费后发送到新的topic。

方式一：KafkaStream

第一步：初始化配置

Properties properties = new Properties();

    // 组id
properties.put(StreamsConfig.APPLICATION_ID_CONFIG, "userfriend");    
    // 设备id
properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
    // key反序列化
properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
    // value反序列化
properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

第二步：创建流构造器

StreamsBuilder builder = new StreamsBuilder();

第三步：从topic中取数据，放入到新的topic中

builder.stream("event_attendees_raw").to("event_attendees");

上面一行相当于下面两行
KStream<Object, Object> mystreamin = builder.stream("event_attendees_raw");
mystreamin.to("event_attendees");

第四步：通过建造者模式，创建kafkaStreams

Topology topo = builder.build();
 
KafkaStreams kafkaStreams = new KafkaStreams(topo, properties);

第五步：启动程序

 kafkaStreams.start();

EventAttendStream

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.*;

import java.util.ArrayList;
import java.util.Properties;

public class EventAttendStream {
    public static void main(String[] args) {

// 1. 初始化配置
        Properties properties = new Properties();
        properties.put(StreamsConfig.APPLICATION_ID_CONFIG, "userfriend");
        properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

// 2. 创建流构造器
        StreamsBuilder builder = new StreamsBuilder();

// 3. 将topic event_attendees_raw的数据取出，处理后发送到topic event_attendees
        builder.stream("event_attendees_raw")
                .flatMap((key,value)->{
                            ArrayList<KeyValue<String, String>> list = new ArrayList<>();
                            String[] fields = value.toString().split(",");
                            String event = fields[0];
                            if(event.trim().length()>0){
                                if (fields.length>=2){ //判断yes的位置是否有值
                                    String[] yes = fields[1].split("\\s+");
                                    for (String y : yes) {
                                        System.out.println(event+" "+y+" yes");
                                        KeyValue<String, String> kv = new KeyValue<>(null, event + "," + y+" yes");
                                        list.add(kv);
                                    }
                                }
                                if (fields.length>=3){ //判断maybe的位置是否有值
                                    String[] maybe = fields[2].split("\\s+");
                                    for (String my : maybe) {
                                        System.out.println(event+" "+my+" maybe");
                                        KeyValue<String, String> kv = new KeyValue<>(null, event + "," + my+" maybe");
                                        list.add(kv);
                                    }
                                }
                                if (fields.length>=4){ //判断invited的位置是否有值
                                    String[] invited  = fields[3].split("\\s+");
                                    for (String in : invited) {
                                        System.out.println(event+" "+in+" invited");
                                        KeyValue<String, String> kv = new KeyValue<>(null, event + "," + in+" invited");
                                        list.add(kv);
                                    }
                                }
                                if (fields.length>=5){  //判断no的位置是否有值
                                    String[] no = fields[4].split("\\s+");
                                    for (String n : no) {
                                        System.out.println(event+" "+n+" no");
                                        KeyValue<String, String> kv = new KeyValue<>(null, event + "," + n+" no");
                                        list.add(kv);
                                    }
                                }
                            }
                            return list;
                        })
                .to("event_attendees");

// 4. kafkaStream需要传递两个参数，Topology topology, Properties props
// props配置通过new Properties()实现
// Topology则通过builder.build()创建

        Topology topo = builder.build();
        KafkaStreams streams = new KafkaStreams(topo, properties);

// 5. 启动程序
        streams.start();
    }
}

 

UserFriendsStream

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.*;

import java.util.ArrayList;
import java.util.Properties;


public class UserFriendsStream {
    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(StreamsConfig.APPLICATION_ID_CONFIG,"userfriend");
        properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.136.20:9092");
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,"false");

        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
        properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,Serdes.String().getClass());

        StreamsBuilder builder = new StreamsBuilder();

        // 主要业务逻辑处理 start
        builder.stream("user_friends_raw")
                .flatMap((key,value)->{
                    ArrayList<KeyValue<String,String>> list = new ArrayList<>();
                    String[] fields = value.toString().split(",");
                    if(fields.length==2){
                        String userid = fields[0];
                        String[] friends = fields[1].split("\\s+");
                        for (String friendid :
                                friends) {
                            System.out.println(userid + " " + friendid);
                            KeyValue kv = new KeyValue<String, String>(null, userid + "," + friendid);
                            list.add(kv);
                        }
                    }

                    return list;
                })
                .to("user_friends");
        // 主要业务逻辑处理 end

        Topology topo = builder.build();
        KafkaStreams streams = new KafkaStreams(topo, properties);

        streams.start();
    }
}

方式二：sparkStreaming

第一步：配置sparkStream

val conf = new SparkConf().setAppName("user_friends_raw").setMaster("local[*]")
val streamingContext = new StreamingContext(conf,Seconds(5))

第二步：kafka配置

val kafkaParams = Map(
    (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> "192.168.136.20:9092"),
    (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
    (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
    (ConsumerConfig.GROUP_ID_CONFIG -> "ea01"),
)

第三步：KafkaStream创建流

val kafkaStream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
    streamingContext,
    LocationStrategies.PreferConsistent,
    ConsumerStrategies.Subscribe(Set("user_friends_raw"), kafkaParams)
)

第四步：对RDD进行处理

kafkaStream.foreachRDD(
    rdd=>{
        rdd.foreachPartition(x=>x处理)
    }
)

第五步：创建Kafka producer并发送

val props = new util.HashMap[String,Object]()
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.136.20:9092")
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer")
 props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer")
val producer = new KafkaProducer[String,String](props)

val record = new ProducerRecord[String,String]("user_friends2",userid+","+friend)
producer.send(record)

第六步：启动采集器

streamingContext.start()
streamingContext.awaitTermination()

EventAttendeesRaw -> EventAttendees

import java.util
import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord}
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.dstream.InputDStream
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}

object SparkStreamEventAttendeesrawToEventAttends {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("user_friends_raw").setMaster("local[*]")
    val streamingContext = new StreamingContext(conf,Seconds(5))

    streamingContext.checkpoint("checkpoint")

    val kafkaParams = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> "192.168.136.20:9092"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG -> "ea01"),
      (ConsumerConfig.AUTO_OFFSET_RESET_CONFIG->"earliest")
    )
    val kafkaStream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      streamingContext,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("event_attendees_raw"), kafkaParams)
    )

    kafkaStream.foreachRDD(rdd=>{
      rdd.foreachPartition(x=>{
        val props = new util.HashMap[String,Object]()
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.136.20:9092")
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer")
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer")
        val producer = new KafkaProducer[String,String](props)

        x.foreach(y=> { // event,yes,maybe,invited,no
          val fields = y.value().split(",")
          if(fields.length>=0) {
            val events = fields(0)
            if (fields.length >= 2) {
              val yess = fields(1).split("\\s+")
              for (yes <- yess) {
                val record = new ProducerRecord[String, String]("event_attendees2", events + "," + yes)
                producer.send(record)
              }
            }
            if (fields.length >= 3) {
              val maybes = fields(2).split("\\s+")
              for (maybe <- maybes) {
                val record = new ProducerRecord[String, String]("event_attendees2", events + "," + maybe)
                producer.send(record)
              }
            }
            if (fields.length >= 4) {
              val inviteds = fields(3).split("\\s+")
              for (invited <- inviteds) {
                val record = new ProducerRecord[String, String]("event_attendees2", events + "," + invited)
                producer.send(record)
              }
            }
            if (fields.length >= 5) {
              val nos = fields(4).split("\\s+")
              for (no <- nos) {
                val record = new ProducerRecord[String, String]("event_attendees2", events + "," + no)
                producer.send(record)
              }
            }
          }
          })
      })
    }
    )

// 启动采集器
      streamingContext.start()
      streamingContext.awaitTermination()

  }

}

UserFriendRaw -> UserFriend

import java.util

import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord}
import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.InputDStream
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}
import org.apache.spark.streaming.{Seconds, StreamingContext}

object SparkStreamUserFriendrawToUserFriend {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("user_friends_raw").setMaster("local[*]")
    val streamingContext = new StreamingContext(conf,Seconds(5))

    streamingContext.checkpoint("checkpoint")

    val kafkaParams = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> "192.168.136.20:9092"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG -> "uf"),
      (ConsumerConfig.AUTO_OFFSET_RESET_CONFIG->"earliest")
    )
    val kafkaStream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      streamingContext,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("user_friends_raw"), kafkaParams)
    )

    kafkaStream.foreachRDD(
      rdd=>{
        rdd.foreachPartition(x=>{
          val props = new util.HashMap[String,Object]()
          props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.136.20:9092")
          props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer")
          props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer")
          val producer = new KafkaProducer[String,String](props)
          x.foreach(y=>{
            val splits = y.value().split(",")
            if(splits.length==2){
              val userid = splits(0)
              val friends = splits(1).split("\\s+")
              for(friend<-friends){
                val record = new ProducerRecord[String,String]("user_friends2",userid+","+friend)
                producer.send(record)
              }

            }
          })
        })
      }
    )


    streamingContext.start()
    streamingContext.awaitTermination()
  }

}

 

 第三步：Kafka topic数据发送到HBase

第一步：初始化配置

Properties properties = new Properties();

properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "event_attendee");

第二步：配置Kafka消费者，消费数据

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
consumer.subscribe(Collections.singleton("event_attendees"));

第三步：配置hbase信息，连接hbase数据库

Configuration conf = HBaseConfiguration.create();

conf.set(HConstants.HBASE_DIR,"hdfs://192.168.136.20:9000/hbase");
conf.set(HConstants.ZOOKEEPER_QUORUM,"192.168.136.20");
conf.set(HConstants.CLIENT_PORT_STR,"2181");

Connection connection = ConnectionFactory.createConnection(conf);
Table eventAttendTable = connection.getTable(TableName.valueOf("events_db:event_attendee"));

第四步：consumer消费数据

ConsumerRecords<String, String> poll = consumer.poll(Duration.ofMillis(100));

第五步：数据拉取出来之后插入到hbase中

ArrayList<Put> datas = new ArrayList<>();

for (ConsumerRecord<String, String> record : poll) {

// 将Kafka中的数据按照","分隔，
    String[] split = record.value().split(",");

    // rowkey
    Put put = new Put(Bytes.toBytes(split[0]));
    // 列族
    put.addColumn("eu".getBytes(), "user".getBytes(), split[0].getBytes());

    // put作为一条数据，插入到datas中
    datas.add(put);
}

EventAttendee -> HBase

package nj.zb.kb21.kafka.kafkatohb;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Properties;

public class EventAttendeToHB {

    static int num = 0; // 计数器

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");  // 手动提交
        properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");   // 自动提交时，提交时间
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "event_attendee");
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // 配置Kafka信息。消费消息
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("event_attendees"));

        // 配置hbase信息，连接hbase数据库
        Configuration conf = HBaseConfiguration.create();
        conf.set(HConstants.HBASE_DIR,"hdfs://192.168.136.20:9000/hbase");
        conf.set(HConstants.ZOOKEEPER_QUORUM,"192.168.136.20");
        conf.set(HConstants.CLIENT_PORT_STR,"2181");

        Connection connection = null;
        try {
            connection = ConnectionFactory.createConnection(conf);
            Table eventAttendTable = connection.getTable(TableName.valueOf("events_db:event_attendee"));

            while (true) {
                ConsumerRecords<String, String> poll = consumer.poll(Duration.ofMillis(100));
                ArrayList<Put> datas = new ArrayList<>();
                for (ConsumerRecord<String, String> record : poll) {
                    System.out.println(record.value()); // userid,friendid
                    String[] eventattend = record.value().split(",");
                    Put put = new Put(Bytes.toBytes(eventattend[0]+eventattend[1].split("\\s")[0]+eventattend[1].split("\\s")[1]));     // rowkey
                    put.addColumn("euat".getBytes(), "eventid".getBytes(), Bytes.toBytes(eventattend[0]));
                    put.addColumn("euat".getBytes(), "friendid".getBytes(), Bytes.toBytes(eventattend[1].split("\\s")[0]));
                    put.addColumn("euat".getBytes(), "state".getBytes(), Bytes.toBytes(eventattend[1].split("\\s")[1]));

                    datas.add(put);
                }
                num = num + datas.size();
                System.out.println("----------num:" + num);
                if (datas.size() != 0)
                    eventAttendTable.put(datas);

                Thread.sleep(10);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

}

Events -> HBase

package nj.zb.kb21.kafka.kafkatohb;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Properties;

public class EventsToHB {
    static int num = 0; // 计数器

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");  // 手动提交
        properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");   // 自动提交时，提交时间
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "events");
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // 配置Kafka信息。消费消息
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("events"));

        // 配置hbase信息，连接hbase数据库
        Configuration conf = HBaseConfiguration.create();
        conf.set(HConstants.HBASE_DIR,"hdfs://192.168.136.20:9000/hbase");
        conf.set(HConstants.ZOOKEEPER_QUORUM,"192.168.136.20");
        conf.set(HConstants.CLIENT_PORT_STR,"2181");

        Connection connection = null;
        try {
            connection = ConnectionFactory.createConnection(conf);
            Table userFriendTable = connection.getTable(TableName.valueOf("events_db:events"));

            while (true){
                ConsumerRecords<String,String> poll = consumer.poll(Duration.ofMillis(100));
                ArrayList<Put> datas = new ArrayList<>();
                for (ConsumerRecord<String, String> record : poll){
                    System.out.println(record.value()); // userid,friendid
                    String[] event = record.value().split(",");
                    Put put = new Put(Bytes.toBytes(event[0]));     // row_key
                    put.addColumn("creator".getBytes(),"userid".getBytes(),event[1].getBytes());
                    put.addColumn("schedule".getBytes(),"starttime".getBytes(),event[2].getBytes());
                    put.addColumn("location".getBytes(),"city".getBytes(),event[3].getBytes());
                    put.addColumn("location".getBytes(),"state".getBytes(),event[4].getBytes());
                    put.addColumn("location".getBytes(),"zip".getBytes(),event[5].getBytes());
                    put.addColumn("location".getBytes(),"country".getBytes(),event[6].getBytes());
                    put.addColumn("location".getBytes(),"lat".getBytes(),event[7].getBytes());
                    put.addColumn("location".getBytes(),"lng".getBytes(),event[8].getBytes());
                    put.addColumn("remark".getBytes(),"commonwords".getBytes(),event[9].getBytes());

                    datas.add(put);
                }
                num = num + datas.size();
                System.out.println("----------num:"+num);
                if (datas.size()!=0)
                    userFriendTable.put(datas);

                Thread.sleep(10);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

Train -> HBase

package nj.zb.kb21.kafka.kafkatohb;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Properties;

public class TrainToHB {
    static int num = 0; // 计数器

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");  // 手动提交
        properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");   // 自动提交时，提交时间
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "train");
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // 配置Kafka信息。消费消息
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("train"));

        // 配置hbase信息，连接hbase数据库
        Configuration conf = HBaseConfiguration.create();
        conf.set(HConstants.HBASE_DIR,"hdfs://192.168.136.20:9000/hbase");
        conf.set(HConstants.ZOOKEEPER_QUORUM,"192.168.136.20");
        conf.set(HConstants.CLIENT_PORT_STR,"2181");

        Connection connection = null;
        try {
            connection = ConnectionFactory.createConnection(conf);
            Table usersTable = connection.getTable(TableName.valueOf("events_db:train"));

            while (true) {
                ConsumerRecords<String, String> poll = consumer.poll(Duration.ofMillis(100));
                ArrayList<Put> datas = new ArrayList<>();
                for (ConsumerRecord<String, String> record : poll) {
                    System.out.println(record.value()); // userid,friendid
                    String[] split = record.value().split(",");
                    Put put = new Put(Bytes.toBytes(split[0]));     // rowkey
                    put.addColumn("eu".getBytes(), "user".getBytes(), split[0].getBytes());
                    put.addColumn("eu".getBytes(), "event".getBytes(), split[1].getBytes());
                    put.addColumn("eu".getBytes(), "invited".getBytes(), split[2].getBytes());
                    put.addColumn("eu".getBytes(), "timestamp".getBytes(), split[3].getBytes());
                    put.addColumn("eu".getBytes(), "interested".getBytes(), split[4].getBytes());
                    put.addColumn("eu".getBytes(), "not_interested".getBytes(), split[5].getBytes());

                    datas.add(put);
                }
                num = num + datas.size();
                System.out.println("----------num:" + num);
                if (datas.size() != 0)
                    usersTable.put(datas);

                Thread.sleep(10);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

UserFriends -> HBase

package nj.zb.kb21.kafka.kafkatohb;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.protocol.types.Field;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Properties;

public class UserFriendsToHB {

    static int num = 0; // 计数器

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");  // 手动提交
        properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");   // 自动提交时，提交时间
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "userfriend_group");
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // 配置Kafka信息。消费消息
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("user_friends"));

        // 配置hbase信息，连接hbase数据库
        Configuration conf = HBaseConfiguration.create();
        conf.set(HConstants.HBASE_DIR,"hdfs://192.168.136.20:9000/hbase");
        conf.set(HConstants.ZOOKEEPER_QUORUM,"192.168.136.20");
        conf.set(HConstants.CLIENT_PORT_STR,"2181");

        Connection connection = null;
        try {
            connection = ConnectionFactory.createConnection(conf);
            Table userFriendTable = connection.getTable(TableName.valueOf("events_db:user_friend"));

            while (true){
                ConsumerRecords<String,String> poll = consumer.poll(Duration.ofMillis(100));
                ArrayList<Put> datas = new ArrayList<>();
                for (ConsumerRecord<String, String> record : poll){
                    System.out.println(record.value()); // userid,friendid
                    String[] split = record.value().split(",");
                    Put put = new Put(Bytes.toBytes((split[0]+split[1]).hashCode()));   // rowkey
                    put.addColumn("uf".getBytes(),"userid".getBytes(),split[0].getBytes());
                    put.addColumn("uf".getBytes(),"friendid".getBytes(),split[1].getBytes());
                    datas.add(put);
                }
                num = num + datas.size();
                System.out.println("----------num:"+num);
                if (datas.size()!=0)
                    userFriendTable.put(datas);

                Thread.sleep(10);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

Users -> HBase

package nj.zb.kb21.kafka.kafkatohb;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Properties;

public class UsersToHB {
    static int num = 0; // 计数器

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.20:9092");
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");  // 手动提交
        properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");   // 自动提交时，提交时间
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "users");
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // 配置Kafka信息。消费消息
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("users"));

        // 配置hbase信息，连接hbase数据库
        Configuration conf = HBaseConfiguration.create();
        conf.set(HConstants.HBASE_DIR,"hdfs://192.168.136.20:9000/hbase");
        conf.set(HConstants.ZOOKEEPER_QUORUM,"192.168.136.20");
        conf.set(HConstants.CLIENT_PORT_STR,"2181");

        Connection connection = null;
        try {
            connection = ConnectionFactory.createConnection(conf);
            Table usersTable = connection.getTable(TableName.valueOf("events_db:users"));

            while (true) {
                ConsumerRecords<String, String> poll = consumer.poll(Duration.ofMillis(100));
                ArrayList<Put> datas = new ArrayList<>();
                for (ConsumerRecord<String, String> record : poll) {
                    System.out.println(record.value()); // userid,friendid
                    String[] user = record.value().split(",");
                    Put put = new Put(Bytes.toBytes(user[0]));  // rowkey
                    put.addColumn("profile".getBytes(), "birthyear".getBytes(), user[2].getBytes());
                    put.addColumn("profile".getBytes(), "gender".getBytes(), user[3].getBytes());
                    put.addColumn("region".getBytes(), "locale".getBytes(), user[1].getBytes());
                    if (user.length > 5)
                        put.addColumn("region".getBytes(), "location".getBytes(), user[5].getBytes());
                    if (user.length > 6)
                        put.addColumn("region".getBytes(), "timezone".getBytes(), user[6].getBytes());
                    if (user.length > 4)
                        put.addColumn("registration".getBytes(), "joinAt".getBytes(), user[4].getBytes());

                    datas.add(put);
                }
                num = num + datas.size();
                System.out.println("----------num:" + num);
                if (datas.size() != 0)
                    usersTable.put(datas);

                Thread.sleep(10);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

five小点心
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
flume将数据发送到kafka、hdfs、hive、http、netcat等模式的使用总结
hadoop菜鸟的博客
07-07 1739
1、source为http模式，sink为logger模式，将数据在控制台打印出来。conf配置文件如下：# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = http #该设置表示接收通过h...
Flume采集数据到Hive&Hbase
m0_73623240的博客
11-11 1077
Flum采集数据到Hive&Hbase
flume-ng 日志收集到hbase
wuguangxue255的专栏
09-15 1215
flume-ng  收集日志 1：设计架构图如下。 根据日志模块的量级，以及解析复杂性，决定是否设置loadbalance，实现负载均衡和failover 2：client-agent 设计 （1）使用tail -F 实时收集固定文件名的日志  （2）对于文件名会变化的日志文件使用log4jappender直接转为avro 3：server端设计 使用sinkgroup设置多个处
flume修改源码读日志到hbase
09-24
flume修改源码读日志到hbase，①日志文件为json数据②修改文件编译打包并替换flumejar中的对应文件
flume日志写入hbase问题记录
zh350229319的专栏
05-02 1642
flume日志写入hbase问题记录 标签（空格分隔）： flume flume使用kafkachannel，source为avro，应用通过logback写入 logback-flume-appender数据发送body内容异常 这个问题在单进程写日志的时候不会出现，在2个以上的进程时出现event的body数据异常，导致flume的自定义sink中无法解析数据 ...
SparkStreaming_HBase:将从Kafka收集过来的数据保存到HBase中
04-20
SparkStreaming_HBase将从Kafka收集过来的数据保存到HBase中数据来源：日志生成器。 编写一个python工程，用于产生行为日志，每运行一次，产生所设定的数量数据，使用Linux的定时器，每隔60s执行一次，行为日志保存...
本科毕业设计项目，基于spark streaming+flume+kafka+hbase的实时日志处理分析系统，大数据处理技术
06-20
本科毕业设计项目，基于spark streaming+flume+kafka+hbase的实时日志处理分析系统 基于spark streaming+flume+kafka+hbase的实时日志处理分析系统 本科毕业设计项目，基于spark streaming+flume+kafka+hbase的...
计算机课程毕设：基于spark+flume+kafka+hbase的实时日志处理分析系统.zip
09-16
综上所述，这个毕业设计项目通过整合Spark、Flume、Kafka和HBase等工具，构建了一个完整的实时日志处理和分析平台，涵盖了数据的采集、传输、处理和存储，体现了大数据技术在现代信息系统中的应用。学生在完成这个...
天气爬虫采集，kafka实时分发，flume 收集数据导入到 Hbase.zip
01-04
《天气爬虫采集，Kafka实时分发，Flume收集数据导入到HBase：构建大数据实时处理系统》 在当今大数据时代，实时数据处理成为关键的一环，本资料集围绕"天气爬虫采集，Kafka实时分发，Flume收集数据导入到HBase"的...
Flume+Kafka+HBase实例
01-11
4. 数据流转：Flume从源收集数据，将其发送到Kafka；Kafka作为中间层，接收并暂存数据，然后由HBase消费者读取并写入HBase。整个过程形成了一个完整的数据处理流水线。 在这个项目中，通过这样的组合，我们可以实现...
Flume 从端口获取数据，输出到多端（Kafka、HDFS）单文件
m0_58027884的博客
08-15 1389
Flume 单文件，单端口输入，多端输出
Flume汇入数据到Hive&Hbase
m0_63982362的博客
11-13 589
Flume汇入数据到Hive&Hbase
flume实现上传nginx日志到kafka
最新发布
chanyue123的博客
01-19 1121
flume上传nginx日志 nginx日志的切分策略实现
Flume实时采集日志到Kafka（极简版）
热门推荐
Abysscarry的博客
04-01 1万+
背景： 由于项目采用微服务架构，业务日志文件数量较多，我做了个简单的日志监控服务，先在此分享下日志采集的简单步骤，没有任何花里胡哨 ~（ps：一切数据到了 kafka就非常好解决了！） 一、Flume安装 Flume的安装使用可以说非常简单，直接进官网：http://flume.apache.org/ 最新是1.9.0版本，我们选择1.8.0版本下载。 然后在Linux下解压： 配置用...
Flume日志采集步骤
weixin_45631366的博客
05-13 1123
** Flume日志采集步骤 ** ①配置jobs/taildir_kafka.conf文件 ②写拦截器代码，导入jar包到flume的lib下 ③分发flume ④kafka创建topic_start topic_event ⑤flume生成数据 ⑥第5步骤会将生成的数据写进kafka中，消费一下测试是否成功 ...
Flume分布式日志采集
敲代码的彭于晏
08-01 1427
Apache Flume 概述 Flume是一种分布式，可靠且可用的服务，用于有效地收集，聚合和移动大量日志数据。Flume构建在日志流之上一个简单灵活的架构。它具有可靠的可靠性机制和许多故障转移和恢复机制，具有强大的容错性。使用Flume这套架构实现对日志流数据的实时在线分析。Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。当前Flume有两个版本Flume 0.9X版本的统称Flume-og，Flume1.X版本的
06_Hudi案例实战、Apache Flume日志采集、SparkSession数据处理、数据入kafka、保存Hudi表、集成Hive指标分析、加载Hudi表数据、FineBI 报表可视化等
涂作权的博客
08-13 966
本文来自"黑马程序员"hudi课程6.第六章 Hudi案例实战 6.1 案例架构 6.2 业务数据 6.2.1 消息数据格式 6.2.2 数据生成 6.3 七陌数据采集 6.3.1 Apache Flume 是什么 6.3.2 Apache Flume 运行机制 6.3.3 Apache Flume 安装部署 6.3.4 Apache Flume 入门程序 6.3.5 七陌社交数据采集 6.3.5 七陌社交数据采集 6.4.1 创建模块 6.4.2 封装实体类 6.4.3 编写流式程序 6.4.3.1 构建
大数据之使用Flume监听端口采集数据流到Kafka
Eternity.Arrebol的博客
04-26 2887
本题来源于全国职业技能大赛之大数据技术赛项电商赛题 - 实时数据采集题目很简单。大数据处理涉及到海量的数据采集和处理，如何高效地采集数据是一个重要的问题。Flume是一个分布式的、可靠的、高可用的海量日志采集工具，可以轻松地采集、聚合和移动大量的数据。在数据采集过程中，Flume可以使用网络协议、文件系统和各种其他数据源来收集数据。Flume设计了一个事件模型来处理数据，使用称为“通道”的可配置缓冲区来存储事件，以确保数据不会丢失。同时，使用Flume将数据流转到Kafka中，可以满足将大量数据快速稳定地传
Flume学习-采集端口数据存入kafka
HaveAGoodDay428的博客
03-15 3170
Flume学习-采集端口数据存入kafka
sparkstreaming读取kafka对接flume抽取到的数据库数据并保存到hbase中，hive映射hbase进行查询
03-16
这个流程大致如下：使用Spark Streaming从Kafka中读取数据，然后使用Flume将数据抽取到数据库中，最后将数据保存到HBase中。同时，可以使用Hive将HBase中的数据映射到Hive中，以便进行查询。 具体实现步骤如下： 1...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值