大数据开发项目-电信项目2-传输数据

1. 配置flume文件

首先在kafka的配置文件下创建flume的配置文件

# 1 agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 2 source +0是从第零行开始
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F -c +0 /opt/jars/calllog.csv
a1.sources.r1.shell = /bin/bash -c

# 3 sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList = hou-01:9092,hou-02:9092,hou-03:9092
a1.sinks.k1.topic = calllog
a1.sinks.k1.batchSize = 20
a1.sinks.k1.requiredAcks = 1

# 4 channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 5 bind
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2.数据采集部分打通

2.1启动zookeeper及集群

2.1.1启动zookeeper命令

zkServer.sh start #每台机器都要开

2.2启动kafka集群

2.2.1 启动集群
2.2.1 创建主题
2.2.3 启动kafka消费者

启动kafka命令

2.2.1个人路径：
启动kafka集群(前提要启动ZK的集群):
/root/soft/kafka/bin/kafka-server-start.sh /root/soft/kafka/config/server.properties &

2.2.2创建主题：
/root/soft/kafka/bin/kafka-topics.sh --zookeeper hou-01:2181 --topic calllog --create --replication-factor 1 --partitions 3

删除主题：
/root/soft/kafka/bin/kafka-topics.sh --zookeeper bigdata11:2181 --delete --topic calllog 

列出所有主题：
/root/soft/kafka/bin/kafka-topics.sh --zookeeper hou-01:2181 --list

2.2.3启动kafka消费者：
/root/soft/kafka/bin/kafka-console-consumer.sh --zookeeper hou-01:2181 --topic calllog --from-beginning

查看消费者组
bin/kafka-consumer-groups.sh --zookeeper bigdata11:2181 --group console-consumer-30191 --describe

bin/flume-ng agent --conf conf --conf-file jobs/kafkaToflume.conf --name agent -Dflume.root.logger=INFO,console

2.3启动flume集群

2.3.1 运行命令

/root/soft/apache-flume-1.6.0-bin/bin/flume-ng agent --conf /root/soft/apache-flume-1.6.0-bin/conf/ --name a1 --conf-file /root/soft/apache-flume-1.6.0-bin/conf/flume-kafka.conf

2.4生产数据

2.4.1 运行脚本

sh produceData.sh

sh脚本文件
[root@hou-01 jars]# cat produceData.sh 
#!/bin/bash
java -cp /root/jars/ct_producer-1.0-SNAPSHOT.jar producer.ProductLog /root/jars/calllog.csv

3 数据消费环境准备

写api用habse接收kafka的数据

3.1添加maven配置

查看自己kafka版本，下载对应配置

2.10是scala版本 0.8.1.1是kafka版本

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>Telecom</artifactId>
        <groupId>com.itstar</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>ct_consumer</artifactId>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.8.1.1</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.3.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-server -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.8.1.1</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.12.4</version>
                <configuration>
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

3.2添加maven配置

3.2.1
hadoop/etc/hadoop/core-site.xml
hadoop/etc/hadoop/hdfs-site.xml
habse/conf/habase-site.xml
habse/conf/log4j.properties
3.2.2
在windows下，修改主机映射hosts
C:\Windows\System32\drivers\etc\hosts

3.2.3 新建habse_consumer.properties
内容如下

# 设置kafka的# 设置kafka的brokerlist
bootstrap.servers=bigdata11:9092,bigdata12:9092,bigdata13:9092
# 设置消费者所属的消费组
group.id=hbase_consumer_group
# 设置是否自动确认offset
enable.auto.commit=true
# 自动确认offset的时间间隔
auto.commit.interval.ms=30000
# 设置key，value的反序列化类的全名
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserialize