[Kafka基础]-- Kafka的2套Consumer API(本文讲解SampleConsumer)

最新推荐文章于 2021-02-17 05:30:41 发布
最新推荐文章于 2021-02-17 05:30:41 发布
阅读量1.7k 收藏 1
点赞数 1
分类专栏： Kafka 文章标签： kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/high2011/article/details/53725736
版权

Kafka提供了两套API给Consumer

  1. The high-level Consumer API
  2. The SimpleConsumer API     

第一种高度抽象的Consumer API，它使用起来简单、方便，但是对于某些特殊的需求我们可能要用到第二种更底层的API，那么先介绍下第二种API能够帮助我们做哪些事情

  • 一个消息读取多次
  • 在一个处理过程中只消费Partition其中的一部分消息
  • 添加事务管理机制以保证消息被处理且仅被处理一次

使用SimpleConsumer有哪些弊端呢？

  • 必须在程序中跟踪offset值
  • 必须找出指定Topic Partition中的lead broker
  • 必须处理broker的变动

使用SimpleConsumer的步骤

  1. 从所有活跃的broker中找出哪个是指定Topic Partition中的leader broker
  2. 找出指定Topic Partition中的所有备份broker
  3. 构造请求
  4. 发送请求查询数据
  5. 处理leader broker变更

代码举例一

package com.ngaa.spark.create;

import kafka.api.PartitionOffsetRequestInfo;
import kafka.common.TopicAndPartition;
import kafka.javaapi.OffsetResponse;
import kafka.javaapi.PartitionMetadata;
import kafka.javaapi.TopicMetadata;
import kafka.javaapi.TopicMetadataRequest;
import kafka.javaapi.consumer.SimpleConsumer;

import java.util.*;
import java.util.Map.Entry;

/**
 * Created by yangjf on 2016/12/18
 * Update date:
 * Time: 11:00
 * Describle :获取Kafka的每个分区的最新offset，以及求和
 * Result of Test:测试通过
 * Command:
 * Email: jifei.yang@ngaa.com.cn
 */
public class KafkaOffsetTools {
    public KafkaOffsetTools() {
    }

    public static void main(String[] args) {
        String topic = "20161121a";      //topic名称
        String seed = "192.168.1.54";    //broker的地址(任意一个地址)
        int port = 9092;                 //broker端口号

        List seeds = new ArrayList();
        seeds.add(seed);
        KafkaOffsetTools kot = new KafkaOffsetTools();

        TreeMap metadatas = kot.findLeader(seeds, port, topic);

        int sum = 0;

        for (Entry entry : metadatas.entrySet()) {
            int partition = entry.getKey();
            String leadBroker = entry.getValue().leader().host();
            String clientName = "Client_" + topic + "_" + partition;
            SimpleConsumer consumer = new SimpleConsumer(leadBroker, port, 100000,
                    64 * 1024, clientName);
            long readOffset = getLastOffset(consumer, topic, partition,
                    kafka.api.OffsetRequest.LatestTime(), clientName);
            sum += readOffset;
            System.out.println("partion is :" + partition + ",offset:" + readOffset);
            if (consumer != null) consumer.close();
        }
        System.out.println("message的总和：" + sum);

    }

    public static long getLastOffset(SimpleConsumer consumer, String topic,
                                     int partition, long whichTime, String clientName) {
        TopicAndPartition topicAndPartition = new TopicAndPartition(topic,
                partition);
        Map requestInfo = new HashMap();
        requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(
                whichTime, 1));
        kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(
                requestInfo, kafka.api.OffsetRequest.CurrentVersion(),
                clientName);
        OffsetResponse response = consumer.getOffsetsBefore(request);

        if (response.hasError()) {
            System.out.println("Error fetching data Offset Data the Broker. Reason: "
                    + response.errorCode(topic, partition));
            return 0;
        }
        long[] offsets = response.offsets(topic, partition);
//		long[] offsets2 = response.offsets(topic, 3);
        return offsets[0];
    }

    //寻找leader
    private TreeMap findLeader(List a_seedBrokers, int a_port, String a_topic) {
        TreeMap map = new TreeMap();
        loop:
        for (String seed : a_seedBrokers) {
            SimpleConsumer consumer = null;
            try {
                consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024,
                        "leaderLookup" + new Date().getTime());
                List topics = Collections.singletonList(a_topic);
                TopicMetadataRequest req = new TopicMetadataRequest(topics);
                kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);

                List metaData = resp.topicsMetadata();
                for (TopicMetadata item : metaData) {
                    for (PartitionMetadata part : item.partitionsMetadata()) {
                        map.put(part.partitionId(), part);
//						if (part.partitionId() == a_partition) {
//							returnMetaData = part;
//							break loop;
//						}
                    }
                }
            } catch (Exception e) {
                System.out.println("Error communicating with Broker [" + seed
                        + "] to find Leader for [" + a_topic + ", ] Reason: " + e);
            } finally {
                if (consumer != null)
                    consumer.close();
            }
        }
//		if (returnMetaData != null) {
//			m_replicaBrokers.clear();
//			for (kafka.cluster.Broker replica : returnMetaData.replicas()) {
//				m_replicaBrokers.add(replica.host());
//			}
//		}
        return map;
    }

}
 seeds = new ArrayList();
        seeds.add(seed);
        KafkaOffsetTools kot = new KafkaOffsetTools();

        TreeMap metadatas = kot.findLeader(seeds, port, topic);

        int sum = 0;

        for (Entry entry : metadatas.entrySet()) {
            int partition = entry.getKey();
            String leadBroker = entry.getValue().leader().host();
            String clientName = "Client_" + topic + "_" + partition;
            SimpleConsumer consumer = new SimpleConsumer(leadBroker, port, 100000,
                    64 * 1024, clientName);
            long readOffset = getLastOffset(consumer, topic, partition,
                    kafka.api.OffsetRequest.LatestTime(), clientName);
            sum += readOffset;
            System.out.println("partion is :" + partition + ",offset:" + readOffset);
            if (consumer != null) consumer.close();
        }
        System.out.println("message的总和：" + sum);

    }

    public static long getLastOffset(SimpleConsumer consumer, String topic,
                                     int partition, long whichTime, String clientName) {
        TopicAndPartition topicAndPartition = new TopicAndPartition(topic,
                partition);
        Map requestInfo = new HashMap();
        requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(
                whichTime, 1));
        kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(
                requestInfo, kafka.api.OffsetRequest.CurrentVersion(),
                clientName);
        OffsetResponse response = consumer.getOffsetsBefore(request);

        if (response.hasError()) {
            System.out.println("Error fetching data Offset Data the Broker. Reason: "
                    + response.errorCode(topic, partition));
            return 0;
        }
        long[] offsets = response.offsets(topic, partition);
//		long[] offsets2 = response.offsets(topic, 3);
        return offsets[0];
    }

    //寻找leader
    private TreeMap findLeader(List a_seedBrokers, int a_port, String a_topic) {
        TreeMap map = new TreeMap();
        loop:
        for (String seed : a_seedBrokers) {
            SimpleConsumer consumer = null;
            try {
                consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024,
                        "leaderLookup" + new Date().getTime());
                List topics = Collections.singletonList(a_topic);
                TopicMetadataRequest req = new TopicMetadataRequest(topics);
                kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);

                List metaData = resp.topicsMetadata();
                for (TopicMetadata item : metaData) {
                    for (PartitionMetadata part : item.partitionsMetadata()) {
                        map.put(part.partitionId(), part);
//						if (part.partitionId() == a_partition) {
//							returnMetaData = part;
//							break loop;
//						}
                    }
                }
            } catch (Exception e) {
                System.out.println("Error communicating with Broker [" + seed
                        + "] to find Leader for [" + a_topic + ", ] Reason: " + e);
            } finally {
                if (consumer != null)
                    consumer.close();
            }
        }
//		if (returnMetaData != null) {
//			m_replicaBrokers.clear();
//			for (kafka.cluster.Broker replica : returnMetaData.replicas()) {
//				m_replicaBrokers.add(replica.host());
//			}
//		}
        return map;
    }

}

代码举例二

import java.nio.ByteBuffer;  
import java.util.ArrayList;  
import java.util.Collections;  
import java.util.HashMap;  
import java.util.List;  
import java.util.Map;  
  
import kafka.api.FetchRequest;  
import kafka.api.FetchRequestBuilder;  
import kafka.api.PartitionOffsetRequestInfo;  
import kafka.common.ErrorMapping;  
import kafka.common.TopicAndPartition;  
import kafka.javaapi.FetchResponse;  
import kafka.javaapi.OffsetResponse;  
import kafka.javaapi.PartitionMetadata;  
import kafka.javaapi.TopicMetadata;  
import kafka.javaapi.TopicMetadataRequest;  
import kafka.javaapi.consumer.SimpleConsumer;  
import kafka.message.MessageAndOffset;  
/**
 * Created by yangjf on 2016/12/18
 * Update date:
 * Time: 10:44
 * Describle :SampleConsumer  API举例  ，获取最新的offset
 * Result of Test:未测试
 * Command:
 * Email: jifei.yang@ngaa.com.cn
 */
public class SampleKafkaConsumer {  
    private List m_replicaBrokers = new ArrayList();  
  
    public SimpleExample() {  
        m_replicaBrokers = new ArrayList();  
    }  
 
    public static void main(String args[]) {  
        SimpleExample example = new SimpleExample();  
        // 最大读取消息数量  
        long maxReads = Long.parseLong("3");  
        // 要订阅的topic  
        String topic = "20161218a";  
        // 要查找的分区  
        int partition = Integer.parseInt("0");  
        // broker节点的ip  
        List seeds = new ArrayList();  
        seeds.add("hadoop1");  //可以是ip或者主机名  
        seeds.add("hadoop2");  
        seeds.add("hadoop3");  
        // 端口  
        int port = Integer.parseInt("9092");  
        try {  
            example.run(maxReads, topic, partition, seeds, port);  
        } catch (Exception e) {  
            System.out.println("Oops:" + e);  
            e.printStackTrace();  
        }  
    }  
  
    public void run(long a_maxReads, String a_topic, int a_partition, List a_seedBrokers, int a_port) throws Exception {  
        // 获取指定Topic partition的元数据  
        PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition);  
        if (metadata == null) {  
            System.out.println("Can't find metadata for Topic and Partition. Exiting");  
            return;  
        }  
        if (metadata.leader() == null) {  
            System.out.println("Can't find Leader for Topic and Partition. Exiting");  
            return;  
        }  
        String leadBroker = metadata.leader().host();  
        String clientName = "Client_" + a_topic + "_" + a_partition;  
  
        SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);  
        long readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);  
        int numErrors = 0;  
        while (a_maxReads > 0) {  
            if (consumer == null) {  
                consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);  
            }  
            FetchRequest req = new FetchRequestBuilder().clientId(clientName).addFetch(a_topic, a_partition, readOffset, 100000).build();  
            FetchResponse fetchResponse = consumer.fetch(req);  
  
            if (fetchResponse.hasError()) {  
                numErrors++;  
                // Something went wrong!  
                short code = fetchResponse.errorCode(a_topic, a_partition);  
                System.out.println("Error fetching data from the Broker:" + leadBroker + " Reason: " + code);  
                if (numErrors > 5)  
                    break;  
                if (code == ErrorMapping.OffsetOutOfRangeCode()) {  
                    // We asked for an invalid offset. For simple case ask for  
                    // the last element to reset  
                    readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);  
                    continue;  
                }  
                consumer.close();  
                consumer = null;  
                leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port);  
                continue;  
            }  
            numErrors = 0;  
  
            long numRead = 0;  
            for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) {  
                long currentOffset = messageAndOffset.offset();  
                if (currentOffset < readOffset) {  
                    System.out.println("Found an old offset: " + currentOffset + " Expecting: " + readOffset);  
                    continue;  
                }  
  
                readOffset = messageAndOffset.nextOffset();  
                ByteBuffer payload = messageAndOffset.message().payload();  
  
                byte[] bytes = new byte[payload.limit()];  
                payload.get(bytes);  
                System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8"));  
                numRead++;  
                a_maxReads--;  
            }  
  
            if (numRead == 0) {  
                try {  
                    Thread.sleep(1000);  
                } catch (InterruptedException ie) {  
                }  
            }  
        }  
        if (consumer != null)  
            consumer.close();  
    }  
  
    public static long getLastOffset(SimpleConsumer consumer, String topic, int partition, long whichTime, String clientName) {  
        TopicAndPartition topicAndPartition = new TopicAndPartition(topic, partition);  
        Map requestInfo = new HashMap();  
        requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 1));  
        kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(requestInfo, kafka.api.OffsetRequest.CurrentVersion(), clientName);  
        OffsetResponse response = consumer.getOffsetsBefore(request);  
  
        if (response.hasError()) {  
            System.out.println("Error fetching data Offset Data the Broker. Reason: " + response.errorCode(topic, partition));  
            return 0;  
        }  
        long[] offsets = response.offsets(topic, partition);  
        return offsets[0];  
    }  
  
    /** 
     * @param a_oldLeader 
     * @param a_topic 
     * @param a_partition 
     * @param a_port 
     * @return String 
     * @throws Exception 
     *             找一个leader broker 
     */  
    private String findNewLeader(String a_oldLeader, String a_topic, int a_partition, int a_port) throws Exception {  
        for (int i = 0; i < 3; i++) {  
            boolean goToSleep = false;  
            PartitionMetadata metadata = findLeader(m_replicaBrokers, a_port, a_topic, a_partition);  
            if (metadata == null) {  
                goToSleep = true;  
            } else if (metadata.leader() == null) {  
                goToSleep = true;  
            } else if (a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) {  
                // first time through if the leader hasn't changed give  
                // ZooKeeper a second to recover  
                // second time, assume the broker did recover before failover,  
                // or it was a non-Broker issue  
                //  
                goToSleep = true;  
            } else {  
                return metadata.leader().host();  
            }  
            if (goToSleep) {  
                try {  
                    Thread.sleep(1000);  
                } catch (InterruptedException ie) {  
                }  
            }  
        }  
        System.out.println("Unable to find new leader after Broker failure. Exiting");  
        throw new Exception("Unable to find new leader after Broker failure. Exiting");  
    }  
  
    private PartitionMetadata findLeader(List a_seedBrokers, int a_port, String a_topic, int a_partition) {  
        PartitionMetadata returnMetaData = null;  
        loop: for (String seed : a_seedBrokers) {  
            SimpleConsumer consumer = null;  
            try {  
                consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024, "leaderLookup");  
                List topics = Collections.singletonList(a_topic);  
                TopicMetadataRequest req = new TopicMetadataRequest(topics);  
                kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);  
  
                List metaData = resp.topicsMetadata();  
                for (TopicMetadata item : metaData) {  
                    for (PartitionMetadata part : item.partitionsMetadata()) {  
                        if (part.partitionId() == a_partition) {  
                            returnMetaData = part;  
                            break loop;  
                        }  
                    }  
                }  
            } catch (Exception e) {  
                System.out.println("Error communicating with Broker [" + seed + "] to find Leader for [" + a_topic + ", " + a_partition + "] Reason: " + e);  
            } finally {  
                if (consumer != null)  
                    consumer.close();  
            }  
        }  
        if (returnMetaData != null) {  
            m_replicaBrokers.clear();  
            for (kafka.cluster.Broker replica : returnMetaData.replicas()) {  
                m_replicaBrokers.add(replica.host());  
            }  
        }  
        return returnMetaData;  
    }  
}
 m_replicaBrokers = new ArrayList();  
  
    public SimpleExample() {  
        m_replicaBrokers = new ArrayList();  
    }  
 
    public static void main(String args[]) {  
        SimpleExample example = new SimpleExample();  
        // 最大读取消息数量  
        long maxReads = Long.parseLong("3");  
        // 要订阅的topic  
        String topic = "20161218a";  
        // 要查找的分区  
        int partition = Integer.parseInt("0");  
        // broker节点的ip  
        List seeds = new ArrayList();  
        seeds.add("hadoop1");  //可以是ip或者主机名  
        seeds.add("hadoop2");  
        seeds.add("hadoop3");  
        // 端口  
        int port = Integer.parseInt("9092");  
        try {  
            example.run(maxReads, topic, partition, seeds, port);  
        } catch (Exception e) {  
            System.out.println("Oops:" + e);  
            e.printStackTrace();  
        }  
    }  
  
    public void run(long a_maxReads, String a_topic, int a_partition, List a_seedBrokers, int a_port) throws Exception {  
        // 获取指定Topic partition的元数据  
        PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition);  
        if (metadata == null) {  
            System.out.println("Can't find metadata for Topic and Partition. Exiting");  
            return;  
        }  
        if (metadata.leader() == null) {  
            System.out.println("Can't find Leader for Topic and Partition. Exiting");  
            return;  
        }  
        String leadBroker = metadata.leader().host();  
        String clientName = "Client_" + a_topic + "_" + a_partition;  
  
        SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);  
        long readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);  
        int numErrors = 0;  
        while (a_maxReads > 0) {  
            if (consumer == null) {  
                consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);  
            }  
            FetchRequest req = new FetchRequestBuilder().clientId(clientName).addFetch(a_topic, a_partition, readOffset, 100000).build();  
            FetchResponse fetchResponse = consumer.fetch(req);  
  
            if (fetchResponse.hasError()) {  
                numErrors++;  
                // Something went wrong!  
                short code = fetchResponse.errorCode(a_topic, a_partition);  
                System.out.println("Error fetching data from the Broker:" + leadBroker + " Reason: " + code);  
                if (numErrors > 5)  
                    break;  
                if (code == ErrorMapping.OffsetOutOfRangeCode()) {  
                    // We asked for an invalid offset. For simple case ask for  
                    // the last element to reset  
                    readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);  
                    continue;  
                }  
                consumer.close();  
                consumer = null;  
                leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port);  
                continue;  
            }  
            numErrors = 0;  
  
            long numRead = 0;  
            for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) {  
                long currentOffset = messageAndOffset.offset();  
                if (currentOffset < readOffset) {  
                    System.out.println("Found an old offset: " + currentOffset + " Expecting: " + readOffset);  
                    continue;  
                }  
  
                readOffset = messageAndOffset.nextOffset();  
                ByteBuffer payload = messageAndOffset.message().payload();  
  
                byte[] bytes = new byte[payload.limit()];  
                payload.get(bytes);  
                System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8"));  
                numRead++;  
                a_maxReads--;  
            }  
  
            if (numRead == 0) {  
                try {  
                    Thread.sleep(1000);  
                } catch (InterruptedException ie) {  
                }  
            }  
        }  
        if (consumer != null)  
            consumer.close();  
    }  
  
    public static long getLastOffset(SimpleConsumer consumer, String topic, int partition, long whichTime, String clientName) {  
        TopicAndPartition topicAndPartition = new TopicAndPartition(topic, partition);  
        Map requestInfo = new HashMap();  
        requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 1));  
        kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(requestInfo, kafka.api.OffsetRequest.CurrentVersion(), clientName);  
        OffsetResponse response = consumer.getOffsetsBefore(request);  
  
        if (response.hasError()) {  
            System.out.println("Error fetching data Offset Data the Broker. Reason: " + response.errorCode(topic, partition));  
            return 0;  
        }  
        long[] offsets = response.offsets(topic, partition);  
        return offsets[0];  
    }  
  
    /** 
     * @param a_oldLeader 
     * @param a_topic 
     * @param a_partition 
     * @param a_port 
     * @return String 
     * @throws Exception 
     *             找一个leader broker 
     */  
    private String findNewLeader(String a_oldLeader, String a_topic, int a_partition, int a_port) throws Exception {  
        for (int i = 0; i < 3; i++) {  
            boolean goToSleep = false;  
            PartitionMetadata metadata = findLeader(m_replicaBrokers, a_port, a_topic, a_partition);  
            if (metadata == null) {  
                goToSleep = true;  
            } else if (metadata.leader() == null) {  
                goToSleep = true;  
            } else if (a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) {  
                // first time through if the leader hasn't changed give  
                // ZooKeeper a second to recover  
                // second time, assume the broker did recover before failover,  
                // or it was a non-Broker issue  
                //  
                goToSleep = true;  
            } else {  
                return metadata.leader().host();  
            }  
            if (goToSleep) {  
                try {  
                    Thread.sleep(1000);  
                } catch (InterruptedException ie) {  
                }  
            }  
        }  
        System.out.println("Unable to find new leader after Broker failure. Exiting");  
        throw new Exception("Unable to find new leader after Broker failure. Exiting");  
    }  
  
    private PartitionMetadata findLeader(List a_seedBrokers, int a_port, String a_topic, int a_partition) {  
        PartitionMetadata returnMetaData = null;  
        loop: for (String seed : a_seedBrokers) {  
            SimpleConsumer consumer = null;  
            try {  
                consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024, "leaderLookup");  
                List topics = Collections.singletonList(a_topic);  
                TopicMetadataRequest req = new TopicMetadataRequest(topics);  
                kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);  
  
                List metaData = resp.topicsMetadata();  
                for (TopicMetadata item : metaData) {  
                    for (PartitionMetadata part : item.partitionsMetadata()) {  
                        if (part.partitionId() == a_partition) {  
                            returnMetaData = part;  
                            break loop;  
                        }  
                    }  
                }  
            } catch (Exception e) {  
                System.out.println("Error communicating with Broker [" + seed + "] to find Leader for [" + a_topic + ", " + a_partition + "] Reason: " + e);  
            } finally {  
                if (consumer != null)  
                    consumer.close();  
            }  
        }  
        if (returnMetaData != null) {  
            m_replicaBrokers.clear();  
            for (kafka.cluster.Broker replica : returnMetaData.replicas()) {  
                m_replicaBrokers.add(replica.host());  
            }  
        }  
        return returnMetaData;  
    }  
}

 

 

官方举例：https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

 

 

往事随风ing
关注
专栏目录
Jmeter Kafka Sampler（Producer）
C'mon的博客
10-11 790
之前写了一个Jmeter kafka 插件是基于公司对kafka二次封装写的，这次基于原生kafka写一个插件，如下，废话不多说，中心思想是一样的，可以参考我上篇文章。Jmeter Kafka插件开发之Sampler篇。
2.2 Consumer API官网剖析（博主推荐）
weixin_33717117的博客
04-26 56
    不多说，直接上干货！   　　一切来源于官网 http://kafka.apache.org/documentation/           2.2 Consumer API 2.2、消费者API     随着0.9.0版本，我们已经增加了一个新的Java消费者替换我们现有的基于zookeeper的高级和低级消费者。这个客户端还是测试...
Kafka Consumer的底层API- SimpleConsumer
chongxungu8275的博客
06-21 222
Kafka Consumer的底层API- SimpleConsumer 1.Kafka提供了两套API给Consumer The high-level Consumer API The SimpleConsumer API...
SampleConsumer的代码
java学习之旅
08-20 118
网址：http://code.google.com/p/openid4java/wiki/SampleConsumer /* * Copyright 2006-2007 Sxip Identity Corporation */ package org.openid4java.consumer; import org.openid4java.discovery.Identifi...
kafka官网示例说明--KafkaConsumer
kw-home
12-18 1万+
Kafka client会消费kafka cluster中的记录。 它将透明地处理Kafka集群中的服务器故障，并透明地适应它在集群内迁移的数据分区。该客户机还与服务器交互，以允许用户组使用消费者组来负载平衡消费(如下所述)。 消费者维护TCP连接到必要的代理来获取数据。使用后未能关闭消费者会泄漏这些连接。消费者不是线程安全的。更多细节见多线程处理。 偏移量 Kafka为分
flink-connector-kafka-base-2.11-1.10.0-API文档-中文版.zip
04-23
赠送原API文档：flink-connector-kafka-base_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-connector-kafka-base_2.11-1.10.0-sources.jar； 赠送Maven依赖信息文件：flink-connector-kafka-base_2.11-1.10.0.pom；...
kafka-clients-0.10.0.1-API文档-中文版.zip
04-23
赠送原API文档：kafka-clients-0.10.0.1-javadoc.jar； 赠送源代码：kafka-clients-0.10.0.1-sources.jar； 赠送Maven依赖信息文件：kafka-clients-0.10.0.1.pom； 包含翻译后的API文档：kafka-clients-0.10.0.1-...
kafka-clients-2.0.1-API文档-中文版.zip
07-14
赠送原API文档：kafka-clients-2.0.1-javadoc.jar； 赠送源代码：kafka-clients-2.0.1-sources.jar； 赠送Maven依赖信息文件：kafka-clients-2.0.1.pom； 包含翻译后的API文档：kafka-clients-2.0.1-javadoc-API文档...
kafka-clients-2.0.0-API文档-中英对照版.zip
06-17
赠送原API文档：kafka-clients-2.0.0-javadoc.jar； 赠送源代码：kafka-clients-2.0.0-sources.jar； 赠送Maven依赖信息文件：kafka-clients-2.0.0.pom； 包含翻译后的API文档：kafka-clients-2.0.0-javadoc-API文档...
kafka-clients-2.2.0-API文档-中文版.zip
07-14
赠送原API文档：kafka-clients-2.2.0-javadoc.jar； 赠送源代码：kafka-clients-2.2.0-sources.jar； 赠送Maven依赖信息文件：kafka-clients-2.2.0.pom； 包含翻译后的API文档：kafka-clients-2.2.0-javadoc-API文档...
kafka全套视频教程 某客学院
03-22
视频教程从入门到源码分析，内容全面，课程质量很高！
余老师带你学习大数据-Spark快速大数据处理第十章Kafka第六节Kafka-Consumer
weixin_45810046的博客
01-25 146
代码操做及解释 1、在运行每个单独class的时，要先检查pom文件中的主程序是否对应的是要执行的class 后面的发送信息的（Producer）运行操作的命令： 编译：mvn compile -Dexec.mainClass="com.kinginsai.bigdata.kafka.producer.ProducerSample" 运行：mvn exec:java -Dexec.mainClass="com.kinginsai.bigdata.kafka.producer.ProducerSample"
JMeter-扩展插件实现Kafka协议的测试(转)
测试小鸟的博客
09-09 2992
摘要## JMeter本身提供了插件机制，允许第三方扩展JMeter以支持JMeter不支持的协议的测试。本文以扩展一个简单的Apache Kafka producer sampler为例描述了如何对JMeter进行扩展，方便地实现对新协议的支持。 Kafka简介## Apache Kafka是由Apache软件基金会开发的一个开源消息系统项目。Kafka最初是由LinkedIn开发，...
kafka 2.10 java api_kafka中常用API的简单JAVA代码
weixin_27960253的博客
02-17 96
通过之前《kafka分布式消息队列介绍以及集群安装》的介绍，对kafka有了初步的了解。本文主要讲述java代码中常用的操作。准备：增加kafka依赖org.apache.kafkakafka-clients0.10.2.0一、kafka中对topic的操作package org.kafka;import kafka.admin.DeleteTopicCommand;import kafka.ad...
通俗易懂的Kafka升级版教程（含配套资料）
08-29
Kafka是一个开源流处理平台，由Scala和Java编写。Kafka是一种高吞吐量的分布式发布订阅消息系统，它可以处理消费者规模的网站中的所有动作流数据。在本课程中，你将学习到，Kafka架构原理、安装配置使用、详细的Kafka写入数据和处理数据以及写出数据的流程、新旧版本对比及运用、分区副本机制的详解、内部存储策略、高阶API直接消费数据、低阶API自行管理Offset消费数据、Kafka拦截器以及KafkaStream流式处理。更新：课件升级、Kafka幂等性、Kafka事务、监控器Eagle的使用、Kafka对接Flume的使用以及企业真实面试题详解
Kafka详解五、Kafka Consumer的底层API- SimpleConsumer
好读书，每有会意，便欣然忘食。
11-17 1116
问题导读 1.Kafka如何实现和Consumer之间的交互？ 2.使用SimpleConsumer有哪些弊端呢？ 1.Kafka提供了两套API给Consumer The high-level Consumer API The SimpleConsumer API 第一种高度抽象的Consumer API，它使用起来简单
关于Kafka 的 consumer 消费者手动提交详解
weixin_34409822的博客
02-08 2246
前言 在上一篇 Kafka使用Java实现数据的生产和消费demo 中介绍如何简单的使用kafka进行数据传输。本篇则重点介绍kafka中的 consumer 消费者的讲解。 应用场景 在上一篇kafka的consumer消费者，我们使用的是自动提交offset下标。 但是offset下标自动提交其实在很多场景都不适用，因为自动提交是在kafka拉取到数据之后就直接提交，这样很容易丢失数据，尤其是...
java自己手动控制kafka的offset
热门推荐
allen的博客
03-21 2万+
之前使用kafka的KafkaStream：http://blog.csdn.net/qq_20641565/article/details/60810174，让每个消费者和对应的patition建立对应的流来读取kafka上面的数据，如果comsumer得到数据，那么kafka就会自动去维护该comsumer的offset，例如在获取到kafka的消息后正准备入库（未入库），但是消费者挂了，那么如
Kafka-Eagle监控工具：安装与使用指南
"本文主要介绍了Kafka监控工具Kafka-Eagle的使用，包括其简介、部署和验证。Kafka-Eagle是一款开源免费的Kafka集群监控工具，能够监控offset、lag变化、partition分布和owner等关键信息。文章还提到了开启Kafka JMX...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

往事随风ing

你的鼓励将是我创作的最大动力!

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值