7 kafka 消费者和生产者实例

最新推荐文章于 2024-03-13 07:30:00 发布
最新推荐文章于 2024-03-13 07:30:00 发布
阅读量1.5k 收藏 1
点赞数 1
分类专栏： 【构建高可用架构】 【大数据】 【kafka】
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_27384769/article/details/80113279
版权

7 kafka消费者和生产者实例

更多干货

一、概述

二、例子代码地址

https://github.com/csy512889371/learndemo/tree/master/kafka

三、Kafka消费者编程模型

四、分区消费模型

image

伪代码描述


main()
   获取分区的size
   for index =0 to size
     create thread(or process) consumer(Index)


第index个线程(进程)
consumer(index)
    创建到kafka broker的连接: KafkaClient(host,port)
    指定消费参数构建consumer: SimpleConsumer(topic, partitions)
    设置消费offset : consumer.seek(offset,0)
    while  True
	  消费指定topic第index个分区的数据
	  处理
     记录当前消息offset
     提交当前offset（可选）

五、组(Group)消费模型

image

按组(Group)消费伪代码描述


main()
   设置需要创建的流数N
   for index =0 to N
     create thread  consumer(Index)

第index个线程
consumer(index)
    创建到kafka broker的连接: KafkaClient(host,port)
    指定消费参数构建consumer: SimpleConsumer(topic, partitions)
    设置从头消费还是从最新的消费(smallest或largest)
    while  True
	    从指定topic的第index个流取数据
	    处理
     （offset会自动提交到zookeeper,无需我们操作）

Consumer分配算法

For each topic T that Ci subscribes to 
let PT be all partitions producing topic T 
let CG be all consumers in the same group as Ci that consume topic T 
sort PT (so partitions on the same broker are clustered together) 
sort CG 
let i be the index position of Ci in CG and let N = size(PT)/size(CG) 
assign partitions from i*N to (i+1)*N - 1 to consumer Ci 
remove current entries owned by Ci from the partition owner registry
add newly assigned partitions to the partition owner registry
   (we may need to re-try this until the original partition owner releases 
   its ownership)

六、两种消费模型对比

分区消费模型更加灵活但是：

  • 1、需要自己处理各种异常情况；
  • 2、需要自己管理offset(以实现消息传递的其他语义)；

组消费模型更加简单，但是不灵活：

  • 1、不需要自己处理异常情况，不需要自己管理offset；
  • 2、只能实现kafka默认的最少一次消息传递语义；

七、Kafka消费者的Python

需要的软件环境:

  • 已搭建好的kafka集群、Linux服务器一台、Python2.7.6 、kafka-Python软件包

Python客户端参数调优

  • fetch_size_bytes: 从服务器获取单包大小;
  • buffer_size: kafka客户端缓冲区大小;
  • Group:分组消费时分组名
  • auto_commit: offset是否自动提交;

分区消费

import logging, time
import partition_consumer


def main():

    threads = []
    partition = 3
    for index in range(partition):
        threads.append(partition_consumer.Consumer(index))

    for t in threads:
        t.start()

    time.sleep(50000)

if __name__ == '__main__':
    #logging.basicConfig(
    #    format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:%(levelname)s:%(process)d:%(message)s',
    #    level=logging.INFO
    #    )
    main()

import threading
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer

class Consumer(threading.Thread):
    daemon = True
    def __init__(self,partition_index):
        threading.Thread.__init__(self)
        self.part = [partition_index]
        self.__offset = 0


    def run(self):
        client = KafkaClient("10.206.216.13:19092,10.206.212.14:19092,10.206.209.25:19092")
        consumer = SimpleConsumer(client, "test-group", "jiketest",auto_commit=False,partitions=self.part)

        consumer.seek(0,0)

        while True:
            message = consumer.get_message(True,60)
            self.__offset = message.offset
            print message.message.value

分组消费

import logging, time
import group_consumer


def main():

    conusmer_thread = group_consumer.Consumer()
    conusmer_thread.start()

    time.sleep(500000)

if __name__ == '__main__':
    #logging.basicConfig(
    #    format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:%(levelname)s:%(process)d:%(message)s',
    #    level=logging.INFO
    #    )
    main()

import threading
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer

class Consumer(threading.Thread):
    daemon = True

    def run(self):
        client = KafkaClient("10.206.216.13:19092,10.206.212.14:19092,10.206.209.25:19092")
        consumer = SimpleConsumer(client, "test-group", "jiketest")

        for message in consumer:
            print(message.message.value)

七、Kafka消费者的Java

需要的软件环境:

  • 已搭建好的kafka集群、Linux服务器一台、Apache Maven

Java客户端参数调优

  • fetchSize: 从服务器获取单包大小;
  • bufferSize: kafka客户端缓冲区大小;
  • group.id: 分组消费时分组名

分区消费

package kafka.consumer.partition;

import kafka.api.FetchRequest;
import kafka.api.FetchRequestBuilder;
import kafka.api.PartitionOffsetRequestInfo;
import kafka.common.ErrorMapping;
import kafka.common.TopicAndPartition;
import kafka.javaapi.*;
import kafka.javaapi.consumer.SimpleConsumer;
import kafka.message.MessageAndOffset;
 
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class PartitionConsumerTest {
	public static void main(String args[]) {
		PartitionConsumerTest example = new PartitionConsumerTest();
        long maxReads = Long.MAX_VALUE;
        String topic = "jiketest";
		if(args.length < 1){
			System.out.println("Please assign partition number.");
		}
		
		List<String> seeds = new ArrayList<String>();
		String hosts="10.206.216.13,10.206.212.14,10.206.209.25";
		String[] hostArr = hosts.split(",");
		for(int index = 0;index < hostArr.length;index++){
			seeds.add(hostArr[index].trim());
		}
		
		int port = 19092;
		 
        int partLen = Integer.parseInt(args[0]);
		for(int index=0;index < partLen;index++){
	        try {
	            example.run(maxReads, topic, index/*partition*/, seeds, port);
	        } catch (Exception e) {
	            System.out.println("Oops:" + e);
	             e.printStackTrace();
	        }
		}
	}
	
	private List<String> m_replicaBrokers = new ArrayList<String>();
	 
	    public PartitionConsumerTest() {
	        m_replicaBrokers = new ArrayList<String>();
	    }
	 
	    public void run(long a_maxReads, String a_topic, int a_partition, List<String> a_seedBrokers, int a_port) throws Exception {
	        // find the meta data about the topic and partition we are interested in
	        //
	        PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition);
	        if (metadata == null) {
	            System.out.println("Can't find metadata for Topic and Partition. Exiting");
	            return;
	        }
	        if (metadata.leader() == null) {
	            System.out.println("Can't find Leader for Topic and Partition. Exiting");
	            return;
	        }
	        String leadBroker = metadata.leader().host();
	        String clientName = "Client_" + a_topic + "_" + a_partition;
	 
	        SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
	        long readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);
	 
	        int numErrors = 0;
	        while (a_maxReads > 0) {
	            if (consumer == null) {
	                consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
	            }
	            FetchRequest req = new FetchRequestBuilder()
	                    .clientId(clientName)
	                    .addFetch(a_topic, a_partition, readOffset, 100000) // Note: this fetchSize of 100000 might need to be increased if large batches are written to Kafka
	                    .build();
	            FetchResponse fetchResponse = consumer.fetch(req);
	 
	            if (fetchResponse.hasError()) {
	                numErrors++;
	                // Something went wrong!
	                short code = fetchResponse.errorCode(a_topic, a_partition);
	                System.out.println("Error fetching data from the Broker:" + leadBroker + " Reason: " + code);
	                if (numErrors > 5) break;
	                if (code == ErrorMapping.OffsetOutOfRangeCode())  {
	                    // We asked for an invalid offset. For simple case ask for the last element to reset
	                    readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);
	                    continue;
	                }
	                consumer.close();
	                consumer = null;
	                leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port);
	                continue;
	            }
	            numErrors = 0;
	 
	            long numRead = 0;
	            for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) {
	                long currentOffset = messageAndOffset.offset();
	                if (currentOffset < readOffset) {
	                    System.out.println("Found an old offset: " + currentOffset + " Expecting: " + readOffset);
	                    continue;
	                }
	                readOffset = messageAndOffset.nextOffset();
	                ByteBuffer payload = messageAndOffset.message().payload();
	 
	                byte[] bytes = new byte[payload.limit()];
	                payload.get(bytes);
	                System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8"));
	                numRead++;
	                a_maxReads--;
	            }
	 
	            if (numRead == 0) {
	                try {
	                    Thread.sleep(1000);
	                } catch (InterruptedException ie) {
	                }
	            }
	        }
	        if (consumer != null) consumer.close();
	    }
	 
	    public static long getLastOffset(SimpleConsumer consumer, String topic, int partition,
	                                     long whichTime, String clientName) {
	        TopicAndPartition topicAndPartition = new TopicAndPartition(topic, partition);
	        Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = new HashMap<TopicAndPartition, PartitionOffsetRequestInfo>();
	        requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 1));
	        kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(
	                requestInfo, kafka.api.OffsetRequest.CurrentVersion(), clientName);
	        OffsetResponse response = consumer.getOffsetsBefore(request);
	 
	        if (response.hasError()) {
	            System.out.println("Error fetching data Offset Data the Broker. Reason: " + response.errorCode(topic, partition) );
	            return 0;
	        }
	        long[] offsets = response.offsets(topic, partition);
	        return offsets[0];
	    }
	 
	    private String findNewLeader(String a_oldLeader, String a_topic, int a_partition, int a_port) throws Exception {
	        for (int i = 0; i < 3; i++) {
	            boolean goToSleep = false;
	            PartitionMetadata metadata = findLeader(m_replicaBrokers, a_port, a_topic, a_partition);
	            if (metadata == null) {
	                goToSleep = true;
	            } else if (metadata.leader() == null) {
	                goToSleep = true;
	            } else if (a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) {
	                // first time through if the leader hasn't changed give ZooKeeper a second to recover
	                // second time, assume the broker did recover before failover, or it was a non-Broker issue
	                //
	                goToSleep = true;
	            } else {
	                return metadata.leader().host();
	            }
	            if (goToSleep) {
	                try {
	                    Thread.sleep(1000);
	                } catch (InterruptedException ie) {
	                }
	            }
	        }
	        System.out.println("Unable to find new leader after Broker failure. Exiting");
	        throw new Exception("Unable to find new leader after Broker failure. Exiting");
	    }
	 
	    private PartitionMetadata findLeader(List<String> a_seedBrokers, int a_port, String a_topic, int a_partition) {
	        PartitionMetadata returnMetaData = null;
	        loop:
	        for (String seed : a_seedBrokers) {
	            SimpleConsumer consumer = null;
	            try {
	                consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024, "leaderLookup");
	                List<String> topics = Collections.singletonList(a_topic);
	                TopicMetadataRequest req = new TopicMetadataRequest(topics);
	                kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);
	 
	                List<TopicMetadata> metaData = resp.topicsMetadata();
	                for (TopicMetadata item : metaData) {
	                    for (PartitionMetadata part : item.partitionsMetadata()) {
	                        if (part.partitionId() == a_partition) {
	                            returnMetaData = part;
	                            break loop;
	                        }
	                    }
	                }
	            } catch (Exception e) {
	                System.out.println("Error communicating with Broker [" + seed + "] to find Leader for [" + a_topic
	                        + ", " + a_partition + "] Reason: " + e);
	            } finally {
	                if (consumer != null) consumer.close();
	            }
	        }
	        if (returnMetaData != null) {
	            m_replicaBrokers.clear();
	            for (kafka.cluster.Broker replica : returnMetaData.replicas()) {
	                m_replicaBrokers.add(replica.host());
	            }
	        }
	        return returnMetaData;
	    }
}

分组消费

package kafka.consumer.group;

import kafka.consumer.ConsumerConfig;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
 
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class GroupConsumerTest extends Thread {
	private final ConsumerConnector consumer;
    private final String topic;
    private  ExecutorService executor;
	
	public GroupConsumerTest(String a_zookeeper, String a_groupId, String a_topic){
		consumer = kafka.consumer.Consumer.createJavaConsumerConnector(
                createConsumerConfig(a_zookeeper, a_groupId));
        this.topic = a_topic;
	}
	
	public void shutdown() {
        if (consumer != null) consumer.shutdown();
        if (executor != null) executor.shutdown();
        try {
            if (!executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS)) {
                System.out.println("Timed out waiting for consumer threads to shut down, exiting uncleanly");
            }
        } catch (InterruptedException e) {
            System.out.println("Interrupted during shutdown, exiting uncleanly");
        }
   }
 
    public void run(int a_numThreads) {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, new Integer(a_numThreads));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
 
        // now launch all the threads
        //
        executor = Executors.newFixedThreadPool(a_numThreads);
 
        // now create an object to consume the messages
        //
        int threadNumber = 0;
        for (final KafkaStream stream : streams) {
            executor.submit(new ConsumerTest(stream, threadNumber));
            threadNumber++;
        }
    }
	private static ConsumerConfig createConsumerConfig(String a_zookeeper, String a_groupId) {
        Properties props = new Properties();
        props.put("zookeeper.connect", a_zookeeper);
        props.put("group.id", a_groupId);
        props.put("zookeeper.session.timeout.ms", "40000");
        props.put("zookeeper.sync.time.ms", "2000");
        props.put("auto.commit.interval.ms", "1000");
 
        return new ConsumerConfig(props);
    }
	
	public static void main(String[] args) {
		if(args.length < 1){
			System.out.println("Please assign partition number.");
		}
		
        String zooKeeper = "10.206.216.13:12181,10.206.212.14:12181,10.206.209.25:12181";
        String groupId = "jikegrouptest";
        String topic = "jiketest";
        int threads = Integer.parseInt(args[0]);
 
		GroupConsumerTest example = new GroupConsumerTest(zooKeeper, groupId, topic);
        example.run(threads);
 
        try {
            Thread.sleep(Long.MAX_VALUE);
        } catch (InterruptedException ie) {
 
        }
        example.shutdown();
    }
}

ConsumerTest


package kafka.consumer.group;

import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
 
public class ConsumerTest implements Runnable {
    private KafkaStream m_stream;
    private int m_threadNumber;
 
    public ConsumerTest(KafkaStream a_stream, int a_threadNumber) {
        m_threadNumber = a_threadNumber;
        m_stream = a_stream;
    }
 
    public void run() {
        ConsumerIterator<byte[], byte[]> it = m_stream.iterator();
        while (it.hasNext()){
            System.out.println("Thread " + m_threadNumber + ": " + new String(it.next().message()));
			
        }
        System.out.println("Shutting down Thread: " + m_threadNumber);
    }
}

afka生产者编程模型

  • 同步生产模型

image

  • 异步生成模型

image

两种生产模型对比

同步生产模型:

  • 1、低消息丢失率；
  • 2、高消息重复率(由于网络原因，回复确认未收到)；
  • 3、高延迟

异步生产模型:

  • 1、低延迟;
  • 2、高发送性能；
  • 3、高消息丢失率(无确认机制，发送端队列满)

九、Kafka生产者编程模型-java实现

  • 已搭建好的kafka集群、Linux服务器一台、Apache Maven 3.2.3、 kafka 0.8.1

参数调优

  • message.send.max.retries: 发送失败重试次数;
  • retry.backoff.ms :未接到确认，认为发送失败的时间;
  • producer.type: 同步发送或者异步发送；
  • batch.num.messages: 异步发送时，累计最大消息数；
  • queue.buffering.max.ms:异步发送时，累计最大时间；
package kafka.producer.partiton;

import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;
 
public class SimplePartitioner implements Partitioner {
    public SimplePartitioner (VerifiableProperties props) {
 
    }
 
    public int partition(Object key, int a_numPartitions) {
        int partition = 0;
        String stringKey = (String) key;
        int offset = stringKey.lastIndexOf('.');
        if (offset > 0) {
           partition = Integer.parseInt( stringKey.substring(offset+1)) % a_numPartitions;
        }
       return partition;
  }
 
}


package kafka.producer.async;

import java.util.*;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;


public class ASyncProduce {
	public static void main(String[] args) {
        long events = Long.MAX_VALUE;
        Random rnd = new Random();
 
        Properties props = new Properties();
        props.put("metadata.broker.list", "10.206.216.13:19092,10.206.212.14:19092,10.206.209.25:19092");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
		//kafka.serializer.DefaultEncoder
        props.put("partitioner.class", "kafka.producer.partiton.SimplePartitioner");
		//kafka.producer.DefaultPartitioner: based on the hash of the key
        //props.put("request.required.acks", "1");
		props.put("producer.type", "async");
		//props.put("producer.type", "1");
		// 1: async 2: sync
 
        ProducerConfig config = new ProducerConfig(props);
 
        Producer<String, String> producer = new Producer<String, String>(config);
 
        for (long nEvents = 0; nEvents < events; nEvents++) { 
               long runtime = new Date().getTime();  
               String ip = "192.168.2." + rnd.nextInt(255); 
               String msg = runtime + ",www.example.com," + ip; 
               KeyedMessage<String, String> data = new KeyedMessage<String, String>("jiketest", ip, msg);
               producer.send(data);
			   try {
                   Thread.sleep(1000);
               } catch (InterruptedException ie) {
               }
        }
        producer.close();
    }
}


package kafka.producer.sync;
import java.util.*;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

public class SyncProduce {
	public static void main(String[] args) {
        long events = Long.MAX_VALUE;
        Random rnd = new Random();
 
        Properties props = new Properties();
        props.put("metadata.broker.list", "10.206.216.13:19092,10.206.212.14:19092,10.206.209.25:19092");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
		//kafka.serializer.DefaultEncoder
        props.put("partitioner.class", "kafka.producer.partiton.SimplePartitioner");
		//kafka.producer.DefaultPartitioner: based on the hash of the key
        props.put("request.required.acks", "1");
		//0;  绝不等确认  1:   leader的一个副本收到这条消息，并发回确认 -1：   leader的所有副本都收到这条消息，并发回确认
 
        ProducerConfig config = new ProducerConfig(props);
 
        Producer<String, String> producer = new Producer<String, String>(config);
 
        for (long nEvents = 0; nEvents < events; nEvents++) { 
               long runtime = new Date().getTime();  
               String ip = "192.168.2." + rnd.nextInt(255); 
               String msg = runtime + ",www.example.com," + ip; 
			   //eventKey必须有（即使自己的分区算法不会用到这个key，也不能设为null或者""）,否者自己的分区算法根本得不到调用
               KeyedMessage<String, String> data = new KeyedMessage<String, String>("jiketest", ip, msg);
			   												//			 eventTopic, eventKey, eventBody
               producer.send(data);
			   try {
                   Thread.sleep(1000);
               } catch (InterruptedException ie) {
               }
        }
        producer.close();
    }
}

十、Kafka生产者编程模型-python实现

需要的软件环境:

  • 已搭建好的kafka集群、Linux服务器一台、Python2.7.6 、kafka-Python软件包

参数调优

  • req_acks:发送失败重试次数;
  • ack_timeout: 未接到确认，认为发送失败的时间;
  • async : 是否异步发送；
  • batch_send_every_n: 异步发送时，累计最大消息数；
  • batch_send_every_t:异步发送时，累计最大时间；
import threading, time

from kafka.client import KafkaClient
from kafka.producer import SimpleProducer

class ASyncProducer(threading.Thread):
    daemon = True

    def run(self):
        client = KafkaClient("10.206.216.13:19092,10.206.212.14:19092,10.206.209.25:19092")
        producer = SimpleProducer(client,async=True)

        while True:
            producer.send_messages('jiketest', "test")
            producer.send_messages('jiketest', "test")

            time.sleep(1)

import threading, time

from kafka.client import KafkaClient
from kafka.producer import SimpleProducer
from kafka.partitioner import HashedPartitioner

class SyncProducer(threading.Thread):
    daemon = True

    def run(self):
        client = KafkaClient("10.206.216.13:19092,10.206.212.14:19092,10.206.209.25:1909")
        producer = SimpleProducer(client)
        #producer = KeyedProducer(client,partitioner=HashedPartitioner)

        while True:
            producer.send_messages('jiketest', "test")
            producer.send_messages('jiketest', "test")

            time.sleep(1)
chenshiying007
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Java实现Kafka生产者消费者代码实例
08-19
主要介绍了Java实现Kafka生产者消费者代码实例,文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值,需要的朋友可以参考下
Kafka 生产者和消费者实例
m0_57781407的博客
09-17 3676
基于命令行使用Kafka 类似scala，mysql等，命令行是初学者操作Kafka的基本方式，kafka的模式是生产者消费者模式，他们之间通讯是通过，一个公共频道完成。指定消费者组ID，在同一时刻同一消费组中只有一个线程可以去消费一个分区数据，不同的消费组可以去消费同一个分区的数据。(查看KafkaConsumerTest的运行框)由以下图可以看出生产者生产消息。在工程里面的pom.xml文件添加Kafka依赖。
Kafka的消费者API
张鑫的博客
08-15 4339
kafka官方文档API http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html 一、高级API kafka自己维护offset分区等等：创建和设置KafkaConsumer对象，记住要记得去订阅topic即调用subscribe方法 publi...
Kafka 实现之接口设计 —— 生产者 API、消费者 API
最新发布
流华追梦的专栏
03-13 855
Kafka 强大的应用程序层的基础是用于访问存储的两个基本 API，即用于写入事件的生产者 API和用于读取事件的消费者 API。在这两个 API 之上构建了用于集成和处理的 API。
kafka消费者简单样例
LLittle.Kevin的博客
04-28 1606
前言这段时间在做一个内部sql转换中间件项目．涉及到从kafka取消息．网上的简单样例大多是单线程的自动获取提交样例．在这里我需要手动提交消息．maven &lt;dependency&gt; &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt; &lt;artifactId&gt;kafka_2.11&lt;/artifactId&gt; ...
Kafka生产者与消费者api示例
m0_53400772的博客
06-04 1329
consumer的消费位移提交方式：全自动定时提交到consumer_offsets半自动然后手动触发提交 consumer.commitSync()；提交到consumer_offsets全手动写自己的代码去把消费位移保存到你自己的地方mysql/zk/redis/提交到自己所涉及的存储；初始化时也需要自己去从自定义存储中查询到消费位移。
消息中间件 - RabbitMQ
Crayon的博客
09-24 383
什么是MQ MQ全称为Message Queue，消息队列是应用程序和应用程序之间的通信方法。 有什么好处 1、异步处理 将不需要同步处理的并且耗时长的操作由消息队列通知消息接收方进行异步处理。提高了应用程序的响应时间。 ------提升用户体验和系统吞吐量 2、程序解耦 MQ相当于一个中介，生产方通过MQ与消费方交互，它将应用程序进行解耦合。 ------提高系统容错性和可维护性 3、削峰填谷 高并发下(1000qps) 通过一次性拉取指定数量(300)的Message 来处理。 --..
go-kafka-example:Golang Kafka消费者和生产者示例
05-06
示例Golang Kafka消费者和生产者 设置 制片人 go run cmd/producer/main.go 消费者 标志： brokerList 话题 分区（默认值：0） offsetType messageCountStart（默认值：0） go run cmd/consumer/main.go ...
kafka模拟生产者消费者(集群模式)实例
04-27
kafka模拟生产者、消费者，集群模式，若是单机版，将ip端口组改为相应ip端口即可；
kafka生产者-实例参数配置.md
07-24
kafka生产者参数配置使用于各种编程语言重要参数已标明kafka生产者参数配置使用于各种编程语言重要参数已标明
kafka实例 消费者生产者
11-03
kafka实例 消费者生产者，实现代码，可使用maven 下载包pom.xml
kafka,zookeeper,librdkafka(用于c++客户端的库)以及生产者消费者等的示例代码
10-26
kafka,zookeeper,librdkafka(用于c++客户端的库)以及生产者消费者等的示例代码。以上安装包均是目前为止最新的安装包，如果需要别的版本安装包，文件中赋有各安装包下载地址，以供下载，希望能够帮助到你
kafka-java-demo 基于java的kafka生产消费者示例
11-24
kafka-java-demo 基于java的kafka生产消费者示例。 mvn
Kafka 生产者和消费者入门代码基础
weixin_30247307的博客
03-15 218
这篇博文讲解Kafka的生产者和消费者实例。 基础版本一 生产者 ProducerFastStart.java package com.xingyun.tutorial_1; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer...
kafka-consumer-消费者代码实例
weixin_43119856的博客
11-25 2486
kafka-consumer-消费者代码实例
Kafka生产与消费--简单示例
wdquan19851029的专栏
05-03 2876
下面用java给出一个kafka生产与消费的简单示例： 运行环境： java：java version "1.8.0_291" kafka： <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>2.1.0</version> </dependency&
Kudu遇到的坑，是真的坑~
热门推荐
留歌__36的博客
09-25 1万+
写在前面 在我使用spark对接kudu的过程中，出现如下错误： Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (192.168.1.117:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableExce...
Kafka消费者和生产者实例的使用步骤
07-28
2. 创建Kafka生产者实例。可以使用以下代码创建一个KafkaProducer的实例： ```java @Configuration public class Config { public final static String bootstrapServers = "127.0.0.1:9092"; @Bean...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值