快速启动
步骤1:下载代码
下载最近的一个稳定版本。> tar xzf kafka-<VERSION>.tgz > cd kafka-<VERSION> > ./sbt update > ./sbt package
步骤2:启动服务器
Kafka brokers and consumers use this for co-ordination.
bin/zookeeper-server-start.sh config/zookeeper.properties [2010-11-21 23:45:02,335] INFO Reading configuration from: config/zookeeper.properties "><b>>斌/ zookeeper-server-start.sh配置/ zookeeper.properties</b> [ 2010-11-21 23:45:02335 ]信息读取配置:配置/ zookeeper.properties …现在开始卡夫卡服务器:
> bin/kafka-server-start.sh config/server.properties jkreps-mn-2:kafka-trunk jkreps$ bin/kafka-server-start.sh config/server.properties [2010-11-21 23:51:39,608] INFO starting log cleaner every 60000 ms (kafka.log.LogManager) [2010-11-21 23:51:39,628] INFO connecting to ZK: localhost:2181 (kafka.server.KafkaZooKeeper) ...
步骤3:发送一些信息
卡夫卡是一个命令行客户端将从标准输入并把它作为消息的卡夫卡群。默认情况下,每个线将作为一个单独的消息发送。这个话题 测试时自动创建的消息发送给它的。省略记录你应该看到这样的事情:> bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test This is a message This is another message
步骤4:启动消费
kafka也有一个命令行的消费者,将把信息到标准输出。下面是一些很简单的用kafka发送消息的例子,更完整的例子可以在实例目录中找到kafka的源代码。> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning This is a message This is another message
生产者代码
生产API
Here are examples of using the producer API -kafka.producer.Producer<T>
-
- 首先,开始的ZooKeeper服务器本地实例
./bin/zookeeper-server-start.sh config/zookeeper.properties
- 接下来,开始kafka配置
./bin/kafka-server-start.sh config/server.properties
- 现在,创建所有配置默认发现使用ZooKeeper生产者。
- 使用异步生产者以及gzip压缩。这个缓冲区写入内存直到
batch.size
或queue.time
达到。然后,数据被发送到kafkaHadoop的消费者
聚合和数据加载到Hadoop提供横向扩展的解决方案是我们的一个基本用例。为了支持这个用例,我们提供了一个基于Hadoop的消费滋生了许多地图任务并行从卡夫卡群拉数据。这提供了非常快速的基于拉的Hadoop数据的负载能力(我们能够完全饱和,只有少数卡夫卡服务器网络)。
在Hadoop的用户使用信息可以发现在这里。
简单的消费
import kafka.api.FetchRequest; import kafka.javaapi.consumer.SimpleConsumer; import kafka.javaapi.message.ByteBufferMessageSet; import kafka.message.Message; import kafka.message.MessageSet; import kafka.utils.Utils; ...
// create a consumer to connect to the kafka server running on localhost, port 9092, socket timeout of 10 secs, socket receive buffer of ~1MB SimpleConsumer consumer = new SimpleConsumer("127.0.0.1", 9092, 10000, 1024000); long offset = 0; while (true) { // create a fetch request for topic “test”, partition 0, current offset, and fetch size of 1MB FetchRequest fetchRequest = new FetchRequest("test", 0, offset, 1000000); // get the message set from the consumer and print them out ByteBufferMessageSet messages = consumer.fetch(fetchRequest); for(MessageAndOffset msg : messages) { System.out.println("consumed: " + Utils.toString(msg.message.payload(), "UTF-8")); // advance the offset after consuming each message offset = msg.offset; } }
官方给的示例并不是很完整,以下代码是经过我补充的并且编译后能运行的。
Producer Code
import java.util.*; import kafka.message.Message; import kafka.producer.ProducerConfig; import kafka.javaapi.producer.Producer; import kafka.javaapi.producer.ProducerData;
public class ProducerSample {
public static void main(String[] args) { ProducerSample ps = new ProducerSample();
Properties props = new Properties(); props.put("zk.connect", "127.0.0.1:2181"); props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props); Producer<String, String> producer = new Producer<String, String>(config); ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message2"); producer.send(data); producer.close(); } }
Consumer Code
import java.nio.ByteBuffer; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import kafka.consumer.Consumer; import kafka.consumer.ConsumerConfig; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import kafka.message.Message; import kafka.message.MessageAndMetadata;
public class ConsumerSample {
public static void main(String[] args) { // specify some consumer properties Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("zk.connectiontimeout.ms", "1000000"); props.put("groupid", "test_group");
// Create the connection to the cluster ConsumerConfig consumerConfig = new ConsumerConfig(props); ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
// create 4 partitions of the stream for topic “test-topic”, to allow 4 threads to consume HashMap<String, Integer> map = new HashMap<String, Integer>(); map.put("test-topic", 4); Map<String, List<KafkaStream<Message>>> topicMessageStreams = consumerConnector.createMessageStreams(map); List<KafkaStream<Message>> streams = topicMessageStreams.get("test-topic");
// create list of 4 threads to consume from each of the partitions ExecutorService executor = Executors.newFixedThreadPool(4);
// consume the messages in the threads for (final KafkaStream<Message> stream : streams) { executor.submit(new Runnable() { public void run() { for (MessageAndMetadata msgAndMetadata : stream) { // process message (msgAndMetadata.message()) System.out.println("topic: " + msgAndMetadata.topic()); Message message = (Message) msgAndMetadata.message(); ByteBuffer buffer = message.payload(); <SPAN style="WHITE-SPACE: pre"> </SPAN>byte[] bytes = new byte[message.payloadSize()]; buffer.get(bytes); String tmp = new String(bytes); System.out.println("message content: " + tmp); } } }); }
} }
分别启动zookeeper,kafka server之后,依次运行Producer,Consumer的代码
运行ProducerSample:
运行ConsumerSample:
<trans data-src="" create="" a="" consumer="" to="" connect="" the="" kafka="" server="" running="" on="" localhost,="" port="" 9092,="" socket="" timeout="" of="" 10="" secs,="" receive="" buffer="" ~1mb="" simpleconsumer="" simpleconsumer("127.0.0.1",="" 10000,="" 1024000);"="" style="background: transparent;">