kafka 生产者和消费者实例

一、环境安装配置

1、去官网下载kafka

我下载的版本是 kafka_2.11-0.10.0.1.tgz，下面的实例也是基于该版本。

2、解压安装

tar -xzf kafka_2.11-0.10.0.1.tgz 
mv kafka_2.11-0.10.0.1    /root

3、修改配置文件

cd /root/kafka_2.11-0.10.0.1/config
cp server.properties server1.properties 
cp server.properties server2.properties 
cp server.properties server3.properties 

修改配置中的三个参数如下：

server1.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs-1

server2.properties

broker.id=2<pre name="code" class="plain"><span style="font-family: Arial, Helvetica, sans-serif;"></span><pre name="code" class="plain">listeners=PLAINTEXT://:9094

log.dirs=/tmp/kafka-logs-2

server3.properties

broker.id=3
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-3

4、启动脚本kafkacluster编写

bin/zookeeper-server-start.sh config/zookeeper.properties &
sleep 3s
bin/kafka-server-start.sh config/server1.properties &
bin/kafka-server-start.sh config/server2.properties &
bin/kafka-server-start.sh config/server3.properties &

启动kafka集群只需要执行./kafkacluster

另外附上一个kill kafka脚本

#!/bin/sh
PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}')

if [ -z "$PIDS" ]; then
    echo "No kafka server to stop"
else 
    for pid in $PIDS
    do  
        echo "$pid"
        kill -s 9 $pid
    done
fi

sleep 3s

PIDS=$(ps ax | grep java | grep -i QuorumPeerMain | grep -v grep | awk '{print $1}')

if [ -z "$PIDS" ]; then
    echo "No zookeeper server to stop"
    exit 1
else
    kill -s TERM $PIDS
fi

5、创建topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic test

二、代码编写

pom.xml

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<spring.version>3.2.14.RELEASE</spring.version>
		<kafka.version>0.10.0.1</kafka.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>log4j</groupId>
			<artifactId>log4j</artifactId>
			<version>1.2.17</version>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-log4j12</artifactId>
			<version>1.7.12</version>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-api</artifactId>
			<version>1.7.12</version>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>3.8.1</version>
			<scope>test</scope>
		</dependency>

		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-context</artifactId>
			<version>${spring.version}</version>
		</dependency>


		<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka_2.11</artifactId>
			<version>${kafka.version}</version>
			<scope>compile</scope>
			<exclusions>
				<exclusion>
					<artifactId>jmxri</artifactId>
					<groupId>com.sun.jmx</groupId>
				</exclusion>
				<exclusion>
					<artifactId>jms</artifactId>
					<groupId>javax.jms</groupId>
				</exclusion