1. 检查service配置文件
修改参数 advertised.listeners=PLAINTEXT://tjtestrac1:9092
注意防火墙对端口的屏蔽
[kafka@tjtestrac1 config]$ vi server.properties
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
#listeners=PLAINTEXT:tjtestrac1:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://tjtestrac1:9092
2. 重启kafka 服务
这里采用最简单粗暴的方式关闭 生产千万不要这样
[kafka@tjtestrac1 config]$ jps
12048 Jps
30323 QuorumPeerMain
4739 Kafka
[kafka@tjtestrac1 config]$ kill -9 30323
[kafka@tjtestrac1 config]$ kill -9 4739
[kafka@tjtestrac1 config]$ jps
12727 Jps
确认关闭干净后 重新启动
[kafka@tjtestrac1 config]$ zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &
[1] 16757
[kafka@tjtestrac1 config]$ jps
17093 Jps
16757 QuorumPeerMain
[kafka@tjtestrac1 config]$ kafka-server-start.sh $KAFKA_HOME/config/server.properties &
[kafka@tjtestrac1 config]$ jps
19858 Kafka
16757 QuorumPeerMain
20600 Jps
2. Kafka producer 的体系结构和基本概念
Producer component
Producer 提供了很多不同场景下的 API 接口:
1. 对于传统的credit card 交易系统是不允许丢失数据以及产生错误的重复数据
1. 对于互联网数据用户行为跟踪等数据的采集,系统是可以容忍数据丢失以及错误数据的产生的
3. Eclipse 构建Kafka 项目 (Maven 构建)
a) 创建名为kafkaClient 的项目
b) 配置pom.xml 文件
https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.12/2.1.0
配置slf4j 服务
https://mvnrepository.com/artifact/org.slf4j/slf4j-api/1.8.0-beta2
https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12/1.8.0-beta2
https://mvnrepository.com/artifact/org.slf4j/slf4j-nop/1.8.0-beta2
https://mvnrepository.com/artifact/commons-logging/commons-logging
https://mvnrepository.com/artifact/org.slf4j/slf4j-simple/1.8.0-beta2
https://mvnrepository.com/artifact/org.slf4j/slf4j-jdk14/1.8.0-beta2
在pom 文件中加入
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
<modelVersion>4.0.0</modelVersion>
<groupId>KafkaClient</groupId>
<artifactId>KafkaClient</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.12</artifactId>
<version>2.1.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.8.0-beta2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.8.0-beta2</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-logging/commons-logging -->
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-nop -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.8.0-beta2</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.8.0-beta2</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-jdk14 -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.8.0-beta2</version>
<scope>test</scope>
</dependency>
</dependencies>
log4j.xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
</layout>
</appender>
<root>
<priority value ="debug" />
<appender-ref ref="console" />
</root>
</log4j:configuration>
c) 创建一个生产者的类 MsgSender
生产者几个配置的必要信息如下:
parameter name | parameter desc |
---|---|
bootstrap.servers | 配置连接Kafka 服务器的连接字符串信息 |
key.serializer | Key 的class: 接口org.apache.kafka.common.serialization.Serializer 的实现类 |
value.serializer | Value 的 class: 生产者指定固定的类的对象向broker 发送信息 |
java 代码:
import java.util.Properties;
import java.util.concurrent.Future;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class MsgSender {
/**
* @param args
*/
public static String TOPIC = "TestMsg";
public static void main(String[] args)throws Exception {
Properties kafkaProps = new Properties();
kafkaProps.setProperty("bootstrap.servers", "tjtestrac1:9092");
kafkaProps.setProperty("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
kafkaProps.setProperty("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<String, String>(kafkaProps);
ProducerRecord<String, String> record = new ProducerRecord<String, String>(TOPIC, "123123123",
"Welcome to my home!!! ");
try {
System.out.print("sending start..............");
Future future = producer.send(record);
future.get();
System.out.print("sending end..............");
} catch (Exception e) {
e.printStackTrace();
}finally {
producer.close();
}
}
}
控制台输出
INFO ProducerConfig - ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [tjtestrac1:9092]
buffer.memory = 33554432
client.dns.lookup = default
client.id =
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
登陆服务器 验证消息是否发送成功
[kafka@tjtestrac1 config]$ kafka-topics.sh --list --zookeeper localhost:2181
TestMsg
__consumer_offsets
test
[kafka@tjtestrac1 config]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TestMsg --from-beginning
Welcome to my home!!!
Welcome to my home!!!
Welcome to my home!!!