kafka作为常用的消息中间件,在各大企业得到广泛应用,笔者从今天开始将陆续对kafka 0.10.1进行源码分析。工欲善其事必先利其器,在阅读Kafka源码之前,首先第一步得搭建源码阅读环境,选择合适的工具可以使我们的学习效率更好,在这里我选用Intellij Idea 作为源码阅读环境的IDEA环境。
环境依赖
jdk
下载jdk 1.8,设置JAVA_HOME
java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
gradle
下载gradle 3.1.6,设置GRADLE_HOME
GRADLE_HOME=/Users/lidongmeng/software/gradle-6.1.1
PATH=$PATH:$GRADLE_HOME/bin
scala
下载scala-2.11.8 ,设置SCALA_HOME
SCALA_HOME=/Users/lidongmeng/software/scala-2.11.8
PATH=$PATH:$SCALA_HOME/bin
zookeeper
下载zookeeper 3.4.9: ,设置ZK_HOME
ZK_HOME=/Users/lidongmeng/software/zookeeper-3.4.9
PATH=$PATH:$ZK_HOME/bin
kafka源码
下载kafka-0.10.1源代码: http://kafka.apache.org/downloads
idea-scala插件
Idea -> preferences -> plugin -> scala
编译源码
构建
使用命令: gradle idea
构建项目,如果是eclipse使用gradle eclipse
> Task :idea
Generated IDEA project at file:///Users/lidongmeng/source_code_read/kafka-0.10.0.1-src/kafka-0.10.0.1-src.ipr
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.6.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 7m 16s
16 actionable tasks: 16 executed
构建错误及解决思路
- scalaBasePlugin缺失
// 问题
* Where:
Build file '/Users/lidongmeng/source_code_read/kafka-0.10.0.1-src/build.gradle' line: 233
* What went wrong:
A problem occurred evaluating root project 'kafka-0.10.0.1-src'.
> Failed to apply plugin [class 'org.gradle.api.plugins.scala.ScalaBasePlugin']
> Could not create task ':core:compileScala'.
> No such property: useAnt for class: org.gradle.api.tasks.scala.ScalaCompileOptions
// 解决方案
ScalaCompileOptions.metaClass.daemonServer = true
ScalaCompileOptions.metaClass.fork = true
ScalaCompileOptions.metaClass.useAnt = false
ScalaCompileOptions.metaClass.useCompileDaemon = false
- org.scoverage plugin出错
// 错误栈
* Where:
Build file '/Users/lidongmeng/source_code_read/kafka-0.10.0.1-src/build.gradle' line: 376
* What went wrong:
A problem occurred evaluating root project 'kafka-0.10.0.1-src'.
> Failed to apply plugin [id 'org.scoverage']
> Could not create an instance of type org.scoverage.ScoverageExtension.
> You can't map a property that does not exist: propertyName=testClassesDir
// 解决方案
classpath 'org.scoverage:gradle-scoverage:2.5.0' ## 将2.0.1修改为2.5.0
运行&验证
server启动
-
log4j文件
将config/log4j.properties文件放置/core/src/main/scala/目录下,并且添加
kafka.logs.dir=logs
配置 -
Server.log文件
将config/server.properties文件中的
log.dirs=xx/kafka-0.10.0.1-src/kafka-logs
更改到指定的文件夹中 -
Run配置
-
运行
[2020-02-24 14:48:11,176] INFO Kafka version : unknown (org.apache.kafka.common.utils.AppInfoParser) [2020-02-24 14:48:11,176] INFO Kafka commitId : unknown (org.apache.kafka.common.utils.AppInfoParser) [2020-02-24 14:48:11,177] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
创建topic
kafka_2.11-0.11.0.1/bin : ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
[2020-02-24 14:27:32,217] INFO Completed load of log test-0 with log end offset 0 (kafka.log.Log)
[2020-02-24 14:27:32,220] INFO Created log for partition [test,0] in /Users/lidongmeng/source_code_read/kafka-0.10.0.1-src/kafka-logs with properties {compression.type -> producer, message.format.version -> 0.10.0-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> true, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> delete, flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
producer & consumer发送接收消息
Producer:
kafka_2.11-0.11.0.1 : bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>afafdsafs
>this is an
Consumer:
kafka_2.11-0.11.0.1 : bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
afafdsafs
this is an