IntelliJ IDEA下构建kafka环境与源码编译

最新推荐文章于 2024-08-02 01:13:56 发布

&捕风的汉子&

最新推荐文章于 2024-08-02 01:13:56 发布

阅读量1.8k

点赞数

分类专栏： kafka scala

本文链接：https://blog.csdn.net/goodstudy168/article/details/89164446

版权

scala 同时被 2 个专栏收录

3 篇文章 0 订阅

订阅专栏

kafka

2 篇文章 0 订阅

订阅专栏

最近希望深入研究下kafka的架构，想了解架构必须去阅读其源码。于是自己动手在IDE下构建一个可运行可调试的kafka环境，下面是构建kafka环境及编译源码。

一、安装jdk

C:\Users\Administrator>java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

二、安装scala

C:\Users\Administrator>scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

三、安装gradle

E:\workspace\javaworkspace\kafka-1.1.1-src\kafka-1.1.1-src>gradle --version

------------------------------------------------------------
Gradle 5.3-rc-2
------------------------------------------------------------

Build time:   2019-03-11 21:07:26 UTC
Revision:     cb19a113017a9351f8d9fa767d50a9a0f590617a

Kotlin:       1.3.21
Groovy:       2.5.4
Ant:          Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM:          1.8.0_172 (Oracle Corporation 25.172-b11)
OS:           Windows 7 6.1 amd64

四、安装mvn

C:\Users\Administrator>mvn -version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:1
4+08:00)
Maven home: D:\Program Files\apache-maven-3.5.4\bin\..
Java version: 1.8.0_172, vendor: Oracle Corporation, runtime: D:\Program Files\J
ava\jdk1.8.0_172\jre
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"

五、将kafka生成idea项目

E:\workspace\javaworkspace\kafka-1.1.1-src\kafka-1.1.1-src>gradle idea

> Configure project :
Building project 'core' with Scala version 2.11.12
<-------------> 2% EXECUTING [22m 54s]
> IDLE
> :ideaProject > Resolve files of :core:detachedConfiguration5 > scala-compiler
> IDLE
> IDLE

> Task :idea
Generated IDEA project at file:///E:/workspace/javaworkspace/kafka-1.1.1-src/kaf
ka-1.1.1-src/kafka-1.1.1-src.ipr

Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.3-rc-2/userguide/command_line_interface.html#sec:c
ommand_line_warnings

BUILD SUCCESSFUL in 1h 50m 31s
24 actionable tasks: 24 executed

六、安装zookeeper

我下载的zookeeper版本为apache-zookeeper-3.5.5-bin.tar.gz，将安装包解压到本地目录，然后调整配置文件，将zoo_sample.cfg复制一份为zoo.cfg，内容如下：

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=D:/APP/apache-zookeeper-3.5.5-bin/data
# the port at which the clients will connect
clientPort=12181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=127.0.0.1:12888:13888

在D:/APP/apache-zookeeper-3.5.5-bin/data目录下增加一个myid的文件，然后内容填写"1"，此是为了构建zookeeper集群时同步提供区分节点的id，此处我们只是单点，并没有部署集群。

启动zookeeper服务，执行zkServer.cmd

D:\APP\apache-zookeeper-3.5.5-bin\bin>zkServer.cmd &

D:\APP\apache-zookeeper-3.5.5-bin\bin>call "D:\APP\jdk1.8.0_101"\bin\java "-Dzookeeper.log.dir=D:\APP\apache-zookeeper-3.5.5-bin\bin\..\logs" "-Dzookeeper.root.logger=INFO,CONSOLE" "-Dzookeeper.log.file=zookeeper-linxiaojie-server-LINXIAOJIE.log" "-XX:+HeapDumpOnOutOfMemoryError" "-XX:OnOutOfMemoryError=cmd /c taskkill /pid %%p /t /f" -cp "D:\APP\apache-zookeeper-3.5.5-bin\bin\..\build\classes;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\build\lib\*;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\*;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\*;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\APP\apache-zookeeper-3.5.5-bin\bin\..\conf\zoo.cfg"
2019-08-12 17:00:09,254 [myid:] - INFO  [main:QuorumPeerConfig@133] - Reading configuration from: D:\APP\apache-zookeeper-3.5.5-bin\bin\..\conf\zoo.cfg
2019-08-12 17:00:09,263 [myid:] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:12181
2019-08-12 17:00:09,263 [myid:] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
2019-08-12 17:00:09,273 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-08-12 17:00:09,275 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-08-12 17:00:09,276 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-08-12 17:00:09,276 [myid:1] - WARN  [main:QuorumPeerMain@125] - Either no config or no quorum defined in config, running  in standalone mode
2019-08-12 17:00:09,278 [myid:1] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-08-12 17:00:09,337 [myid:1] - INFO  [main:QuorumPeerConfig@133] - Reading configuration from: D:\APP\apache-zookeeper-3.5.5-bin\bin\..\conf\zoo.cfg
2019-08-12 17:00:09,338 [myid:1] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:12181
2019-08-12 17:00:09,341 [myid:1] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
2019-08-12 17:00:09,346 [myid:1] - INFO  [main:ZooKeeperServerMain@117] - Starting server
2019-08-12 17:00:18,387 [myid:1] - INFO  [main:Environment@109] - Server environment:zookeeper.version=3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT
2019-08-12 17:00:18,388 [myid:1] - INFO  [main:Environment@109] - Server environment:host.name=linxiaojie
2019-08-12 17:00:18,398 [myid:1] - INFO  [main:Environment@109] - Server environment:java.version=1.8.0_101
2019-08-12 17:00:18,404 [myid:1] - INFO  [main:Environment@109] - Server environment:java.vendor=Oracle Corporation
2019-08-12 17:00:18,413 [myid:1] - INFO  [main:Environment@109] - Server environment:java.home=D:\APP\jdk1.8.0_101\jre
2019-08-12 17:00:18,414 [myid:1] - INFO  [main:Environment@109] - Server environment:java.class.path=D:\APP\apache-zookeeper-3.5.5-bin\bin\..\build\classes;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\build\lib\*;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\*;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\audience-annotations-0.5.0.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\commons-cli-1.2.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jackson-annotations-2.9.0.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jackson-core-2.9.8.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jackson-databind-2.9.8.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\javax.servlet-api-3.1.0.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jetty-http-9.4.17.v20190418.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jetty-io-9.4.17.v20190418.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jetty-security-9.4.17.v20190418.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jetty-server-9.4.17.v20190418.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jetty-servlet-9.4.17.v20190418.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jetty-util-9.4.17.v20190418.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\jline-2.11.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\json-simple-1.1.1.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\log4j-1.2.17.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\netty-all-4.1.29.Final.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\slf4j-api-1.7.25.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\slf4j-log4j12-1.7.25.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\zookeeper-3.5.5.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\lib\zookeeper-jute-3.5.5.jar;D:\APP\apache-zookeeper-3.5.5-bin\bin\..\conf
2019-08-12 17:00:18,416 [myid:1] - INFO  [main:Environment@109] - Server environment:java.library.path=D:\APP\jdk1.8.0_101\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;D:\APP\Anaconda3;D:\APP\Anaconda3\Library\mingw-w64\bin;D:\APP\Anaconda3\Library\usr\bin;D:\APP\Anaconda3\Library\bin;D:\APP\Anaconda3\Scripts;C:\Perl64\site\bin;C:\Perl64\bin;D:\app\linxiaojie\product\11.2.0\dbhome_1\bin;D:\APP\jdk1.8.0_101\bin\;D:\APP\jdk1.8.0_101\lib\;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\MySQL\MySQL Fabric 1.5 & MySQL Utilities 1.5\;C:\Program Files (x86)\MySQL\MySQL Fabric 1.5 & MySQL Utilities 1.5\Doctrine extensions for PHP\;C:\Program Files\CMake\bin;C:\Program Files\Microsoft Network Monitor 3\;D:\APP\Apache Software Foundation\Tomcat 8.0\bin;.;D:\APP\scala\bin;D:\APP\Git\cmd;C:\Go\bin;D:\APP\hadoop-common-2.2.0\bin;C:\Program Files\TortoiseSVN\bin;D:\APP\MATLAB\R2017b\runtime\win64;D:\APP\MATLAB\R2017b\bin;C:\WINDOWS\System32\OpenSSH\;D:\APP\apache-maven-3.5.4\bin;D:\APP\gradle-3.5.1-bin\gradle-3.5.1\bin;C:\Users\linxiaojie\AppData\Local\Microsoft\WindowsApps;D:\APP\Nmap;C:\Users\linxiaojie\AppData\Local\GitHubDesktop\bin;C:\Users\linxiaojie\AppData\Local\Programs\Fiddler;D:\APP\Microsoft VS Code\bin;.
2019-08-12 17:00:18,417 [myid:1] - INFO  [main:Environment@109] - Server environment:java.io.tmpdir=C:\Users\LINXIA~1\AppData\Local\Temp\
2019-08-12 17:00:18,424 [myid:1] - INFO  [main:Environment@109] - Server environment:java.compiler=<NA>
2019-08-12 17:00:18,427 [myid:1] - INFO  [main:Environment@109] - Server environment:os.name=Windows 10
2019-08-12 17:00:18,428 [myid:1] - INFO  [main:Environment@109] - Server environment:os.arch=amd64
2019-08-12 17:00:18,428 [myid:1] - INFO  [main:Environment@109] - Server environment:os.version=10.0
2019-08-12 17:00:18,429 [myid:1] - INFO  [main:Environment@109] - Server environment:user.name=linxiaojie
2019-08-12 17:00:18,429 [myid:1] - INFO  [main:Environment@109] - Server environment:user.home=C:\Users\linxiaojie
2019-08-12 17:00:18,429 [myid:1] - INFO  [main:Environment@109] - Server environment:user.dir=D:\APP\apache-zookeeper-3.5.5-bin\bin
2019-08-12 17:00:18,432 [myid:1] - INFO  [main:Environment@109] - Server environment:os.memory.free=358MB
2019-08-12 17:00:18,437 [myid:1] - INFO  [main:Environment@109] - Server environment:os.memory.max=5440MB
2019-08-12 17:00:18,437 [myid:1] - INFO  [main:Environment@109] - Server environment:os.memory.total=368MB
2019-08-12 17:00:18,439 [myid:1] - INFO  [main:ZooKeeperServer@938] - minSessionTimeout set to 4000
2019-08-12 17:00:18,439 [myid:1] - INFO  [main:ZooKeeperServer@947] - maxSessionTimeout set to 40000
2019-08-12 17:00:18,441 [myid:1] - INFO  [main:ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir D:\APP\apache-zookeeper-3.5.5-bin\data\version-2 snapdir D:\APP\apache-zookeeper-3.5.5-bin\data\version-2
2019-08-12 17:00:18,484 [myid:1] - INFO  [main:Log@193] - Logging initialized @9393ms to org.eclipse.jetty.util.log.Slf4jLog
2019-08-12 17:00:18,625 [myid:1] - WARN  [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@1794d431{/,null,UNAVAILABLE} contextPath ends with /*
2019-08-12 17:00:18,626 [myid:1] - WARN  [main:ContextHandler@1599] - Empty contextPath
2019-08-12 17:00:18,648 [myid:1] - INFO  [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_101-b13
2019-08-12 17:00:18,696 [myid:1] - INFO  [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0
2019-08-12 17:00:18,697 [myid:1] - INFO  [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaults
2019-08-12 17:00:18,709 [myid:1] - INFO  [main:HouseKeeper@149] - node0 Scavenging every 660000ms
2019-08-12 17:00:18,720 [myid:1] - INFO  [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@1794d431{/,null,AVAILABLE}
2019-08-12 17:00:18,828 [myid:1] - INFO  [main:AbstractConnector@292] - Started ServerConnector@50f8360d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2019-08-12 17:00:18,855 [myid:1] - INFO  [main:Server@410] - Started @9770ms
2019-08-12 17:00:18,858 [myid:1] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-08-12 17:00:18,868 [myid:1] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-08-12 17:00:18,872 [myid:1] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2019-08-12 17:00:18,876 [myid:1] - INFO  [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:12181
2019-08-12 17:00:18,915 [myid:1] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-08-12 17:00:18,917 [myid:1] - INFO  [main:FileTxnSnapLog@372] - Snapshotting: 0x0 to D:\APP\apache-zookeeper-3.5.5-bin\data\version-2\snapshot.0
2019-08-12 17:00:18,921 [myid:1] - INFO  [main:FileTxnSnapLog@372] - Snapshotting: 0x0 to D:\APP\apache-zookeeper-3.5.5-bin\data\version-2\snapshot.0
2019-08-12 17:00:18,940 [myid:1] - INFO  [main:ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000

七、配置Kafka

将server.properties中的log.dirs修改为本地磁盘。

log.dirs=D:/tmp/kafka-logs

将config目录下的log4j.properties文件拷贝到core\src\main\scala\目录下，方便我们后面查看日志。其他地方暂时不需要变更，毕竟此项目是为了进一步研究kafka运行原理作用的，主要kafka能够跑起来，我们可以调试代码即可。

八、启动kafka

配置启动参数：

然后执行启动，我们来确认下服务是否已启动：

C:\Users\Administrator>netstat -ano | findstr "9092"
  TCP    0.0.0.0:9092           0.0.0.0:0              LISTENING       31380
  TCP    192.168.211.1:1172     192.168.211.1:9092     ESTABLISHED     31380
  TCP    192.168.211.1:9092     192.168.211.1:1172     ESTABLISHED     31380
  TCP    [::]:9092              [::]:0                 LISTENING       31380

从上面看到kafka绑定的端口已经启动了，说明kafka broker已运行。调试就是后续事情啦。