Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十七）：kafka manager安装

最新推荐文章于 2020-12-28 16:01:31 发布
最新推荐文章于 2020-12-28 16:01:31 发布
阅读量701 收藏
点赞数
文章标签： scala 数据库 运维

一、kafka-manager简介

       为了简化开发者和服务工程师维护Kafka集群的工作，yahoo构建了一个叫做Kafka管理器的基于Web工具，叫做 Kafka Manager。这个管理工具可以很容易地发现分布在集群中的哪些topic分布不均匀，或者是分区在整个集群分布不均匀的的情况。它支持管理多个集群、选择副本、副本重新分配以及创建Topic。同时，这个管理工具也是一个非常好的可以快速浏览这个集群的工具，有如下功能：

  • 管理多个集群
  • 轻松检查群集状态（主题，消费者，偏移，代理，副本分发，分区分发）
  • 运行首选副本选举
  • 使用选项生成分区分配以选择要使用的代理
  • 运行分区重新分配（基于生成的分配）
  • 使用可选主题配置创建主题（0.8.1.1具有与0.8.2+不同的配置）
  • 删除主题（仅支持0.8.2+并记住在代理配​​置中设置delete.topic.enable = true）
  • 主题列表现在指示标记为删除的主题（仅支持0.8.2+）
  • 批量生成多个主题的分区分配，并可选择要使用的代理
  • 批量运行重新分配多个主题的分区
  • 将分区添加到现有主题
  • 更新现有主题的配置
  • kafka-manager 项目地址：https://github.com/yahoo/kafka-manager/

二、kafka-manager安装

1.下载安装包

使用Git或者直接从Releases中下载，这里下载 1.3.3.18 版本：https://github.com/yahoo/kafka-manager/releases

 wget https://github.com/yahoo/kafka-manager/archive/1.3.3.18.zip

2.解压安装包

Last login: Thu Sep  6 20:51:48 2018 from 192.168.0.103
[spark@master ~]$ cd /opt/
[spark@master opt]$ wget https://github.com/yahoo/kafka-manager/archive/1.3.3.18.zip
[spark@master opt]$ ll
total 1076
drwxr-xr-x.  6 root   root        99 Jul  5 07:00 apache-maven-3.5.4
drwxr-xr-x.  9 spark  spark      155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark     4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark      172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root       153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143     255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root       101 Jul 17 07:01 kafka_2.11-1.1.0
-rw-r--r--.  1 root   root   1075857 Sep 11 21:32 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark      232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root      9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark      252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark       67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark     4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000      50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop     236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark     4096 Aug  8 00:23 zookeeper-3.4.12
[spark@master opt]$ su root
Password: 
[root@master opt]# unzip kafka-manager-1.3.3.18.zip 
Archive:  kafka-manager-1.3.3.18.zip
8dcdbf8fabb0001691c9b52b447b656f498b4d7b
   creating: kafka-manager-1.3.3.18/
   。。。。
  inflating: kafka-manager-1.3.3.18/test/kafka/test/SeededBroker.scala  
[root@master opt]# ls
apache-maven-3.5.4         jdk1.8.0_171                mysql57-community-release-el7-8.noarch.rpm  spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2        kafka_2.11-1.1.0            nifi-1.7.1                                  zookeeper-3.4.12
elasticsearch-head-master  kafka-manager-1.3.3.18      node-8.9.1
hadoop-2.9.0               kafka-manager-1.3.3.18.zip  node-v8.9.1
hdfs-over-ftp-master       kibana-6.2.2-linux-x86_64   scala-2.11.0

3.sbt编译

1）yum安装sbt(因为kafka-manager需要sbt编译)

[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ls
app  build.sbt  conf  img  LICENCE  project  public  README.md  sbt  src  test
[root@master kafka-manager-1.3.3.18]# sbt
bash: sbt: command not found
[root@master kafka-manager-1.3.3.18]# cd ..
[root@master opt]#  curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   160    0   160    0     0     21      0 --:--:--  0:00:07 --:--:--    40
[root@master opt]# ll
total 1080
drwxr-xr-x.  6 root   root        99 Jul  5 07:00 apache-maven-3.5.4
-rw-r--r--.  1 root   root       160 Sep 11 21:38 bintray-sbt-rpm.repo
drwxr-xr-x.  9 spark  spark      155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark     4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark      172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root       153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143     255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root       101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x.  9 root   root       189 Jul  7 04:44 kafka-manager-1.3.3.18
-rw-r--r--.  1 root   root   1075857 Sep 11 21:32 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark      232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root      9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark      252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark       67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark     4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000      50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop     236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark     4096 Aug  8 00:23 zookeeper-3.4.12
[root@master opt]# mv bintray-sbt-rpm.repo /etc/yum.repos.d/
[root@master opt]#  yum install sbt
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirrors.zju.edu.cn
 * extras: mirrors.shu.edu.cn
 * updates: mirrors.zju.edu.cn
base                                                                                                         | 3.6 kB  00:00:00     
bintray--sbt-rpm                                                                                             | 1.3 kB  00:00:00     
extras                                                                                                       | 3.4 kB  00:00:00     
mysql-connectors-community                                                                                   | 2.5 kB  00:00:00     
mysql-tools-community                                                                                        | 2.5 kB  00:00:00     
mysql57-community                                                                                            | 2.5 kB  00:00:00     
updates                                                                                                      | 3.4 kB  00:00:00     
(1/4): extras/7/x86_64/primary_db                                                                            | 187 kB  00:00:00     
(2/4): mysql-connectors-community/x86_64/primary_db                                                          |  26 kB  00:00:00     
(3/4): bintray--sbt-rpm/primary                                                                              | 3.8 kB  00:00:02     
(4/4): updates/7/x86_64/primary_db                                                                           | 5.2 MB  00:00:02     
bintray--sbt-rpm                                                                                                              32/32
Resolving Dependencies
--> Running transaction check
---> Package sbt.noarch 0:1.2.1-0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================
 Package                   Arch                         Version                        Repository                              Size
====================================================================================================================================
Installing:
 sbt                       noarch                       1.2.1-0                        bintray--sbt-rpm                       1.1 M

Transaction Summary
====================================================================================================================================
Install  1 Package

Total download size: 1.1 M
Installed size: 1.2 M
Is this ok [y/d/N]: y
Downloading packages:
sbt-1.2.1.rpm                                                                                                | 1.1 MB  00:00:19     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : sbt-1.2.1-0.noarch                                                                                               1/1 
  Verifying  : sbt-1.2.1-0.noarch                                                                                               1/1 

Installed:
  sbt.noarch 0:1.2.1-0                                                                                                              

Complete!

改仓库地址：（sbt 默认下载库文件很慢, 还时不时被打断），我们可以在用户目录下创建 touch ~/.sbt/repositories, 填上阿里云的镜像   # vi ~/.sbt/repositories  

[root@master opt]# ls
apache-maven-3.5.4         jdk1.8.0_171                mysql57-community-release-el7-8.noarch.rpm  spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2        kafka_2.11-1.1.0            nifi-1.7.1                                  zookeeper-3.4.12
elasticsearch-head-master  kafka-manager-1.3.3.18      node-8.9.1
hadoop-2.9.0               kafka-manager-1.3.3.18.zip  node-v8.9.1
hdfs-over-ftp-master       kibana-6.2.2-linux-x86_64   scala-2.11.0
[root@master opt]# sbt
Getting org.scala-sbt sbt 1.2.1  (this may take some time)...
^C[root@master opt]# sbt -version
Getting org.scala-sbt sbt 1.2.1  (this may take some time)...
^C[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ./sbt clean dist
Downloading sbt launcher for 0.13.9:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar
    To  /root/.sbt/launchers/0.13.9/sbt-launch.jar
Download failed. Obtain the jar manually and place it at /root/.sbt/launchers/0.13.9/sbt-launch.jar
[root@master kafka-manager-1.3.3.18]# vi ~/.sbt/repositories

[repositories]
  local
  aliyun-nexus: http://maven.aliyun.com/nexus/content/groups/public/
  jcenter: https://jcenter.bintray.com/
  typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[c
lassifier]).[ext], bootOnly
  maven-central
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~

2）编译kafka-manager

[root@master kafka-manager-1.3.3.18]# ./sbt clean dist
Downloading sbt launcher for 0.13.9:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar
    To  /root/.sbt/launchers/0.13.9/sbt-launch.jar
[info] Loading project definition from /opt/kafka-manager-1.3.3.18/project
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[info] Set current project to kafka-manager (in build file:/opt/kafka-manager-1.3.3.18/)
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[warn] Credentials file /root/.bintray/.credentials does not exist
[success] Total time: 0 s, completed Sep 11, 2018 10:54:50 PM
。。。。。
[info]  [SUCCESSFUL ] jline#jline;2.12.1!jline.jar (340ms)
[info] Done updating.
[warn] Scala version was updated by one of library dependencies:
[warn]  * org.scala-lang:scala-library:(2.11.6, 2.11.8, 2.11.11, 2.11.7, 2.11.5, 2.11.0) -> 2.11.12
[warn] To force scalaVersion, add the following:
[warn]  ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) }
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * org.webjars:jquery:1.11.1 -> 2.1.4
[warn] Run 'evicted' to see detailed eviction warnings
[info] Wrote /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18.pom
[info] Compiling 127 Scala sources and 2 Java sources to /opt/kafka-manager-1.3.3.18/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info] Main Scala API documentation to /opt/kafka-manager-1.3.3.18/target/scala-2.11/api...
[info]   Compilation completed in 8.323 s
model contains 701 documentable templates
[info] Main Scala API documentation successful.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-javadoc.jar ...
[info] Done packaging.
[info] LESS compiling on 1 source(s)
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-web-assets.jar ...
[info] Done packaging.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18.jar ...
[info] Done packaging.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-sans-externalized.jar ...
[info] Done packaging.
[info] 
[info] Your package is ready in /opt/kafka-manager-1.3.3.18/target/universal/kafka-manager-1.3.3.18.zip
[info] 
[success] Total time: 376 s, completed Sep 11, 2018 11:01:06 PM
[root@master kafka-manager-1.3.3.18]# ls
app  build.sbt  conf  img  LICENCE  project  public  README.md  sbt  src  target  test
[root@master kafka-manager-1.3.3.18]# cd target/universal
[root@master universal]# ls
kafka-manager-1.3.3.18.zip  scripts
[root@master universal]# scp kafka-manager-1.3.3.18.zip /opt/kafka-manager-1.3.3.18.zip

4.安装

重新解压编译好的kafka-manager-1.3.3.18.zip&修改配置文件

[root@master universal]# cd /opt/
[root@master opt]# ls
apache-maven-3.5.4         hadoop-2.9.0          kafka_2.11-1.1.0            kibana-6.2.2-linux-x86_64                   node-8.9.1    spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2        hdfs-over-ftp-master  kafka-manager-1.3.3.18      mysql57-community-release-el7-8.noarch.rpm  node-v8.9.1   zookeeper-3.4.12
elasticsearch-head-master  jdk1.8.0_171          kafka-manager-1.3.3.18.zip  nifi-1.7.1                                  scala-2.11.0
[root@master opt]# mv kafka-manager-1.3.3.18 kafka-manager-1.3.3.18-source
[root@master opt]# ll
total 77696
drwxr-xr-x.  6 root   root         99 Jul  5 07:00 apache-maven-3.5.4
drwxr-xr-x.  9 spark  spark       155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark      4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark       172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root        153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143      255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root        101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x. 10 root   root        203 Sep 11 22:54 kafka-manager-1.3.3.18-source
-rw-r--r--.  1 root   root   79534088 Sep 11 23:01 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark       232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root       9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark       252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark        67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark      4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000       50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop      236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark      4096 Aug  8 00:23 zookeeper-3.4.12
[root@master opt]# unzip kafka-manager-1.3.3.18.zip 
Archive:  kafka-manager-1.3.3.18.zip
  inflating: kafka-manager-1.3.3.18/lib/kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar  
  。。。。
  inflating: kafka-manager-1.3.3.18/share/doc/api/index/index-d.html  
  inflating: kafka-manager-1.3.3.18/README.md  
[root@master opt]# ll
total 77696
drwxr-xr-x.  6 root   root         99 Jul  5 07:00 apache-maven-3.5.4
drwxr-xr-x.  9 spark  spark       155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark      4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark       172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root        153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143      255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root        101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x.  6 root   root         70 Sep 11 23:02 kafka-manager-1.3.3.18
drwxr-xr-x. 10 root   root        203 Sep 11 22:54 kafka-manager-1.3.3.18-source
-rw-r--r--.  1 root   root   79534088 Sep 11 23:01 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark       232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root       9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark       252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark        67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark      4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000       50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop      236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark      4096 Aug  8 00:23 zookeeper-3.4.12
[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ls
bin  conf  lib  README.md  share
[root@master kafka-manager-1.3.3.18]# cd conf/
[root@master conf]# ls
application.conf  consumer.properties  logback.xml  logger.xml  routes
[root@master conf]# vim application.conf 

# Secret key
# ~~~~~
# The secret key is used to secure cryptographics functions.
# If you deploy your application to several instances be sure to use the same key!
play.crypto.secret="^<csmm5Fx4d=r2HEX8pelM3iBkFVv?k[mc;IZE<_Qoq8EkX_/7@Zt6dP05Pzea3U"
play.crypto.secret=${?APPLICATION_SECRET}

# The application languages
# ~~~~~
play.i18n.langs=["en"]

play.http.requestHandler = "play.http.DefaultHttpRequestHandler"
play.http.context = "/"
play.application.loader=loader.KafkaManagerLoader

#kafka-manager.zkhosts="kafka-manager-zookeeper:2181"
kafka-manager.zkhosts="192.168.0.120:2181,192.168.0.121:2181,192.168.0.122:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
pinned-dispatcher.type="PinnedDispatcher"
pinned-dispatcher.executor="thread-pool-executor"
application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFea
ture"]

akka {
  loggers = ["akka.event.slf4j.Slf4jLogger"]
  loglevel = "INFO"
}

akka.logger-startup-timeout = 60s

basicAuthentication.enabled=false
basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED}
basicAuthentication.username="admin"
basicAuthentication.username=${?KAFKA_MANAGER_USERNAME}
basicAuthentication.password="password"
basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}
basicAuthentication.realm="Kafka-Manager"
basicAuthentication.excluded=["/api/health"] # ping the health of your instance without authentification

kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE}
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
"application.conf" 46L, 1682C written

5.启动服务

启动zk集群，kafka集群，再启动kafka-manager服务。

bin/kafka-manager 默认的端口是9000，可通过 -Dhttp.port，指定端口; -Dconfig.file=conf/application.conf指定配置文件:

[root@master conf]# cd ..
[root@master kafka-manager-1.3.3.18]# ls
bin  conf  lib  README.md  share
[root@master kafka-manager-1.3.3.18]# 
[root@master kafka-manager-1.3.3.18]# nohup bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=19093 &

jps查看：

[root@master spark]# jps
4596 QuorumPeerMain
4956 Kafka
4476 ProdServerStart
5116 Jps
[root@master spark]#

WebUI查看：http://192.168.0.120:19093/ 出现如下界面则启动成功。

其他就是在ui上新建cluster，新建topic，查看topic等操作。

 

参考：

http://www.cnblogs.com/frankdeng/p/9584870.html

https://www.cnblogs.com/dadonggg/p/8205302.html

 

weixin_34212189
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Kafka：ZK+Kafka+Spark Streaming集群环境搭建（一）VMW安装四台CentOS，并实现本机与它们能交互，虚拟机内部实现可以上网。...
weixin_34331102的博客
06-26 156
使用VMW安装四台CentOS-7-x86_64-DVD-1804.iso虚拟机： 计划配置三台centos虚拟机： master：192.168.0.120 slave1：192.168.0.121 slave2：192.168.0.122slave3：192.168.0.123 资源分配情况： 每台虚拟机分配：内存2G，CPU核数1个，磁盘20G。 VMW配置及本地IPv...
Kafka：ZK+Kafka+Spark Streaming集群环境搭建（四）针对hadoop2.9.0启动执行start-all.sh出现异常：failed to launch: nice -n 0...
weixin_34258838的博客
06-30 186
启动问题： 执行start-all.sh出现以下异常信息： failed to launch: nice -n 0 /bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master:7077 解决方案： Just added export JAVA_HOME=/usr/ja...
Kafka监控：Kafka-Manager
weixin_33762321的博客
05-03 346
2019独角兽企业重金招聘Python工程师标准>>> ...
解决jar包下载不下来的问题
分享日常bug
12-28 991
mvn install:install-file -Dfile=需要操作的jar包地址 -DgroupId=junit -DartifactId=junit -Dversion=4.12 -Dpackaging=jar -DgeneratePom=true -DcreateChecksum=true cmd运行上面配置好的代码！！！需要配置maven
（一）离线安装部署kafka（zookeeper、kafka集群与伪集群）
不知不觉地……
02-20 1914
目录 写在前面 1.zookeeper集群安装 2.zookeeper伪分布式安装（在同一台机器安装不同实例） 3.kafka集群安装 4.kafka伪分布式安装 5.Java API 6.关于kafka相关文章 写在前面 (1)本例使用三台服务器(VMware虚拟机搭建，centos7)，已在hosts中配置主机与IP映射 主机 IP 部署内容 my108 ...
Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+ES+Redash等详细安装部署
06-22
在大数据领域，构建一个完整的生态系统是至关重要的，其中包括多个组件，如Hadoop、Spark、Hive、HBase、Oozie、Kafka、Flume、Flink、Elasticsearch和Redash。这些组件协同工作，提供了数据存储、处理、调度、流...
使用Flume+Logstash+Kafka+Spark Streaming进行实时日志处理分析【大数据】
10-29
flume+Logstash+Kafka+Spark Streaming进行实时日志处理分析【大数据】
Spark踩坑记——SparkStreaming+Kafka
02-25
本文首先对sparkstreaming嵌入kafka的方式进行归纳总结，之后简单阐述Sparkstreaming+kafka在舆情项目中的应用，最后将自己在SparkStreaming+kafka的实际优化中的一些经验进行归纳总结。（如有任何纰漏欢迎补充来踩...
SparkStreaming：Spark Streaming + Flume + Kafka + HBase + Hadoop + Zookeeper实现实时日志分析统计； SpringBoot + Echarts实现数据可视化展示
02-05
/ class”开头的表示实战课程，然后通过流水线Flume + Kafka + SparkStreaming进行实时日志的收集，HBase来存储数据）*注意事项（使用的软件工具及环境配置） hadoop-2.6.0-cdh5.7.0 hbase-1.2.0-cdh5.7.0 zookeeper...
aarch64架构（ARMv8）系统环境下编译 Ambari
Yore - Home
05-20 8140
本文主要介绍了在 aarch64架构系统（ARM v8）下的 Ambari和 HDP的移植，重点对 Ambari 的编译及中间出现的问题的解决方法，并提供了最终的华为官方的镜像资源包的下载连接。
Kafka：ZK+Kafka+Spark Streaming集群环境搭建（九）安装kafka_2.11-1.1.0
weixin_33859665的博客
06-30 279
如何搭建配置centos虚拟机请参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（一）VMW安装四台CentOS，并实现本机与它们能交互，虚拟机内部实现可以上网。》 如何安装hadoop2.9.0请参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二）安装hadoop2.9.0》 如何配置hadoop2.9.0 HA 请参考《Kafk...
sbt编译kafka-manager项目的苦难经历
u013905744的专栏
12-14 2676
基础背景 scala scala也是生成JVM的字节码，依赖JVM来跑代码。   相比较java，主要优势有两点： 1. 函数化编程 Function Programming FP 如果用多了 Spring 中大量的 template 接口, 你就会觉得 FP 其实还是蛮好用的. 而这仅仅是 FP 好处的冰山一角. 函数其实就是一个 input -&gt; output (scal...
spark worker 启动失败 failed to launch org.apache.spark.deploy.worker.Worker
热门推荐
hzdxw的专栏
06-18 1万+
failed to launch org.apache.spark.deploy.worker.Worker
sbt学习笔记
anjsoft's it blog
04-25 1024
1、自动生成build.sbt文件 $sbt >set name := "anjsoft" （:= 表示 sbt.Setting["name"]="anjsoft"） >session save 2、 ModuleID = groupId % artifact % version % configuration "org.specs2" %% "specs2" % "1.6.1" %
VS2017 Linux项目 Bulid时提示"Could not connect to the remote system"解决方法
fw_wyc的博客
05-20 2861
重启VS
sbt安装与配置
weixin_33790053的博客
11-02 974
下载地址：http://www.scala-sbt.org/download.html 当前版本：sbt-0.13.13.tgz 安装 1.解压并赋予权限 [root@hidden util]# tar -zxvf sbt-0.13.13.tgz sbt-launcher-packaging-0.13.13/ sbt-...
python爬虫等获取实时数据+flume+kafka+spark streaming+mysql+echarts
最新发布
09-13
Spark Streaming可以与Kafka等数据源集成，以实时的方式处理来自不同源的数据，并进行转换、分析和存储。 MySQL是一种关系型数据库管理系统，它被广泛用于存储和管理结构化数据。在上述技术栈中，MySQL可以被用作...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值