Heron 0.17.1单节点本地环境配置（CentOS）

最新推荐文章于 2023-04-12 17:24:32 发布

Ethan_pika

最新推荐文章于 2023-04-12 17:24:32 发布

阅读量382

点赞数

文章标签： Heron Centos Local

本文链接：https://blog.csdn.net/yitian_z/article/details/102956857

版权

系统环境

VMware Workstation：CentOS 7 64bit（CentOS 6版本运行示例时有问题）
Python：Python 2.7（系统自带，负责需要安装）
JDK：Java 1.8（需要配置JDK）
使用用户：root
CentOS配置：4g内存，4CPU核心，20g硬盘

Heron下载

下载地址：https://github.com/twitter/heron/releases

Heron文档快速开始：https://twitter.github.io/heron/docs/getting-started/

这里选择下载与系统环境相同的安装文件：heron-client-install-0.17.1.sh与heron-tool-install-0.17.1.sh。当前的最新版本为0.17.1。下载完成后，将安装文件复制到CentOS中，以备后面运行安装。

Heron安装

1. 配置JDK环境变量（用户切换到root）

JDK配置详情见：CentOS下安装和配置JDK1.8。这里的配置与该内容中相同。（注意JDK版本的下载，这里是linux 64bit版本）

注：在解压jdk.tar.gz文件时，遇到的错误与解决方法：http://blog.csdn.net/u014686180/article/details/44703809

/etc/profile配置文件中环境变量配置，如下：

# jdk
export JAVA_HOME=/usr/java/jdk1.8.0_151
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

使配置文件生效：

source /etc/profile

2. 检查python安装情况

[root@localhost Heron Files]# python
Python 2.7.5 (default, Aug  4 2017, 00:39:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

3. 准备工作

复制Heron安装文件到/usr/heron目录（需要创建该目录，该步骤不是必要）

cp /home/yitian/Desktop/Heron Files/heron-client-install-0.17.1.sh /usr/heron/
cp /home/yitian/Desktop/Heron Files/heron-tool-install-0.17.1.sh /usr/heron/

更改用户组并修改运行权限：

chown root:root /user/heron/heron-client-install-0.17.1.sh
chown root:root /user/heron/heron-tool-install-0.17.1.sh

修改文件权限：

chmod 777 /user/heron/heron-client-install-0.17.1.sh
chmod 777 /user/heron/heron-tool-install-0.17.1.sh

3. 运行安装文件

运行heron-client-install-0.17.1.sh：

[root@localhost heron]# ./heron-client-install-0.17.1-centos.sh --user
--warning=no-timestamp
Heron client installer
----------------------
Uncompressing..tar xfz /root/.heron/heron-client.tar.gz -C /root/.heron --warning=no-timestamp
...
Heron is now installed!
Make sure you have "/root/bin" in your path.
See http://heronstreaming.io/docs/getting-started for how to use Heron.
heron.build.version : '0.17.1'
heron.build.time : Sat Nov 18 01:08:27 UTC 2017
heron.build.timestamp : 1510967307000
heron.build.host : ci-server-01
heron.build.user : release-agent1
heron.build.git.revision : 874feb31b1ad9df6ea4a51d58b573750468ad28d
heron.build.git.status : Clean

运行heron-tool-install-0.17.1.sh：

[root@localhost heron]# ./heron-tools-install-0.17.1-centos.sh --user
--warning=no-timestamp
Heron tools installer
---------------------
Uncompressing..tar xfz /root/.herontools/heron-tools.tar.gz -C /root/.herontools --warning=no-timestamp
....
Heron Tools is now installed!
Make sure you have "/root/bin" in your path.
See http://heronstreaming.io/docs/getting-started for how to use Heron.
heron.build.version : '0.17.1'
heron.build.time : Sat Nov 18 01:08:27 UTC 2017
heron.build.timestamp : 1510967307000
heron.build.host : ci-server-01
heron.build.user : release-agent1
heron.build.git.revision : 874feb31b1ad9df6ea4a51d58b573750468ad28d
heron.build.git.status : Clean
[root@localhost heron]#

3. 配置环境变量

# heron
export PATH=~/bin:$PATH

使配置文件生效：（如果重新打开一个Centos命令行终端检查heron安装情况时显示找不到该命令的错误，运行该命令）

source /etc/profile

4. 检查Heron安装情况：

[root@localhost examples]# heron
usage: heron [-h] <command> <options> ...
optional arguments:
  -h, --help           show this help message and exit
Available commands:
    activate           Activate a topology
    deactivate         Deactivate a topology
    help               Prints help for commands
    kill               Kill a topology
    restart            Restart a topology
    submit             Submit a topology
    update             Update a topology
    version            Print version of heron-cli
    config             Config properties for a cluster
Getting more help:
  heron help <command> Prints help and options for <command>
For detailed documentation, go to http://heronstreaming.io
 
[root@localhost examples]# heron version
heron.build.git.revision : 874feb31b1ad9df6ea4a51d58b573750468ad28d
heron.build.git.status : Clean
heron.build.host : ci-server-01
heron.build.time : Sat Nov 18 01:08:27 UTC 2017
heron.build.timestamp : 1510967307000
heron.build.user : release-agent1
heron.build.version : 0.17.1
[root@localhost examples]#

到这里，heron单节点本地简易安装完成。

示例运行

1. 发布示例

[root@localhost examples]# heron submit local ~/.heron/examples/heron-api-examples.jar com.twitter.heron.examples.api.WordCountTopology WordCountTopology --deploy-deactivated
[2017-12-11 04:42:08 +0000] [INFO]: Using cluster definition in /root/.heron/conf/local
[2017-12-11 04:42:10 +0000] [INFO]: Launching topology: 'WordCountTopology'
[2017-12-11 04:42:14 -0800] [INFO] com.twitter.heron.scheduler.local.LocalLauncher: core release package uri: file:///root/.heron/dist/heron-core.tar.gz 
[2017-12-11 04:42:32 +0000] [INFO]: Successfully launched topology 'WordCountTopology'

2. 运行heron-tracker

[root@localhost examples]# heron-tracker 
[2017-12-11 04:42:59 +0000] [INFO]: Connecting to file state with rootpath: /root/.herondata/repository/state/local
[2017-12-11 04:42:59 +0000] [INFO]: State watch triggered for topologies.
[2017-12-11 04:42:59 +0000] [INFO]: Adding new topology: WordCountTopology, state_manager: local
[2017-12-11 04:42:59 +0000] [INFO]: Registering a watch with uid: d2eae06c-1e89-4c1f-bf62-002244de4f45
[2017-12-11 04:42:59 +0000] [INFO]: No execution state found for: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Watch triggered for topology execution state: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Setting topology info for topology: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Tracker has started
[2017-12-11 04:42:59 +0000] [INFO]: Running on port: 8888
[2017-12-11 04:42:59 +0000] [INFO]: Using config file: /root/.herontools/conf/heron_tracker.yaml
[2017-12-11 04:42:59 +0000] [INFO]: Using state manager:
    type: file
    name: local
    rootpath: ~/.herondata/repository/state/local
    tunnelhost: 127.0.0.1
[2017-12-11 04:42:59 +0000] [INFO]: Watch triggered for topology pplan: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Setting topology info for topology: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Watch triggered for topology tmaster: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Setting topology info for topology: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Watch triggered for topology scheduler location: WordCountTopology
[2017-12-11 04:42:59 +0000] [INFO]: Setting topology info for topology: WordCountTopology

在浏览器中查看：

3. 运行heron-ui

[root@localhost .heron]# heron-ui
[2017-12-11 04:45:59 +0000] [INFO]: Listening at http://0.0.0.0:8889
[2017-12-11 04:45:59 +0000] [INFO]: Using tracker url: http://127.0.0.1:8888
[2017-12-11 04:45:59 +0000] [INFO]: Using base url:
[2017-12-11 04:45:59 +0000] [INFO]: static/

注意：运行heron-ui命令之前，首先需要运行heron-tracker命令（不能关闭，新打开一个命令行终端运行heron-ui），否则会出现错误。原因是，heron-ui服务需要调用heron-tracker提供的JSON REST API。

在浏览器中查看：

4. 查看提交的拓扑目录

[root@localhost yitian]# ls -al ~/.herondata/topologies/local/root/WordCountTopology
total 3484
drwxr-xr-x. 6 root root    4096 Dec 11 04:47 .
drwxr-xr-x. 3 root root      31 Dec 11 04:42 ..
-rw-r--r--. 1 root root       5 Dec 11 04:42 container_1_consumer_2.pid
-rw-r--r--. 1 root root       5 Dec 11 04:42 container_1_word_1.pid
-rwxr-xr-x. 1 root root 3230820 Dec 31  1969 heron-api-examples.jar
drwxr-xr-x. 2 root root     247 Dec 31  1969 heron-conf
drwxr-xr-x. 4 root root      28 Dec 31  1969 heron-core
-rw-r--r--. 1 root root       5 Dec 11 04:42 heron-executor-0.pid
-rw-r--r--. 1 root root       0 Dec 11 04:42 heron-executor-0.stderr
-rw-r--r--. 1 root root    6059 Dec 11 04:42 heron-executor-0.stdout
-rw-r--r--. 1 root root       5 Dec 11 04:42 heron-executor-1.pid
-rw-r--r--. 1 root root       0 Dec 11 04:42 heron-executor-1.stderr
-rw-r--r--. 1 root root    8874 Dec 11 04:42 heron-executor-1.stdout
-rw-r--r--. 1 root root       5 Dec 11 04:42 heron-metricscache.pid
-rw-r--r--. 1 root root       5 Dec 11 04:42 heron-shell-0.pid
-rw-r--r--. 1 root root       5 Dec 11 04:42 heron-shell-1.pid
-rw-r--r--. 1 root root       5 Dec 11 04:42 heron-tmaster.pid
-rw-r--r--. 1 root root       0 Dec 11 04:42 java-325627ca-started.stderr
drwxr-xr-x. 2 root root    4096 Dec 11 04:42 log-files
-r--r--r--. 1 root root    4823 Dec 11 04:48 metrics.json.metricsmgr-0.0
-r--r--r--. 1 root root    4746 Dec 11 04:44 metrics.json.metricsmgr-0.1
-r--r--r--. 1 root root    4820 Dec 11 04:45 metrics.json.metricsmgr-0.2
-r--r--r--. 1 root root    4862 Dec 11 04:46 metrics.json.metricsmgr-0.3
-r--r--r--. 1 root root    4820 Dec 11 04:47 metrics.json.metricsmgr-0.4
-r--r--r--. 1 root root   13825 Dec 11 04:48 metrics.json.metricsmgr-1.0
-r--r--r--. 1 root root   13679 Dec 11 04:44 metrics.json.metricsmgr-1.1
-r--r--r--. 1 root root   13758 Dec 11 04:45 metrics.json.metricsmgr-1.2
-r--r--r--. 1 root root   13756 Dec 11 04:46 metrics.json.metricsmgr-1.3
-r--r--r--. 1 root root   13785 Dec 11 04:47 metrics.json.metricsmgr-1.4
-rw-r--r--. 1 root root       5 Dec 11 04:42 metricsmgr-0.pid
-rw-r--r--. 1 root root       5 Dec 11 04:42 metricsmgr-1.pid
drwxr-xr-x. 3 root root      21 Dec 11 04:42 .pex
-r-xr-xr-x. 1 root root     282 Dec 31  1969 release.yaml
-rw-r--r--. 1 root root       5 Dec 11 04:42 stmgr-1.pid
-rw-r--r--. 1 root root  132554 Dec 11 04:42 WordCountTopology.defn
[root@localhost yitian]#

--------------------------
[root@localhost yitian]# ls -al ~/.herondata/topologies/local/root/WordCountTopology/log-files/
total 196
drwxr-xr-x. 2 root root  4096 Dec 11 04:42 .
drwxr-xr-x. 6 root root  4096 Dec 11 04:47 ..
-rw-r--r--. 1 root root 10346 Dec 11 04:42 container_1_consumer_2.log.0
-rw-r--r--. 1 root root     0 Dec 11 04:42 container_1_consumer_2.log.0.lck
-rw-r--r--. 1 root root 10548 Dec 11 04:42 container_1_word_1.log.0
-rw-r--r--. 1 root root     0 Dec 11 04:42 container_1_word_1.log.0.lck
-rw-r--r--. 1 root root     0 Dec 11 04:42 gc.container_1_consumer_2.log.0.current
-rw-r--r--. 1 root root     0 Dec 11 04:42 gc.container_1_word_1.log.0.current
-rw-r--r--. 1 root root  3568 Dec 11 04:46 gc.metricscache.log.0.current
-rw-r--r--. 1 root root  5352 Dec 11 04:43 gc.metricsmgr.log.0.current
-rw-r--r--. 1 root root   164 Dec 11 04:42 heron-shell-0.log
-rw-r--r--. 1 root root   164 Dec 11 04:42 heron-shell-1.log
lrwxrwxrwx. 1 root root    77 Dec 11 04:42 heron-stmgr-stmgr-1.INFO -> heron-stmgr-stmgr-1.localhost.localdomain.root.log.INFO.20171211-044237.47663
-rw-r--r--. 1 root root 10108 Dec 11 04:49 heron-stmgr-stmgr-1.localhost.localdomain.root.log.INFO.20171211-044237.47663
lrwxrwxrwx. 1 root root   125 Dec 11 04:42 heron-tmaster-WordCountTopology2bd695b6-c905-4b11-bb58-cefb2e43bdb9.INFO -> heron-tmaster-WordCountTopology2bd695b6-c905-4b11-bb58-cefb2e43bdb9.localhost.localdomain.root.log.INFO.20171211-044240.47703
-rw-r--r--. 1 root root  4450 Dec 11 04:42 heron-tmaster-WordCountTopology2bd695b6-c905-4b11-bb58-cefb2e43bdb9.localhost.localdomain.root.log.INFO.20171211-044240.47703
-rw-r--r--. 1 root root 11234 Dec 11 04:42 heron-WordCountTopology-scheduler.log.0
-rw-r--r--. 1 root root     0 Dec 11 04:42 heron-WordCountTopology-scheduler.log.0.lck
-rw-r--r--. 1 root root 51207 Dec 11 04:49 metricscache-0.log.0
-rw-r--r--. 1 root root     0 Dec 11 04:42 metricscache-0.log.0.lck
-rw-r--r--. 1 root root 19758 Dec 11 04:48 metricsmgr-0.log.0
-rw-r--r--. 1 root root     0 Dec 11 04:42 metricsmgr-0.log.0.lck
-rw-r--r--. 1 root root 37511 Dec 11 04:49 metricsmgr-1.log.0
-rw-r--r--. 1 root root     0 Dec 11 04:42 metricsmgr-1.log.0.lck

激活拓扑（拓扑管理）

拓扑管理的命令：

heron activate local WordCountTopology
heron deactivate local WordCountTopology
heron kill local WordCountTopology

1. 激活拓扑：

[root@localhost yitian]# heron activate local WordCountTopology
[2017-12-11 04:52:25 +0000] [INFO]: Using cluster definition in /root/.heron/conf/local
[2017-12-11 04:52:30 -0800] [INFO] com.twitter.heron.spi.utils.TMasterUtils: Topology command ACTIVATE completed successfully. 
[2017-12-11 04:52:33 +0000] [INFO]: Successfully activate topology: WordCountTopology

注意：在虚拟机中运行较慢，内存占用情况严重，出现卡顿现象。

拓扑激活后，可以再heron-ui的页面中查看拓扑的逻辑计划和物理计划，已经拓扑的运行情况：

2. deactivate拓扑

[root@localhost Heron Files]# heron deactivate local WordCountTopology
[2017-12-11 06:02:41 +0000] [INFO]: Using cluster definition in /root/.heron/conf/local
[2017-12-11 06:02:50 -0800] [INFO] com.twitter.heron.spi.utils.TMasterUtils: Topology command DEACTIVATE completed successfully. 
[2017-12-11 06:02:50 +0000] [INFO]: Successfully deactivate topology: WordCountTopology

3. 显示heron命令帮助

[root@localhost Heron Files]# heron help submit
usage: heron submit [options] cluster/[role]/[env] topology-file-name topology-class-name [topology-args]
Required arguments:
  cluster/[role]/[env]  Cluster, role, and environment to run topology
  topology-file-name    Topology jar/tar/zip file
  topology-class-name   Topology class name
Optional arguments:
  -h, --help            show this help message and exit
  --config-path CONFIG_PATH
                        Path to cluster configuration files
  --config-property PROPERTY=VALUE
                        Configuration properties that overrides default
                        options
  --deploy-deactivated DEPLOY_DEACTIVATED
                        Deploy topology in deactivated mode
  --dry-run             Enable dry-run mode. Information about the command
                        will print but no action will be taken on the topology
  --dry-run-format DRY_RUN_FORMAT
                        The format of the dry-run output
                        ([raw|table|colored_table], default=table). Ignored
                        when dry-run mode is not enabled
  --extra-launch-classpath CLASS_PATH
                        Additional JVM class path for launching topology
  --service-url SERVICE_URL
                        API service end point
  --topology-main-jvm-property PROPERTY=VALUE
                        JVM system property for executing topology main
  --verbose VERBOSE     Verbose mode. Increases logging level to show debug
                        messages

该帮助也可以在heron document中获取：https://twitter.github.io/heron/docs/getting-started/

Troubleshooting

Heron Document给出的一些问题的解决方案： Quick Start Troubleshooting.

探索其他示例

在上述使用–user运行heron安装命令之后，heron的示例jar文件所在目录为：

[root@localhost Heron Files]# cd ~/.heron/examples/
[root@localhost examples]# ll
total 14360
-rwxr-xr-x. 1 root root  3230820 Dec 31  1969 heron-api-examples.jar
-rwxr-xr-x. 1 root root 11470867 Dec 31  1969 heron-streamlet-examples.jar

注意：在heron document中给出的运行示例为0.17.0版本的，这里使用0.17.1版本中的示例，两者在示例中有所差别。

Heron 0.17.1版本中的示例jar如下：

常见问题

在运行heron activate local WordCountTopology命令时，出现如下错误提示：

[root@localhost Desktop]# heron activate local WordCountTopology
[2017-12-02 05:23:32 +0000] [INFO]: Using cluster definition in /usr/local/heron/conf/local
[2017-12-02 05:23:32 +0000] [ERROR]: cluster/role/environ does not match. Topology 'WordCountTopology' is running at local/root/default, not local/yitian/default
[2017-12-02 05:23:32 +0000] [ERROR]: Failed to activate topology: WordCountTopology

解决问题：

heron activate Error: cluster/role/environ does not match. Topology ‘WordCountTopology’ is running at local/root/default, not local/yitian/default