Ambari自定义服务集成–Azkaban
参考资料
Ambari集成组件官方文档:Stacks and Services
文档目录
一. 准备工作
1.1 安装Ambari
Ambari源码编译安装可参考我之前写的博客:
Ambari2.6.0源码编译安装及遇到的问题解决(超详细文档)
Ambari离线安装也可以参考我之前写的博客:
1.2 源码编译配置Azkaban
1.2.1 Azkaban源码编译
下载解压压缩包:
# 下载指定版本的压缩包
wget https://github.com/azkaban/azkaban/archive/3.90.0.tar.gz
# 解压
tar -zxvf 3.90.0.tar.gz
源码编译:
# 进入到azkaban解压后的根目录,后续该目录用$AZKABAN_HOME替代,使用gradlew方式编译
./gradlew build -x test
如下图所示,编译成功:
$AZKABAN_HOME/azkaban-?/build/distributions/ 目录下会生成相应的压缩包
源码配置
- 创建目录
mkdir -p var/www/html/azkaban
-
解压azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz得到文件create-all-sql-0.1.0-SNAPSHOT.sql
-
将以下两个文件放入到var/www/html/azkaban目录下
a. $AZKABAN_HOME/azkaban-web-server/build/distributions目录下的azkaban-web-server-0.1.0-SNAPSHOT.tar.gz文件
b. $AZKABAN_HOME/azkaban-exec-server/build/distributions目录下的azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz文件
以上文件可以通过web ui的方式访问,如下图
1.2.2 Azkaban数据库配置(mysql)
创建azkaban数据库
create database azkaban character set utf8;
创建azkaban用户
create user 'azkaban'@'%' identified by 'yourpassword';
授予权限
grant all privileges on azkaban.* to azkaban;
flush privileges;
导入azkaban使用的表
# 使用创建的azkaban用户登录mysql
mysql -uazkaban -pyourpassword;
use azkaban;
# 导入azkaban使用的表,该命令需要在该sql脚本所在路径下执行
source create-all-sql-0.1.0-SNAPSHOT.sql
show tables;
二. Ambari集成Azkaban原理及配置
2.1 Ambari自定义服务集成原理
对于未受 ambari 界面管理的服务，比如 Elasticsearch、Kylin、甚至是一个 jar 包，都可以利用 自定义服务集成相关技术 将 服务 集成到 ambari 界面里。
这样，就可以通过 ambari 实现对 自定义服务 的 安装、配置、启动、监听启动状态、停止、指标监控、告警、快速链接 等很多操作，极其方便。
自定义服务集成,是通过stack来进行,stack的版本目录在 /ambari-server/src/main/resources/stacks下
当我们初始化了ambari-server之后,stack的定义也可以在/var/lib/ambari-server/resources/stacks目录下找到
stacks目录如下图,自定义服务集成的时候,按照如下目录树配置即可:
|_ stacks
|_ <stack_name>
|_ <stack_version>
metainfo.xml
|_ hooks
|_ repos
repoinfo.xml
|_ services
|_ <service_name>
metainfo.xml
metrics.json
|_ configuration
{configuration files}
|_ package
{files, scripts, templates}
2.2 Ambari集成Azkaban配置
首先在
/var/lib/ambari-server/resources/stacks/HDP/2.6/services下新建目录AZKABAN
2.2.1 metainfo.xml文件配置
进入目录Azkaban,创建metainfo.xml文件,文件内容如下:
<?xml version="1.0"?>
<!--
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<metainfo>
<schemaVersion>2.0</schemaVersion>
<services>
<service>
<name>AZKABAN</name>
<displayName>Azkaban</displayName>
<comment>Azkaban是用于运行Hadoop job的批处理工作流作业调度程序,Azkaban致力于通过作业依赖性进行排序,并提供易于使用的web用户界面来维护和跟踪工作流程.
</comment>
<version>3.90.0</version>
<quickLinksConfigurations>
<quickLinksConfiguration>
<fileName>quicklinks.json</fileName>
<default>true</default>
</quickLinksConfiguration>
</quickLinksConfigurations>
<components>
<component>
<name>AZKABAN_WEB</name>
<displayName>Azkaban Web Server</displayName>
<category>MASTER</category>
<cardinality>1</cardinality>
<versionAdvertised>true</versionAdvertised>
<recovery_enabled>true</recovery_enabled>
<commandScript>
<script>scripts/azkaban_web.py</script>
<scriptType>PYTHON</scriptType>
<timeout>300</timeout>
</commandScript>
</component>
<component>
<name>AZKABAN_EXECUTOR</name>
<displayName>Azkaban Executor Server</displayName>
<category>SLAVE</category>
<cardinality>0+</cardinality>
<versionAdvertised>true</versionAdvertised>
<recovery_enabled>true</recovery_enabled>
<commandScript>
<script>scripts/azkaban_executor.py</script>
<scriptType>PYTHON</scriptType>
</commandScript>
</component>
</components>
<configuration-dependencies>
<config-type>azkaban-common</config-type>
<config-type>azkaban-executor.properties</config-type>
<config-type>azkaban-users</config-type>
<config-type>azkaban-web.properties</config-type>
<config-type>global.properties</config-type>
<config-type>log4j.properties</config-type>
</configuration-dependencies>
</service>
</services>
</metainfo>
2.2.2 configuration目录配置
2.2.2.1 azkaban-users.xml配置
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>content</name>
<display-name>azkaban-users.xml template</display-name>
<description>Custom azkaban-users.xml</description>
<value>
<![CDATA[
<azkaban-users>
<user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
<user password="metrics" roles="metrics" username="metrics"/>
<role name="admin" permissions="ADMIN"/>
<role name="metrics" permissions="METRICS"/>
</azkaban-users>
]]>
</value>
<value-attributes>
<type>content</type>
<show-property-name>false</show-property-name>
</value-attributes>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
2.2.2.2 global.properties.xml配置
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>content</name>
<display-name>azkaban-users.xml template</display-name>
<description>Custom azkaban-users.xml</description>
<value>
<![CDATA[
retries=10
retry.backoff=10000
failure.emails=
]]>
</value>
<value-attributes>
<type>content</type>
<show-property-name>false</show-property-name>
</value-attributes>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
2.2.2.3 azkaban-executor.properties.xml文件配置
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>executor.port</name>
<value>12321</value>
</property>
<property>
<name>azkaban.jobtype.plugin.dir</name>
<value>plugins/jobtypes</value>
</property>
<property>
<name>executor.maxThreads</name>
<value>100</value>
</property>
<property>
<name>executor.flow.threads</name>
<value>100</value>
</property>
<property require-input="true">
<name>azkaban.webserver.url</name>
<value>http://localhost:10100</value>
<description>
example: http://localhost:10100
</description>
</property>
</configuration>
2.2.2.4 azkaban-web.properties.xml配置
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>mail.sender</name>
<value>xxx@163.com</value>
</property>
<property>
<name>mail.host</name>
<value>smtp.163.com</value>
</property>
<property>
<name>mail.user</name>
<value>xxx@163.com</value>
</property>
<property>
<name>mail.password</name>
<value>xxx</value>
</property>
<property>
<name>azkaban.use.multiple.executors</name>
<value>true</value>
</property>
<property>
<name>azkaban.executorselector.filters</name>
<value>StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus</value>
</property>
<property>
<name>azkaban.executorselector.comparator.NumberOfAssignedFlowComparator</name>
<value>1</value>
</property>
<property>
<name>azkaban.executorselector.comparator.Memory</name>
<value>1</value>
</property>
<property>
<name>azkaban.executorselector.comparator.LastDispatched</name>
<value>1</value>
</property>
<property>
<name>azkaban.executorselector.comparator.CpuUsage</name>
<value>1</value>
</property>
</configuration>
2.2.2.5 log4j.properties.xml配置
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>content</name>
<display-name>log4j.properties template</display-name>
<description>Custom log4j.properties</description>
<value>
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.File=logs/azkaban.log
log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %5p [%c{1}] [%t] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=1024MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %5p [%c{1}] [%t] [Azkaban]
%m%n
</value>
<value-attributes>
<type>content</type>
<show-property-name>false</show-property-name>
</value-attributes>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
2.2.2.6 azkaban-common.xml配置
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>azkaban.name</name>
<value>Prod</value>
</property>
<property>
<name>azkaban.label</name>
<value>Prod Azkaban</value>
</property>
<property>
<name>azkaban.color</name>
<value>#FF3601</value>
</property>
<property>
<name>azkaban.default.servlet.path</name>
<value>/index</value>
</property>
<property>
<name>web.resource.dir</name>
<value>web/</value>
</property>
<property>
<name>default.timezone.id</name>
<value>Asia/Shanghai</value>
</property>
<property>
<name>user.manager.class</name>
<value>azkaban.user.XmlUserManager</value>
</property>
<property>
<name>user.manager.xml.file</name>
<value>conf/azkaban-users.xml</value>
</property>
<property>
<name>executor.global.properties</name>
<value>conf/global.properties</value>
</property>
<property>
<name>azkaban.project.dir</name>
<value>projects</value>
</property>
<property>
<name>velocity.dev.mode</name>
<value>false</value>
</property>
<property>
<name>jetty.use.ssl</name>
<value>false</value>
</property>
<property>
<name>jetty.maxThreads</name>
<value>100</value>
</property>
<property>
<name>jetty.port</name>
<value>10200</value>
</property>
<property>
<name>lockdown.create.projects</name>
<value>false</value>
</property>
<property>
<name>cache.directory</name>
<value>cache</value>
</property>
<property>
<name>jetty.connector.stats</name>
<value>true</value>
</property>
<property>
<name>executor.connector.stats</name>
<value>true</value>
</property>
<property>
<name>database.type</name>
<value>mysql</value>
</property>
<property require-input="true">
<name>mysql.host</name>
<value></value>
</property>
<property require-input="true">
<name>mysql.port</name>
<value>3306</value>
</property>
<property require-input="true">
<name>mysql.database</name>
<value>azkaban</value>
</property>
<property require-input="true">
<name>mysql.user</name>
<value>azkaban</value>
</property>
<property require-input="true">
<name>mysql.password</name>
<value></value>
</property>
<property>
<name>mysql.numconnections</name>
<value>200</value>
</property>
</configuration>
2.2.3 package/scripts配置
2.2.3.1 azkaban_executor.py配置
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import os.path as path
import time
from common import azkabanHome, azkabanExecTarUrl, azkabanExecTarName, azkabanConfPath
from resource_management.core.exceptions import ExecutionFailed, ComponentIsNotRunning
from resource_management.core.resources.system import Execute
from resource_management.libraries.script.script import Script
class ExecutorServer(Script):
def install(self, env):
Execute('yum install -y python-requests')
tmpAzkabanExecTarPath = '/tmp/' + azkabanExecTarName
Execute('mkdir -p {0}'.format(azkabanHome))
Execute('wget --no-check-certificate {0} -O {1}'.format(azkabanExecTarUrl, tmpAzkabanExecTarPath))
Execute('tar -xf {0} -C {1} --strip-components=1'.format(tmpAzkabanExecTarPath, azkabanHome))
Execute(
'cd {0} && '
'chmod +x bin/start-exec.sh && '
'chmod +x bin/shutdown-exec.sh && '
'chmod +s bin/internal/internal-start-executor.sh'.format(azkabanHome)
)
self.configure(env)
def stop(self, env):
Execute('cd {0} && ./bin/shutdown-exec.sh'.format(azkabanHome))
def start(self, env):
self.configure(env)
Execute('cd {0} && ./bin/start-exec.sh'.format(azkabanHome))
from params import azkaban_executor_properties
executor_port = int(azkaban_executor_properties['executor.port'])
url = 'http://127.0.0.1:{0}/executor?action=ping'.format(executor_port)
maxRetryCount = 10
retryCount = 0
import requests
while True:
try:
resp = requests.get(url)
print(resp.text)
if json.loads(resp.text)['status'] == 'alive':
print('executor is alive')
Execute('curl -G "localhost:{0}/executor?action=activate" && echo'.format(executor_port))
print('after activate')
break
except:
print('executor is not alive')
time.sleep(1)
retryCount += 1
if retryCount > maxRetryCount:
raise Exception('web start failed')
def status(self, env):
# TODO 可优化,和start逻辑一致
try:
Execute(
'export AZ_CNT=`ps -ef |grep -v grep |grep azkaban-exec-server | wc -l` && `if [ $AZ_CNT -ne 0 ];then exit 0;else exit 3;fi `'
)
except ExecutionFailed as ef:
if ef.code == 3:
raise ComponentIsNotRunning("ComponentIsNotRunning")
else:
raise ef
def configure(self, env):
from params import azkaban_executor_properties, log4j_properties, azkaban_common, global_properties
key_val_template = '{0}={1}\n'
with open(path.join(azkabanConfPath, 'azkaban.properties'), 'w') as f:
for key, value in azkaban_common.iteritems():
f.write(key_val_template.format(key, value))
for key, value in azkaban_executor_properties.iteritems():
if key != 'content':
f.write(key_val_template.format(key, value))
if azkaban_executor_properties.has_key('content'):
f.write(str(azkaban_executor_properties['content']))
with open(path.join(azkabanConfPath, 'log4j.properties'), 'w') as f:
if log4j_properties.has_key('content'):
f.write(str(log4j_properties['content']))
with open(path.join(azkabanConfPath, 'global.properties'), 'w') as f:
if global_properties.has_key('content'):
f.write(str(global_properties['content']))
if __name__ == '__main__':
ExecutorServer().execute()
2.2.3.2 azkaban_web.py配置
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import os.path as path
import time
from common import azkabanHome, azkabanWebTarUrl, azkabanWebTarName, \
azkabanConfPath
from resource_management.core.exceptions import ExecutionFailed, ComponentIsNotRunning
from resource_management.core.resources.system import Execute
from resource_management.libraries.script.script import Script
class WebServer(Script):
def install(self, env):
Execute('yum install -y python-requests')
tmpAzkabanWebTarPath = '/tmp/' + azkabanWebTarName
Execute('mkdir -p {0}'.format(azkabanHome))
Execute('wget --no-check-certificate {0} -O {1}'.format(azkabanWebTarUrl, tmpAzkabanWebTarPath))
Execute('tar -xf {0} -C {1} --strip-components=1'.format(tmpAzkabanWebTarPath, azkabanHome))
Execute(
'cd {0} && '
'chmod +x bin/start-web.sh && '
'chmod +x bin/shutdown-web.sh && '
'chmod +s bin/internal/internal-start-web.sh'.format(azkabanHome)
)
self.configure(env)
def stop(self, env):
Execute('cd {0} && ./bin/shutdown-web.sh'.format(azkabanHome))
def start(self, env):
self.configure(env)
maxRetryCount = 10
retryCount = 0
from params import host_info, azkaban_executor_properties
executor_port = int(azkaban_executor_properties['executor.port'])
execHosts = host_info['azkaban_executor_hosts']
urlTmpl = 'http://{0}:{1}/executor?action=getStatus'
import requests
while True:
for execHost in execHosts:
try:
url = urlTmpl.format(execHost, executor_port)
print(url)
resp = requests.get(url)
print(resp.text)
if json.loads(resp.text)['isActive'] == 'true':
Execute('cd {0} && ./bin/start-web.sh'.format(azkabanHome))
return
except:
print('web is not alive')
time.sleep(0.5)
time.sleep(1)
retryCount += 1
if retryCount > maxRetryCount:
raise Exception('web start failed')
def status(self, env):
try:
Execute(
'export AZ_CNT=`ps -ef |grep -v grep |grep azkaban-web-server | wc -l` && `if [ $AZ_CNT -ne 0 ];then exit 0;else exit 3;fi `'
)
except ExecutionFailed as ef:
if ef.code == 3:
raise ComponentIsNotRunning("ComponentIsNotRunning")
else:
raise ef
def configure(self, env):
from params import azkaban_common, azkaban_web_properties, azkaban_users, global_properties, log4j_properties
key_val_template = '{0}={1}\n'
with open(path.join(azkabanConfPath, 'azkaban.properties'), 'w') as f:
for key, value in azkaban_common.iteritems():
f.write(key_val_template.format(key, value))
for key, value in azkaban_web_properties.iteritems():
if key != 'content':
f.write(key_val_template.format(key, value))
if azkaban_web_properties.has_key('content'):
f.write(str(azkaban_web_properties['content']))
with open(path.join(azkabanConfPath, 'azkaban-users.xml'), 'w') as f:
if azkaban_users.has_key('content'):
f.write(str(azkaban_users['content']))
with open(path.join(azkabanConfPath, 'global.properties'), 'w') as f:
if global_properties.has_key('content'):
f.write(str(global_properties['content']))
with open(path.join(azkabanConfPath, 'log4j.properties'), 'w') as f:
if log4j_properties.has_key('content'):
f.write(str(log4j_properties['content']))
if __name__ == '__main__':
WebServer().execute()
2.2.3.3 common.py配置
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import ConfigParser
script_dir = os.path.dirname(os.path.realpath(__file__))
config = ConfigParser.ConfigParser()
config.readfp(open(os.path.join(script_dir, 'download.ini')))
azkabanHome = '/data/azkaban'
azkabanConfPath = azkabanHome + '/conf'
azkabanWebTarUrl = config.get('download', 'azkaban_web_tar_url')
azkabanExecTarUrl = config.get('download', 'azkaban_executor_tar_url')
azkabanWebTarName = azkabanWebTarUrl.split('/')[-1]
azkabanExecTarName = azkabanExecTarUrl.split('/')[-1]
2.2.3.4 params.py配置
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from resource_management.libraries.script.script import Script
# config object that holds the configurations declared in the config xml file
config = Script.get_config()
azkaban_web_properties = config['configurations']['azkaban-web.properties']
azkaban_executor_properties = config['configurations']['azkaban-executor.properties']
azkaban_users = config['configurations']['azkaban-users']
azkaban_common = config['configurations']['azkaban-common']
global_properties = config['configurations']['global.properties']
log4j_properties = config['configurations']['log4j.properties']
host_info = config['clusterHostInfo']
host_level_params = config['hostLevelParams']
java_home = host_level_params['java_home']
2.2.3.5 download.ini配置
# 这里配置的为1.2.1过程中放入/var/www/html/azkaban目录下的两个文件
[download]
azkaban_web_tar_url = http://ip/azkaban/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
azkaban_executor_tar_url = http://ip/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
2.2.4 quicklinks.json文件配置
{
"name": "default",
"description": "default quick links configuration",
"configuration": {
"protocol": {
"type": "HTTP_ONLY"
},
"links": [
{
"name": "azkaban_web",
"label": "Azkaban Web",
"component_name": "AZKABAN_WEB",
"requires_user_name": "false",
"url": "%@://%@:%@:10200",
"port": {
"http_property": "jetty.port",
"regex": "^(\\d+)$",
"site": "azkaban-web.properties"
}
}
]
}
}
2.3 重启ambari-server
ambari-server restart
登录ambari后,已经可以在add service界面查找到Azkaban服务
三. 根据界面提示安装启动Azkaban
根据界面提示逐步操作,即可初始化Azkaban
此时,遇到错误按照日志提示逐步排查即可:
如下图所示,Azkaban安装启动成功!
根据 2.2.2.3 azkaban-executor.properties.xml配置项azkaban.webserver.url配置的value值访问web界面
根据 2.2.2.1 azkaban-users.xml配置的账户名和密码登录azkaban
如下图所示,配置完成!
四. Ambari集成Azkaban过程中遇到的问题
4.1 azkaban源码clean操作报错
./gradlew clean Exception in thread "main"
java.lang.RuntimeException: Timeout of 120000 reached waiting for exclusive access to file:
/root/.gradle/wrapper/dists/gradle-4.1-all/bzyivzo6n839fup2jbap0tjew/gradle-4.1-all.zip at
org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65) at
org.gradle.wrapper.Install.createDist(Install.java:48) at
org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
解决办法:
- 删除报错指向的文件夹
/root/.gradle/wrapper/dists/gradle-4.1-all
- 重新执行命令
问题解决:
4.2 azkaban安装初始化报错:
File "/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/AZKABAN/package/scripts/azkaban_web.py", line 67
self.configure(env)
^
IndentationError: expected an indented block
类似的问题大多数是python脚本缩进问题导致的,python对缩进的要求十分严格,按照报错提示修改对应位置的脚本即可
1805
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
