CDH Parcel包制作流程详解

CDH Parcel包制作流程详解

• 背景介绍

CDH可以很方便的添加一些大数据相关服务，但这仅限于cloudera公司提供。若想将第三方服务（如公司自己开发的组件）添加到CDH集群（托管在CDH上），需要按照一定的规则流程制作相关程序包，最后发布到CDH上。

本文以开源项目dolphinscheduler的master服务打包成parcel包部署为例，详解cdh parcel包的制作详细流程。

• 相关规则介绍

一个完整的集成到CDH的parcel包应包含如下几个包：

 

● parcel：以“.parcel”结尾的gz格式的压缩文件。它必须为一个固定规则的文件名。

parcel包内包含了你的服务组件，同时包含一个重要的描述性文件parcel.json；这个文件记录了你的服务信息，如版本、所属用户、适用的CDH平台版本等parcel必须包置于/opt/cloudera/parcel-repo/目录下才可以被CDH发布程序时识别到。

● parcel命名规则必须如下：

文件名称格式为三段，第一段是包名，第二段是版本号，第三段是运行平台。

例如：DSMASTER-1.3.3.dsmaster.p0.5-el7.parcel

包名：DSMASTER

版本号：1.3.3.dsmaster.p0.5

运行环境：el7

el6是代表centos6系统，centos7则用el7表示

● DSMASTER-1.3.3.jar文件是一个jar包，它记录了你的服务在CDH上的管理规则，如你的服务在CDH页面上显示的图标、依赖的服务、暴露的端口、启动规则等。csd的jar包必须置于/opt/cloudera/csd/目录才可以在添加集群服务时被识别到。

• 相关下载

制作完成的parcel包和csd jar包需要校验其合法性，cdh提供了jar方便我们对制作完成parcel包和csd jar进行校验。下载地址如下：

https://github.com/cloudera/cm_ext

 

 

• 制作步骤

1、制作parcel包

1.1parcel包目录结构

parcel包的目录结构由

 

parcel包目录结构由你的服务目录(lib/dsmaster)和一个meta目录组成。meta目录组成文件如下：

 

1.2dsmaster_env.sh

声明你的服务运行时的所需的一些变量环境，可以根据你的服务需要可以自行添加设置。

#!/bin/bash

export DSMASTER_HOME=$PARCELS_ROOT/$PARCEL_DIRNAME

 

1.3parcel.json

Parcel.json是一个重要性的描述文件，这个文件记录了你的服务信息，如版本、所属用户、适用的CDH平台版本等。

{
  "schema_version":     1,
  "name":               "DSMASTER",
  "version":            "1.3.3.dsmaster.p0.5",
  "setActiveSymlink":   true,
  "depends": "CDH (>= 5.0), CDH (<< 6.0)",
  "provides": ["dsmaster"],
  "scripts": {
    "defines": "es_env.sh"
  },

  "packages": [{
    "name": "dsmaster",
    "version": "1.3.3.dsmaster.p0.5"
  }],

  "components": [{
    "name":    "dsmaster",
    "version": "1.3.3",
    "pkg_version": "1.3.3"
  }],

  "users": {
    "dsmaster": {
      "longname"    : "DSMASTER",
      "home"       : "/var/lib/dsmaster",
 	  "shell"       : "/bin/bash",
      "extra_groups": []
    }
  },

  "groups": ["dsmaster"]
}

 

 

1.4打包parcel包

执行打包命令打成parcel包。

tar -zcvf DSMASTER-1.3.3.dsmaster.p0.5-el7.parcel DSMASTER-1.3.3.dsmaster.p0.5
java -jar /root/github/cloudera/cm_ext/validator/target/validator.jar -f DSMASTER-1.3.3.dsmaster.p0.5-el7.parcel

生成.sha哈希校验文件。

vi DSMASTER-0.0.5.dsmaster.p0.5-el7.parcel.sha
sha1sum DSMASTER-0.0.5.dsmaster.p0.5-el7.parcel

 

 

1. 制作csd jar

2.1目录结构

csd包的目录如下：

● descriptor：放置服务的规则描述文件service.sdl。

● images：放置服务的图标文件，png格式。不放图标文件，则CDH页面不显示图标。

● scripts ：放置服务的启停脚本，服务的启停通过该目录下的control.sh脚本来实现。

2.2service.sdl

{
	"name": "DSMASTER",
	"label": "DS Master",
	"description": "master service",
	"version": 1,
	"runAs": {
		"user": "root",
		"group": "root"
	},
	"maxInstances": 1,
	"icon": "images/icon.png",
	"compatibility": {
		"generation": 1,
		"cdhVersion": {
			"min": 5,
			"max": 5
		}
	},
	"parcel": {
		"requiredTags": ["cdh"],
		"optionalTags": ["dsmaster-plugin"]
	},
	"commands" : [
		{
		"name" : "ForceStop",
		"label" : "Force stop",
		"description" : "Force stop of DSm Nodes",
		"roleCommand" : "ForceStopRole",
		"roleName" : "DSMASTER_NODE",
		"runMode" : "all"
		}
	],
	"roles": [{
		"name": "DSMASTER_NODE",
		"label": "DSMASTER Node",
		"pluralLabel": "DSMASTER Nodes",
		"jvmBased": true,
		"startRunner": {
			"program": "scripts/control.sh",
			"args": ["start"]
		},
		"stopRunner" : {"timeout" : "90000",
			"runner" : {
				"program" : "scripts/control.sh",
				"args" : ["stop"]
				}
		},
		"commands": [
			{
			"name": "ForceStopRole",
			"label": "Force Stop Role",
			"description": "Force stop of DSMASTER Node",
			"expectedExitCodes": [0],
			"requiredRoleState": "stopped",
			"commandRunner": {
				"program": "scripts/control.sh",
				"args": ["stop"]
				}
			}
		],
		"configWriter": {
            "generators": [{
                "filename": "dsmaster-conf.properties",
                "configFormat": "properties",
                "includeParams": [
                    "dbhost",
                    "dhname",
                    "username",
                    "password",
                    "zkQuorum"
                ]
            }]
        }
	}],	
"parameters": [
        {
            "name": "dbhost",
            "label": "dbhost",
            "description": "dbhost",
            "configName": "dbhost",
            "required": "true",
            "type": "string",
            "configurableInWizard": true,
            "default": "localhost"
        },
        {
            "name": "dbname",
            "label": "dbname",
            "description": "dbname",
            "configName": "dbname",
            "required": "true",
            "type": "string",
            "configurableInWizard": true,
            "default": "dolphinscheduler"
        },
        {
            "name": "username",
            "label": "username",
            "description": "username",
            "configName": "username",
            "required": "true",
            "type": "string",
            "configurableInWizard": true,
            "default": "root"
        },
        {
            "name": "password",
            "label": "password",
            "description": "password",
            "configName": "password",
            "required": "true",
            "type": "string",
            "configurableInWizard": true,
            "default": "*******"
        },
        {
            "name": "zkQuorum",
            "label": "zkQuorum",
            "description": "zkQuorum",
            "configName": "zkQuorum",
            "required": "true",
            "type": "string",
            "configurableInWizard": true,
            "default": "server12:2181,server13:2181,server14:2181"
        }

	]
}

 

2.3control.sh

#!/bin/sh
export ES_HOME=/opt/cloudera/parcels/DSMASTER/lib/dsmaster

PROGNAME=`basename "$0"`

warn() {
    echo "${PROGNAME}: $*"
}

die() {
    warn "$*"
    exit 1
}

locateJava() {
	echo
   # export JAVA_HOME=/usr/java/latest
    echo "Changing Java Home to: $JAVA_HOME"
	export JAVA="$JAVA_HOME/bin/java"
	echo "Changing Java to: $JAVA"
    echo
}

config() {
	echo
	echo "Creating config"
}
init() {
   echo "init"
   source $CONF_DIR/dsmaster-conf.properties
   echo "dbhost: $dbhost"
   echo "dbname: $dbname"
   echo "username: $username"
   echo "password: $password"
   echo "zkQuorum: $zkQuorum"
   sed -i  "s#spring.datasource.url.*#spring.datasource.url=jdbc:mysql://$dbhost/$dbname?characterEncoding=UTF-8\&allowMultiQueries=true#g" $ES_HOME/conf/datasource.properties
   sed -i  "s#spring.datasource.username.*#spring.datasource.username=$username#g" $ES_HOME/conf/datasource.properties
   sed -i  "s#spring.datasource.password.*#spring.datasource.password=$password#g" $ES_HOME/conf/datasource.properties
   sed -i  "s#zookeeper.quorum.*#zookeeper.quorum=$zkQuorum#g" $ES_HOME/conf/zookeeper.properties
   
}

start() {
	echo "Running DSMASTER"
    sh $ES_HOME/bin/dolphinscheduler-daemon.sh stop master-server
    exec sh $ES_HOME/bin/dolphinscheduler-daemon-cdh.sh start master-server
}

stop() {
    echo "stop DSMASTER"
    exec sh $ES_HOME/bin/dolphinscheduler-daemon-cdh.sh stop master-server
}


init
case "$1" in
    start)
        start
        ;;
	stop)
		stop
		;;
    restart)
		stop
        start
		;;
    *)
        echo "Usage DSMASTER {start|stop|restart}"
        ;;
esac

2.4打包成csd jar

 

java -jar /root/github/cloudera/cm_ext/validator/target/validator.jar -s descriptor/service.sdl
jar -cvf DSMASTER-1.3.3.jar *

• 发布到CDH

1、分发激活parcel包

将parcel包和parcel.sha拷贝到cm节点的/opt/cloudera/parcel-repo/目录下，将csd jar包拷贝到/opt/cloudera/csd目录下。

点击【主机】【Parcel】【检查新Parcel】可以识别出刚放入的parcel包。

 

点击【分配】【激活】

 

 

成功后执行命令重启cloudera-scm-server。

systemctl restart cloudera-scm-server

 

2、添加服务

首页添加服务。

 

选择需要添加的服务。

 

选择服务部署的主机。

更改服务所需的配置。

 

 

最后等待执行成功即可。

 

最后执行成功后在首页可以看到新添加的服务。