【Azkaban】 Missing required property ‘azkaban.native.lib‘ cause: null

1.使用环境

版本：Azkaban3.X

部署模式：集群

2.问题描述

Azkaban一直执行都没有问题，最近执行job的时候全部都包以下错误

azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib'
	at azkaban.utils.Props.getString(Props.java:454)
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:242)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:823)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:602)
	at azkaban.execapp.JobRunner.run(JobRunner.java:563)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
21-04-2021 11:09:11 CST pod-mes-pull-order ERROR - Missing required property 'azkaban.native.lib' cause: null
21-04-2021 11:09:11 CST pod-mes-pull-order INFO - Finishing job pod-mes-pull-order at 1618974551015 with status FAILED

3.解决问题过程

3.1根据网上通俗的解决方法

1.查看目录:

conf/azkaban.properties

2.查看选项

azkaban.jobtype.plugin.dir=plugins/jobtypes

3.在配置的目录 plugins/jobtype 下的 commonprivate.properties 

# set execute-as-user
execute.as.user=false
azkaban.native.lib=false

4.然后重启 azkaban 

但是最后还是报同样错误，后来才知道这是Azkaban单机部署解决方式，不适合集群部署，因为集群部署azkaban.native.lib是需要配置，不能配置为false

4.参考集群部署解决

4.1验证azkaban.native.lib路径问题

后来参考了集群部署的方式(Azkaban集群部署的配置)，解决问题的关键是execute-as-user 这个文件

查看plugins/jobtype 下的 commonprivate.properties ，azkaban.native.lib路径是能对应上execute-as-user这个文件

## hadoop security manager setting common to all hadoop jobs
hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0

## hadoop security related settings

# proxy.keytab.location=
# proxy.user=

# azkaban.should.proxy=true
# obtain.binary.token=true
# obtain.namenode.token=true
# obtain.jobtracker.token=true

# global classpath items for all jobs. e.g. hadoop-core jar, hadoop conf
#jobtype.global.classpath=${hadoop.home}/*,${hadoop.home}/conf

# global jvm args for all jobs. e.g. java.io.temp.dir, java.library.path
#jobtype.global.jvm.args=

# configs for jobtype security settings
execute.as.user=false
# 后面的路径是你放置execute-as-user这个文件的路径
azkaban.native.lib=/opt/azkaban/azkaban-exec/execute-as-user
# 使用系统用户提交的时候，azkaban默认把它们放入azkaban组
# 所以你需要提前创建好azkaban这个组或者修改为一个已存在的组
azkaban.group.name=azkaban

4.2 重新编译execute-as-user

既然路径是对的上，那问题就还剩下在execute-as-user 这个文件，于是去了官网下载了execute-as-user.c来代替现有的execute-as-user(记得将现在execute-as-user文件保存一份)

execute-as-user.c文件地址：https://github.com/azkaban/azkaban/blob/3.89.0/az-exec-util/src/main/c/execute-as-user.c(记得与官网保持最新的代码)

/*
 * Copyright 2017 LinkedIn Corp.
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not
 * use this file except in compliance with the License. You may obtain a copy of
 * the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */

#include <dirent.h>
#include <fcntl.h>
#include <fts.h>
#include <errno.h>
#include <grp.h>
#include <unistd.h>
#include <signal.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <sys/stat.h>
#include <sys/mount.h>
#include <sys/types.h>
#include <pwd.h>

FILE *LOGFILE = NULL;
FILE *ERRORFILE = NULL;
int SETUID_OPER_FAILED = 10;
int USER_NOT_FOUND = 20;
int INVALID_INPUT = 30;

/*
 *  Change the real and effective user and group from super user to the specified user
 *
 *  Adopted from:
 *  ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
 *
 */

int change_user(char *username, uid_t user, gid_t group) {
    if (user == getuid() && user == geteuid() &&
            group == getgid() && group == getegid()) {
        return 0;
    }

    if (initgroups(username, group) != 0) {
        fprintf(LOGFILE, "Error setting supplementary groups for user %s: %s\n",
            username, strerror(errno));
        return SETUID_OPER_FAILED;
    }
    if (seteuid(0) != 0) {
        fprintf(LOGFILE, "unable to reacquire root - %s\n", strerror(errno));
        fprintf(LOGFILE, "Real: %d:%d; Effective: %d:%d\n",
                getuid(), getgid(), geteuid(), getegid());
        return SETUID_OPER_FAILED;
    }
    if (setgid(group) != 0) {
        fprintf(LOGFILE, "unable to set group to %d - %s\n", group,
                strerror(errno));
        fprintf(LOGFILE, "Real: %d:%d; Effective: %d:%d\n",
                getuid(), getgid(), geteuid(), getegid());
        return SETUID_OPER_FAILED;
    }
    if (setuid(user) != 0) {
        fprintf(LOGFILE, "unable to set user to %d - %s\n", user, strerror(errno));
        fprintf(LOGFILE, "Real: %d:%d; Effective: %d:%d\n",
                getuid(), getgid(), geteuid(), getegid());
        return SETUID_OPER_FAILED;
    }

    return 0;
}

int main(int argc, char **argv){

    // set up the logging stream
    if (!LOGFILE){
        LOGFILE=stdout;
    }
    if (!ERRORFILE){
        ERRORFILE=stderr;
    }

    if (argc < 3) {
        fprintf(ERRORFILE, "Requires at least 3 variables: ./execute-as-user username command [args]");
        return INVALID_INPUT;
    }

    char *username = argv[1];

    // gather information about user
    struct passwd *user_info = getpwnam(username);
    if (user_info == NULL){
        fprintf(LOGFILE, "user does not exist: %s", username);
        return USER_NOT_FOUND;
    }

    // try to change user
    fprintf(LOGFILE, "Changing user: user: %s, uid: %d, gid: %d\n", username, user_info->pw_uid, user_info->pw_gid);
    int retval = change_user(username, user_info->pw_uid, user_info->pw_gid);
    if (retval != 0){
        fprintf(LOGFILE, "Error changing user to %s\n", username);
        return SETUID_OPER_FAILED;
    }

    // execute the command
    char **user_argv = &argv[2];
    fprintf(LOGFILE, "user command starting from: %s\n", user_argv[0]);
    fflush(LOGFILE);
    retval = execvp(*user_argv, user_argv);
    fprintf(LOGFILE, "system call return value: %d\n", retval);

    // sometimes system(cmd) returns 256, which is interpreted to 0, making a failed job a successful job
    // hence this goofy piece of if statement.
    if (retval != 0){
        return 1;
    }
    else{
        return 0;
    }

}

创建文件execute-as-user.c，并把上面的代码帖进去，然后上传到服务器/opt/azkaban/azkaban-exec(azkaban.native.lib配置的目录)，然后执行以下命令：

gcc execute-as-user.c -o execute-as-user
chown root execute-as-user
chmod 6050 execute-as-user

命令解释：

1.使用 gcc execute-as-user.c -o execute-as-user 命令编译
2.然后使用chown root execute-as-user 和 chmod 6050 execute-as-user 设置权限
设置完权限后，ls -l 看下，这个文件的属性这样的

---Sr-s--- 1 root root 10185 Aug  3 13:02 execute-as-user

如果我们要把这个文件复制到其他目录下，就又会变成普通文件，记得复制完了再做一次 chmod 6050 execute-as-user

最后重新executor服务

#Executor服务关闭
sh /opt/azkaban/azkaban-exec/bin/shutdown-exec.sh
#Executor服务启动
sh /opt/azkaban/azkaban-exec/bin/start-exec.sh

然后重新执行Job

参考资料：

Azkaban集群模式安装与execute-as-user配置：Azkaban集群模式安装与execute-as-user配置 - 简书

Azkaban编译安装配置文档：azkaban编译安装配置文档 - 丹江湖畔养蜂子赵大爹 - 博客园