【PowerJob】从源码编译到k8s部署

太空眼睛

已于 2024-05-18 20:56:23 修改

阅读量995

点赞数 23

分类专栏： Spring Boot 文章标签： PowerJob JDK17 k8s PostgreSQL PG 定时任务

于 2024-05-04 12:08:42 首次发布

本文链接：https://blog.csdn.net/friendlytkyj/article/details/138438061

版权

Spring Boot 专栏收录该内容

34 篇文章 10 订阅

订阅专栏

前言

虽然PowerJob官方说支持JPA各种数据源，但在PG数据库的兼容性上，确实存在小问题，issue也有相关原理描述，官方采用的优雅方式并未真正解决问题，因为只解决了从@Lob字段读取的时候，自动建表的时候还是会生成oid类型。本文从0到1，从源码开始一步步解决编译、运行、部署过程中遇到的问题，并分享出来，希望对大家有所帮助。

本地开发环境说明

开发依赖	版本
数据库	PostgreSQL
JDK	17

源码编译、启动

调度中心（powerjob-server）:
- https://github.com/PowerJob/PowerJob.git
- https://gitee.com/KFCFans/PowerJob.git
修改源码tech.powerjob.server.persistence.config.dialect.PowerJobPGDialect

package tech.powerjob.server.persistence.config.dialect;

import org.hibernate.dialect.PostgreSQL10Dialect;
import org.hibernate.type.descriptor.sql.LongVarbinaryTypeDescriptor;
import org.hibernate.type.descriptor.sql.LongVarcharTypeDescriptor;
import org.hibernate.type.descriptor.sql.SqlTypeDescriptor;

import java.sql.Types;

/**
 * PostgreSQL 数据库支持，需要在 application.properties 中添加以下配置项进行激活
 * spring.datasource.remote.hibernate.properties.hibernate.dialect=tech.powerjob.server.persistence.config.dialect.PowerJobPGDialect
 *
 * @author Kung Yao
 * @author Echo009
 * 2021/3/24 下午 04:23
 * 1074_King
 */
public class PowerJobPGDialect extends PostgreSQL10Dialect {

    /**解决建表时，@Lob字段会生成oid类型，需要改成text类型*/
    public PowerJobPGDialect() {
        super();
        registerColumnType(Types.BLOB, "bytea");
        registerColumnType(Types.CLOB, "text");
    }

    /**解决读取@Lob字段时，使用Text类型*/
    @Override
    public SqlTypeDescriptor getSqlTypeDescriptorOverride(int sqlCode) {
        switch (sqlCode) {
            case Types.CLOB:
                return LongVarcharTypeDescriptor.INSTANCE;
            case Types.NCLOB:
            case Types.BLOB:
                return LongVarbinaryTypeDescriptor.INSTANCE;
        }
        return super.getSqlTypeDescriptorOverride(sqlCode);
    }
}

编译

JDK17+

修改lombok版本

<properties>
    <lombok.version>1.18.28</lombok.version>
</properties>

增加依赖powerjob-server/powerjob-server-starter/pom.xml

    <dependency>
        <groupId>org.jboss.xnio</groupId>
        <artifactId>xnio-nio</artifactId>
        <version>3.8.7.Final</version>
        <scope>runtime</scope>
    </dependency>

编译: mvn compile -Pdev -DskipTests=true -e

PG数据库需要增加配置:

spring.datasource.remote.hibernate.properties.hibernate.dialect=tech.powerjob.server.persistence.config.dialect.PowerJobPGDialect

相关问题描述: https://github.com/PowerJob/PowerJob/issues/153#issuecomment-812771783

####### Database properties(Configure according to the the environment) #######
spring.datasource.remote.hibernate.properties.hibernate.dialect=tech.powerjob.server.persistence.config.dialect.PowerJobPGDialect
spring.datasource.core.driver-class-name=org.postgresql.Driver
spring.datasource.core.jdbc-url=jdbc:postgresql://localhost:5432/powerjob-product?autoReconnect=true&zeroDateTimeBehavior=convertToNull&useUnicode=true&characterEncoding=utf8&useNewIO=true&rewriteBatchedStatements=true&CharSet=utf8&serverTimezone=GMT&autoReconnection=true&remarks=true&useSSL=false
spring.datasource.core.username=powerjob
spring.datasource.core.password=powerjob
spring.datasource.core.maximum-pool-size=20
spring.datasource.core.minimum-idle=5

数据库初始化

CREATE DATABASE "powerjob-product" WITH ENCODING = 'UTF8' CONNECTION LIMIT = -1 IS_TEMPLATE = False
CREATE ROLE powerjob WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOREPLICATION NOBYPASSRLS CONNECTION LIMIT -1 PASSWORD 'powerjob';
COMMENT ON ROLE powerjob IS 'powerjob';
GRANT ALL PRIVILEGES ON DATABASE "powerjob-product" TO powerjob;
GRANT ALL ON SCHEMA PUBLIC TO powerjob;

启动参数增加: --spring.profiles.active=daily
浏览器访问: http://127.0.0.1:7700/
4.x: 注册应用，然后登录
5.x: 超级管理员登录: ADMIN/powerjob_admin

前端页面（powerjob-console）:
- https://github.com/PowerJob/PowerJob-Console.git
- https://gitee.com/KFCFans/PowerJob-Console.git

执行器（powerjob-worker）

参考资料: https://www.yuque.com/powerjob/guidence/deploy_worker
pom依赖

<dependency>
    <groupId>tech.powerjob</groupId>
    <artifactId>powerjob-worker-spring-boot-starter</artifactId>
    <version>${powerjob.version}</version>
</dependency>

处理器（Processor）

package com.wen3.powerjob.demo.jobs;

import jakarta.annotation.Resource;
import org.springframework.stereotype.Component;
import tech.powerjob.worker.annotation.PowerJobHandler;
import tech.powerjob.worker.core.processor.ProcessResult;
import tech.powerjob.worker.core.processor.TaskContext;
import tech.powerjob.worker.core.processor.sdk.BasicProcessor;
import tech.powerjob.worker.log.OmsLogger;

/**
 * 单机处理器：BasicProcessor
 *
 * 单机执行的策略下，server 会在所有可用 worker 中选取健康度最佳的机器进行执行。单机执行任务需要实现接口 BasicProcessor，代码示例如下：
 */
// 支持 SpringBean 的形式
@Component
public class BasicProcessorDemo implements BasicProcessor {

    @Override
    public ProcessResult process(TaskContext context) throws Exception {
        System.out.println("BasicProcessorDemo");
        // 在线日志功能，可以直接在控制台查看任务日志，非常便捷
        OmsLogger omsLogger = context.getOmsLogger();
        omsLogger.info("BasicProcessorDemo start to process, current JobParams is {}.", context.getJobParams());

        return new ProcessResult(true, "result is xxx");
    }

    @PowerJobHandler(name = "xxx1")
    public void xx1(TaskContext context) throws Exception {
        System.out.println("xxx1");
        // 在线日志功能，可以直接在控制台查看任务日志，非常便捷
        OmsLogger omsLogger = context.getOmsLogger();
        omsLogger.info("xx1");
    }

    @PowerJobHandler(name = "xxx2")
    public void xx2(TaskContext context) throws Exception {
        System.out.println("xxx2");
        // 在线日志功能，可以直接在控制台查看任务日志，非常便捷
        OmsLogger omsLogger = context.getOmsLogger();
        omsLogger.info("xx2");
    }
}

在调度中心配置basicProcessorDemo，则执行的是接口BasicProcessor对应的process方法
在调度中心配置basicProcessorDemo#xxx1，则执行的是注解@PowerJobHandler(name = "xxx1")对应的方法

docker镜像制作

java打包: mvn package -Pdev -DskipTests=true -e
把打包后的powerjob-server-starter-4.3.9.jar、构建脚本、Dockerfile放到同一个目录

-rwxr-xr-x 1 root root       268 May  4 11:05 docker-build.sh
-rw-r--r-- 1 root root      2712 May  4 10:43 Dockerfile
-rw-r--r-- 1 root root 135117797 May  4 11:03 powerjob-server-starter-4.3.9.jar

执行构建脚本: ./docker-build.sh

构建脚本

version=4.3.9

# 删除旧镜像
docker rmi -f imgsreg.ipipa.cn:20443/base/powerjob-server:$version
# 构建 powerjob-server 镜像
docker build -t imgsreg.ipipa.cn:20443/base/powerjob-server:$version .
docker push imgsreg.ipipa.cn:20443/base/powerjob-server:$version

k8s部署yaml文件制作

pv制作

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  finalizers:
    - kubernetes.io/pv-protection
  name: nfs-pv-public
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 10Ti
  nfs:
    path: /xxx/
    server: 192.168.xx.xx
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

pvc制作

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
  finalizers:
    - kubernetes.io/pvc-protection
  name: nfs-pvc-public
  namespace: kube-public
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Ti
  volumeMode: Filesystem
  volumeName: nfs-pv-public

service制作

apiVersion: v1
kind: Service
metadata:
  name: powerjob
  namespace: kube-public
  labels:
    app: vxiao-powerjob
spec:
  type: NodePort
  ports:
    - name: http
      port: 17700
      targetPort: 7700
      nodePort: 17700
    - name: akka
      port: 10086
      targetPort: 10086
      nodePort: 10086
    - name: oms-http
      port: 10010
      targetPort: 10010
      nodePort: 10010
  selector:
    app: powerjob

deployment制作

apiVersion: apps/v1
kind: Deployment
metadata:
  name: powerjob
  namespace: kube-public
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: powerjob
  template:
    metadata:
      labels:
        app: powerjob
    spec:
      containers:
        - name: powerjob
          imagePullPolicy: Always
          image: xxxx:port/base/powerjob-server:4.3.9
          env:
            - name: JAVA_PROGRAM_ARGS
              value: >-
                --spring.profiles.active=product
                --spring.datasource.remote.hibernate.properties.hibernate.dialect=tech.powerjob.server.persistence.config.dialect.PowerJobPGDialect
                --spring.datasource.core.driver-class-name=org.postgresql.Driver
                --spring.datasource.core.jdbc-url=jdbc:postgresql://xxx:5432/powerjob-product?autoReconnect=true&zeroDateTimeBehavior=convertToNull&useUnicode=true&characterEncoding=utf8&useNewIO=true&rewriteBatchedStatements=true&CharSet=utf8&serverTimezone=GMT&autoReconnection=true&remarks=true&useSSL=false
                --spring.datasource.core.username=powerjob
                --spring.datasource.core.password=powerjob
          ports:
            - containerPort: 7700
            - containerPort: 10086
            - containerPort: 10010
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 7700
            initialDelaySeconds: 30
            failureThreshold: 2
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 7700
            initialDelaySeconds: 15
            periodSeconds: 15
          startupProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 7700
            failureThreshold: 30
            periodSeconds: 15
          resources:
            requests:
              cpu: 0.5
              memory: 500Mi
            limits:
              cpu: 1
              memory: 1Gi
          volumeMounts:
            - name: nfs-pvc
              mountPath: /root/powerjob
              subPath: powerjob
      volumes:
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc-public

k8s部署

kubectl apply -f powerjob.yaml -n kube-public

使用注意事项

必须先在调度中心管理控制台注册应用名，比如: PowerJobDemo
然后在执行器配置powerjob.worker.app-name=PowerJobDemo
这样启动执行器的时候才能注册到调度中心
打印日志配置:
- logback配置: powerjob-server/powerjob-server-starter/src/main/resources/logback-product.xml
- 在线日志路径: tech.powerjob.server.common.utils.OmsFileUtils代码写死在${user.home}/powerjob/server/online_log

运行截图

在首页先注册应用
使用注册的应用名称和密码进行登录
在任务管理中新建任务

参考资料

官网: http://www.powerjob.tech/
在线文档: https://www.yuque.com/powerjob/guidence/intro
快速开始: https://www.yuque.com/powerjob/guidence/quick_start
PowerJob作者博客

太空眼睛

关注

23
点赞
踩
9

收藏

觉得还不错? 一键收藏
打赏
2
评论
【PowerJob】从源码编译到k8s部署

虽然PowerJob官方说支持JPA各种数据源，但在PG数据库的兼容性上，确实存在小问题，issue也有相关原理描述，官方采用的优雅方式并未真正解决问题，因为只解决了从@Lob字段读取的时候，自动建表的时候还是会生成oid类型。本文从0到1，从源码开始一步步解决编译、运行、部署过程中遇到的问题，并分享出来，希望对大家有所帮助。
复制链接

扫一扫