Oralce增量同步方案-定时任务

最新推荐文章于 2023-12-21 14:06:02 发布

仰望星空的我

最新推荐文章于 2023-12-21 14:06:02 发布

阅读量877

点赞数

分类专栏： kettle任务调度

kettle任务调度专栏收录该内容

4 篇文章 0 订阅

订阅专栏

使用Quarz调度Kettle执行定时抽取

以下只罗列一些关键部分。重点关注java代码模块；

1.工程Maven依赖;重点关注意红色字体标注内容。

<?xml version="1.0"?>
<project
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.topsci</groupId>
<artifactId>fqs-sync</artifactId>
<version>1.0.8</version>
</parent>
<artifactId>fqs-sync-etl</artifactId>
<version>1.0.8</version>
<packaging>war</packaging>
<name>fqs-sync-etl Maven Webapp</name>
<url>http://maven.apache.org</url>
<repositories>
<repository>
<id>pentaho-releases</id>
<url>http://maven-repository.com/artifact/pentaho-kettle</url>
</repository>
<repository>
<id>people.apache.snapshots</id>
<url> http://repository.apache.org/content/groups/snapshots-group/ </url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-core</artifactId>
<version>7.1.0.0-12</version>
</dependency>
<dependency>
<groupId>com.verhas </groupId>
<artifactId>license3j</artifactId>
<version>1.0.7</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-dbdialog</artifactId>
<version>7.1.0.0-12</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-engine</artifactId>
<version>7.1.0.0-12</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>metastore</artifactId>
<version>7.1.0.0-12</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>


</dependency>
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
</dependency>
<dependency>
<groupId>com.topsci</groupId>
<artifactId>fqs-commons</artifactId>
</dependency>
<dependency>
<groupId>com.topsci</groupId>
<artifactId>fqs-enty</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
</dependency>

<dependency>
<groupId>com.github.pagehelper</groupId>
<artifactId>pagehelper</artifactId>
<version>4.0.1</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.8.4</version>
</dependency>

<dependency>
<groupId>com.jolbox</groupId>
<artifactId>bonecp-spring</artifactId>
<version>0.8.0.RELEASE</version>
</dependency>

<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.7.25</version>
</dependency>

<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
</dependency>
</dependencies>

<pluginRepositories>
<pluginRepository>
<id>apache.snapshots</id>
<name>Apache Snapshots</name>
<url> http://repository.apache.org/content/groups/snapshots-group/ </url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
<build>
<finalName>fqs-sync-etl</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<version>2.0-SNAPSHOT</version>
<configuration>
<url>http://192.168.6.221:8580/manager/text</url>
<server>192.168.1.22_tomcat7</server>
<username>admin</username>
<password>admin</password>
</configuration>
</plugin>
</plugins>
</build>
</project>

2.采用spring调度任务,本例采用手动触发机制。既用户会手动点击一次数据同步按钮触发任务

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx" xmlns:context="http://www.springframework.org/schema/context"
xmlns:cache="http://www.springframework.org/schema/cache"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/cache http://www.springframework.org/schema/cache/spring-cache.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

<bean id="scheduler" autowire="no"
class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="schedulerName" value="scheduler" />
<property name="jobFactory">
<bean class="com.atsig.etl.job.SpringQuarzFactory" />
</property>
<property name="applicationContextSchedulerContextKey" value="applicationContextKey" />

<property name="startupDelay" value="10" />

<property name="overwriteExistingJobs" value="true" />

<property name="autoStartup" value="true" />
<property name="waitForJobsToCompleteOnShutdown" value="true" />

<property name="jobDetails">
<list>
<ref bean="historyTaskDetail" />
<ref bean="baseTaskDetail" />
<ref bean="todayTaskDetail" />
</list>
</property>
</bean>
<bean id="todayTaskDetail"
class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.atsig.etl.job.TodayTask" />

<property name="durability" value="true" />

<property name="requestsRecovery" value="true" />
</bean>
<bean id="historyTaskDetail"
class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.atsig.etl.job.HistoryTask" />

<property name="durability" value="true" />

<property name="requestsRecovery" value="true" />
</bean>
<bean id="baseTaskDetail"
class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.atsig.etl.job.BaseTask" />

<property name="durability" value="true" />

<property name="requestsRecovery" value="true" />
</bean>

import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.BrokenBarrierException;
import javax.annotation.Resource;
import org.apache.commons.lang.StringUtils;
import org.canmeng.utils.BeanUtils;
import org.canmeng.utils.JsonUtil;
import org.canmeng.utils.PropertiesUtil;
import org.canmeng.utils.TransResultUtil;
import org.canmeng.utils.date.DateUtil;
import org.pentaho.di.core.KettleEnvironment;
import org.pentaho.di.core.util.EnvUtil;
import org.pentaho.di.trans.Trans;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
import com.fasterxml.jackson.annotation.JsonInclude.Include;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
/**
* @ClassName: TodayTask
* @Description: 当日数据同步任务
* @author tzhang
* @date 2017年9月4日上午9:48:24
*
*/
@Component
@Transactional
public class BaseTask implements Job {
@Autowired
private FqsDeptSubairlineMapper deptSubairlineMapper;
@Autowired
private FqsDeptMapper deptMapper;
@Autowired
private AommsServiceConfigMapper serviceConfigMapper;
private static PropertiesUtil config = PropertiesUtil.getInstance("config.properties");
private ObjectMapper objectMapper = new ObjectMapper();
@Resource
private AmqpTemplate innerRbmqTemplate;
@Resource
private TopicExchange ampTopic;
//权限消息变更通知
private final static String PERMS_ROUTINGKEY=config.getPropsValue("notification.perms.routingkey");
private static List<Trans> transList=new ArrayList<Trans>();
public static Logger logger = LoggerFactory.getLogger(BaseTask.class);
{
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
// 序列化的时候跳过null字段
objectMapper.setSerializationInclusion(Include.NON_NULL);
// objectMapper.configure(SerializationFeature.WRITE_NULL_MAP_VALUES, false);
objectMapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));
}
public void initBaseKtr(){
try {
// transList=LoadKtrUtil.loadKtr(SyncTableType.BASE);
//手工替换飞机注册号数据库连接
} catch (Exception e) {
logger.error("加载基础数据模型转换出错:", e);
}
}
private static JobExecutionException exception;
public JobExecutionException getException() {
return exception;
}
public void clearException(){
exception=null;
}
public void execute(JobExecutionContext context) throws JobExecutionException {
long startTime=System.currentTimeMillis();
try {
boolean isNotify=false;
transList=LoadKtrUtil.loadKtr(SyncTableType.BASE);
KettleEnvironment.init();
EnvUtil.environmentInit();
for (Trans trans :transList) {
trans.execute(null); // You can pass arguments instead of null.
trans.waitUntilFinished();
if (trans.getErrors()>0) {
this.exception= new JobExecutionException(trans.getName()+"执行转换出错,请联系管理员查看日志文件!");
break;
}
/*if(StringUtils.indexOf(trans.getName(), "AOMMS_SERVICE_DETAIL")>=0||) {
}*/
List<Map<String, Object>> transResults=TransResultUtil.loadResult(trans);
logger.info("运行基础数据抽取作业{}",trans.getName());
if(transResults.size()>0) {
isNotify=true;
String transName=trans.getName();
if(StringUtils.indexOf(transName, "AOMMS_SERVICE_DETAIL")>=0) {
updateOrDelDeptService(transResults);
}else if(StringUtils.indexOf(transName, "SSUBAIRLINE")>=0) {
updateOrDelDeptSubline(transResults);
}
}
}
if(isNotify) {
notifyIaETLModify(new MQProtocol<String>(MessageType.IA,DataFlag.U,PERMS_ROUTINGKEY));
}
long endTime=System.currentTimeMillis();
logger.info("{}耗时:{}s",context.getTrigger().getDescription(), (endTime-startTime)/1000);
} catch (Exception e) {
e.printStackTrace();
// exception=new JobExecutionException("执行转换出错,请联系管理员查看日志文件!");
}finally{
try {
SyncDataResources.waitSyncJob.await();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (BrokenBarrierException e) {
e.printStackTrace();
}
}
}
}

仰望星空的我

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Oralce增量同步方案-定时任务

使用Quarz调度Kettle执行定时抽取以下只罗列一些关键部分。重点关注java代码模块；1.工程Maven依赖;重点关注意红色字体标注内容。 &lt;?xml version="1.0"?&gt; &lt;project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://m...
复制链接

扫一扫