使用mossrose构建分布式作业集群

本文档介绍了如何使用Mossrose构建分布式作业集群,涵盖了社区资源、安装步骤、核心概念,如SimpleJob、DistributedJob和MapReduceJob等,并提供了快速入门示例,包括实现简单任务、分布式任务和流式任务。
摘要由CSDN通过智能技术生成

使用mossrose构建分布式作业集群


社区

文档

Requirement

Installation

<dependency>
  <groupId>com.jiuxian</groupId>
  <artifactId>mossrose</artifactId>
  <version>2.2.0-RELEASE</version>
</dependency>

Key concept

SimpleJob

简单任务

DistributedJob

分布式任务,通过Slicer将作业分隔成多个子任务,子任务在集群内分布执行

StreamingJob

分布式流式任务,解决分片非常多时DistributedJob内存占用大的问题

MapReduceJob

MapReduce任务

MossroseProcess

多个MossroseProcess组成集群,集群保证有且只有一个节点竞选成为主节点,主节点负责触发作业;所有节点都是工作节点,主节点触发的任务会在所有工作节点上分布执行

MossroseConfig

Mossrose配置,包括集群元信息和任务元信息

Quick Start

Implement a simple job
public class SimpleExampleJob implements SimpleJob {

    @Override
    public Executor executor() {
        return new Executor() {

            @Override
            public void execute() {
                LOGGER.info("SimpleJob");
            }
        };
    }

}
Config the job in spring
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" 
    xmlns:mossrose="https://jiuxiantuan.github.io/mossrose"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        https://jiuxiantuan.github.io/mossrose https://jiuxiantuan.github.io/mossrose/mossrose.xsd">

    <bean class="com.jiuxian.jobs.bean.BusinessBean" />
    <bean id="simpleExampleJob" class="com.jiuxian.jobs.job.SimpleExampleJob" />
    <bean id="distributedExampleJob" class="com.jiuxian.jobs.job.DistributedExampleJob" />
    <bean id="streamingExampleJob" class="com.jiuxian.jobs.job.StreamingExampleJob" />

    <mossrose:springholder />
    <mossrose:config>
        <mossrose:cluster name="mossrose-example" discovery-zk="localhost:2181" />
        <mossrose:jobs>
            <mossrose:job id="SimpleExampleJob" cron="0/5 * * * * ?" job-bean-name="simpleExampleJob" group="example" />
            <mossrose:job id="DistributedExampleJob" cron="0/15 * * * * ?" job-bean-name="distributedExampleJob" group="example" />
            <mossrose:job id="StreamingExampleJob" cron="0/20 * * * * ?" job-bean-name="streamingExampleJob" group="example"
                description="分布式流式任务示例" />
        </mossrose:jobs>
    </mossrose:config>
    <mossrose:process />
    <mossrose:ui />

</beans>
Start the job
applicationContext.getBean(MossroseProcess.class).run();

Distributed Job

Implement a distributed job
public class SomeDistributedJob implements DistributedJob<String> {

    private static final Logger LOGGER = LoggerFactory.getLogger(SomeDistributedJob.class);

    @Override
    public Slicer<String> slicer() {
        return new Slicer<String>() {

            @Override
            public List<String> slice() {
                return Splitter.on(" ").splitToList("This is a test on the mossrose distributed job, how are you feeling?");
            }
        };
    }

    @Override
    public com.jiuxian.mossrose.job.DistributedJob.Executor<String> executor() {
        return new Executor<String>() {

            @Override
            public void execute(String item) {
                LOGGER.info(Thread.currentThread() + " DistributedJob: " + item);
            }
        };
    }

}

Streaming Job

Implement a streaming job

DistributedJob需要把需要分布式执行的任务集合一次性的返回,在集合非常大的时候会存在内存的问题,StreamingJob解决了这个问题,任务可以以流的方式不断输出,以保证内存可以及时释放。

public class SomeStreamingJob implements StreamingJob<String> {

    private static final Logger LOGGER = LoggerFactory.getLogger(SomeStreamingJob.class);

    @Override
    public Streamer<String> streamer() {
        return new Streamer<String>() {

            private List<String> list = Lists.newArrayList("Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday");

            private int index = 0;

            @Override
            public boolean hasNext() {
                return index < list.size();
            }

            @Override
            public String next() {
                return list.get(index++);
            }
        };
    }

    @Override
    public Executor<String> executor() {
        return new Executor<String>() {

            @Override
            public void execute(String item) {
                LOGGER.info(Thread.currentThread() + " StreamingJob: " + item);
            }
        };
    }

}

MapReduce Job

Implement a map/reduce job
public class MapReduceExampleJob implements MapReduceJob<Integer, Integer> {

    @Override
    public com.jiuxian.mossrose.job.MapReduceJob.Mapper<Integer> mapper() {
        return new Mapper<Integer>() {

            @Override
            public List<Integer> map() {
                return Lists.newArrayList(1, 2, 3, 4, 5, 6, 7);
            }
        };
    }

    @Override
    public com.jiuxian.mossrose.job.MapReduceJob.Executor<Integer, Integer> executor() {
        return new Executor<Integer, Integer>() {

            @Override
            public Integer execute(Integer item) {
                return item * 2;
            }
        };
    }

    @Override
    public com.jiuxian.mossrose.job.MapReduceJob.Reducer<Integer> reducer() {
        return new Reducer<Integer>() {

            @Override
            public void reduce(List<Integer> rs) {
                LOGGER.info("Reduce result : {}", rs);
            }
        };
    }

}
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值