RockerMq集群部署及简单使用

RockerMq分布式存储架构

1. Name Server
   • NameServer是RockerMQ的寻址服务，用于聚合和管理Broker的路由信息，客户端持有Topic通过NameServer获取对应的Broker路由信息，从而与其Broker做连接
   • NameServer集群本身没有状态，各NameServer节点也不做通信，仅提供负载均衡效果
   • NamsServer通过Broker上报状态将Broker路由清单存储至内存中（类似于Eureka）
   • 如果NameServer服务都停止了，仅会影响路由的更新，而如果客户端已和Broker做了长连接，则不会被影响之间的通信
2. Broker节点
   • 是消息存储、消费和转发的服务器
   • Broker集群通过Group进行组区分，每个组有一个Master节点和若干个Slave节点，其中只有Master具有写操作
   • Slave会从Master同步数据，同步策略由Master配置决定，可采用同步双写和异步复制两种
   • 客户端默认从Master中消费数据，如果NameServer感知Broker挂机后，则会从Slave中消费
   • Broker会和所有NameServer节点做长连接，并注册Topic信息

RockerMq集群部署

注意：本次部署通过docker-compose一台机部署，生产环境可分开

1. 创建name server文件目录

mkdir -p nameserver/{nameserver1,nameserver2}/{logs,store}

2. 创建broker server文件目录

mkdir -p broker/{broker-master,broker-salve1,broker-salve2}/{logs,store,conf}

3. 编写主节点配置文件：vi broker/broker-master/conf/broker.conf

#集群名称
brokerClusterName=DefaultCluster
#broker名称
brokerName=broker-master
#对外网暴露的IP地址，brokerIP1是外界访问IP，brokerIP2是Slave同步master所使用的，如果是从节点配置，则不需要brokerIP2
brokerIP1=192.168.142.31
brokerIP2=192.168.142.31
# broker节点Id，0是主节点
brokerId=0
#默认的Topic队列数
defaultTopicQueueNums=4
#自动创建Topic，建议线上关闭，线下开启
autoCreateTopicEnable=true
#自动创建订阅组，建议线上关闭，线下开启
autoCreateSubscriptionGroup=true
#broker对外服务监听端口
listenPort=10911
#默认0时删除文件
deleteWhen=00
#文件保留120小时
fileReservedTime=120
#commitLog default size=1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue per file=30W
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比，超过则报警
#diskMaxUsedSpaceRatio=88
#store path
storePathRootDir=/home/rocketmq/store
#commitLog path
storePathCommitLog=/home/rocketmq/store/commitlog
#consumer path
storePathConsumeQueue=/home/rocketmq/store/consumequeue
#index path
storePathIndex=/home/rocketmq/store/store/index
#checkpoint file path
storeCheckpoint=/home/rocketmq/store/checkpoint
#abort path
abortFile=/home/rocketmq/store/abort
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
# Broker role 
#-ASYNC_MASTER 异步复制Master   - SYNC_MASTER 同步双写Master
# 从节点的话：brokerRole=SLAVE
brokerRole=SYNC_MASTER
flushDiskType=SYNC_FLUSH
#checkTransactionMessageEnable=false
#sendMessageThreadPoolNums=128
#pullMessageThreadPoolNums=128
#磁盘满、且无过期文件情况下 TRUE 表示强制删除文件,优先保证服务可用;FALSE 标记服务不可用,文件 不删除 
cleanFileForciblyEnable=true

4. 将配置文件拷贝到从节点中
   • 修改节点名称：brokerName
   • 修改节点ID：brokerId
   • 去掉BrokerIP2配置（此配置是master提供给slave同步用的）
   • 修改节点Role：brokerRole=SLAVE

cp broker/broker-master/conf/broker.conf broker/broker-salve1/conf/broker.conf;
cp broker/broker-master/conf/broker.conf broker/broker-salve2/conf/broker.conf

5. 编写启动docker-compose配置文件：vi docker-compose.yml

version: '2'
services:
  namesrv1:
    image: rocketmqinc/rocketmq
    container_name: rmqnamesrv1  
    ports:
      - 9876:9876
    volumes:
      - ./nameserver1/logs:/home/rocketmq/logs
      - ./nameserver1/store:/home/rocketmq/store
    command: sh mqnamesrv
  namesrv2:
    image: rocketmqinc/rocketmq
    container_name: rmqnamesrv2   
    ports:
      - 9877:9876
    volumes:
      - ./nameserver2/logs:/home/rocketmq/logs
      - ./nameserver2/store:/home/rocketmq/store
    command: sh mqnamesrv    

  broker1:
    image: rocketmqinc/rocketmq
    container_name: rmqbroker1   
    ports:
      - 10909:10909
      - 10911:10911
      - 10912:10912
    volumes:
      - ./broker1/logs:/home/rocketmq/logs
      - ./broker1/store:/home/rocketmq/store
      - ./broker1/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf
    command: sh mqbroker -n namesrv1:9876,namesrv2:9876 -c ./broker1/conf/broker.conf
    depends_on:
      - namesrv1
      - namesrv2
    environment:
      - JAVA_HOME=/opt/module/java/jdk1.8.0_161
  broker2:
    image: rocketmqinc/rocketmq
    container_name: rmqbroker2   
    ports:
      - 20909:10909
      - 20911:10911
      - 20912:10912
    volumes:
      - ./broker2/logs:/home/rocketmq/logs
      - ./broker2/store:/home/rocketmq/store
      - ./broker2/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf
    command: sh mqbroker -n namesrv1:9876,namesrv2:9876 -c ./broker2/conf/broker.conf
    depends_on:
      - namesrv1
      - namesrv2
    environment:
      - JAVA_HOME=/opt/module/java/jdk1.8.0_161      
  console:
    image: styletang/rocketmq-console-ng
    container_name: rocketmq-console-ng    
    ports:
      - 8076:8080
    depends_on:
      - namesrv1
      - namesrv2
    environment:
      - JAVA_OPTS= -Dlogging.level.root=info   -Drocketmq.namesrv.addr=rmqnamesrv1:9876,rmqnamesrv2:9876 
      - Dcom.rocketmq.sendMessageWithVIPChannel=false

6. 启动服务：docker-compose up -d
7. 网页查看：http://192.168.142.31:8076/#/
   

SpringBoot简单案例

1. pom导入依赖

<dependencies>
    <dependency>
        <groupId>com.alibaba.rocketmq</groupId>
        <artifactId>rocketmq-client</artifactId>
        <version>3.0.10</version>
    </dependency>
    <dependency>
        <groupId>com.alibaba.rocketmq</groupId>
        <artifactId>rocketmq-all</artifactId>
        <version>3.0.10</version>
        <type>pom</type>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>1.1.1</version>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-core</artifactId>
        <version>1.1.1</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.10</version>
        <scope>test</scope>
    </dependency>
</dependencies>

2. 编写生产者代码

public class TestProducer {
    public static void main(String[] args) throws Exception {
        String producerGroup = "my-rmq-group";
        String namesrvAddr = "192.168.142.31:9876;192.168.142.31:9877;";
        String producerName = "producer";
        String topic = "test-topic";
        String tags = "TagA";
        DefaultMQProducer producer = new DefaultMQProducer(producerGroup);
        producer.setNamesrvAddr(namesrvAddr);
        producer.setInstanceName(producerName);
        // 启动生产者
        producer.start();
        for(int i = 0; i < 10; i++) {
            Thread.sleep(1000);
            String body = "这是一条生产信息，信息编号是：" + (i + 1);
            Message message = new Message(topic, tags, body.getBytes());
            // 使用生产者生产信息
            SendResult sendResult = producer.send(message);
            System.out.println(sendResult);
        }
        // 关闭生产者
//        producer.shutdown();
    }
}

3. 编写消费者代码

public class TestConsumer {
    public static void main(String[] args) throws Exception {
        String group = "my-rmq-group";
        String namesrvAddr = "192.168.142.31:9876;192.168.142.31:9877;";
        String consumerName = "consumer";
        String topic = "test-topic";
        String tags= "TagA";
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group);
        consumer.setNamesrvAddr(namesrvAddr);
        consumer.setInstanceName(consumerName);
        consumer.subscribe(topic, tags);
        // 事件监听器
        consumer.registerMessageListener((MessageListenerConcurrently) (list, consumeConcurrentlyContext) -> {
            for (MessageExt msg : list) {
                System.out.println("消费的消息：" + msg.getMsgId() + "，消息内容：" + new String(msg.getBody()));
            }
            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        });
        consumer.start();
        System.out.println("Rocket消费者客户端启动……");
    }
}