java整合NSQ实时分布式消息平台

kiss strong

已于 2024-01-30 10:37:21 修改

阅读量1.1k

点赞数 8

文章标签： java 分布式开发语言

于 2024-01-26 11:46:25 首次发布

本文链接：https://blog.csdn.net/qq_62408075/article/details/135859493

版权

NSQ简介

NSQ是什么

NSQ是一个实时分布式消息平台

特征

支持无单点故障的分布式拓扑
水平可扩展（无代理，无缝添加更多节点到集群）
基于低延迟推送的消息传递（性能）
组合负载平衡和多播样式消息路由
擅长流式处理（高吞吐量）和面向作业（低吞吐量）的工作负载
主要在内存中（超过高水位线消息将透明地保存在磁盘上）
供消费者查找生产者的运行时发现服务（nsqlookupd）
传输层安全 (TLS)
数据格式不可知
很少的依赖项（易于部署）和健全的、有限的默认配置
支持任何语言的客户端库的简单 TCP 协议
用于统计、管理操作和生产者的 HTTP 接口（无需发布客户端库）
与statsd集成以实现实时检测
强大的集群管理界面（nsqadmin）

保证

与任何分布式系统一样，实现目标需要做出明智的权衡。通过对这些权衡的现实保持透明，我们希望对 NSQ在生产中部署时的表现设定预期。

消息不持久（默认情况下）：尽管系统支持“释放阀”( --mem-queue-size)，在此之后消息将透明地保存在磁盘上，但它主要是一个内存中消息传递平台。--mem-queue-size可以设置为 0 以确保所有传入消息都保留到磁盘。在这种情况下，如果节点发生故障，您很容易受到故障面减少的影响（即操作系统或底层 IO 子系统发生故障）。没有内置复制。然而，有多种方法可以管理这种权衡，例如部署拓扑和以容错方式主动从动并将主题保留到磁盘的技术。
消息至少传递一次

与上面密切相关，这假设给定nsqd节点没有失败。

这意味着，由于各种原因，消息可以多次传递（客户端超时、断开连接、重新排队等）。执行幂等操作或重复数据删除是客户的责任。
收到的消息是无序

您不能依赖传递给消费者的消息的顺序。

nsqd与消息传递语义类似，这是重新排队、内存中和磁盘存储相结合以及每个节点不共享任何内容的结果。

通过在消费者中引入一个延迟窗口来接受消息并在处理之前对它们进行排序（尽管，为了保持这一不变性必须丢弃落在该窗口之外的消息）。
消费者最终找到所有主题生产者：发现服务 ( nsqlookupd ) 被设计为最终一致。 nsqlookupd节点不协调来维护状态或回答查询。网络分区不会影响可用性，因为分区的两侧仍然可以回答查询。部署拓扑对于缓解此类问题具有最显着的效果。

下载及安装

下载地址 NSQ Docs 1.3.0 - Installing

如果无法打开网站，也可以使用本文的资源

我这边选择的是nsq-1.3.0.linux-amd64.go1.21.5.tar.gz

把文件放到服务器上

解压文件到当前文件夹tar -xf nsq-1.3.0.linux-amd64.go1.21.5.tar.gz

查看安装包中有哪些文件

进入bin目录查看

这里面都是命令，其中常用的命令时 nsqadmin、nsqd、nsqlookupd，具体功能如下

常用命令及解释

nsqd

nsqd是接收消息、排队消息并将消息传递给客户端的守护进程。

它可以独立运行，但通常在具有实例的集群中配置nsqlookupd （在这种情况下，它将宣布用于发现的主题和通道）。

它监听两个 TCP 端口，一个用于客户端，另一个用于 HTTP API。它可以选择侦听第三个端口的 HTTPS

命令行选项

-auth-http-address value
    <addr>:<port> to query auth server (may be given multiple times)
-broadcast-address string
    address that will be registered with lookupd (defaults to the OS hostname) (default "yourhost.local")
-broadcast-http-port int
    HTTP port that will be registered with lookupd (defaults to the HTTP port that this nsqd is listening to)
-broadcast-tcp-port int
    TCP port that will be registered with lookupd (defaults to the TCP port that this nsqd is listening to)
-config string
    path to config file
-data-path string
    path to store disk-backed messages
-deflate
    enable deflate feature negotiation (client compression) (default true)
-e2e-processing-latency-percentile value
    message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)
-e2e-processing-latency-window-time duration
    calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds) (default 10m0s)
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4151")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-https-address string
    <addr>:<port> to listen on for HTTPS clients (default "0.0.0.0:4152")
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqd] ")
-lookupd-tcp-address value
    lookupd TCP address (may be given multiple times)
-max-body-size int
    maximum size of a single command body (default 5242880)
-max-bytes-per-file int
    number of bytes per diskqueue file before rolling (default 104857600)
-max-channel-consumers int
    maximum channel consumer connection count per nsqd instance (default 0, i.e., unlimited)
-max-deflate-level int
    max deflate compression level a client can negotiate (> values == > nsqd CPU usage) (default 6)
-max-heartbeat-interval duration
    maximum client configurable duration of time between client heartbeats (default 1m0s)
-max-msg-size int
    maximum size of a single message in bytes (default 1048576)
-max-msg-timeout duration
    maximum duration before a message will timeout (default 15m0s)
-max-output-buffer-size int
    maximum client configurable size (in bytes) for a client output buffer (default 65536)
-max-output-buffer-timeout duration
    maximum client configurable duration of time between flushing to a client (default 30s)
-max-rdy-count int
    maximum RDY count for a client (default 2500)
-max-req-timeout duration
    maximum requeuing timeout for a message (default 1h0m0s)
-mem-queue-size int
    number of messages to keep in memory (per topic/channel) (default 10000)
-min-output-buffer-timeout duration
    minimum client configurable duration of time between flushing to a client (default 25ms)
-msg-timeout duration
    default duration to wait before auto-requeing a message (default 1m0s)
-node-id int
    unique part for message IDs, (int) in range [0,1024) (default is hash of hostname) (default 248)
-output-buffer-timeout duration
    default duration of time between flushing data to clients (default 250ms)
-snappy
    enable snappy feature negotiation (client compression) (default true)
-statsd-address string
    UDP <addr>:<port> of a statsd daemon for pushing stats
-statsd-interval duration
    duration between pushing to statsd (default 1m0s)
-statsd-mem-stats
    toggle sending memory and GC stats to statsd (default true)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement) (default "nsq.%s")
-statsd-udp-packet-size int
    the size in bytes of statsd UDP packets (default 508)
-sync-every int
    number of messages per diskqueue fsync (default 2500)
-sync-timeout duration
    duration of time per diskqueue fsync (default 2s)
-tcp-address string
    <addr>:<port> to listen on for TCP clients (default "0.0.0.0:4150")
-tls-cert string
    path to certificate file
-tls-client-auth-policy string
    client certificate auth policy ('require' or 'require-verify')
-tls-key string
    path to key file
-tls-min-version value
    minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', or 'tls1.2') (default 769)
-tls-required
    require TLS for client connections (true, false, tcp-https)
-tls-root-ca-file string
    path to certificate authority file
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string
-worker-id
    [deprecated] use --node-id

nsqlookupd

nsqlookupd是管理拓扑信息的守护进程。客户端查询nsqlookupd以发现 nsqd特定主题的生产者，nsqd节点广播主题和频道信息。

有两个接口：用于nsqd广播的 TCP 接口和用于客户端执行发现和管理操作的 HTTP 接口。

命令行选项

-broadcast-address string
    address of this lookupd node, (default to the OS hostname) (default "yourhost.local")
-config string
    path to config file
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4161")
-inactive-producer-timeout duration
    duration of time a producer will remain in the active list since its last ping (default 5m0s)
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqlookupd] ")
-tcp-address string
    <addr>:<port> to listen on for TCP clients (default "0.0.0.0:4160")
-tombstone-lifetime duration
    duration of time a producer will remain tombstoned if registration remains (default 45s)
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string

nsqadmin

nsqadmin是一个 Web UI，用于实时查看聚合的集群统计信息并执行各种管理任务。

命令行选项

-acl-http-header string
    HTTP header to check for authenticated admin users (default "X-Forwarded-User")
-admin-user value
    admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions; acl-http-header is used to determine the authenticated user)
-allow-config-from-cidr string
    A CIDR from which to allow HTTP requests to the /config endpoint (default "127.0.0.1/8")
-base-path string
    URL base path (default "/")
-config string
    path to config file
-graphite-url string
    graphite HTTP address
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4171")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-http-client-tls-cert string
    path to certificate file for the HTTP client
-http-client-tls-insecure-skip-verify
    configure the HTTP client to skip verification of TLS certificates
-http-client-tls-key string
    path to key file for the HTTP client
-http-client-tls-root-ca-file string
    path to CA file for the HTTP client
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqadmin] ")
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-notification-http-endpoint string
    HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent
-nsqd-http-address value
    nsqd HTTP address (may be given multiple times)
-proxy-graphite
    proxy HTTP requests to graphite
-statsd-counter-format string
    The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.counters.%s.count")
-statsd-gauge-format string
    The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.gauges.%s")
-statsd-interval duration
    time interval nsqd is configured to push to statsd (must match nsqd) (default 1m0s)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement, must match nsqd) (default "nsq.%s")
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string

部署

单点部署

在一个 shell 中，启动nsqlookupd

进入bin目录执行 ./nsqlookupd

在另一个 shell 中，启动nsqd

执行命令 ./nsqd --lookupd-tcp-address=127.0.0.1:4160

在另一个 shell 中，启动nsqadmin

执行命令 ./nsqadmin --lookupd-http-address=127.0.0.1:4161

测试

打开浏览器输入http://192.168.31.128:4171/ 出现如下界面即可

集群部署

二进制部署

一台机器部署

解压压缩包三次，分别命名如下

进入nsq-node1的bin目录下面执行

./nsqlookupd

打开另一个shell，进入nsq-node1的bin目录下面执行

./nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128 --tcp-address=:4150 --http-address=:4151

打开另一个shell，进入nsq-node1的bin目录下面执行

./nsqadmin --lookupd-http-address=192.168.31.128:4161

这时查看浏览器打开浏览器输入http://192.168.31.128:4171/ 发现已经部署好了一个节点

此时只有一个节点

向集群中添加节点

进入nsq-node2的bin目录下面，执行

./nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128 --tcp-address=:4250 --http-address=:4251

进入nsq-node3的bin目录下面，执行

./nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128 --tcp-address=:4350 --http-address=:4351

这时查看控制台界面，发现有三个节点，集群已经部署成功

多台机器部署

机器192.168.31.128，192.168.31.129

在192.168.31.128上执行如下命令

./nsqlookupd

./nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128

./nsqadmin --lookupd-http-address=192.168.31.128:4161

在192.168.31.129上执行如下命令

./nsqlookupd

./nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.129
查看浏览器

docke-compose部署

docker-compose.yml

version: '3'
services:
  nsqlookupd:
    image: nsqio/nsq
    command: /nsqlookupd --broadcast-address=192.168.31.128
    ports:
      - "4160:4160"
      - "4161:4161"
  nsqd:
    image: nsqio/nsq
    command: /nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128 --tcp-address=:4150 --http-address=:4151
    depends_on:
      - nsqlookupd
    ports:
      - "4150:4150"
      - "4151:4151"
  nsqd2:
    image: nsqio/nsq
    command: /nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128 --tcp-address=:4250 --http-address=:4251
    depends_on:
      - nsqlookupd
    ports:
      - "4250:4250"
      - "4251:4251"
  nsqd3:
    image: nsqio/nsq
    command: /nsqd --lookupd-tcp-address=192.168.31.128:4160 --broadcast-address=192.168.31.128 --tcp-address=:4350 --http-address=:4351
    depends_on:
      - nsqlookupd
    ports:
      - "4350:4350"
      - "4351:4351"
  nsqadmin:
    image: nsqio/nsq
    command: /nsqadmin --lookupd-http-address=192.168.31.128:4161
    depends_on:
      - nsqlookupd
    ports:
      - "4171:4171"

启动执行 docker-compose up -d

查看浏览器

整合java（需要配置域名）

选择客户端

官网客户端如下

可以用java连接的只有几个，我们选一个功能最多的nsq-j测试

引入依赖

<dependency>
  <groupId>com.sproutsocial</groupId>
  <artifactId>nsq-j</artifactId>
  <version>1.4.1</version>
</dependency>

发送端

package com.cyz.nsqjlocal;

import com.sproutsocial.nsq.Publisher;

import java.util.concurrent.TimeUnit;

/**
 * @author cyz
 * @since 2024/1/25 9:44
 */
public class MqPublisher {
    public static void main(String[] args) {
        Publisher publisher = new Publisher("192.168.31.128:4150");
        for(int i=0;i<5;i++){
            System.out.println("===> "+i);
            byte[] data = ("Hello cyz nsq"+i).getBytes();
            publisher.publishBuffered("cyz_topic", data);
        }

    }
}

消费端

package com.cyz.nsqjlocal;

import com.sproutsocial.nsq.MessageHandler;
import com.sproutsocial.nsq.Subscriber;

/**
 * @author cyz
 * @since 2024/1/25 9:45
 */
public class MqConsumer {
    public static void main(String[] args) {
        Subscriber subscriber = new Subscriber("192.168.31.128:4161");
        subscriber.subscribe("cyz_topic", "test_channel", (MessageHandler) message -> {
            byte[] data = message.getData();
            System.out.println("Received:" + new String(data));
            message.finish();
        });
    }
}

测试代码

测试发送端

看界面

证明发送成功

测试消费端

这时发现没有收到消息，也没有报错，查阅网上资料也没有答案，没办法，只能源码断点查看了

源码查看nsq-j为什么无法消费

源码地址： https://github.com/sproutsocial/nsq-j

下载好解压，用idea打开

写好测试类，TestCyzCom

启动测试

发现连接被拒绝，使用debug查看

这边发现host竟然是域名，连接肯定报错啊，那这个host为什么是这个域名呢

我们再次debug

发现这个host是查询出来的，我们去页面看看

发现一模一样

总结：我们要的是ip，他给我们的是域名，那我们只能在host中配置域名映射了

打开C:\Windows\System32\drivers\etc\hosts文件，加入映射

再次启动测试消费

整合（java）不配置域名消费

只需要启动nsqd时设置broadcast即可

执行如下命令

  ./nsqd --lookupd-tcp-address=127.0.0.1:4160 -broadcast-address 192.168.31.128

再次消费，查看获取到的host

发现是ip，这下即使不配置域名也可以了

集群整合

发布消息

package com.cyz.cluster;

import com.sproutsocial.nsq.Client;
import com.sproutsocial.nsq.ListBasedBalanceStrategy;
import com.sproutsocial.nsq.Publisher;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;

/**
 * @author cyz
 * @since 2024/1/30 9:04
 */
public class Pub {
    public static void main(String[] args) {
        Client client = new Client();
        List<String> nsqList = Arrays.asList("192.168.31.128:4150", "192.168.31.129:4150");
        Publisher publisher = new Publisher(client, ListBasedBalanceStrategy.getRoundRobinStrategyBuilder(nsqList));
        for (int i = 0; i < 10; i++) {
            String msg = "hello cluster nsq " + i;
            System.out.println(msg);
            publisher.publish("cyz_topic_cluster", msg.getBytes(StandardCharsets.UTF_8));
        }
    }
}

消费消息

package com.cyz.cluster;

import com.sproutsocial.nsq.MessageHandler;
import com.sproutsocial.nsq.Subscriber;

/**
 * @author cyz
 * @since 2024/1/30 9:04
 */
public class Com {
    public static void main(String[] args) {
        Subscriber subscriber = new Subscriber("192.168.31.128:4161");
        subscriber.subscribe("cyz_topic_cluster", "test_channel", (MessageHandler) message -> {
            System.out.println(message.getId());
            byte[] data = message.getData();
            System.out.println("Received:" + new String(data));
            message.finish();
        });
    }
}

会消费集群里面的消息