nsq详细教程2 各组件详细配置及功能

上一篇中我们了解了nsq的基本功能，点击查看
这里我们比较全面和深入的了解其部分工作机制,更多信息请参考nsq官网

NSQD

nsqd是一个守护进程，负责接收、排队、分发消息到消费者。可以单独运行也可以通过 注册到 nsqlookupd 服务实现集群部署。
默认监听俩端口 4150 （tcp客户端） 4151 （http api 接口） 。也可以配置端口支持https 服务端口

配置项

-auth-http-address value
    <addr>:<port> to query auth server (may be given multiple times)
-broadcast-address string
    address that will be registered with lookupd (defaults to the OS hostname) (default "yourhost.local")
-broadcast-http-port int
    HTTP port that will be registered with lookupd (defaults to the HTTP port that this nsqd is listening to)
-broadcast-tcp-port int
    TCP port that will be registered with lookupd (defaults to the TCP port that this nsqd is listening to)
-config string
    path to config file
-data-path string
    path to store disk-backed messages
-deflate
    enable deflate feature negotiation (client compression) (default true)
-e2e-processing-latency-percentile value
    message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)
-e2e-processing-latency-window-time duration
    calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds) (default 10m0s)
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4151")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-https-address string
    <addr>:<port> to listen on for HTTPS clients (default "0.0.0.0:4152")
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqd] ")
-lookupd-tcp-address value
    lookupd TCP address (may be given multiple times)
-max-body-size int
    maximum size of a single command body (default 5242880)
-max-bytes-per-file int
    number of bytes per diskqueue file before rolling (default 104857600)
-max-channel-consumers int
    maximum channel consumer connection count per nsqd instance (default 0, i.e., unlimited)
-max-deflate-level int
    max deflate compression level a client can negotiate (> values == > nsqd CPU usage) (default 6)
-max-heartbeat-interval duration
    maximum client configurable duration of time between client heartbeats (default 1m0s)
-max-msg-size int
    maximum size of a single message in bytes (default 1048576)
-max-msg-timeout duration
    maximum duration before a message will timeout (default 15m0s)
-max-output-buffer-size int
    maximum client configurable size (in bytes) for a client output buffer (default 65536)
-max-output-buffer-timeout duration
    maximum client configurable duration of time between flushing to a client (default 30s)
-max-rdy-count int
    maximum RDY count for a client (default 2500)
-max-req-timeout duration
    maximum requeuing timeout for a message (default 1h0m0s)
-mem-queue-size int
    number of messages to keep in memory (per topic/channel) (default 10000)
-min-output-buffer-timeout duration
    minimum client configurable duration of time between flushing to a client (default 25ms)
-msg-timeout duration
    default duration to wait before auto-requeing a message (default 1m0s)
-node-id int
    unique part for message IDs, (int) in range [0,1024) (default is hash of hostname) (default 248)
-output-buffer-timeout duration
    default duration of time between flushing data to clients (default 250ms)
-snappy
    enable snappy feature negotiation (client compression) (default true)
-statsd-address string
    UDP <addr>:<port> of a statsd daemon for pushing stats
-statsd-interval duration
    duration between pushing to statsd (default 1m0s)
-statsd-mem-stats
    toggle sending memory and GC stats to statsd (default true)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement) (default "nsq.%s")
-statsd-udp-packet-size int
    the size in bytes of statsd UDP packets (default 508)
-sync-every int
    number of messages per diskqueue fsync (default 2500)
-sync-timeout duration
    duration of time per diskqueue fsync (default 2s)
-tcp-address string
    <addr>:<port> to listen on for TCP clients (default "0.0.0.0:4150")
-tls-cert string
    path to certificate file
-tls-client-auth-policy string
    client certificate auth policy ('require' or 'require-verify')
-tls-key string
    path to key file
-tls-min-version value
    minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', or 'tls1.2') (default 769)
-tls-required
    require TLS for client connections (true, false, tcp-https)
-tls-root-ca-file string
    path to certificate authority file
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string
-worker-id
    [deprecated] use --node-id

http接口
nsqd默认提供了很多http接口，包括发送消息、创建/删除/清空 topic 等。具体如下：

/ping - liveness
/info - version
/stats - comprehensive runtime telemetry
/pub - publish a message to a topic
/mpub - publish multiple messages to a topic
/config - configure nsqd
/debug/pprof - pprof debugging portal
/debug/pprof/profile - generate pprof CPU profile
/debug/pprof/goroutine - generate pprof goroutine profile
/debug/pprof/heap - generate pprof heap profile
/debug/pprof/block - generate pprof blocking profile
/debug/pprof/threadcreate - generate pprof OS thread profile
v1 namespace (as of nsqd v0.2.29+):

/topic/create - create a new topic
/topic/delete - delete a topic
/topic/empty - empty a topic
/topic/pause - pause message flow for a topic
/topic/unpause - unpause message flow for a topic
/channel/create - create a new channel
/channel/delete - delete a channel
/channel/empty - empty a channel
/channel/pause - pause message flow for a channel
/channel/unpause - unpause message flow for a channel

接口调用实例参考如下，其他详情参考官方文档 官网

调试分析
nsqd提供了一套调试分析接口，可以直接和go的pprof进行集成，只需要运行如下命令既可以进行cpu和内存分析

# memory profiling
$ go tool pprof http://localhost:4151/debug/pprof/heap

# cpu profiling
$ go tool pprof http://localhost:4151/debug/pprof/profile

TLS

如果nsqd配置了 --tls-cert 和 --tls-key 选项，那么客户端可以通过tls 来和nsqd通信，更加安全。可以通过–tls-required (as of nsqd v0.2.28+).配置要求客户端必须使用 tls 来通讯。 这是一种客户端身份验证方式。
可以通过 --tls-client-auth-policy (require or require-verify) 选项配置客户端证书策略。

require - 客户端必须提供一个证书，否则拒绝请求
require-verify - 客户端必须提供一个CA（或下级CA） 签发的证书,否则请求会被拒绝。
可以通过如下命令签发证书

$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes

AUTH
可以通过 -auth-http-address=host:port 参数来指定一个认证服务地址，指定后nsqd就需要做身份认证了。这个参数配置的地址必须是符合 http认证协议。由于nsq认证只对tcp协议有效，http协议无效，us哦一开启认证后，nsqd最好只对外暴露tcp服务，http服务不对外暴露。
认证服务必须提供如下地址来接收http请求如下，就是必须自己发布一个 授权服务，这个授权服务必须满足nsq的授权规范，而且官方给出了实例的基于python授权服务。网上有人实现了基于go的授权服务. nsq-auth

/auth?remote_ip=...&tls=...&auth_secret=...

并且返回一下响应

{
  "ttl": 3600,
  "identity": "username",
  "identity_url": "https://....",
  "authorizations": [
    {
      "permissions": [
        "subscribe",
        "publish"
      ],
      "topic": ".*",
      "channels": [
        ".*"
      ]
    }
  ]
}

nsqd服务将缓存这个认证结果，直到达到超时时间，重新请求并缓存。
使用命令行 工具时 可以使用–reader-opt 来穿密码

$ nsq_tail ... -reader-opt="tls_v1,true" -reader-opt="auth_secret,$SECRET"

认证机制测是测试实战 见下一篇文章

NSQLOOKUPD

这个服务是用来管理nsqd的注册信息，同时供客户端查询指定topic和channle 所在节点。信息。
监听两个端口 4160 tcp 给nsqd 注册调用。4161http端口 供客户端调用

配置参数
支持的参数信息如下：

-broadcast-address string
    address of this lookupd node, (default to the OS hostname) (default "yourhost.local")
-config string
    path to config file
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4161")
-inactive-producer-timeout duration
    duration of time a producer will remain in the active list since its last ping (default 5m0s)
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqlookupd] ")
-tcp-address string
    <addr>:<port> to listen on for TCP clients (default "0.0.0.0:4160")
-tombstone-lifetime duration
    duration of time a producer will remain tombstoned if registration remains (default 45s)
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string

http接口
提供了 各种http接口供调用，详情参考官网，这里仅做个别测试
/nodes :显示集群节点信息
/info ：显示版本信息
/ping ：验证服务状态
/topics：显示所有的topic
/channel/create:创建channel
/channel/delete:删除channel
/topic/create:创建topic
/topic/delete:删除topic
/lookup ：查找指定topic 的生产者

/topic/tombstone：删除指定topic的生产者。

NSQADMIN

一个web管理工具，可以展示、管理nsq集群的各种信息。

-acl-http-header string
    HTTP header to check for authenticated admin users (default "X-Forwarded-User")
-admin-user value
    admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions; acl-http-header is used to determine the authenticated user)
-allow-config-from-cidr string
    A CIDR from which to allow HTTP requests to the /config endpoint (default "127.0.0.1/8")
-base-path string
    URL base path (default "/")
-config string
    path to config file
-graphite-url string
    graphite HTTP address
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4171")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-http-client-tls-cert string
    path to certificate file for the HTTP client
-http-client-tls-insecure-skip-verify
    configure the HTTP client to skip verification of TLS certificates
-http-client-tls-key string
    path to key file for the HTTP client
-http-client-tls-root-ca-file string
    path to CA file for the HTTP client
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqadmin] ")
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-notification-http-endpoint string
    HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent
-nsqd-http-address value
    nsqd HTTP address (may be given multiple times)
-proxy-graphite
    proxy HTTP requests to graphite
-statsd-counter-format string
    The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.counters.%s.count")
-statsd-gauge-format string
    The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.gauges.%s")
-statsd-interval duration
    time interval nsqd is configured to push to statsd (must match nsqd) (default 1m0s)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement, must match nsqd) (default "nsq.%s")
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string

可以通过 notification-http-endpoint 参数配置一个地址，当集群有管理操作发生时为这个url推送通知。通知内容包括 时间 动作等。
管理界面展示的信息，请自行查看。