nsq 笔记

1. nsq 笔记

1.1. nsq 组件

• nsqd is the daemon that receives, queues, and delivers messages to clients.
• nsqlookupd is the daemon that manages topology information. Clients query nsqlookupd to discover nsqd producers for a specific topic and nsqd nodes broadcast topics and channel information.
• nsqadmin is a Web UI to view aggregated cluster stats in realtime and perform various administrative tasks.

1.1.1. 默认端口

nsqlookupd 监听两个端口

• 4160 TCP 用于接收 nsqd 的广播, 记录 nsqd 的地址以及监听 TCP/HTTP 端口等;
• 4161 HTTP 用于接收客户端发送的管理和发现操作请求(增删话题, 节点等管理查看性操作等)。当 Consumer 进行连接时, 返回对应存在 Topic 的 nsqd 列表;

nsqd 监听两个端口

• 4151 HTTP Producer 使用 HTTP 协议的 curl 等工具生产数据; Consumer 使用 HTTP 协议的 curl 等工具消费数据;
• 4150 TCP Producer 使用 TCP 协议的 nsq-j 等工具生产数据; Consumer 使用 TCP 协议的 nsq-j 等工具消费数据;

nsqadmin 监听一个端口

• 4171 HTTP 用于管理页面

1.2. docker

https://nsq.io/deployment/docker.html

1.3. 源码示例

1.3.1. docker-compose.yml

version: '3'
services:
  nsqlookupd:
    image: nsqio/nsq
    command: /nsqlookupd
    ports:
      - "4160:4160"
      - "4161:4161"
  nsqd:
    image: nsqio/nsq
    command:  /nsqd --broadcast-address=nsqd --lookupd-tcp-address=nsqlookupd:4160
    depends_on:
      - nsqlookupd
    ports:
      - "4151:4151"
      - "4150:4150"
  nsqadmin:
    image: nsqio/nsq
    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
    depends_on:
      - nsqlookupd  
    ports:
      - "4171:4171"

Notes: Don’t forget to add nsqd as 127.0.0.1 at your /etc/hosts
Since our Producer and Consumer are located in our localhost and the NSQ environment runs in docker, this will provide a workaround so the nsqd service in docker will be accessible both from the docker side and localhost side.

To view the running containers status and mapped ports.

docker-compose ps

To see the logs from the running containers.

docker-compose logs

1.3.2. producer.go

package main

import (
	"encoding/json"
	"log"
	"time"

	"github.com/nsqio/go-nsq"
)

type Message struct {
	Name      string
	Content   string
	Timestamp string
}

func main() {
	//The only valid way to instantiate the Config
	config := nsq.NewConfig()
	//Creating the Producer using NSQD Address
	producer, err := nsq.NewProducer(":4150", config)
	if err != nil {
		log.Fatal(err)
	}
	defer producer.Stop()
	//Init topic name and message
	topic := "Topic_Example"
	msg := Message{
		Name:      "Message Name Example",
		Content:   "Message Content Example",
		Timestamp: time.Now().String(),
	}
	//Convert message as []byte
	payload, err := json.Marshal(msg)
	if err != nil {
		log.Println(err)
	}

	for i := 0; i < 24; i++ {
		//Publish the Message
		err = producer.Publish(topic, payload)
		if err != nil {
			log.Println(err)
		}

		log.Println("publish succeeded.")
		time.Sleep(time.Second)
	}
}

1.3.3. consumer.go

package main

import (
	"encoding/json"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/nsqio/go-nsq"
)

// type messageHandler struct{}
type Message struct {
	Name      string
	Content   string
	Timestamp string
}

func main() {
	//The only valid way to instantiate the Config
	config := nsq.NewConfig()
	//Tweak several common setup in config
	// Maximum number of times this consumer will attempt to process a message before giving up
	config.MaxAttempts = 10
	// Maximum number of messages to allow in flight
	config.MaxInFlight = 5
	// Maximum duration when REQueueing
	config.MaxRequeueDelay = time.Second * 900
	config.DefaultRequeueDelay = time.Second * 0
	//Init topic name and channel
	topic := "Topic_Example"
	channel := "Channel_Example"
	//Creating the consumer
	consumer, err := nsq.NewConsumer(topic, channel, config)
	if err != nil {
		log.Fatal(err)
	}
	// Set the Handler for messages received by this Consumer.
	// consumer.AddHandler(&messageHandler{})
	consumer.AddHandler(nsq.HandlerFunc(func(message *nsq.Message) error {
		//Process the Message
		var request Message
		if err := json.Unmarshal(message.Body, &request); err != nil {
			log.Println("Error when Unmarshaling the message body, Err : ", err)
			// Returning a non-nil error will automatically send a REQ command to NSQ to re-queue the message.
			return err
		}
		//Print the Message
		log.Println("Message")
		log.Println("--------------------")
		log.Println("Name : ", request.Name)
		log.Println("Content : ", request.Content)
		log.Println("Timestamp : ", request.Timestamp)
		log.Println("--------------------")
		log.Println("")
		// Will automatically set the message as finish
		return nil
	}))
	//Use nsqlookupd to find nsqd instances
	consumer.ConnectToNSQLookupd("127.0.0.1:4161")
	// wait for signal to exit
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	<-sigChan
	// Gracefully stop the consumer.
	consumer.Stop()
}

// HandleMessage implements the Handler interface.
// func (h *messageHandler) HandleMessage(m *nsq.Message) error {
// 	//Process the Message
// 	var request Message
// 	if err := json.Unmarshal(m.Body, &request); err != nil {
// 		log.Println("Error when Unmarshaling the message body, Err : ", err)
// 		// Returning a non-nil error will automatically send a REQ command to NSQ to re-queue the message.
// 		return err
// 	}
// 	//Print the Message
// 	log.Println("Message")
// 	log.Println("--------------------")
// 	log.Println("Name : ", request.Name)
// 	log.Println("Content : ", request.Content)
// 	log.Println("Timestamp : ", request.Timestamp)
// 	log.Println("--------------------")
// 	log.Println("")
// 	// Will automatically set the message as finish
// 	return nil
// }

The above example hardcoded the ip of nsqd into the consumer code, which is not a best practice. A better way to go about it is to point the consumer at nsqlookupd, which will transparently connect to the appropriate nsqd that happens to be publishing that topic.

In our example, we only have a single nsqd, so it’s an extraneous lookup. But it’s good to get into the right habits early, especially if you are a habitual copy/paster.

The consumer example only needs a one-line change to get this enhancement:

err := q.ConnectToNSQLookupd("127.0.0.1:4161")

Which will connect to the HTTP port of nsqlookupd.

1.3.4. 运行

• Start the docker using the command

docker-compose up -d

• Build the producer.go and consumer.go

go build producer.go
go build consumer.go

• Execute the consumer binary first then proceed to execute the producer binary multiple times

./consumer
./producer

Notes: The consumer will need to sometimes to query the nsqlookupd

You could check the NSQadmin by :

• Open http://localhost:4171 at your browser

It will list down all the available topics