Nginx与ZeroMQ的结合

最近在Nginx wiki上看到两个与zmq结合的nginx第三方模块，顿时眼前一亮。在此记录一下，有时间再摸索一下怎么个玩

一、LogZMQ模块

ZeroMQ, zero-em-queue, is a protocol for messages exchange. It’s a easy way to communicate using any language or platform via inproc, IPC, TCP, TPIC or multicast. It’s asynchronous and only requires a small library.

Note

This module is not distributed with the nginx source. See the installation instructions.

Status

This module is already production ready.

Description

This is a nginx logger module integrated with ZeroMQ library.

nginx-log-zmq provides a very efficient way to log data for one or more PUB/SUB subscribers, over one or more different endpoints. This can be useful for data gathering and processing.

The message format can be the same as the tradicional log format which gives a interesting way to tail data via the network or exploring other text formats like JSON. As with the traditional log, it’s possible to use nginx variables updated each request.

All messages are sent asynchronously and do not block the normal behaviour of the nginx server. As expected, the connections are resilient to network failures.

Synopsis

http {
    # simple message to an IPC endpoint with 4 threads and 1000 queue elements

    log_zmq_server main "/tmp/main.ipc" ipc 4 1000;
    log_zmq_endpoint  main "/topic/";

    log_zmq_format main '{"remote_addr":"$remote_addr"}'

    # send messages to a subscriber listening at 127.0.0.1:5556

    log_zmq_server secondary 127.0.0.1:5556 tcp 4 1000;

    # set secondary endpoint
    log_zmq_endpoint secondary "/endpoint/";

    # set format using multiline
    log_zmq_format secondary '{"request_uri":"$request_uri",'
                             ' "status":"$status"}';


    server {

        location /status {
            # mute all messages from log_zmq for this location

            log_zmq_off all;
        }

        location /endpoint {
            # mute main messages from log_zmq for this location

            log_zmq_off main;
        }
    }
}

Directives

log_zmq_server

Syntax:	log_zmq_server <definition_name> <address> <ipc|tcp> <threads> <queue size>
Default:	no
Context:	http

Configures a server (PUB/SUB subscriber) to connect to.

The following options are required:

definition_name <name> - the name that nginx will use to identify this logger instance.

address <path>|<ipaddress>:<port> - the subscriber’s address. If you are using the IPC protocol, you should specify the <path> for the unix socket. If you are using the TCP protocol, you should specify the <ipaddress>and <port> where your ZeroMQ subscriber is listening.

protocol <ipc|tcp> - the protocol to be used for communication.

threads <integer> - the number of I/O threads to be used.

queue_size <integer> - the maximum queue size for messages waiting to be sent.

log_zmq_endpoint

Syntax:	log_zmq_endpoint <definition_name> “<topic>”
Default:	no
Context:	http

Configures the topic for the ZeroMQ messages.

definition_name <name> - the name that nginx will use to identify this logger instance.

topic <topic> - the topic for the messages. This is a string (which can be a nginx variable) prepended to every sent message. For example, if you send the message “hello” to the “/talk:” topic, the message will end up as “/talk:hello”.

Example:

http {
    log_zmq_server main "/tmp/example.ipc" 4 1000;

    # send a message for for an topic based on response status

    log_zmq_endpoint main "/remote/$status";
}

log_zmq_format

Syntax:	log_zmq_format <definition_name> “<format>”
Default:	no
Context:	http

Configures the ZeroMQ message format.

definition_name <name> - the name that nginx will use to identify this logger instance.

format <format> - the format for the messages. This defines the actual messages sent to the PUB/SUB subscriber. It follows the sames rules as the standard log_format directive. It is possible to use nginx variables here, and also to break it over multiple lines.

http {
    log_zmq_format main '{"line1": value,'
                        ' "line2": value}';
}

log_zmq_off

Syntax:	log_zmq_off<definition_name>|all
Default:	no
Context:	location

Turn off ZeroMQ logging in the current context.

definition_name <name> the name of the logger instance to be muted. If the special all name is used, all logger instances are muted.

Installation

To build a nginx binary containting this module:

• Download the latest version of this module from githup
• Grab the nginx source code from nginx.org, for example, version 1.6.2 (see nginx compatibility), and then build it like so:

./configure --prefix=/usr/local/nginx --add-module=/path/to/nginx-log-zmq

make
make install

NGINX Compatibility

The following versions of nginx are known to work with this module:

• 1.8.0
• 1.6.x (last tested: 1.6.2)
• 1.5.x
• 1.4.x (last tested: 1.4.4)

二、ngx_zeromq模块

About

ngx_zeromq is a transport module which allows nginx to use ZeroMQ message-oriented transport layer when communicating with upstream servers.

It's level 7 protocol agnostic, which means that it can be used with any well-behaving upstream modules (proxy, fastcgi,uwsgi, scgi, etc.).

Status

This is experimental module.

Caveats

At this time, ngx_zeromq support only REQ/REP ZeroMQ sockets, which means that upstream response must be delivered in a single ZeroMQ message (although it can be multipart message).

Each message part must fit into upstream buffer (as defined by proxy_buffer_size, proxy_buffers and other directives).

ZeroMQ is hungry for file descriptors, it will crash and bring worker process down with it once it runs out of them. In order to guarantee that it doesn't happen, worker connections limit (see: worker_connections) should be set few times lower than the file descriptors limit (see: worker_rlimit_nofile).

SSL is another transport module, which means that it cannot be combined with ngx_zeromq on the same connection (but this matters only for communication with upstream servers, clients can still use SSL when connecting to nginx).

Configuration directives

zeromq_threads

• syntax: zeromq_threads <number>
• default: 1
• context: main

Configure number of ZeroMQ I/O threads to be used by each worker process.

zeromq_local

• syntax: zeromq_local <socket_type> <local_endpoint>
• default: none
• context: upstream

Configure local ZeroMQ endpoint (must use random port numbers).

zeromq_remote

• syntax: zeromq_remote <socket_type> <remote_endpoint>
• default: none
• context: upstream

Configure remote ZeroMQ endpoint (cannot use random port numbers).

zeromq_single

• syntax: zeromq_single on|off
• default: off
• context: upstream

Enable single mode which allows use of predefined port numbers for local endpoints. It comes at a price (only one worker can bind to such endpoint and it will stop working after nginx reload) and it's supposed to be used only during testing and development. Do not use this mode in production.

Sample configuration

Use HTTP over ZeroMQ (using standard proxy module).

http {
    upstream blackhole {
        zeromq_remote   REQ tcp://127.0.0.1:5555;
    }

    server {
        location / {
            proxy_pass  http://blackhole;
        }
    }
}

模块的使用示例可以参考以下博文

Nginx使用ngx_zeromq模块返回502错误的解决方法

当使用了ngx_zeromq模块在nginx中，使用以下命令进行编译安装：

./configure --with-debug --add-module=./zeromq_module/ngx_zeromq/ --with-http_ssl_module

开启一个zero_mq的ZMQ_REP模式的服务端程序，那么在web中输入http://127.0.0.1/时，返回502页面，再查看默认编译安装目录/usr/local/nginx/logs/error.log日志中，可以看到以下的错误导致的：

2015/06/22 23:46:28 [alert] 43143#0: *1 connect() failed (88: Socket operation on non-socket) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://tcp://127.0.0.1:5555/", host: "127.0.0.1"

找到此行错误的代码所在文件为:

src/http/ngx_http_upstream.c中的ngx_http_upstream_test_connect函数中，原来是由于getsockopt造成的，那么如何解决这个问题呢？

经过自己写zeromq的REQ和REP模式的方法后，同样使用zeromq创建的连接的socketfd来使用getsockopt来进行使用，也会出现这样的错误，但是这并没有影响zeromq本身的通信，那也即是说nginx中的upstream模块中的getsockopt这个方法出错了对于zeromq来说无关紧要，那么将ngx_http_upstream_test_connect函数直接返回NGX_OK即可解决这个问题，再进行安装测试即可正常看到页面了。

REF:

ngx_zeromq模块：

https://github.com/FRiCKLE/ngx_zeromq

测试ngx_zeromq工程：

https://github.com/jamesmarlowe/ngx-zmq-sample