Linux下使用hiredis库与libevent实现异步接口的I/O复用

1 前言

    之前的一篇文章《Linux下使用hiredis库实现优先级队列》，用的同步的接口实践；

    后来遇到一个场景，同时需要处理Redis订阅的消息，又需要处理其他网络socket操作、定时器操作，当然多线程是一个思路，本文尝试从Reactive模式上解决这个问题，即用redis的异步接口，与libevent进行对接。

    其实最终目的就是就是Redis接入到这篇文章编程实例里面：《Linux下使用libevent库实现服务器端编程》

2.接口

2.1 async.h

/usr/include/hiredis/aync.h里面，定义了异步接口:

连接与释放的接口：

/* Functions that proxy to hiredis */
redisAsyncContext *redisAsyncConnect(const char *ip, int port);
void redisAsyncDisconnect(redisAsyncContext *ac);

可以注册连接、断连接的回调函数：
（上述异步connect是直接返回成功的，需要在callback里面判断是否连接redis成功）

int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);

读写处理，具体怎么用先不纠结：

/* Handle read/write events */
void redisAsyncHandleRead(redisAsyncContext *ac);
void redisAsyncHandleWrite(redisAsyncContext *ac);

与同步接口用法类似redisCommand，异步命令用如下接口，但是通过redisCallbackFn获取操作的结果：

/* Command functions for an async context. Write the command to the
 * output buffer and register the provided callback. */
int redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *format, ...);

异步实例的结构体定义：

/* Context for an async connection to Redis */
typedef struct redisAsyncContext {
    /* Hold the regular context, so it can be realloc'ed. */
    redisContext c;

    /* Setup error flags so they can be used directly. */
    int err;
    char *errstr;

    /* Not used by hiredis */
    void *data;

    /* Event library data and hooks */
    struct {
        void *data;

        /* Hooks that are called when the library expects to start
         * reading/writing. These functions should be idempotent. */
        void (*addRead)(void *privdata);
        void (*delRead)(void *privdata);
        void (*addWrite)(void *privdata);
        void (*delWrite)(void *privdata);
        void (*cleanup)(void *privdata);
    } ev; 

    /* Called when either the connection is terminated due to an error or per
     * user request. The status is set accordingly (REDIS_OK, REDIS_ERR). */
    redisDisconnectCallback *onDisconnect;

    /* Called when the first write event was received. */
    redisConnectCallback *onConnect;

    /* Regular command callbacks */
    redisCallbackList replies;

    /* Subscription callbacks */
    struct {
        redisCallbackList invalid;
        struct dict *channels;
        struct dict *patterns;
    } sub;
} redisAsyncContext;

2.2 adapters接口

接口然后下一步看看怎么跟libevent结合，能在结构体看到里面有个ev结构，这个地方就是事件触发的核心。
这块hiredis已经帮我们考虑到了，直接用async.h比较麻烦，所以他都给适配了大部分事件库的接口：

/usr/include/hiredis/adapters
├── ae.h
├── glib.h
├── ivykis.h
├── libevent.h
├── libev.h
├── libuv.h
├── macosx.h
└── qt.h

与libevent关联，我们需要额外调用一个接口redisLibeventAttach：

static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
    redisContext *c = &(ac->c);
    redisLibeventEvents *e; 

    /* Nothing should be attached when something is already attached */
    if (ac->ev.data != NULL)
        return REDIS_ERR;

    /* Create container for context and r/w events */
    e = (redisLibeventEvents*)malloc(sizeof(*e));
    e->context = ac; 

    /* Register functions to start/stop listening for events */
    ac->ev.addRead = redisLibeventAddRead;
    ac->ev.delRead = redisLibeventDelRead;
    ac->ev.addWrite = redisLibeventAddWrite;
    ac->ev.delWrite = redisLibeventDelWrite;
    ac->ev.cleanup = redisLibeventCleanup;
    ac->ev.data = e;

    /* Initialize and install read/write events */
    e->rev = event_new(base, c->fd, EV_READ, redisLibeventReadEvent, e); 
    e->wev = event_new(base, c->fd, EV_WRITE, redisLibeventWriteEvent, e); 
    event_add(e->rev, NULL);
    event_add(e->wev, NULL);
    return REDIS_OK;
}

3 编程实例

以下编程实例准备实现：

1. Redis异步连接，断线重连机制（定时器）；
2. Redis-LIST-POP功能，外部有事件或超时了，进行触发；
3. Libevent定时器，模拟其他业务功能（定时器）；
4. Libevent信号处理；

3.1 类定义

定义类class mod_redisev如下：

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <errno.h>
#include <hiredis/hiredis.h>
#include <hiredis/async.h>
#include <hiredis/adapters/libevent.h>
#include <event2/event.h>
#include <string>

class mod_redisev
{
private:
    uint16_t _port;
    std::string _addr;
    int _status; // 标识是否连接成功

    struct redisAsyncContext *_rc; // Redis异步实例
    struct event_base *_base; // libevent实例

    struct event *_ev_quit; // 信号处理
    struct event *_ev_timer; // 业务定时器
    struct event *_ev_kalive; // 保活定时器
public:

    mod_redisev(const std::string &addr = "127.0.0.1", uint16_t port = 6379)
        : _port(port)
        , _addr(addr)
        , _status(REDIS_ERR)
        , _rc(NULL)
    {
        _base = event_base_new();
        if (!_base) {
            throw errno;
        }

        _ev_kalive = event_new(_base, -1, EV_PERSIST, on_keepalive, this);
        _ev_timer = event_new(_base, -1, EV_PERSIST, on_do_something, this);
        _ev_quit = evsignal_new(_base, SIGINT, on_signal_quit, this);
        if (!_ev_kalive  || !_ev_quit || !_ev_timer) {
            throw errno;
        }
    }

    ~mod_redisev()
    {
        event_base_free(_base);
    }

    int redis_connect(); 
    int dispatch(); 
    void redis_close();

private:
    void __do_keepalive();
    void __on_auth(struct redisReply *reply);
    void __on_pop(struct redisReply *reply);
    void __on_connect(int status);
    void __on_quit()
    {
        printf("quit...\n");
        event_base_loopbreak(_base);
    }

private:
    static void on_do_something(int fd, short events, void *args)
    {
        printf("Do something...\n");
    }

    static void on_keepalive(int fd, short events, void *args)
    {
        ((class mod_redisev *)args)->__do_keepalive();
    }

    static void on_signal_quit(int fd, short events, void *args)
    {
        ((class mod_redisev *)args)->__on_quit();
    }

    static void on_redis_connect(const struct redisAsyncContext *rc, int status)
    {
        ((class mod_redisev *)rc->data)->__on_connect(status);
    }

    static void on_redis_close(const struct redisAsyncContext *rc, int status)
    {
        printf("Redis disconnect...\n");
    }

    static void on_redis_auth(struct redisAsyncContext *rc, void *reply, void *args)
    {
        ((class mod_redisev *)args)->__on_auth((struct redisReply *)reply);
    }

    static void on_redis_pop(struct redisAsyncContext *rc, void *reply, void *args)
    {
        ((class mod_redisev *)args)->__on_pop((struct redisReply *)reply);
    }
};

轻松愉快的main入口：

int main(int argc, char *argv[])
{
    class mod_redisev redisev;
    redisev.redis_connect();
    redisev.dispatch();
    redisev.redis_close();
    exit(EXIT_SUCCESS);
}

3.2 Redis连接

异步连接功能，用redisLibeventAttach与libevent绑定：
设置两个回调函数：on_redis_connect、on_redis_close

int mod_redisev::redis_connect()
{
    int res = -1;

    _rc = redisAsyncConnect(_addr.c_str(), _port);
    if (!_rc || _rc->err) {
        printf("redisAsyncConnect: %s\n", _rc ? _rc->errstr : "error");
        return -1;
    }
    _rc->data = this; // attch to class

    res = redisLibeventAttach(_rc, _base);
    if (0 != res) {
        printf("redisLibeventAttach\n");
        return -1;
    }

    (void)redisAsyncSetConnectCallback(_rc, on_redis_connect);
    (void)redisAsyncSetDisconnectCallback(_rc, on_redis_close);
    return 0;
}

连接的流程为：触发连接成功了，设置REDIS_OK标识:

void mod_redisev::__on_connect(int status)
{
    _status = status;

    switch (_status) {
    case REDIS_OK:
        printf("Redis connected...\n");
        (void)redisAsyncCommand(_rc, on_redis_auth, this, "AUTH 123456");
        break;

    case REDIS_ERR:
        printf("Redis reconnecting...\n");
        break;

    default:
        break;
    }
}

Redis的连接保活借助了定时器对 _status的检查，如果连接不正常，再次进行connect操作:

void mod_redisev::__do_keepalive()
{
    if (_status != REDIS_OK) {
        printf("Reconect...\n");
        redis_connect();
    }
}

3.2 认证

上述__on_connnect函数中，如果连接成功了，则发起认证指令AUTH
AUTH并不是可选的，由于我redis-server配置了密码：

/etc/redis.conf
requirepass 123456

认证结果的处理，如果成功，那么进入下一个逻辑，BRPOP获取数据

void mod_redisev::__on_auth(struct redisReply *reply)
{
    if (!reply || reply->type == REDIS_REPLY_ERROR) {
        printf("Reply: %s\n", reply ? reply->str : "error");
        _status = REDIS_ERR;
        return;
    }
    else if (reply->type != REDIS_REPLY_STATUS) {
        printf("Reply unknown: %d\n", reply->type);
        _status = REDIS_ERR;
        return;
    }
    printf("AUTH success...\n");
    (void)redisAsyncCommand(_rc, on_redis_pop, this, "BRPOP queue1 queue2 queue3 10");
}

注意常见的认证错误有下面这种：

Reply: NOAUTH Authentication required.  // 没有配置密码，无须认证
Reply: ERR invalid password // 错误的密码

3.3 拉取数据

认证成功后，进入拉取数据阶段，这里用了循环拉取的逻辑：
以阻塞10秒的形式，分别看 queue1 queue2 queue3 是否有数据过来：

BRPOP queue1 queue2 queue3 10

void mod_redisev::__on_pop(struct redisReply *reply)
{
    if (!reply || reply->type == REDIS_REPLY_ERROR) {
        printf("Reply: %s\n", reply ? reply->str : "error");
        _status = REDIS_ERR;
        return;
    }

    if (reply->type == REDIS_REPLY_NIL) {
        printf("BRPOP: empty...\n");
    }
    else if (reply->type == REDIS_REPLY_ARRAY) {
        if (reply->elements > 0) {
            struct redisReply *rs = reply->element[1];
            printf("BRPOP: %s\n", rs->str);
        }
    }
    redisAsyncCommand(_rc, on_redis_pop, this, "BRPOP queue1 queue2 queue3 10");
}

这里就体现出异步事件的优势出来了，如果使用同步的请求接口，那么就得阻塞10秒，期间什么也不能干；

每秒1次的，假装干活的接口：

class mod_redisev {
	...
    static void on_do_something(int fd, short events, void *args)
    {
        printf("Do something...\n");
    }
    ...
};

3.4 信号处理

class mod_redisev {
	...
    static void on_signal_quit(int fd, short events, void *args)
    {
        ((class mod_redisev *)args)->__on_quit();
    }
    void __on_quit()
    {
        printf("quit...\n");
        event_base_loopbreak(_base);
    }
    ...
};

最后剩下的两个处理，dispatch与连接关闭：

int mod_redisev::dispatch()
{
    struct timeval tv = {1};
    event_add(_ev_timer, &tv);
    event_add(_ev_kalive, &tv);
    event_add(_ev_quit, NULL);
    return event_base_dispatch(_base);
}

void mod_redisev::redis_close()
{
    if (_rc && _status == REDIS_OK) {
        printf("Redis disconnect...\n");
        redisAsyncDisconnect(_rc); // __redisAsyncFree() called
        _rc = NULL;
    }
}

3.5 执行效果

编译方法记得把 -lhiredis、-levent 加进来

g++ -o redis_async -std=c++11 -Wall -g2 -O3 -Os redis_async.cc -levent -lhiredis

Redis connected...  // redis连接成功
AUTH success... // auth成功，发起BRPOP请求
Do something... // 每秒1次的 do-something
Do something...
Do something...
Do something...
Do something...
Do something...
Do something...
Do something...
Do something...
Do something...
BRPOP: empty... // BRPOP 10秒超时到，没有数据 
Do something... // 每秒1次的、执着的 do-something
Do something...
Do something...
Do something...
Do something...
Do something...
BRPOP: a         // 外部我执行了命令，LPUSH queue1 a b c d e
BRPOP: b         // BRPOP函数连续触发出来，获取 a b c d e
BRPOP: c
BRPOP: d
BRPOP: e
Do something...
Do something...
Do something...
Do something...
quit...          // 外部 ctrl + c
Redis disconnect... // 回收资源，disconnect

4. 总结

    本文通过libhiredis库提供的async异步接口，利用了adapters/libevent.h实现快速与libevent衔接，主要适合的场景，还是单线程下的事件触发模型；

    主要是几个心得：

1. 需要习惯的地方，异步回调的编程思路，每个命令都需要一个回调函数处理；
2. 如果中间还想穿插直接用同步接口，如redisCommand，似乎_rc->c是不能直接用；
3. hiredis容易用错出现double-free，需要外部_status进行综合判断多多检查；
4. valgrind检查内存，中间过程倒是不泄露，但最后redis-libevent释放有不干净的地方；

参考文章:
[1] https://github.com/redis/hiredis/blob/master/README.md