ioqueue是pjlib的一个精华部分,实现基于Proactor模式的IO复用模型,关于Proactor与Reactor模式,可以参考IO模型。简单来讲,Reactor是监控到事件后,应用自己去读写;Proactor模式是检测到事件后,底层去读写,完成后通知应用。其本质区别就是谁去读写数据。ioqueue的实现在ioqueue.h ioqueue_common_abs.c ioqueue_comon_abs.h ioqueue_select.c/ioqueue_epoll.c(两种实现方式选择一种编译)。
数据结构
1、队列
队列结构体有3个pj_ioqueue_key_t链表,一个监控socke对应一个key,当没有socket绑定时,该key添加到free_list,当开始监控时,从free_list移到到active_list,当socket关闭时,从active_list移到到closing_list。两个读写监控集wfdset合wfdset。
struct pj_ioqueue_t
{
//DECLARE_COMMON_IOQUEUE
pj_lock_t *lock; \
pj_bool_t auto_delete_lock; \
pj_bool_t default_concurrency;
unsigned max, count; /* Max and current key count */
int nfds; /* The largest fd value (for select)*/
pj_ioqueue_key_t active_list; /* List of active keys. */
pj_fd_set_t rfdset;
pj_fd_set_t wfdset;
#if PJ_HAS_TCP
pj_fd_set_t xfdset;
#endif
#if PJ_IOQUEUE_HAS_SAFE_UNREG
pj_mutex_t *ref_cnt_mutex;
pj_ioqueue_key_t closing_list;
pj_ioqueue_key_t free_list;
#endif
};
2、队列key
pj_ioqueue_key_t绑定一个监控socket,key是个链表,fd是socket的fd,cb是完成事的回调函数,三个事件操作结构体,都是由链表组成。
struct pj_ioqueue_key_t
{
//DECLARE_COMMON_KEY
PJ_DECL_LIST_MEMBER(struct pj_ioqueue_key_t);
pj_ioqueue_t *ioqueue; \
pj_grp_lock_t *grp_lock; \
pj_lock_t *lock; \
pj_bool_t inside_callback; \
pj_bool_t destroy_requested; \
pj_bool_t allow_concurrent; \
pj_sock_t fd; \
int fd_type; \
void *user_data; \
pj_ioqueue_callback cb; \
int connecting; \
struct read_operation read_list; \
struct write_operation write_list; \
struct accept_operation accept_list; \
#if PJ_IOQUEUE_HAS_SAFE_UNREG
# define UNREG_FIELDS \
unsigned ref_count; \
pj_bool_t closing; \
pj_time_val free_time; \
#else
# define UNREG_FIELDS
#endif
};
3、操作对象
当要开始读写数据时,需要传递操作对象,其统一抽象为pj_ioqueue_op_key_t,真正使用的是read_operation、write_operation
等。op操作码,读数据到buf缓冲区,大小是size。
typedef struct pj_ioqueue_op_key_t
{
void *internal__[32]; /**< Internal I/O Queue data. */
void *activesock_data; /**< Active socket data. */
void *user_data; /**< Application data. */
} pj_ioqueue_op_key_t;
struct read_operation
{
PJ_DECL_LIST_MEMBER(struct read_operation);
pj_ioqueue_operation_e op;
void *buf;
pj_size_t size;
unsigned flags;
pj_sockaddr_t *rmt_addr;
int *rmt_addrlen;
};
struct write_operation
{
PJ_DECL_LIST_MEMBER(struct write_operation);
pj_ioqueue_operation_e op;
char *buf;
pj_size_t size;
pj_ssize_t written;
unsigned flags;
pj_sockaddr_in rmt_addr;
int rmt_addrlen;
};
创建队列
/**
* Create a new I/O Queue framework.
*
* @param pool The pool to allocate the I/O queue structure.
* @param max_fd The maximum number of handles to be supported, which
* should not exceed PJ_IOQUEUE_MAX_HANDLES.
* @param ioqueue Pointer to hold the newly created I/O Queue.
*
* @return PJ_SUCCESS on success.
*/
pj_ioqueue_create( pj_pool_t *pool, pj_size_t max_fd,pj_ioqueue_t **p_ioqueue)
函数传入最大监控句柄大小,传出ioqueue结构体,函数内部流程:
1、申请内存空间
2、初始化读写fd_set和三个链表
3、创建max_fd个pj_ioqueue_key_t对象,并添加到free_list链表
注册socket到队列
/**
* Register a socket to the I/O queue framework.
* When a socket is registered to the IOQueue, it may be modified to use
* non-blocking IO. If it is modified, there is no guarantee that this
* modification will be restored after the socket is unregistered.
*
* @param pool To allocate the resource for the specified handle,
* which must be valid until the handle/key is unregistered
* from I/O Queue.
* @param ioque The I/O Queue.
* @param sock The socket.
* @param user_data User data to be associated with the key, which can be
* retrieved later.
* @param cb Callback to be called when I/O operation completes.
* @param key Pointer to receive the key to be associated with this
* socket. Subsequent I/O queue operation will need this
* key.
*
* @return PJ_SUCCESS on success, or the error code.
*/
PJ_DECL(pj_status_t) pj_ioqueue_register_sock( pj_pool_t *pool,
pj_ioqueue_t *ioque,
pj_sock_t sock,
void *user_data,
const pj_ioqueue_callback *cb,
pj_ioqueue_key_t **key )
传入队列,socket,完成回调函数,传出绑定socket的key,函数内部流程:
1、从free_list移出一个空闲的key
2、初始化pj_ioqueue_key_t,把sock、user_data、cb填到key,其中cb是将来事件触发时的回调函数。
3、设置socket为非阻塞
4、把该key添加到active_list。
5、调用rescan_fdset,计算最大的监控nfds。注意,这里并没有设置fd到读写事件集。
6、把key返回给应用。
poll轮询事件
/**
* Poll the I/O Queue for completed events.
*
* Note: polling the ioqueue is not necessary in Symbian. Please see
* @ref PJ_SYMBIAN_OS for more info.
*
* @param ioque the I/O Queue.
* @param timeout polling timeout, or NULL if the thread wishes to wait
* indefinetely for the event.
*
* @return
* - zero if timed out (no event).
* - (<0) if error occured during polling. Callback will NOT be called.
* - (>1) to indicate numbers of events. Callbacks have been called.
*/
PJ_DECL(int) pj_ioqueue_poll( pj_ioqueue_t *ioque,
const pj_time_val *timeout);
应用需在另一条工作线程(或者多条线程池)调用poll进行监控
pj_ioqueue_poll( pj_ioqueue_t *ioqueue, const pj_time_val *timeout)
1、调用pj_sock_select检测是否有事件发生
2、通过调用key_has_pending_write、key_has_pending_read等查看是否有应用在等待读写,其本质是判断key中的链表是不是空。
PJ_INLINE(int) key_has_pending_write(pj_ioqueue_key_t *key)
{
return !pj_list_empty(&key->write_list);
}
PJ_INLINE(int) key_has_pending_read(pj_ioqueue_key_t *key)
{
return !pj_list_empty(&key->read_list);
}
这里以读流程为例,注册socket设置回调还不够,还需要在某个时候添加到key的read_list链表,才能够让ioqueue去帮应用读取数据,并调用回调。那什么时候去添加到struct read_operation read_list呢?应用需要调用pj_ioqueue_recvfrom或pj_ioqueue_recv,把缓冲区添加到key的read_list,并且把fd添加到监控集合ioqueue->rfdset,这样才能真正收到数据。
所以pj_ioqueue_recvfrom实际上是一个启动开关。
当调用pj_ioqueue_recvfrom启动后,工作线程poll就能监控到事件,然后根据事件类型调用
ioqueue_dispatch_read_event
1、从key的read_list拿到一个对象read_operation
2、调用pj_sock_recvfrom真正读数据到应用设置的缓冲区
3、调用回调h->cb.on_read_complete,通知应用读完成。
总结ioqueue使用流程
ioqueue的使用示例参考pjlib-test/ioq_udp.c
1、主动创建读写缓冲区
2、被动创建ioqueue,调用pj_ioqueue_create
3、创建socket和读完成回调函数
4、注册socket到io队列,pj_ioqueue_register_sock,传入socket和读完成回调函数,被动创建pj_ioqueue_key_t
5、调用pj_ioqueue_recvfrom启动监控,传入读缓冲区地址和第4步的pj_ioqueue_key_t,被动创建pj_ioqueue_op_key_t(可以强转为read_operation、write_operation等)
6、poll检测到事件,派发事件,在读事件中读取数据到5步设置的读缓冲区地址,并调用读完成函数
另外,pjlib还包装了ioqueue实现了一个activesock,更高抽象级别的对象,由于不是很复杂,基本是把一些步骤再封装,核心还是ioqueue,所以不展开讲。