json libcurld 构造_常用libcurl异步使用方法

u=3918275517,691077552&fm=26&gp=0.jpg

前言: 需要了解的基础知识

1 基本网络通信cs模式,select 框架

网上例子很多.

2 epoll 用法

2.1 基础知识:

(1) epoll in/out 与socket io 缓冲区关系？

socket 可读可写是指io 缓冲区的情况.这层由内核控制.socket io 对应的epoll in/out 是应用层控制网络层.

epoll 通常采用ET模型.ET触发方式是指当fd到状态发生变化时通知,如read buffer从无到有,write buffer从满到不满才会通知.

所以用while 不停的读,read完缓冲区。下次来数据的时候，缓冲区又是由无到有,又会触发epoll in.

while 不停的写直到返回缓冲区满返回eagain,然后os立刻会发送,“发送缓冲区的”数据.

“发送缓冲区的”由满变成不满,再次触发epoll out.进入epoll out分支,开始下一轮while写.

如果没写满,但是也写完了,不会再次进入epoll out分支.

(2) epoll out的使用时机?

在自己端准备write之前，通过epoll ctrl设置成epoll out.

epoll in 是被动监听接收，epoll out是主动设置.

2.2 epoll 模型

网上例子很多.

正文:

1 libcurl 基础知识

(1) libcurl作用:用于文件上传下载.和socket的read，write交互不同，curl是单向行为.libcurl:不是一个简单的api,是一组api实现的模块,有自己的使用steps。理解为默认curl是get url 网页，与write function连动，写入文件。

(2) 上传和下载

curl_multi_setopt(ionmulti, CURLMOPT_SOCKETDATA, &setup)使用，现在的例子，全是recv，没有send的

全网：没有CURLMOPT_SOCKETDATA的使用例子

(3) 两种方案:

curl_multi_socket_action()：通常和select/poll/epoll/libev 连用.

curl_multi_perform() + curl_multi_wait()

本文讨论方案1

2 两种模式

2.1 easy 模式

调用curl_easy_setopt函数设置传输的一些基本参数,CULROPT_URL必填.设置完成后,调用curl_easy_perform函数发送数据.

CURLcode curl_easy_setopt(CURL *handle, CURLOPT_WRITEDATA, void *pointer);

curl_easy_setopt(conn->easy, CURLOPT_URL, conn->url);

curl_easy_setopt(conn->easy, CURLOPT_WRITEFUNCTION, write_cb);

curl_easy_setopt(conn->easy, CURLOPT_WRITEDATA, conn);

第一句 curl建立与url 连接，第二句将respond 发送到write_cb,第三句是将write_cb处理内容写入conn指向的文件

2.2 multi 模式

2.2.1 常用函数介绍(注意参数)

(1) CURLMOPT_SOCKETFUNCTION:设置socket的回调函数.是所有socket 变化都会调用callback(不仅仅是socket connect，read/write也调用).

int socket_callback(CURL easy, / easy handle /

curl_socket_t s, / socket /

int what, / describes the socket */

void userp, / private callback pointer */

void socketp); / private socket pointer */

CURLMOPT_SOCKETDATA: it is params userp

对于接收的话，CURLMOPT_SOCKETDATA没用。socketp是返回值。

CURLMOPT_TIMERDATA and CURLOPT_WRITEDATA 都是对应 curl function的入参.

(2) CURLMOPT_TIMERFUNCTION :set callback to receive timeout values

int timer_callback(CURLM multi, / multi handle /

long timeout_ms, / timeout in number of ms */

void userp); / private callback pointer */

You can also use the curl_multi_timeout function to poll the value at any given time,

but for an event-based system using the callback is far better than relying on polling the timeout value.

系统超时操作，类似单片机踢狗操作.

(3) CURLOPT_WRITEFUNCTION:set callback for writing received data

size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata);

CURLOPT_WRITEDATA:custom pointer passed to the write callback

设置CURLOPT_WRITEFUNCTION的参数4,把参数1 ptr指向的数据拷到参数4 userdata

(4) curl_multi_setopt

CURLMcode curl_multi_setopt(CURLM *handle, CURLMOPT_TIMERFUNCTION, timer_callback);

CURLMOPT_TIMERDATA:The userp pointer is set with CURLMOPT_TIMERDATA. 入参

curl_multi_setopt is used to tell a libcurl multi handle how to behave.

CURLMOPT_SOCKETFUNCTION,CURLMOPT_SOCKETDATA,CURLMOPT_TIMERFUNCTION,CURLMOPT_TIMERDATA

(5) curl_multi_assign

curl_multi_assign(CURLM *multi_handle, curl_socket_t sockfd, void *sockptr);

set association sockfd with sockptr ;curl系统中将sockfd和sockpt结构体绑定

(6) curl_multi_info_read

Use curl_multi_info_read(3) to figure out which easy handle that completed. 当前只发现这一个功能

注意这个函数不是read data，而是read info ,具体说就是completed flag，是结束检测函数.

(7) curl_multi_socket_action .

curl_multi_socket_action=curl_multi_perform：reads/writes available data given an action.

调用CURLOPT_WRITEDATA将download data写入file.

(8) 函数关系:CURLMOPT_SOCKETFUNCTION和epoll_wait作用区别？这个问题比较关键

根据https://blog.csdn.net/big_yellow_duck/article/details/78013082 中的说明是,CURLMOPT_SOCKETFUNCTION 是socket变成read or write的情况，切换变化的情况才触发回调.一直在read的状态不触发这个回调. curl有自己的curl select,因为是select所以仅仅支持1024个文件描述符.所以不能做高并发.只能做epoll ctrl的epoll in,epoll out 切换.

epoll_wait 则只要socket 有数据就读写操作，不是切换的时候.

3 curl实例:

curl同时和socket，epoll，file 同时打交道.

curl精华：curl_multi_socket_action可以自动get file from server(CURLOPT_URL) and 自动write file to local(CURLOPT_WRITEFUNCTION)

3.1 multi 模式使用(没有epoll，比如curl+select 等)

multi+select :https://curl.haxx.se/libcurl/c/curl_multi_perform.html

C语言libcurl例程：multi 多线程，多任务

https://blog.csdn.net/Rong_Toa/article/details/105712677

3.2 curl+ epol

CURLOPT_URL=connect，CURLMOPT_SOCKETFUNCTION=epoll_ctrl, curl_multi_socket_action=get remote file and write file to local ,epoll==select.

steps and 例子时序图

https://curl.haxx.se/libcurl/c/ephiperfifo.html

实现框架

(1) Create a multi handle

(2) Set the socket callback with CURLMOPT_SOCKETFUNCTION

(3) Set the timeout callback with CURLMOPT_TIMERFUNCTION, to get to know what timeout value to use when waiting for socket activities.

(4) Add easy handles with curl_multi_add_handle()

(5) Provide some means to manage the sockets libcurl is using, so you can check them for activity.

This can be done through your application code, or by way of an external library such as libevent or glib.

(6) Call curl_multi_socket_action(..., CURL_SOCKET_TIMEOUT, 0, ...) to kickstart everything.

To get one or more callbacks called.

(7) Wait for activity on any of libcurl's sockets, use the timeout value your callback has been told.

(8) When activity is detected, call curl_multi_socket_action() for the socket(s) that got action.

If no activity is detected and the timeout expires, call curl_multi_socket_action with CURL_SOCKET_TIMEOUT.

(9) Use curl_multi_info_read to check CURLMSG_DONE

call ladder:

(1) curl_multi_add_handle 因为CURLOPT_URL执行connect. default 执行epoll_in get download file,如果有upload 等执行epoll_out.

(2) CURLMOPT_SOCKETFUNCTION callback epoll ctrl to epoll_in

(3) epoll_wait

(4) curl_multi_socket_action write download file to local file.

(5) curl_multi_info_read check if it is completed.

代码:

epoll_ctl(curl_epoll_fd, EPOLL_CTL_ADD, curl_timer_fd, &ev);//put timer into epoll

curl_easy_setopt(curl, CURLOPT_URL, request_uri);

curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_read_http_body);

curl_easy_setopt(curl, CURLOPT_WRITEDATA, relay_session);

CURLMcode rc = curl_multi_add_handle(curl_multi, curl);

curl_multi_setopt(curl_multi, CURLMOPT_TIMERFUNCTION, curl_timer_setup);//call back for timer

curl_multi_setopt(curl_multi, CURLMOPT_SOCKETFUNCTION, curl_sock_setup);//socket fd 有变化,会调用curl_sock_setup

curl_multi_thread = g_thread_try_new("curl multi thread", &curl_multi_thread_run, NULL, NULL);

static void* curl_multi_thread_run(void* user_data) {

int i, nfds;

struct epoll_event *ev = (struct epoll_event *)malloc(sizeof(struct epoll_event) * MAX_EPOLL_FDS_NUM);

gint64 start_time, end_time;

while (!janus_is_stopping()) {

nfds = epoll_wait(curl_epoll_fd, ev, MAX_EPOLL_FDS_NUM, 3000);//wait 3 seconds

if (nfds > 0) {

for (i = 0; i < nfds; i++) {

curl_event(ev[i].data.fd, ev[i].events);//curl_event//curl_event

}

}

}

}

static void curl_event(int fd, int revents) {

CURLMcode rc;

int still_running = -1;

int action = ((revents & EPOLLIN) ? CURL_CSELECT_IN : 0) |

((revents & EPOLLOUT) ? CURL_CSELECT_OUT : 0);

//call CURLOPT_WRITEFUNCTION callback;read/write curl data

rc = curl_multi_socket_action(curl_multi, fd, action, &still_running);

if (rc == CURLM_OK) {

check_multi_info();//check result

}

}

static void check_multi_info(void) {

CURLMsg *msg = NULL;

int msgs_left;

CURL *curl = NULL;

CURLcode res;

janus_session* session = NULL;

json_error_t error;

json_t *root = NULL;

//从curl msg queue中读取

while ((msg = curl_multi_info_read(curl_multi, &msgs_left))) {

curl = msg->easy_handle;

res = msg->data.result;

curl_easy_getinfo(curl, CURLINFO_PRIVATE, &session);//it is from CURLOPT_PRIVATE

curl_multi_remove_handle(curl_multi, curl);

curl_easy_cleanup(curl);

}

}