【verbs】IBV_WR API(3) Libibverbs Programmer’s Manual

阅读源码的时候，发现一些陌生 api，如ibv_wr_send。搜索后才发现是一些dev的api

 / testing / libibverbs-dev / ibv_wr_send(3)

源文：ibv_wr_send(3) — libibverbs-dev — Debian testing — Debian Manpages

描述

verbs   API (ibv_wr_*) 允许使用函数调用而不是基于结构的 ibv_post_send() 方案将工作高效地发布到发送队列。这种方法旨在最大限度地减少发布过程中的 CPU 分支和锁定。

此 API 旨在用于访问 ibv_post_send() 提供的功能之外的其他功能。
ibv_post_send() 的 WR 批次和此 API WR 批次可以交织在一起，只要它们不在彼此的关键区域内发布。 （此 API 中的一个关键区由 ibv_wr_start() 和 ibv_wr_complete()/ibv_wr_abort() 界定）

用法

要使用这些 API，必须使用 ibv_create_qp_ex() 创建 QP，它允许在 comp_mask 中设置 IBV_QP_INIT_ATTR_SEND_OPS_FLAGS。 send_ops_flags 应设置为将发布到 QP 的工作请求类型的 OR。

如果 QP 不支持所有请求的工作请求类型，则 QP 创建将失败。
向 QP 发布工作请求是在 ibv_wr_start() 和 ibv_wr_complete()/ibv_wr_abort() 形成的关键区域内完成的（参见下面的 CONCURRENCY）。

每个工作请求都是通过调用 WR 构建器函数（请参阅下面的表列 WR 构建器）来开始创建工作请求，然后是下面描述的允许/必需的 setter 函数来创建的。

可以多次调用 WR builder 和 setter 组合以在单个关键区域内有效地发布多个工作请求。
每个 WR builder 都会使用 struct ibv_qp_ex 的 wr_id 成员来设置完成中要返回的值。某些操作还将使用 wr_flags 成员来影响操作（请参阅下面的标志）。这些值

qpx->wr_id = 1;
ibv_wr_send(qpx);
ibv_wr_set_sge(qpx, lkey, &data, sizeof(data));

工作请求部分详细描述了各种 WR 构建器和设置器。
发布工作通过调用 ibv_wr_complete() 或 ibv_wr_abort() 完成。在 ibv_wr_complete() 返回成功之前，不会对队列执行任何工作。 ibv_wr_abort() 将丢弃自 ibv_wr_start() 以来准备的所有工作。

WORK REQUESTS

Operation	WR builder	QP Type Supported	setters
ATOMIC_CMP_AND_SWP	ibv_wr_atomic_cmp_swp()	RC, XRC_SEND	DATA, QP
ATOMIC_FETCH_AND_ADD	ibv_wr_atomic_fetch_add()	RC, XRC_SEND	DATA, QP
BIND_MW	ibv_wr_bind_mw()	UC, RC, XRC_SEND	NONE
LOCAL_INV	ibv_wr_local_inv()	UC, RC, XRC_SEND	NONE
RDMA_READ	ibv_wr_rdma_read()	RC, XRC_SEND	DATA, QP
RDMA_WRITE	ibv_wr_rdma_write()	UC, RC, XRC_SEND	DATA, QP
RDMA_WRITE_WITH_IMM	ibv_wr_rdma_write_imm()	UC, RC, XRC_SEND	DATA, QP
SEND	ibv_wr_send()	UD, UC, RC, XRC_SEND, RAW_PACKET	DATA, QP
SEND_WITH_IMM	ibv_wr_send_imm()	UD, UC, RC, SRC SEND	DATA, QP
SEND_WITH_INV	ibv_wr_send_inv()	UC, RC, XRC_SEND	DATA, QP
TSO	ibv_wr_send_tso()	UD, RAW_PACKET	DATA, QP

Atomic operations¶

Atomic operations are only atomic so long as all writes to memory go only through the same RDMA hardware. It is not atomic with writes performed by the CPU, or by other RDMA hardware in the system.

ibv_wr_atomic_cmp_swp()

If the remote 64 bit memory location specified by rkey and remote_addr equals compare then set it to swap.

ibv_wr_atomic_fetch_add()

Add add to the 64 bit memory location specified rkey and remote_addr.

Memory Windows

Memory window type 2 operations (See man page for ibv_alloc_mw).

ibv_wr_bind_mw()

Bind a MW type 2 specified by mw, set a new rkey and set its properties by bind_info.

ibv_wr_local_inv()

Invalidate a MW type 2 which is associated with rkey.

RDMA

ibv_wr_rdma_read()

Read from the remote memory location specified rkey and remote_addr. The number of bytes to read, and the local location to store the data, is determined by the DATA buffers set after this call.

ibv_wr_rdma_write(), ibv_wr_rdma_write_imm()

Write to the remote memory location specified rkey and remote_addr. The number of bytes to read, and the local location to get the data, is determined by the DATA buffers set after this call.

The _imm version causes the remote side to get a IBV_WC_RECV_RDMA_WITH_IMM containing the 32 bits of immediate data.

Message Send

ibv_wr_send(), ibv_wr_send_imm()

Send a message. The number of bytes to send, and the local location to get the data, is determined by the DATA buffers set after this call.

The _imm version causes the remote side to get a IBV_WC_RECV_RDMA_WITH_IMM containing the 32 bits of immediate data.

ibv_wr_send_inv()

The data transfer is the same as for ibv_wr_send(), however the remote side will invalidate the MR specified by invalidate_rkey before delivering a completion.

ibv_wr_send_tso()

Produce multiple SEND messages using TCP Segmentation Offload. The SGE points to a TCP Stream buffer which will be segmented into MSS size SENDs. The hdr includes the entire network headers up to and including the TCP header and is prefixed before each segment.

QP Specific setters

Certain QP types require each post to be accompanied by additional setters, these setters are mandatory for any operation listing a QP setter in the above table.

UD QPs

ibv_wr_set_ud_addr() must be called to set the destination address of the work.

XRC_SEND QPs

ibv_wr_set_xrc_srqn() must be called to set the destination SRQN field.

DATA transfer setters

For work that requires to transfer data one of the following setters should be called once after the WR builder:

ibv_wr_set_sge()

Transfer data to/from a single buffer given by the lkey, addr and length. This is equivalent to ibv_wr_set_sge_list() with a single element.

ibv_wr_set_sge_list()

Transfer data to/from a list of buffers, logically concatenated together. Each buffer is specified by an element in an array of struct ibv_sge.

Inline setters will copy the send data during the setter and allows the caller to immediately re-use the buffer. This behavior is identical to the IBV_SEND_INLINE flag. Generally this copy is done in a way that optimizes SEND latency and is suitable for small messages. The provider will limit the amount of data it can support in a single operation. This limit is requested in the max_inline_data member of struct ibv_qp_init_attr. Valid only for SEND and RDMA_WRITE.

ibv_wr_set_inline_data()

Copy send data from a single buffer given by the addr and length. This is equivalent to ibv_wr_set_inline_data_list() with a single element.

ibv_wr_set_inline_data_list()

Copy send data from a list of buffers, logically concatenated together. Each buffer is specified by an element in an array of struct ibv_inl_data.

Flags

A bit mask of flags may be specified in wr_flags to control the behavior of the work request.

IBV_SEND_FENCE

Do not start this work request until prior work has completed.

IBV_SEND_IP_CSUM

Offload the IPv4 and TCP/UDP checksum calculation

IBV_SEND_SIGNALED

A completion will be generated in the completion queue for the operation.

IBV_SEND_SOLICTED

Set the solicted bit in the RDMA packet. This informs the other side to generate a completion event upon receiving the RDMA operation.

CONCURRENCY

The provider will provide locking to ensure that ibv_wr_start() and ibv_wr_complete()/abort() form a per-QP critical section where no other threads can enter.

If an ibv_td is provided during QP creation then no locking will be performed and it is up to the caller to ensure that only one thread can be within the critical region at a time.

RETURN VALUE

Applications should use this API in a way that does not create failures. The individual APIs do not return a failure indication to avoid branching.

If a failure is detected during operation, for instance due to an invalid argument, then ibv_wr_complete() will return failure and the entire posting will be aborted.

例子

/* create RC QP type and specify the required send opcodes */
qp_init_attr_ex.qp_type = IBV_QPT_RC;
qp_init_attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
qp_init_attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
qp_init_attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE_WITH_IMM;
ibv_qp *qp = ibv_create_qp_ex(ctx, qp_init_attr_ex);
ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
ibv_wr_start(qpx);
/* create 1st WRITE WR entry */
qpx->wr_id = my_wr_id_1;
ibv_wr_rdma_write(qpx, rkey, remote_addr_1);
ibv_wr_set_sge(qpx, lkey, local_addr_1, length_1);
/* create 2nd WRITE_WITH_IMM WR entry */
qpx->wr_id = my_wr_id_2;
qpx->wr_flags = IBV_SEND_SIGNALED;
ibv_wr_rdma_write_imm(qpx, rkey, remote_addr_2, htonl(0x1234));
ibv_set_wr_sge(qpx, lkey, local_addr_2, length_2);
/* Begin processing WRs */
ret = ibv_wr_complete(qpx);

名称

ibv_wr_abort、ibv_wr_complete、ibv_wr_start - 管理允许发布工作的区域
ibv_wr_atomic_cmp_swp, ibv_wr_atomic_fetch_add - 发布远程原子操作工作请求
ibv_wr_bind_mw, ibv_wr_local_inv - 发布内存窗口的工作请求
ibv_wr_rdma_read、ibv_wr_rdma_write、ibv_wr_rdma_write_imm - 发布 RDMA 工作请求
ibv_wr_send、ibv_wr_send_imm、ibv_wr_send_inv - 发布发送工作请求
ibv_wr_send_tso - 后分段卸载工作请求
ibv_wr_set_inline_data, ibv_wr_set_inline_data_list - 将内联数据附加到最后一个工作请求
ibv_wr_set_sge, ibv_wr_set_sge_list - 将数据附加到最后一个工作请求
ibv_wr_set_ud_addr - 将 UD 寻址信息附加到最后一个工作请求
ibv_wr_set_xrc_srqn - 将 XRC SRQN 附加到最后一个工作请求

概要

#include <infiniband/verbs.h>
void ibv_wr_abort(struct ibv_qp_ex *qp);
int ibv_wr_complete(struct ibv_qp_ex *qp);
void ibv_wr_start(struct ibv_qp_ex *qp);
void ibv_wr_atomic_cmp_swp(struct ibv_qp_ex *qp, uint32_t rkey,


                           uint64_t remote_addr, uint64_t compare,


                           uint64_t swap);
void ibv_wr_atomic_fetch_add(struct ibv_qp_ex *qp, uint32_t rkey,


                             uint64_t remote_addr, uint64_t add);
void ibv_wr_bind_mw(struct ibv_qp_ex *qp, struct ibv_mw *mw, uint32_t rkey,


                    const struct ibv_mw_bind_info *bind_info);
void ibv_wr_local_inv(struct ibv_qp_ex *qp, uint32_t invalidate_rkey);
void ibv_wr_rdma_read(struct ibv_qp_ex *qp, uint32_t rkey,


                      uint64_t remote_addr);
void ibv_wr_rdma_write(struct ibv_qp_ex *qp, uint32_t rkey,


                       uint64_t remote_addr);
void ibv_wr_rdma_write_imm(struct ibv_qp_ex *qp, uint32_t rkey,


                           uint64_t remote_addr, __be32 imm_data);
void ibv_wr_send(struct ibv_qp_ex *qp);
void ibv_wr_send_imm(struct ibv_qp_ex *qp, __be32 imm_data);
void ibv_wr_send_inv(struct ibv_qp_ex *qp, uint32_t invalidate_rkey);
void ibv_wr_send_tso(struct ibv_qp_ex *qp, void *hdr, uint16_t hdr_sz,


                     uint16_t mss);
void ibv_wr_set_inline_data(struct ibv_qp_ex *qp, void *addr, size_t length);
void ibv_wr_set_inline_data_list(struct ibv_qp_ex *qp, size_t num_buf,


                                 const struct ibv_data_buf *buf_list);
void ibv_wr_set_sge(struct ibv_qp_ex *qp, uint32_t lkey, uint64_t addr,


                    uint32_t length);
void ibv_wr_set_sge_list(struct ibv_qp_ex *qp, size_t num_sge,


                         const struct ibv_sge *sg_list);
void ibv_wr_set_ud_addr(struct ibv_qp_ex *qp, struct ibv_ah *ah,


                        uint32_t remote_qpn, uint32_t remote_qkey);
void ibv_wr_set_xrc_srqn(struct ibv_qp_ex *qp, uint32_t remote_srqn);