【InfiniBand】IBV_WR_RDMA_WRITE_WITH_IMM / IBV_WC_RECV_RDMA_WITH_IMM

IB卡带立即数的写操作IBV_WR_RDMA_WRITE_WITH_IMM ，对应的远端(remote)接收操作是 IBV_WC_RECV_RDMA_WITH_IMM。

例：

本端(local)写发送：

  wr.opcode = IBV_WR_RDMA_WRITE_WITH_IMM;
  wr.send_flags = IBV_SEND_SIGNALED;
  wr.imm_data = htonl(len);

  ...

  ibv_post_send(id->qp, &wr, &bad_wr));

对端(remote)用ibv_poll_cq接收到后：

if (wc->opcode == IBV_WC_RECV_RDMA_WITH_IMM) 
{
    uint32_t size = ntohl(wc->imm_data);

    ...

}

注：

本端(local)带立即数写操作IBV_WR_RDMA_WRITE_WITH_IMM，对应的远端(remote)必须运行ibv_post_recv发布Recv Request。

 参考

【vbers】ibv_post_send|IBV_SEND_SOLICITED|RDMA_bdview的博客-CSDN博客

【RDMA】ibv_poll_cq()_bdview的博客-CSDN博客

关于IBV_WR_RDMA_WRITE_WITH_IMM的理解-CSDN博客

ChatGPT

In the context of InfiniBand (IB) and the verbs API which is used to interact with IB hardware, IBV_WR_RDMA_WRITE_WITH_IMM and IBV_WC_RECV_RDMA_WITH_IMM refer to specific operations and completion statuses involving RDMA (Remote Direct Memory Access) transfers with immediate data.
1. IBV_WR_RDMA_WRITE_WITH_IMM:
This is a Work Request (WR) type. When you post a work request of this type to a send queue of a Queue Pair (QP), you're instructing the InfiniBand hardware to perform an RDMA write operation to a remote memory region and to include immediate data with that operation. Immediate data is a small amount of extra information (32 bits) that is transmitted alongside the RDMA operation. It's not written into the memory of the remote system but rather is passed with the completion notification to the remote side. This can be useful for signaling to the remote side information about the data or the operation that was just performed.
2. IBV_WC_RECV_RDMA_WITH_IMM:
This is a Work Completion (WC) event. When you receive this type of event on the Completion Queue (CQ), it signifies that an RDMA operation that included immediate data was performed by a remote QP to a memory region associated with your QP and that you have received that immediate data. This type of event will typically trigger handling logic to process the data that was written to memory and to take any action that is needed based on the context contained within the immediate data.
These mechanisms are part of how RDMA enables low-latency, high-throughput networking, which is essential for many high-performance computing and enterprise data center applications. RDMA operations bypass the operating system and allow the network adapter to directly read or write to an application's memory, which removes many of the traditional bottlenecks associated with data transfer.
When implementing systems that use these RDMA operations, it's essential for developers to understand the details of the verbs API and ensure they synchronize memory operations correctly. This helps avoid race conditions and ensure data integrity across the network.

在InfiniBand（IB）的上下文及其verbs API中，`IBV_WR_RDMA_WRITE_WITH_IMM`和`IBV_WC_RECV_RDMA_WITH_IMM`分别指的是涉及RDMA（远程直接内存访问）传输的特定操作和完成状态，并且包括即时数据。
1. IBV_WR_RDMA_WRITE_WITH_IMM：
这是一种工作请求（Work Request, WR）类型。当您将这类工作请求发布到队列对（Queue Pair, QP）的发送队列时，您指示InfiniBand硬件执行远程内存区域的RDMA写操作，并随该操作包含即时数据。即时数据是传输过程中附加的小量数据（32位），这些数据不会被写入远程系统的内存中，而是与完成通知一起传递给远端。这有助于向远端传达有关刚刚执行的数据或操作的信息。
2. IBV_WC_RECV_RDMA_WITH_IMM：
这是一个工作完成（Work Completion, WC）事件。当您在完成队列（Completion Queue, CQ）上收到这种类型的事件时，意味着远程QP执行了包含即时数据的RDMA操作，并且该即时数据被发送到与您的QP关联的内存区域。这种类型的事件通常会触发处理逻辑，以处理写入内存的数据，并根据即时数据中包含的上下文采取所需的行动。
这些机制都是RDMA实现低延迟、高吞吐网络的一部分，这对于许多高性能计算和企业数据中心应用来说至关重要。RDMA操作绕过了操作系统，允许网络适配器直接读取或写入应用程序的内存，消除了与数据传输相关的许多传统瓶颈。
实现使用这些RDMA操作的系统时，开发人员需要了解verbs API的细节，并确保正确同步内存操作。这有助于避免竞态条件，并确保跨网络的数据完整性。