What is RDMA

papaofdoudou

已于 2024-02-25 00:15:08 修改

阅读量128

点赞数 1

分类专栏： RDMA 文章标签：服务器

于 2023-10-16 13:53:57 首次发布

本文链接：https://blog.csdn.net/tugouxp/article/details/133856836

版权

RDMA 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

远程直接内存访问(RDMA)提供了从一台计算机的内存到另一台计算机的内存的直接访问，而无需通过任何一台计算机的操作系统。该技术支持具有低CPU利用率的高吞吐量、低延迟网络，这在大规模并行计算集群中特别有用。

RDMA和DMA解决的问题都是在不打断CPU的情况下进行模块之间的数据交换，只不过DMA发生的媒介是片内系统总线，而RDMA则是通过以太网总线。

rdma out of tree code

https://gitee.com/tugouxp/nvidia-open-source.git

https://kernel.googlesource.com/pub/scm/linux/kernel/git/leon/linux-rdma

GitHub - ROCm/ROCK-Kernel-Driver: AMDGPU Driver with KFD used by the ROCm project. Also contains the current Linux Kernel that matches this base driver

Author: Jason Gunthorpe <jgg@nvidia.com>
Date:   Thu Nov 26 10:42:11 2020 -0400

    RDMA/core: Introduce peer memory interface

    The peer_memory_client scheme allows a driver to register with the ib_umem
    system that it has the ability to understand user virtual address ranges
    that are not compatible with get_user_pages(). For instance VMAs created
    with io_remap_pfn_range(), or other driver special VMA.

    For ranges the interface understands it can provide a DMA mapped sg_table
    for use by the ib_umem, allowing user virtual ranges that cannot be
    supported by get_user_pages() to be used as umems for RDMA.

    This is designed to preserve the kABI, no functions or structures are
    changed, only new symbols are added:

     ib_register_peer_memory_client
     ib_unregister_peer_memory_client
     ib_umem_activate_invalidation_notifier
     ib_umem_get_peer

    And a bitfield in struct ib_umem uses more bits.

    This interface is compatible with the two out of tree GPU drivers:
     https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/blob/master/drivers/gpu/drm/amd/amdkfd/kfd_peerdirect.c
     https://github.com/Mellanox/nv_peer_memory/blob/master/nv_peer_mem.c

    Signed-off-by: Yishai Hadas <yishaih@mellanox.com>
    Signed-off-by: Feras Daoud <ferasda@mellanox.com>
    Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
    Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

什么是RDMA技术

RDMA(remote direct memory access)即远端直接内存访问，是一种高性能网络通信技术，具有高带宽、低延迟、无CPU消耗、零拷贝等优点。相比kernel TCP、DPDK等传统通信手段，RDMA在延迟、吞吐和CPU消耗方面均有明显优势。

RDMA的verbs编程接口是真正的异步非阻塞接口，完全不需要任何CPU参与，不需要系统调用或陷入内核空间，更不会有上线文切换！

随着RDMA技术在数据中心的普及，了解RDMA技术和熟悉verbs编程会逐渐像TCP socket编程一样成为底层开发人员的必备技能。

Verbs是RDMA的编程API的统称，全套包含几十个API，可分为两大类：

控制类API：用于创建/销毁资源，比如打开网卡设备，创建链接，内存注册等。
数据类API：用于数据收发和事件通知，比如ibv_post_send, ibv_post_recv, ibv_poll_cq等。

RDMA(remote direct memory access)是一种硬件offload的网络技术，无需操作系统内核参与，没有系统调用和上下文切换，不消耗CPU，也没有用户空间和内核空间的来回内存拷贝（彻底的零拷贝）。

RoCE(RDMA over Converged Ethernet)是一种基于以太网的RDMA实现方案，相比于InfiniBand网络RoCE不需要昂贵的专用的网络设备（网卡，交换机），而是基于数据中心已有的以太网架构工作，因此基于RoCE的RDMA技术成为数据中心广泛使用的高性能网络方案（微软、阿里等）。RoCE仍然依赖于数据中心网络，对个人来讲还是不容易接触。

Soft-RoCE(RXE)是一种基于纯软件实现的RoCE，可以让RoCE可以跑在任何类型的网卡上，当然也包括我们的桌面网卡。这使得Soft-RoCE成为学习RDMA编程的首选方案。由于Soft RoCE是纯软件实现，仅作为一种学习和测试方案，生产中几乎没人使用.

RDMA有四种实现方案，分别是InfiniBand，RoCE v1， RoCE v2，iWARP。

其中InfiniBand需要专门的硬件（网卡和交换机）支持，造价较贵，通常用于高性能计算领域。
RoCE是基于以太网的RDMA方案，不需要专门的硬件而是基于数据中心现有的网络就可以工作，所以当前数据中心大多采用该方案来实现RDMA通信。RoCE有v1和v2两个版本，v2使用更加广泛。
iWARP性能不佳。

RDMA通信语义

RDMA的编程接口叫做verbs，它比socket接口语义更加丰富，二者的区别如下：

RDMA提供面向消息的可靠传输，作为对比TCP是面向流的可靠传输，UDP是面向消息的不可靠传输。
RDMA既提供双边语义，也提供单边语义。所谓双边语义是指数据收发时需要通信双方共同参与；所谓单边语义是指数据收发过程只需要一方参与，另一方完全无感知。按照这个定义，TCP和UDP提供的语义都是双边语义。
RDMA支持gather list和scatter list，即允许不连续的数据发送到连续的地址空间，或者将连续的数据发送到不连续的地址空间。这个能力tcp也可以可以通过readv或者writev做到，但是RDMA结合单边语义做到gather/scatter就非常的强大了。
另外，RDMA还支持远程的原子操作。

个人搭建Soft-RoCE环境

搭建Soft RoCE环境最简单的方案就是使用Ubuntu 20.04，因为他已经内置了Soft RoCE。可以建立两台虚拟机来模拟机间RoCE通信。Soft RoCE对应的内核模块叫做 rdma_rxe.

1.install virtualbox&ubuntu20.04.5

2.install virtualbox extensions.

3.配置虚拟机内的Soft RoCE环境，具体命令如下：

# 安装必要的组件
sudo apt-get install libibverbs1 ibverbs-utils librdmacm1 libibumad3 ibverbs-providers rdma-core
# 启动对应的内核模块
sudo modprobe rdma_rxe rdma_cm ib_umad
# 添加一个Soft RoCE设备，注意请把your_nic_name 替换成你的网卡名字，形如enp0s3
sudo rdma link add rxe_0 type rxe netdev enp0s3

上述步骤执行之后，就可以在/dev/infiniband/目录下看到对应的RDMA设备：

设备名是由如下函数决定的，据了解，一个/dev/infinite/uverbsX设备对应一个支持RDMA设备的网卡，如果系统插入了多块支持RDMA功能的网卡，将会多次调用ib_uverbs_add_one添加网卡设备。

corrspond kernel callstack as follows:

root@zlcao-VirtualBox:/sys/kernel/debug/tracing# cat trace
# tracer: function
#
# entries-in-buffer/entries-written: 18/18   #P:1
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
        modprobe-2115    [000] .....    79.084551: device_add <-device_register
        modprobe-2115    [000] .....    79.084554: <stack trace>
 => device_add
 => device_register
 => workqueue_sysfs_register
 => alloc_workqueue
 => udp_sock_create4
 => do_one_initcall
 => do_init_module
 => load_module
 => __do_sys_finit_module
 => __x64_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
        modprobe-2115    [000] .....    79.086334: device_add <-device_register
        modprobe-2115    [000] .....    79.086337: <stack trace>
 => device_add
 => device_register
 => workqueue_sysfs_register
 => alloc_workqueue
 => udp_sock_create4
 => do_one_initcall
 => do_init_module
 => load_module
 => __do_sys_finit_module
 => __x64_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
            rdma-2242    [000] .....    90.534110: device_add <-ib_register_device
            rdma-2242    [000] .....    90.534115: <stack trace>
 => device_add
 => ib_register_device
 => rxe_register_device
 => rxe_add
 => rxe_net_add
 => rxe_newlink
 => nldev_newlink
 => rdma_nl_rcv_msg
 => rdma_nl_rcv
 => netlink_unicast
 => netlink_sendmsg
 => __sock_sendmsg
 => __sys_sendto
 => __x64_sys_sendto
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
            rdma-2242    [000] .....    90.535018: device_add <-cdev_device_add
            rdma-2242    [000] .....    90.535020: <stack trace>
 => device_add
 => cdev_device_add
 => ib_uverbs_add_one
 => add_client_context
 => enable_device_and_get
 => ib_register_device
 => rxe_register_device
 => rxe_add
 => rxe_net_add
 => rxe_newlink
 => nldev_newlink
 => rdma_nl_rcv_msg
 => rdma_nl_rcv
 => netlink_unicast
 => netlink_sendmsg
 => __sock_sendmsg
 => __sys_sendto
 => __x64_sys_sendto
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
            rdma-2242    [000] .....    90.537896: device_add <-add_one_compat_dev
            rdma-2242    [000] .....    90.537900: <stack trace>
 => device_add
 => add_one_compat_dev
 => enable_device_and_get
 => ib_register_device
 => rxe_register_device
 => rxe_add
 => rxe_net_add
 => rxe_newlink
 => nldev_newlink
 => rdma_nl_rcv_msg
 => rdma_nl_rcv
 => netlink_unicast
 => netlink_sendmsg
 => __sock_sendmsg
 => __sys_sendto
 => __x64_sys_sendto
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 systemd-modules-2246    [000] .....    90.591664: device_add <-device_register
 systemd-modules-2246    [000] .....    90.591667: <stack trace>
 => device_add
 => device_register
 => workqueue_sysfs_register
 => alloc_workqueue
 => iser_create_ib_conn_res
 => do_one_initcall
 => do_init_module
 => load_module
 => __do_sys_finit_module
 => __x64_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 systemd-modules-2246    [000] .....    90.592209: device_add <-device_register
 systemd-modules-2246    [000] .....    90.592211: <stack trace>
 => device_add
 => device_register
 => workqueue_sysfs_register
 => alloc_workqueue
 => iser_create_ib_conn_res
 => do_one_initcall
 => do_init_module
 => load_module
 => __do_sys_finit_module
 => __x64_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 systemd-modules-2246    [000] .....    90.595851: device_add <-device_register
 systemd-modules-2246    [000] .....    90.595853: <stack trace>
 => device_add
 => device_register
 => iscsi_register_transport
 => ucma_open
 => do_one_initcall
 => do_init_module
 => load_module
 => __do_sys_finit_module
 => __x64_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 systemd-modules-2246    [000] .....    90.598329: device_add <-device_create_groups_vargs
 systemd-modules-2246    [000] .....    90.598331: <stack trace>
 => device_add
 => device_create_groups_vargs
 => device_create_with_groups
 => misc_register
 => ucma_init
 => do_one_initcall
 => do_init_module
 => load_module
 => __do_sys_finit_module
 => __x64_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe

此时用ibv_devinfo/ibv_devices查看支持RDMA的网卡设备，可以看到一个HCA，就是RXE网卡设备：

4.perftest 验证

perftest工具是一个RDMA性能测试工具，可以用它验证Soft RoCE是否能正常工作。

sudo apt install perftest

# 终端A
ib_write_bw --report_gbits
# 终端B
ib_write_bw localhost --report_gbits

work fine~! here is the kernel callstack corrspond the test,you can see all the invoke session are come from struct file_operations:

# tracer: function
#
# entries-in-buffer/entries-written: 10036/10036   #P:1
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
     ib_write_bw-4804    [000] .....  2006.005831: rxe_query_port <-ib_query_port
     ib_write_bw-4804    [000] .....  2006.005836: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2013.955717: rxe_query_port <-ib_query_port
     ib_write_bw-4813    [000] .....  2013.955723: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.957360: rxe_query_port <-ib_query_port
     ib_write_bw-4804    [000] .....  2014.957380: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.957445: rxe_query_port <-ib_query_port
     ib_write_bw-4813    [000] .....  2014.957448: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.960045: rxe_create_cq <-ib_uverbs_handler_UVERBS_METHOD_CQ_CREATE
     ib_write_bw-4804    [000] .....  2014.960049: <stack trace>
 => rxe_create_cq
 => ib_uverbs_handler_UVERBS_METHOD_CQ_CREATE
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.960083: rxe_mmap <-ib_uverbs_mmap
     ib_write_bw-4804    [000] .....  2014.960085: <stack trace>
 => rxe_mmap
 => ib_uverbs_mmap
 => mmap_region
 => do_mmap
 => vm_mmap_pgoff
 => ksys_mmap_pgoff
 => __x64_sys_mmap
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.960100: rxe_create_qp <-create_qp
     ib_write_bw-4804    [000] .....  2014.960102: <stack trace>
 => rxe_create_qp
 => create_qp
 => ib_create_qp_user
 => create_qp
 => ib_uverbs_create_qp
 => ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.960144: rxe_mmap <-ib_uverbs_mmap
     ib_write_bw-4804    [000] .....  2014.960146: <stack trace>
 => rxe_mmap
 => ib_uverbs_mmap
 => mmap_region
 => do_mmap
 => vm_mmap_pgoff
 => ksys_mmap_pgoff
 => __x64_sys_mmap
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.960151: rxe_mmap <-ib_uverbs_mmap
     ib_write_bw-4804    [000] .....  2014.960153: <stack trace>
 => rxe_mmap
 => ib_uverbs_mmap
 => mmap_region
 => do_mmap
 => vm_mmap_pgoff
 => ksys_mmap_pgoff
 => __x64_sys_mmap
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.960166: rxe_query_port <-ib_query_port
     ib_write_bw-4804    [000] .....  2014.960168: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.962582: rxe_create_cq <-ib_uverbs_handler_UVERBS_METHOD_CQ_CREATE
     ib_write_bw-4813    [000] .....  2014.962586: <stack trace>
 => rxe_create_cq
 => ib_uverbs_handler_UVERBS_METHOD_CQ_CREATE
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.962604: rxe_mmap <-ib_uverbs_mmap
     ib_write_bw-4813    [000] .....  2014.962605: <stack trace>
 => rxe_mmap
 => ib_uverbs_mmap
 => mmap_region
 => do_mmap
 => vm_mmap_pgoff
 => ksys_mmap_pgoff
 => __x64_sys_mmap
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.962626: rxe_create_qp <-create_qp
     ib_write_bw-4813    [000] .....  2014.962628: <stack trace>
 => rxe_create_qp
 => create_qp
 => ib_create_qp_user
 => create_qp
 => ib_uverbs_create_qp
 => ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.962656: rxe_mmap <-ib_uverbs_mmap
     ib_write_bw-4813    [000] .....  2014.962657: <stack trace>
 => rxe_mmap
 => ib_uverbs_mmap
 => mmap_region
 => do_mmap
 => vm_mmap_pgoff
 => ksys_mmap_pgoff
 => __x64_sys_mmap
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.962661: rxe_mmap <-ib_uverbs_mmap
     ib_write_bw-4813    [000] .....  2014.962663: <stack trace>
 => rxe_mmap
 => ib_uverbs_mmap
 => mmap_region
 => do_mmap
 => vm_mmap_pgoff
 => ksys_mmap_pgoff
 => __x64_sys_mmap
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.962673: rxe_query_port <-ib_query_port
     ib_write_bw-4813    [000] .....  2014.962675: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4804    [000] .....  2014.990497: rxe_query_port <-ib_query_port
     ib_write_bw-4804    [000] .....  2014.990499: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.991110: rxe_query_port <-ib_query_port
     ib_write_bw-4813    [000] .....  2014.991112: <stack trace>
 => rxe_query_port
 => ib_query_port
 => ib_uverbs_handler_UVERBS_METHOD_QUERY_PORT
 => ib_uverbs_run_method
 => ib_uverbs_cmd_verbs
 => ib_uverbs_ioctl
 => __x64_sys_ioctl
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.991963: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.991965: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992074: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992076: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992158: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992159: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992226: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992226: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992279: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992279: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992328: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992328: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992379: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992380: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992431: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992432: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992484: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992485: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992536: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992536: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992588: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992588: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992637: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992638: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992689: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992690: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992742: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992742: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992794: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992795: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992847: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992847: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992899: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992900: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992951: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.992951: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.992999: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993000: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993052: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993052: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993104: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993104: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993156: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993157: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993209: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993210: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993262: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993262: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993311: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993311: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993363: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993364: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993415: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993416: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993467: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993468: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993520: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993520: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993572: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993572: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993625: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993626: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993674: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993675: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993727: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993727: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993779: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993780: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993832: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993833: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993884: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993885: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993938: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993938: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.993990: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.993991: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994039: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994040: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994092: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994092: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994144: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994145: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994205: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994205: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994329: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994330: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994385: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994385: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994449: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994450: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994505: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994506: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994558: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994558: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994610: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994611: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994662: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994662: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994711: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994712: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994763: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994763: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994815: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994816: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994867: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994868: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994919: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994920: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.994971: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.994972: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995021: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995021: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995072: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995073: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995124: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995125: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995177: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995177: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995230: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995230: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995288: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995288: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995340: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995341: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995392: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995393: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995442: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995443: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995494: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995494: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995546: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995546: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995598: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995599: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995650: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995651: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995703: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995703: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995752: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995753: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995804: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995805: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995856: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995857: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995909: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995909: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.995961: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.995962: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996013: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996014: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996066: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996066: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996115: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996116: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996167: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996168: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996220: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996220: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996272: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996273: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996325: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996325: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996377: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996377: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996428: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996429: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996478: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996478: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996530: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996530: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996582: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996582: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996634: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996635: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996686: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996687: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996738: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996739: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996788: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996788: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996840: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996841: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.996967: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.996968: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997028: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997029: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997081: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997082: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997134: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997135: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997187: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997187: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997239: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997240: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997291: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997292: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997340: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997340: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997392: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997393: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997445: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997445: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997497: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997498: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997549: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997550: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997601: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997602: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997654: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997654: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997703: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997703: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997755: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997756: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997807: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997808: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997860: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997860: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997912: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997912: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.997964: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.997964: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998016: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998017: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998065: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998066: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998117: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998118: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998170: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998170: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998222: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998223: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998293: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998294: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998348: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998348: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998400: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998401: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998458: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998459: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998511: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998511: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998560: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998560: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998612: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998612: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998664: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998665: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998716: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998717: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998768: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998769: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998820: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998821: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998873: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998873: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998922: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998923: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.998976: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.998976: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999028: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999028: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999080: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999080: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999132: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999132: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999184: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999184: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999236: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999236: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999285: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999285: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999337: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999337: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999389: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999389: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999441: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999441: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999492: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999493: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999544: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999545: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999594: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999594: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999645: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999646: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999812: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999814: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999873: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999873: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999926: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999926: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2014.999978: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2014.999978: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000029: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000030: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000078: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000078: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000130: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000131: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000182: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000182: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000234: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000234: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000285: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000286: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000337: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000338: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000387: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000388: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000440: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000440: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000492: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000492: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000544: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000544: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000596: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000597: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000648: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000649: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000701: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000702: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000750: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000751: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000802: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000803: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000854: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000855: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000906: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000907: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.000960: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.000960: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.001012: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.001013: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.001062: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.001062: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010331: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010337: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010419: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010420: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010478: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010479: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010530: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010531: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010579: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010580: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010632: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010632: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010684: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010685: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010736: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010737: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010788: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010788: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010840: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010840: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010889: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010892: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010941: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010941: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.010992: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.010993: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011044: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011045: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011096: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011097: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011148: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011149: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011201: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011201: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011249: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011250: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011301: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011302: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011353: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011354: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011405: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011406: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011457: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011458: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011509: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011510: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011558: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011559: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011610: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011611: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011663: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011663: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011715: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011715: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011767: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011767: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011818: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011819: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011871: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011872: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011920: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011921: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.011972: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.011973: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012025: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012026: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012077: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012078: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012130: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012131: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012182: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012182: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012231: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012232: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012283: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012284: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012336: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012337: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012389: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012389: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012441: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012442: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012494: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012494: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012546: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012547: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012595: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012596: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012648: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012649: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012700: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012701: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012753: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012753: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012805: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012805: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012857: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012857: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012908: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012909: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.012957: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.012958: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013009: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013010: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013062: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013062: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013114: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013114: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013166: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013167: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013218: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013219: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013267: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013268: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013319: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013320: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013371: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013372: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013423: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013424: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013476: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013477: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013529: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013529: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013581: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013581: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013629: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013630: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013681: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013682: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013734: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013735: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013786: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013787: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013838: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013839: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013890: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013891: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013939: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013940: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.013992: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.013992: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.014044: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.014044: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016472: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016474: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016533: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016534: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016586: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016587: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016638: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016638: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016687: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016690: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016740: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016740: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016792: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016792: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016853: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016854: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016906: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016907: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.016959: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.016959: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017011: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017011: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017063: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017064: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017112: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017112: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017164: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017165: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017217: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017217: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017269: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017270: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017322: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017322: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017374: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017375: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017426: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017427: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017475: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017476: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017527: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017528: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017580: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017580: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017632: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017633: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017684: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017685: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017737: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017737: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017786: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017789: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017838: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017838: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017890: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017890: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017943: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017944: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.017995: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.017996: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018048: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018048: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018100: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018101: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018149: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018150: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018202: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018203: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018255: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018255: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018319: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018319: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018372: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018373: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018438: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018438: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018491: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018492: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018544: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018545: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018596: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018597: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018649: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018649: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018698: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018699: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018751: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018752: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018804: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018805: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018857: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018857: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018909: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018910: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.018961: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.018962: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019013: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019014: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019062: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019063: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019114: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019115: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019166: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019167: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019218: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019219: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019270: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019271: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019322: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019323: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019371: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019371: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019423: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019424: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019475: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019475: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019527: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019527: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019580: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019580: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019632: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019632: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019684: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019684: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019733: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019733: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019785: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019786: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019837: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019839: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019890: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019890: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019942: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019942: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.019994: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.019995: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020043: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020043: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020095: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020096: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020147: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020148: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020200: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020200: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020252: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020252: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020304: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020304: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020356: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020357: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020405: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020406: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020457: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020457: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020509: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020509: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020561: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020561: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020613: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020614: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020666: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020666: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020715: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020715: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020767: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020767: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020819: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020820: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020872: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020872: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020924: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020924: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.020976: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.020977: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021029: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021029: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021078: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021079: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021131: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021132: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021183: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021184: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021235: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021236: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021287: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021288: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021339: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021340: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021392: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021392: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021441: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021442: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021493: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021494: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021546: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021546: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021598: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021598: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021650: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021651: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021703: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021704: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021752: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021756: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021805: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021805: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021857: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021858: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021909: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021910: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.021961: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.021962: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.022013: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.022014: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.022066: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.022066: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.022115: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.022116: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.022168: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.022168: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.022220: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.022220: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.022272: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.022273: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028015: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028024: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028116: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028117: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028173: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028174: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028232: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028232: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028281: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028282: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028333: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028334: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028386: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028386: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028439: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028439: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028491: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028492: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028544: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028545: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028597: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028597: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028646: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028647: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028699: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028700: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028751: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028751: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028832: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028843: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028897: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028898: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.028947: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.028948: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.029006: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.029007: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.029059: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.029059: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.029111: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.029111: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.029164: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.029164: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
     ib_write_bw-4813    [000] .....  2015.029216: rxe_post_send <-ib_uverbs_post_send
     ib_write_bw-4813    [000] .....  2015.029216: <stack trace>
 => rxe_post_send
 => ib_uverbs_post_send
 => ib_uverbs_write
 => vfs_write
 => ksys_write
 => __x64_sys_write
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe

如果要进行verbs编程，还需要安装相应的dev包，命令如下：

sudo apt install libibverbs-dev librdmacm-dev

Soft-RoCE(RXE)内核模块路径

enable the configuration CONFIG_RDMA_RXE and then re-compile the linux kernel:

help message:

一个RDMA的调用堆栈：

/dev/infiniband/uverbsX:
=>SYSCALL_DEFINE3
=>do_vfs_ioctl
=>ib_uverbs_ioctl
=>ib_uverbs_cmd_verbs
=>ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE
=>ib_uverbs_reg_mr
=>mlx5_ib_reg_user_mr
=>ib_umem_get_peer
=>__ib_umem_get
=>ib_peer_umem_get
=>xxx_peer_memory_client_acquire(peer memory client ops.)
=>......

Documents

【研究综述】浅谈GPU通信和PCIe P2P DMA - 知乎

RDMA入门介绍 - 知乎

https://www.snia.org/sites/default/files/ESF/How_Ethernet_RDMA_Protocols_Support_NVMe_over_Fabrics_Final.pdf

ESPCommunity

Which RDMA? (chelsio.com)

WP_RoCE_vs_iWARP.pdf (nvidia.com)

个人RDMA开发环境搭建 - 知乎

结束

papaofdoudou

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
What is RDMA

RDMA(remote direct memory access)即远端直接内存访问，是一种高性能网络通信技术，具有高带宽、低延迟、无CPU消耗、零拷贝等优点。相比kernel TCP、DPDK等传统通信手段，RDMA在延迟、吞吐和CPU消耗方面均有明显优势。RDMA的verbs编程接口是真正的异步非阻塞接口，完全不需要任何CPU参与，不需要系统调用或陷入内核空间，更不会有上线文切换！
复制链接

扫一扫