Centos下编译dpdk，mtcp

一、内核版本
    使用uname -a查看内核，版本必须大于3.13
    查看内核启动顺序，发现新内核顺序为0
    init 6或者reboot
    重启成功后uname -r

二、升级gcc
# 安装gcc9 tooset
yum install centos-release-scl
yum install devtoolset-9-gcc*
scl enable devtoolset-9 bash
# which gcc
# gcc --version

三、安装libnuma
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/numactl-2.0.12-5.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/numactl-devel-2.0.12-5.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/numactl-libs-2.0.12-5.el7.x86_64.rpm

rpm -qa | grep numa
如果已经安装numa，则rpm -Uvh *.rpm，否则rpm -ivh *.rpm，完成后/usr/include/numa.h

四、获取mTCP源码
git clone https://github.com/mtcp-stack/mtcp.git
获取mtcp目录下dpdk源码
cd mtcp
git submodule init
git submodule update

./setup_mtcp_dpdk_env.sh，依次选择15，18，22，24，35
输入35后，会问：Are you using an Intel NIC (y/n)?，没有nic时输入n

五、设置环境变量
export RTE_SDK=`echo $PWD`/dpdk
export RTE_TARGET=x86_64-native-linuxapp-gcc

六、编译mtcp
autoreconf -f -i
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET
make

编译成功后，生成./mtcp/mtcp/lib/libmtcp.a

七、有Intel NIC的运行./setup_mtcp_onvm_env.sh
没有的需要配置
具体源代码、测试的例子在mtcp/apps/example路径下epserver是server端，epwget是client端

八、配置docket，
1）编写Dockerfile文件
FROM debian:stretch

MAINTAINER yanhai<yh@chinabluedon.cn>

COPY sources.list /etc/apt/
COPY dpdk-18.11.tar.xz /opt/

# 解压xz格式的文件，需要先安装xz-utils
RUN apt-get update && \
    apt-get install -y xz-utils && \
    apt-get install -y gcc make && \
    apt-get install -y libnuma-dev && \
    apt-get install -y libpcap-dev && \
    rm -rf /var/lib/apt/lists/*

# 关闭掉一些编译开关，因为我们不需要这些功能，打开会由于某些原因编译失败
# 如果使能PMD_PCAP，需要先安装libpcap开发包
RUN cd /opt && tar -xf dpdk-18.11.tar.xz && \
    cd dpdk-18.11 && \
    export RTE_SDK=/opt/dpdk-18.11 && \
    export RTE_TARGET=x86_64-native-linuxapp-gcc && \
    sed -i 's/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/' config/common_linuxapp && \
    sed -i 's/CONFIG_RTE_LIBRTE_KNI=y/CONFIG_RTE_LIBRTE_KNI=n/' config/common_linuxapp && \
    sed -i 's/CONFIG_RTE_KNI_KMOD=y/CONFIG_RTE_KNI_KMOD=n/' config/common_linuxapp && \
    sed -i 's/CONFIG_RTE_APP_TEST=y/CONFIG_RTE_APP_TEST=n/' config/common_base && \
    sed -i 's/CONFIG_RTE_TEST_PMD=y/CONFIG_RTE_TEST_PMD=n/' config/common_base && \
    sed -i 's/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/' config/common_base && \
    sed -i 's/CONFIG_RTE_LIBRTE_IGB_PMD=y/CONFIG_RTE_LIBRTE_IGB_PMD=n/' config/common_base && \
    sed -i 's/CONFIG_RTE_LIBRTE_IXGBE_PMD=y/CONFIG_RTE_LIBRTE_IXGBE_PMD=n/' config/common_base && \
    sed -i 's/CONFIG_RTE_LIBRTE_I40E_PMD=y/CONFIG_RTE_LIBRTE_I40E_PMD=n/' config/common_base && \
    sed -ri 's,(PMD_PCAP=).*,\1y,' config/common_base && \
    make config T=$RTE_TARGET && \
    make -j4 install T=$RTE_TARGET DESTDIR=/usr/local && \
    cd / && \
    rm -rf /opt/dpdk-18.11.tar.xz && \
    rm -rf /opt/dpdk-18.11
2）build，我们将镜像命名为dpdk-dev
docker build -t dpdk-dev .
3）测试
在宿主机上已经编译了dpdk源码，在/opt/dpdk-18.11/下

安装igb驱动
modprobe uio
insmod /opt/dpdk-18.11/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

配置大页内存
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge

# 分配系统总内存的4分之1用作2MB大页内存
total_mem=`cat /proc/meminfo | grep MemTotal | awk '{print $2}'`
if [[ -d /sys/devices/system/node/node1 ]]; then
    hugepages_mem=$(($total_mem/8000000*1024/2))
    echo $hugepages_mem > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
    echo $hugepages_mem > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
else
    hugepages_mem=$(($total_mem/8000000*1024))
    echo $hugepages_mem > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
fi

创建容器
docker run -it -v /sys/bus/pci/devices:/sys/bus/pci/devices -v /sys/kernel/mm/hugepages:/sys/kernel/mm/hugepages -v /sys/devices/system/node:/sys/devices/system/node -v /dev:/dev dpdk-dev

进入examples目录
这里编译运行helloworld程序
export RTE_SDK=/opt/dpdk-18.11
export RTE_TARGET=x86_64-native-linuxapp-gcc
cd /opt/dpdk-18.11/examples/helloworld
make
./build/app/helloworld