遍搜全网，竟然关于openbmc mctp over pcie的支持说明情况了无文章，这不是一个艰难的问题，服务器BMC也不是一个超级精尖的产品，想当年分享stm32资料，都是满天飞。可能服务器市场大家对于文章上的分享并无兴趣。
此篇文章只是一个最基本的简单的说明文章，弥补一下openbmc资源匮乏吧。另外，本片文章只是一个简单的开篇文章，有错漏，有知识不全面，请尽管评论。
注意: 本章涉及的代码可以查看代码仓库:https://gitee.com/wit_yuan/yuan_mctpd/tree/yuan_modified_mctpd
1.官方活跃的mctp支持情况
官方活跃的2个仓库有:https://github.com/CodeConstruct/mctp和https://github.com/openbmc/libmctp,这一套下来还需要kernel的支持, 代码的提交可以参考如下链接:
https://lore.kernel.org/netdev/20210729022053.134453-1-jk@codeconstruct.com.au/
简单来讲，可以参考内核这几个文件:
build/ast2600-default/workspace/sources/linux-aspeed/net/mctp/af_mctp.c
build/ast2600-default/workspace/sources/linux-aspeed/net/mctp/device.c
build/ast2600-default/workspace/sources/linux-aspeed/net/mctp/neigh.c
build/ast2600-default/workspace/sources/linux-aspeed/net/mctp/route.c
build/ast2600-default/workspace/sources/linux-aspeed/net/socket.c
这几个文件是核心的整个mctp架构代码。
如下几个文件是驱动代码，对接到硬件上:
build/ast2600-default/workspace/sources/linux-aspeed/drivers/net/mctp/mctp-i2c.c
build/ast2600-default/workspace/sources/linux-aspeed/drivers/net/mctp/mctp-i3c.c
build/ast2600-default/workspace/sources/linux-aspeed/drivers/net/mctp/mctp-serial.c
可以看到，这里面没有一个直接的针对mctp-pcie.c文件，粗浅看，这个架构是不支持mctp over pcie的。
补充一点，文件:build/ast2600-default/workspace/sources/linux-aspeed/drivers/soc/aspeed/aspeed-mctp.c是内核的mctp over pcie驱动文件，但是说了与不说一样，因为架构上没支持(差一个驱动程序)。
简单看一下文件:build/ast2600-default/workspace/sources/linux-aspeed/drivers/net/mctp/mctp-i2c.c中的mctp_i2c_add_netdev()函数:
static int mctp_i2c_add_netdev(struct mctp_i2c_client *mcli,
struct i2c_adapter *adap)
---> struct net_device *ndev = NULL;
---> ndev = alloc_netdev(sizeof(*midev), namebuf, NET_NAME_ENUM, mctp_i2c_net_setup);
---> rc = mctp_register_netdev(ndev, &mctp_i2c_mctp_ops);
可以反推，目前的架构是不支持mctp over pcie的。当前的架构情况是kernel mctp + lib + dbus服务组成。
2.intel-mctp的支持情况
留意: intel已经对这一块不再维护了。
可以在intel的openbmc仓库里面，找到链接:
- 1.
libmctp链接:https://github.com/Intel-BMC/libmctp - 2.
pmci链接:https://github.com/Intel-BMC/pmci - 3.
mctpd链接:https://github.com/Intel-BMC/pmci/tree/master/mctpd
从mctpd仓库代码，基本上看到是支持mctp over pcie的并且BMC是支持作为endpoint的。这符合我的需求。
我的需求如下:
1.bios作为bus owner,负责分配eid
2.bmc作为endpoint，访问其他pcie设备
2.1 libmctp分析
libmctp的代码仓库如下:
https://github.com/Intel-BMC/libmctp
对于mctp over pcie的支持，可查看文件:
build/ast2600-default/workspace/sources/libmctp-intel/libmctp.h
build/ast2600-default/workspace/sources/libmctp-intel/astpcie.c
库的功能实现用的是.c文件，实现基本的ioctl()，对于C++来说不好和kernel交互的代码。这里是实现就没什么可说的。
2.2.intel-bmc 的mctpd的实现
mctpd的代码仓库如下:
https://github.com/Intel-BMC/pmci/tree/master/mctpd
这一块的作用是作为一个dbus service，供其他client调用接口。代码结构比较清晰。
注意，这里面有一个yaml的文件需要导入，作为server与client等的支持的头文件。
文件可以参考一个patch: https://github.com/Intel-BMC/openbmc/blob/ac27a5095790a2d49a44d7c46440e375a8f84812/meta-openbmc-mods/meta-common/recipes-phosphor/dbus/phosphor-dbus-interfaces/0032-update-meson-build-for-MCTP-interfaces.patch#L33,具体需要修改:
与:
2.3 导入intel bmc mctp over pcie的实现
注意:这里导入只是一个基本的功能测试mctp over pcie，不是实际功能开发。
参考git仓库上的.bb文件，资源路径:
https://github.com/Intel-BMC/openbmc/tree/1-release/meta-openbmc-mods/meta-common/recipes-phosphor/pmci.
在目录: meta-aspeed-sdk/recipes-phosphor/mctpd/中导入https://github.com/Intel-BMC/openbmc/blob/1-release/meta-openbmc-mods/meta-common/recipes-phosphor/pmci/mctpd.bb文件。
修改mctpd.bb文件名，改为:mctpd_1.0.bb，内容如:
SUMMARY = "MCTP Daemon"
DESCRIPTION = "Implementation of MCTP (DTMF DSP0236)"
LICENSE = "CLOSED"
SRC_URI = "file://mctpd.tar.gz"
S = "${WORKDIR}"
PV = "1.0+git${SRCPV}"
OECMAKE_SOURCEPATH = "${S}"
inherit cmake systemd
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
DEPENDS += " \
libmctp-intel \
systemd \
sdbusplus \
phosphor-logging \
boost \
i2c-tools \
cli11 \
nlohmann-json \
gtest \
phosphor-dbus-interfaces \
udev \
"
PACKAGECONFIG[systemd] = ",,systemd,libsystemd"
SYSTEMD_AUTO_ENABLE:${PN} = "enable"
do_install:append(){
install -d ${D}/${systemd_system_unitdir}
install -d ${D}/${bindir}
install -d ${D}/${datadir}/mctp/
install -m 755 ${WORKDIR}/${PN}-${PV}/mctpd ${D}/${bindir}
install -m 0644 ${S}/configurations/mctp_config.json ${D}/${datadir}/mctp/mctp_config.json
install -m 0644 ${S}/service_files/xyz.openbmc_project.mctpd@.service \
${D}/${systemd_system_unitdir}/xyz.openbmc_project.mctpd@.service
}
FILES:${PN} += "${systemd_system_unitdir}"
FILES:${PN} += "${datadir}/mctp/"
FILES:${PN} += "${bindir}"
如上，是因为当前使用的openbmc代码未加入intel mctp的.bb文件，故把mctpd的代码从intel-bmc网站上下载下来之后，打包成:mctpd.tar.gz，整个目录结构如下:
采取命令:
# devtool modify mctpd
之后，需要修改build/ast2600-default/tmp/work/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/mctpd/1.0+git/CMakeLists.txt中的内容:
set(CMAKE_CXX_STANDARD 23)
-Wnull-dereference \ //需要去掉
install(FILES ${SERVICE_FILES} DESTINATION /usr/lib/systemd/system/) //修改为/usr目录下
另外，修改文件build/ast2600-default/workspace/sources/mctpd/src/SMBusBinding.cpp,注释掉其中的内容:
void SMBusBinding::readResponse()
{
// smbusReceiverFd.async_wait(
// boost::asio::posix::descriptor_base::wait_error, [this](auto& ec) {
// if (ec)
// {
// phosphor::logging::log<phosphor::logging::level::ERR>(
// "Error: mctp_smbus_read()");
// readResponse();
// }
// // through libmctp this will invoke rxMessage and message assembly
// mctp_smbus_read(smbus);
// readResponse();
// });
phosphor::logging::log<phosphor::logging::level::ERR>(
"mctpd yuan modify it, Error: SMBusBinding::readResponse()");
}
另外，修改文件:build/ast2600-default/workspace/sources/mctpd/include/hw/aspeed/PCIeMonitor.hpp中的:
static constexpr const char* astUdevPath =
"/sys/devices/platform/ahb/ahb:apb/1e6e8000.mctp/misc/aspeed-mctp0";
修改文件:meta-aspeed-sdk/recipes-aspeed/packagegroups/packagegroup-oss.bb中的内容：
# delete mctp , wityuan noted 2024/08/25.
SUMMARY:${PN}-apps = "Open Source Applications"
RDEPENDS:${PN}-apps = " \
mdio-tool \
gperf \
iperf3 \
pciutils \
ethtool \
mmc-utils \
i3c-tools \
i2c-tools \
xdma-test \
libpeci \
dhrystone \
nbd-client \
iozone3 \
ncsi-netlink \
hdparm \
stressapptest \
e2fsprogs-mke2fs \
nvme-cli \
${@d.getVar('PREFERRED_PROVIDER_u-boot-fw-utils', True) or 'u-boot-fw-utils'} \
aer-inject \
fio \
memtester \
coremark \
"
删掉了其中的mctp.
之后，执行如下命令，编译mctpd:
# bitbake -c build mctpd
2.4 导入intel-bmc libmctp pcie的实现
当前的openbmc代码实际上是包含libmctp-intel的实现的。修改libmctp-intel的bb文件:
修改meta-aspeed-sdk/recipes-phosphor/pmci/libmctp-intel_git.bb文件，内容如:
SUMMARY = "libmctp:intel"
DESCRIPTION = "Implementation of MCTP(DMTF DSP0236)"
SRC_URI = "git://github.com/Intel-BMC/libmctp.git;protocol=https;branch=master"
SRCREV = "21dc38e911a27af2e914f834b2e2b775f7dad520"
S = "${WORKDIR}/git"
PV = "1.0+git"
LICENSE = "Apache-2.0 | GPL-2.0-or-later"
LIC_FILES_CHKSUM = "file://LICENSE;md5=0d30807bb7a4f16d36e96b78f9ed8fae"
inherit cmake systemd
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
DEPENDS += "i2c-tools"
#FILES:${PN}-dev += ""
FILES:${PN} += " ${libdir}"
LIBVER = "0.1.0"
do_install:append() {
install -d ${D}${libdir}
install -m 755 ${WORKDIR}/${PN}-${PV}/libmctp_intel.so.${LIBVER} ${D}/${libdir}/
}
修改build/ast2600-default/workspace/sources/libmctp-intel/CMakeLists.txt文件，增加内容:
SET_TARGET_PROPERTIES(mctp_intel PROPERTIES VERSION 0.1.0 SOVERSION 0)
修改内容:
add_library (mctp_intel SHARED alloc.c asti3c.c astlpc.c core.c log.c libmctp.h serial.c astpcie.c smbus.c)
接着修改文件:build/ast2600-default/workspace/sources/libmctp-intel/astpcie.h中的内容:
/* driver device file */
#define AST_DRV_FILE "/dev/aspeed-mctp0"
注意，还需要修改文件:meta-aspeed-sdk/recipes-core/images/obmc-phosphor-image.bbappend:
IMAGE_INSTALL:append = " \
webui-vue \
libmctp \
entity-manager \
dbus-sensors \
"
将libmctp改为libmctp-intel.
执行编译操作:
# bitbake -c build libmctp-intel
编译成功后，启动BMC,或者在文件:build/ast2600-default/workspace/sources/libmctp-intel/oe-workdir/libmctp-intel-1.0+git中可以看到如下信息:
# ls /usr/lib/libm* -al
-rwxr-xr-x 1 root root 271756 Mar 9 2018 /usr/lib/libm.so.6
lrwxrwxrwx 1 root root 16 Mar 9 2018 /usr/lib/libmapper.so.1 -> libmapper.so.1.0
-rwxr-xr-x 1 root root 13548 Mar 9 2018 /usr/lib/libmapper.so.1.0
lrwxrwxrwx 1 root root 22 Mar 9 2018 /usr/lib/libmctp_intel.so.0 -> libmctp_intel.so.0.1.0
-rwxr-xr-x 1 root root 34064 Mar 9 2018 /usr/lib/libmctp_intel.so.0.1.0
lrwxrwxrwx 1 root root 15 Mar 9 2018 /usr/lib/libmnl.so.0 -> libmnl.so.0.2.0
-rwxr-xr-x 1 root root 17748 Mar 9 2018 /usr/lib/libmnl.so.0.2.0
lrwxrwxrwx 1 root root 17 Mar 9 2018 /usr/lib/libmount.so.1 -> libmount.so.1.1.0
-rwxr-xr-x 1 root root 407264 Mar 9 2018 /usr/lib/libmount.so.1.1.0
注意:
BMC的驱动名称已经改变:
# ls /dev/aspeed-mctp0 -al
crw------- 1 root root 10, 111 Sep 20 11:23 /dev/aspeed-mctp0
接着修改文件:build/ast2600-default/conf/local.conf，添加内容:
...
IMAGE_INSTALL:append = " mctpd"
以上，改完之后，编译生成镜像:
# bitbake obmc-phosphor-image
最终，启动BMC，在BMC串口下执行命令:
# /usr/bin/mctpd -b pcie &
然后检查是否成功启动dbus mctp服务。
- 1.查看
mctpd服务
# busctl | grep mctp
:1.71 425 mctpd root :1.71 serial-getty@ttyS4.service -
xyz.openbmc_project.MCTP-pcie 425 mctpd root :1.71 serial-getty@ttyS4.service
- 2.查看检查到哪些
endpoint
# busctl tree xyz.openbmc_project.MCTP-pcie
`- /xyz
`- /xyz/openbmc_project
`- /xyz/openbmc_project/mctp
`- /xyz/openbmc_project/mctp/device
|- /xyz/openbmc_project/mctp/device/10
|- /xyz/openbmc_project/mctp/device/29
|- /xyz/openbmc_project/mctp/device/61
`- /xyz/openbmc_project/mctp/device/8
- 3.检查
endpoint属性信息
# busctl introspect xyz.openbmc_project.MCTP-pcie /xyz/openbmc_project/mctp/device/10
NAME TYPE SIGNATURE RESULT/VALUE FLAGS
org.freedesktop.DBus.Introspectable interface - - -
.Introspect method - s -
org.freedesktop.DBus.Peer interface - - -
.GetMachineId method - s -
.Ping method - - -
org.freedesktop.DBus.Properties interface - - -
.Get method ss v -
.GetAll method s a{sv} -
.Set method ssv - -
.PropertiesChanged signal sa{sv}as - -
xyz.openbmc_project.Common.UUID interface - - -
.UUID property s "bca4af17-8995-5632-be7d-53f1697c7912" emits-change
xyz.openbmc_project.Inventory.Decorator.LocationCode interface - - -
.LocationCode property s "" emits-change
xyz.openbmc_project.Inventory.Decorator.PCIDevice interface - - -
.Bus property y 57 emits-change
.Device property y 0 emits-change
.Function property y 0 emits-change
xyz.openbmc_project.MCTP.Endpoint interface - - -
.Mode property s "xyz.openbmc_project.MCTP.Base.Bindin... emits-change
.NetworkId property q 0 emits-change
xyz.openbmc_project.MCTP.PCIVendorDefined interface - - -
.MessageTypeProperty property aq 1 83 emits-change
.VendorID property s "0x590" emits-change
xyz.openbmc_project.MCTP.SupportedMessageTypes interface - - -
.Ethernet property b false emits-change
.MctpControl property b true emits-change
.NCSI property b false emits-change
.NVMeMgmtMsg property b false emits-change
.PLDM property b true emits-change
.SPDM property b false emits-change
.VDIANA property b false emits-change
.VDPCI property b true emits-change
root@ast2600-default:~#
- 4.检查另外一个
endpoint:
# busctl introspect xyz.openbmc_project.MCTP-pcie /xyz/openbmc_project/mctp/device/8
NAME TYPE SIGNATURE RESULT/VALUE FLAGS
org.freedesktop.DBus.Introspectable interface - - -
.Introspect method - s -
org.freedesktop.DBus.Peer interface - - -
.GetMachineId method - s -
.Ping method - - -
org.freedesktop.DBus.Properties interface - - -
.Get method ss v -
.GetAll method s a{sv} -
.Set method ssv - -
.PropertiesChanged signal sa{sv}as - -
xyz.openbmc_project.Common.UUID interface - - -
.UUID property s "00000000-0000-0000-0000-000000000000" emits-change
xyz.openbmc_project.Inventory.Decorator.LocationCode interface - - -
.LocationCode property s "" emits-change
xyz.openbmc_project.Inventory.Decorator.PCIDevice interface - - -
.Bus property y 0 emits-change
.Device property y 20 emits-change
.Function property y 6 emits-change
xyz.openbmc_project.MCTP.Endpoint interface - - -
.Mode property s "xyz.openbmc_project.MCTP.Base.Bindin... emits-change
.NetworkId property q 0 emits-change
xyz.openbmc_project.MCTP.PCIVendorDefined interface - - -
.MessageTypeProperty property aq 1 256 emits-change
.VendorID property s "0x8086" emits-change
xyz.openbmc_project.MCTP.SupportedMessageTypes interface - - -
.Ethernet property b false emits-change
.MctpControl property b true emits-change
.NCSI property b false emits-change
.NVMeMgmtMsg property b false emits-change
.PLDM property b false emits-change
.SPDM property b false emits-change
.VDIANA property b false emits-change
.VDPCI property b true emits-change
如此，基本的mctp over pcie代码逻辑基本上跑通了。
3.问题与解决
3.1 报错libmctp-intel rdepends on libmctp-intel-dev
出现报错信息如下:
ERROR: libmctp-intel-1.0+git-r0 do_package_qa: QA Issue: libmctp-intel rdepends on libmctp-intel-dev [dev-deps]
ERROR: libmctp-intel-1.0+git-r0 do_package_qa: QA Issue: -dev package libmctp-intel-dev contains non-symlink .so '/usr/lib/libmctp_intel.so' [dev-elf]
针对此问题，解决办法如:
在CmakeFile中添加:
SET_TARGET_PROPERTIES(mctp_intel PROPERTIES VERSION 0.1.0 SOVERSION 0)
这样可以生成:
lrwxrwxrwx 1 wityuan wityuan 18 8月 26 23:19 libmctp_intel.so -> libmctp_intel.so.0
lrwxrwxrwx 1 wityuan wityuan 22 8月 26 23:19 libmctp_intel.so.0 -> libmctp_intel.so.0.1.0
-rwxr-xr-x 1 wityuan wityuan 150972 8月 26 23:19 libmctp_intel.so.0.1.0
然后: .bb 中添加:
do_install:append() {
install -d ${D}${libdir}
install -m 755 ${WORKDIR}/${PN}-${PV}/libmctp_intel.so.${LIBVER} ${D}/${libdir}/
}
或者使用静态库？!!, 暂时改为动态库了。
3.2 报错Pseudo log: path mismatch
报错信息如下:
ERROR: Task (/home/wityuan/Desktop/sdk_v09.01/openbmc/meta-aspeed-sdk/recipes-phosphor/mctpd/mctpd_1.0.bb:do_install) failed with exit code '1'
Pseudo log:
path mismatch [3 links]: ino 21059040 db '/home/wityuan/Desktop/sdk_v09.01/openbmc/build/ast2600-default/tmp/work/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/mctpd/1.0+git/package/usr/src/debug/mctpd/1.0+git/src/main.cpp' req '/home/wityuan/Desktop/sdk_v09.01/openbmc/build/ast2600-default/workspace/sources/mctpd/src/main.cpp'.
可以删除文件:
build/ast2600-default/tmp/work/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/mctpd/1.0+git/pseudo/files.db
然后再次编译即可。目前比较好的办法还没找到。^_ ^
参考资料:https://wiki.yoctoproject.org/wiki/Pseudo_Abort,
解释信息如下:
Pseudo can suffer from problems where files are created or modified in pseudo's fakeroot context, then deleted outside of it.
In particular, the inode number for that file may be reused.
If some later access under pseudo uses that inode number, pseudo can see this and become confused about whether the files are the same file or not.
Under older behaviour for pseudo, the modes of the old file may be applied to the new one, causing mode permission corruption.
3.3 报错package is not obeying usrmerge distro feature
报错信息如下:
<packagename> package is not obeying usrmerge distro feature. /<path> should be relocated to /usr. [usrmerge]
在yocto上看到解释如下:
If usrmerge is in DISTRO_FEATURES, this check will ensure that no package installs files to root (/bin, /sbin, /lib, /lib64) directories.
If you are seeing this message, it indicates that the do_install step (or perhaps the build process that do_install is calling into,
e.g. make install is using hardcoded paths instead of the variables set up for this (bindir, sbindir, etc.), and should be changed so that it does.
修改办法直接是修改cmakelist中，将install目录改为/usr下。
3.4 报错Files/directories were installed but not shipped in any package
报错信息具体内容如:
<recipename>: Files/directories were installed but not shipped in any package [installed-vs-shipped]
具体解释可以查看文档:https://docs.yoctoproject.org/ref-manual/qa-checks.html，重点是:
Files have been installed within the do_install task but have not been included in any package by way of the FILES variable
修改办法:
Add the files to FILES for the package you want them to appear in (e.g. FILES:${PN} for the main package).
Delete the files at the end of the do_install task if the files are not needed in any package.
也就是需要对应修改:
FILES:${PN} += "..."
do_install:append() {
...
}
最后，解决问题，需要多看官方文档:https://docs.yoctoproject.org/ref-manual/，一遍不够，多看几遍吧。另外，多分析bitbake源码。
3366
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
