MTK T750平台:CCCI驱动调试
- 1. T750简介
- 2. CCCI驱动调试
- 2.1 调试环境
- 2.2 Release Note
- 2.3 make
- 2.4 error: ‘struct dev_pm_info’ has no member named ‘driver_flags’
- 2.5 error: inlining failed in call to always_inline
- 2.6 error: ‘struct pci_dev’ has no member named ‘priv_flags’
- 2.7 error: missing binary operator before token "("
- 2.8 error: ‘struct net_device’ has no member named ‘max_mtu’
- 2.9 error: ‘atomic_t {aka struct }’ has no member named ‘refs’
- 2.10 Unknown symbol in module
- 3. 验证
- 4. 附件
- 5. 总结
1. T750简介
MediaTek T750 是一款面向新一代5G CPE无线产品,可应用于5G固定无线接入(FWA)和移动热点(MiFi)等设备,为家庭、企业和移动用户带来高速5G连接,芯片平台采用 7nm 制程工艺,高度集成 5G NR FR1 调制解调器,4 核 Arm Cortex-A55 CPU 可提供完整的功能和配置,支持 5G NR Sub-6GHz 下双载波聚合(2CC CA)200MHz 频率,不仅拥有更大的信号覆盖范围,同时也让 5G 的下行速度大幅提升。
在独立组网(SA)中,T750 芯片平台最大下行峰值可达到 4.76Gbps ,上行峰值可达到 1.25Gbps ,可支持两个 2.5Gbps SGMII 接口、多种 LAN 端口配置、多种 Wi-Fi 配置包括单颗 5G 4×4 ,单颗 2.4G 4×4 和 5G 2×2 + 2.4G 2×2 双频 Wi-Fi 6 芯片,在保证网络高速连接的同时,也确保了 Wi-Fi 的稳定性。
2. CCCI驱动调试
2.1 调试环境
主控:NVIDIA XAVIER (ARM)
系统:Ubuntu 16.04
内核:Linux 4.9.0
Modem:MTK T750
驱动:CCCI (Cross Core Communication Interface)
2.2 Release Note
CCCI驱动Release_note:
The host CCCI driver provides standard linux device nodes, which can be find in directory /dev/. User can use this device nodes with linux standard API--open/close and read/write.
You can execute command "make" in driver top directory, then will get driver object files(*.ko) in folder /out, and the directory structure is shown as following.
(Note kernel version: 4.15.0-29-generic, gcc version: 5.4.0 20160609)
从Release_note可以看出MTK仅适配了4.15.0内核版本。为了更快的完成调试工作,我们需要快速的发现和解决问题。
2.3 make
拷贝CCCI驱动到NVIDIA XAVIER (ARM)后,解压直接执行make进行编译时报错:
make[1]: Entering directory '/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/kernel-4.9'
CC [M] /home/sdk/Linux_PCIe_Driver_v1.0.11/./PCIE/core/mtk-pcie.o
gcc: error: unrecognized argument in option ‘-mcmodel=kernel’
gcc: note: valid arguments to ‘-mcmodel=’ are: large small tiny
gcc: error: unrecognized command line option ‘-mno-sse’; did you mean ‘-fno-dse’?
gcc: error: unrecognized command line option ‘-mno-mmx’
gcc: error: unrecognized command line option ‘-mno-sse2’; did you mean ‘-fno-dse’?
gcc: error: unrecognized command line option ‘-mno-3dnow’; did you mean ‘-fno-doc’?
gcc: error: unrecognized command line option ‘-m64’
gcc: error: unrecognized command line option ‘-mno-red-zone’; did you mean ‘-fno-regmove’?
scripts/Makefile.build:335: recipe for target '/home/sdk/Linux_PCIe_Driver_v1.0.11/./PCIE/core/mtk-pcie.o' failed
分析与解决:
- 通过分析,我们发现在顶级目录Makefile中目标架构写成了固定的x86_64,这就导致在编译源码时,编译器无法寻找到合适的交叉编译链,导致编译失败。
COMPILER = "gcc"
LINKER = "ld"
TARGET_ARCH := x86_64
DATE = "BUILD_TIME="
LABEL = "LABEL="
- 针对该问题有如下解决方案:修改TARGET_ARCH 为 arm64,注意,这里只能写成arm64,非aarch64。
COMPILER = "gcc"
LINKER = "ld"
TARGET_ARCH := arm64
DATE = "BUILD_TIME="
LABEL = "LABEL="
2.4 error: ‘struct dev_pm_info’ has no member named ‘driver_flags’
通过上述方法的修改,可以开始编译驱动。出现了诸多“has no member named”错误:
./PCIE/core/mtk-pci.c:374:9: error: implicit declaration of function ‘dev_pm_set_driver_flags’; did you mean ‘devm_get_free_pages’? [-Werror=implicit-function-declaration]
./PCIE/core/mtk-pci.c:374:74: error: ‘DPM_FLAG_NEVER_SKIP’ undeclared (first use in this function); did you mean ‘PF_FREEZER_SKIP’?
./PCIE/core/mtk-pci.c:374:60: error: ‘struct dev_pm_info’ has no member named ‘driver_flags’
./PCIE/core/mtk-pci.c:1391:3: error: ‘const struct pci_error_handlers’ has no member named ‘reset_prepare’
./PCIE/core/mtk-pci.c:1391:19: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
./PCIE/core/mtk-pci.c:1392:3: error: ‘const struct pci_error_handlers’ has no member named ‘reset_done’; did you mean ‘reset_notify’?
./PCIE/core/mtk-pci.c:1478:12: error: ‘struct pci_dev’ has no member named ‘ltr_path’
…
由于NVIDIA XAVIER (ARM)环境的Linux内核版本是4.9.0,而驱动中申明支持4.15.0,怀疑为内核版本差异导致。对比4.9.0和4.15.0版本相关文件:
- struct dev_pm_info结构体:
上图是struct dev_pm_info结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。
static int mtk_pci_pm_init(struct mtk_pci_dev *mtk_dev)
{
…
device_init_wakeup(&pdev->dev, true);
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
if(!suspend_direct_complete)
dev_pm_set_driver_flags(&pdev->dev, pdev->dev.power.driver_flags|DPM_FLAG_NEVER_SKIP);
#endif
…
}
- pci_error_handlers结构体
上图是pci_error_handlers结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以看到在4.15.0中是将4.0.9中的“void (*reset_notify)(struct pci_dev *dev, bool prepare);”函数拆分为两个“void (*reset_prepare)(struct pci_dev *dev);void (*reset_done)(struct pci_dev *dev);”,作用实质上是一致的。进一步阅读CCCI源码对这一块的处理:
static const struct pci_error_handlers mtk_pci_err_handler = {
.error_detected = mtk_pci_error_detected,
.slot_reset = mtk_pci_slot_reset,
.reset_prepare = mtk_pci_reset_prepare,
.reset_done = mtk_pci_reset_done,
.mmio_enabled = mtk_pcie_mmio_enabled,
.resume = mtk_pci_resume,
};
static void mtk_pci_reset_prepare(struct pci_dev *pdev)
{
MTK_PCI_ERS_FUNC_ENTER();
}
#define MTK_PCI_ERS_FUNC_ENTER() \
MTK_INFO_LOG("ERS", "%s Enter\n", __func__)
通过阅读CCCI代码,我们发现,实际处理仅仅是一个log的打印,因此毫无意义,无需再修改CCCI对这块的处理,直接进行版本的区分即可。
static const struct pci_error_handlers mtk_pci_err_handler = {
.error_detected = mtk_pci_error_detected,
.slot_reset = mtk_pci_slot_reset,
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
.reset_prepare = mtk_pci_reset_prepare,
.reset_done = mtk_pci_reset_done,
#endif
.mmio_enabled = mtk_pcie_mmio_enabled,
.resume = mtk_pci_resume,
};
- struct pci_dev
上图是struct pci_dev结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。
static void mtk_pci_bridge_configure(struct mtk_pci_dev *mtk_dev, struct pci_dev *dev)
{
…
//re-configure bridge ltr to satisfy remove-rescan case
#ifdef CONFIG_PCIEASPM
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
if (bridge->ltr_path == 1) {
pcie_capability_read_word(bridge, PCI_EXP_DEVCTL2, &cap);
if(!(cap&PCI_EXP_DEVCTL2_LTR_EN))
pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_LTR_EN);
}
MTK_INFO_LOG(TAG, "Bridge LTR_Path %x/%x cap %x flag %x\n", bridge->ltr_path, dev->ltr_path, cap, cap&PCI_EXP_DEVCTL2_LTR_EN);
#endif
#endif
}
static void mtk_pci_configure_ltr(struct pci_dev *dev)
{
//re-configure device ltr to satisfy remove-rescan case
#ifdef CONFIG_PCIEASPM
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
if (dev->ltr_path == 1) {
pcie_capability_clear_word(dev, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_LTR_EN);
pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_LTR_EN);
}
#endif
#endif
}
2.5 error: inlining failed in call to always_inline
通过修改上述问题,再次进行编译出现如下错误:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-pci.c: In function ‘mtk_pcie_interrupt_reinit’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/platform/M8X/75/mtk-pcie-mac.h:109:13: error: inlining failed in call to always_inline ‘mtk_pcie_intr_enable’: function body not available
inline void mtk_pcie_intr_enable(PPCIE_MAC_IREG_REGS pbase, bool enable);
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-pci.c:958:3: note: called from here
mtk_pcie_intr_enable(mtk_dev->base_addr.pcie_mac_ireg_base, TRUE);
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-isr.c: In function ‘mtk_pcie_msix_irq’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/platform/M8X/75/mtk-pcie-mac.h:132:21: error: inlining failed in call to always_inline ‘mtk_pcie_msix_status_get’: function body not available
inline unsigned int mtk_pcie_msix_status_get(PPCIE_MAC_IREG_REGS pbase);
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-isr.c:100:14: note: called from here
msix_status = mtk_pcie_msix_status_get(mtk_dev->base_addr.pcie_mac_ireg_base);
开始我们以为inline函数的作用域只是当前C文件,而不能被别人C文件调用。但深入学习C语言的inline关键字http://hi.baidu.com/serial_story/blog/item/d9f4c2fa9abf11d4b58f31bf.html后,发现上面的inline,是可以被外部所引用的,也就是说,上面的代码,是可以正常编译的。因此我们尝试通过添加CFLAGS进行规避问题:
COMPILE_FLAGS := -g
COMPILE_FLAGS += -O3 \
-U_FORTIFY_SOURCE \
-fno-stack-protector
COMPILE_FLAGS += -Wall
COMPILE_FLAGS += $(INCLUDE_DIRS)
COMPILE_FLAGS += $(MICRO_DEF)
COMPILE_FLAGS += -Wno-date-time \
-Wno-implicit-function-declaration \
-Wno-misleading-indentation \
-Wno-incompatible-pointer-types
-O3是恢复默认优化所必需的,这是避免许多故障所必需的;-U_FORTIFY_SOURCE需要避免gcc …/sysdeps/unix/sysv/linux/syslog.c失败(在函数’vsyslog_chk’中:[…]内联调用’syslog’失败);-fno-stack-protector需要避免gcc […] elf / dl-allobjs.os libc_pic.a((init-first.os)😦.data 0x0)的失败:’ libc_multiple_libcs’的多重定义)。
通过上述修改,问题仍然存在,通过再次深入阅读代码,发现inline在此可直接省去,非必要,直接删除问题解决。
2.6 error: ‘struct pci_dev’ has no member named ‘priv_flags’
通过修改上述问题,再次进行编译出现如下错误:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./pcie_drv/mtk-pcie-drv-test.c: In function ‘t_pcie_lp_show_lp_info’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./pcie_drv/mtk-pcie-drv-test.c:562:31: error: ‘struct pci_dev’ has no member named ‘priv_flags’; did you mean ‘dev_flags’?
test_bit(0, &mtk_dev->pdev->priv_flags));
对比4.9.0和4.15.0版本相关文件:
上图是struct pci_dev结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。
int t_pcie_lp_show_lp_info(struct mtk_pci_dev *mtk_dev, int argc, char **argv)
{
…
MTK_INFO_LOG(TAG, "runtime usage_count=%d\n",
atomic_read(&(mtk_dev->pdev->dev.power.usage_count)));
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
MTK_INFO_LOG(TAG, "dev disconnected=%d\n",
test_bit(0, &mtk_dev->pdev->priv_flags));
#endif
…
}
2.7 error: missing binary operator before token “(”
通过修改上述问题,再次进行编译出现如下错误:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./pcie_drv/mtk-pcie-drv-test.c:561:48: error: missing binary operator before token "("
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
分析发现是未识别LINUX_VERSION_CODE和KERNEL_VERSION宏,为确实头文件导致。
#include <linux/version.h>
2.8 error: ‘struct net_device’ has no member named ‘max_mtu’
通过修改上述问题,再次进行编译出现如下错误:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c: In function ‘ccmni_dev_init’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c:695:5: error: ‘struct net_device’ has no member named ‘max_mtu’ dev->max_mtu = CCMNI_MTU_MAX;
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c:728:5: error: ‘struct net_device’ has no member named ‘needs_free_netdev’ dev->needs_free_netdev = true;
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c:731:7: error: ‘struct net_device’ has no member named ‘priv_destructor’; did you mean ‘destructor’?dev->priv_destructor = NULL;
scripts/Makefile.build:335: recipe for target '/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.o' failed
对比4.9.0和4.15.0版本相关文件:
上图是struct net_device结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。另外4.15.0中修改destructor函数名为priv_destructor,按照内核版本区分即可。
static inline void ccmni_dev_init(int md_id,struct ccmni_ctl_block *ctlb, struct net_device *dev)
{
…
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
dev->max_mtu = CCMNI_MTU_MAX;
#endif
…
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
/* use kernel default free_netdev function */
dev->needs_free_netdev = true;
/* don't need to free again, because last step has called free_netdev */
dev->priv_destructor = NULL;
#else
dev->destructor = NULL;
#endif
…
}
2.9 error: ‘atomic_t {aka struct }’ has no member named ‘refs’
通过修改上述问题,再次进行编译出现如下错误:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/eccci/port/port_char.c: In function ‘port_char_uninit’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/eccci/port/port_char.c:111:50: error: ‘atomic_t {aka struct <anonymous>}’ has no member named ‘refs’
atomic_read(&port->cdev->kobj.kref.refcount.refs));
对比4.9.0和4.15.0版本相关文件:
上图是struct kref结构体截图,左侧为4.9.0,右侧为4.15.0,可以看到4.15.0中使用的refcount_t结构体作为自己的成员,再进一步使用atomic_t结构体,而在4.9.0中是直接将atomic_t结构体作为自己的成员,两者间存在差异,所以修改CCCI驱动源码,将该结构体相关按照内核版本区分即可。
static void port_char_uninit(struct port_t *port)
{
…
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
MTK_DEBUG_LOG(CHAR, "%s,kobject cnt %d\n",port->name,
atomic_read(&port->cdev->kobj.kref.refcount.refs));
#else
MTK_DEBUG_LOG(CHAR, "%s,kobject cnt %d\n",port->name,
atomic_read(&port->cdev->kobj.kref.refcount));
#endif
…
}
经过以上的修改,所有由于内核差异导致的编译问题全部解决,可编译成功,进一步加载驱动进行测试。
2.10 Unknown symbol in module
通过对内核版本的对比和适配,完成了驱动在客户环境上的编译,现进行加载验证:
root@:/home/sdk/Linux_PCIe_Driver_v1.0.26/out/script# ./setup.sh
insmod: ERROR: could not insert module mtk_pcie_wwan_m80.ko: Unknown symbol in module
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
cat: '/sys/kernel/mtk_wwan_*_pcie/build_time': No such file or directory
cat: '/sys/kernel/mtk_wwan_*_pcie/label': No such file or directory
加载时报如下错误,通过dmesg log我们可以看到是timer_setup函数存在问题:
[24380.191354] mtk_pcie_wwan_m80: Unknown symbol timer_setup (err 0)
进一步对比4.9.0和4.15.0内核之间timer的差异:
上图是timer.h部分截图,左侧为4.9.0,右侧为4.15.0,可以看到4.15.0中修改了timer中函数名称和实现,两者间存在差异,所以修改CCCI驱动源码,将该结构体相关按照内核版本区分即可。
#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
//timer_setup(&dpmaif_ctrl->traffic_monitor,dpmaif_traffic_monitor_func, 0);
#else
__setup_timer(&dpmaif_ctrl->traffic_monitor,
dpmaif_traffic_monitor_func,&dpmaif_ctrl->traffic_monitor, 0);
#endif
3. 验证
3.1 加载成功
通过上述修改,再次编译进行加载,成功。
3.2 端口、网卡枚举情况
查看设备是否正常枚举端口,并发送AT进行测试,均ok:
查看网卡枚举是否正常,结果ok:
会枚举出20个网卡,这个可由加载脚本nic_interface_num参数控制:
#!/bin/sh
# install pcie driver and enable all pcie logs
insmod mtk_pcie_wwan_*.ko dynamic_log_level=6 nic_interface_num=21
# set max log level
echo 7 > /proc/sys/kernel/printk
# set system max network buffer
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
cat /sys/kernel/mtk_wwan_*_pcie/build_time
cat /sys/kernel/mtk_wwan_*_pcie/label
3.3 端口、网卡枚举情况 拨号测试
拨号成功后,配置IP、路由,并进行ping包测试:
Ping ip和域名均ok,功能正常,调试完成。
4. 附件
在安装PCIE驱动时,可以用shell命令dmesg –w查看PCIE驱动安装过程日志,在日志的末尾看到有“md_state change from 3 to 4”和“Create ccmni netdev successfully”字样,则说明PCIE驱动的安装状态正常,并且ccmni网络设备成功生成。
[13098.660843] [MTK_PCIE_WWAN][CCCI_FSM][fsm_append_command][958]:command 3 is appended 0 from fsm_main_thread [mtk_pcie_wwan_m80]
[13098.660851] [MTK_PCIE_WWAN][CCCI_FSM][fsm_finish_command][981]:command 2 is completed 1 by fsm_main_thread [mtk_pcie_wwan_m80]
[13098.660852] [MTK_PCIE_WWAN][HIF_CLDMA][cldma_gpd_tx_collect][743]:txq:0,16,21c,80011001
[13098.660855] [MTK_PCIE_WWAN][CCCI_FSM][fsm_main_thread][881]:command 3 process
[13098.660856] [MTK_PCIE_WWAN][CCCI_FSM][fsm_broadcast_state][232]:md_state change from 3 to 4
[13098.661250] [MTK_PCIE_WWAN][DATA][ccmni_netdev_create][904]:ccmni_netdev_create--nic_num: 21
[13098.663100] [MTK_PCIE_WWAN][DATA][ccmni_md_state_callback][1001]:Create ccmni netdev successfully
[13098.663122] [MTK_PCIE_WWAN][CCCI_FSM][fsm_finish_command][981]:command 3 is completed 1 by fsm_main_thread [mtk_pcie_wwan_m80]
[13098.641689] [MTK_PCIE_WWAN][PCIE][mtk_pci_bridge_configure][1405]:Parent [Bus:0 Devfn:e8 Vendor:8086 Device:9d18]
5. 总结
驱动在新环境的适配,除了架构的差异,更多的是内核上的差异,所以我们需要去按照编译报错信息对比内核相关文件,按照差异去找是否有替换方案,如果没有,为新增内容,最好的方式便是用内核版本进行控制,便于管理和维护。