IPQ4xx Ethernet Analysis
Table of Contents
主要的驱动文件目录如下,同时请务必认真阅读DTS文件。
Device tree
- QCA Ethernet DMA
Documentation/devicetree/bindings/net/qca-ess.txt
- QCA Ethernet Switch ESS-SWITCH is used to forward the packet among LAN, WAN and Host processor.
Documentation/devicetree/bindings/arm/msm/ess-switch.txt
- QCA Ethernet PHY mode
ess-psgmii
- MDIO Interface
ipq40xx-mdio
The three *_bmp
seem to be which physical ports are associated with each of two VLANs:
0x01 0000 0001 -- CPU 0x1e 0001 1110 -- LAN 0x20 0010 0000 -- WAN 上面的是系统默认的端口以及VLAN ID的配置,
edma@c080000 { compatible = "qcom,ess-edma"; reg = <0xc080000 0x8000>; qcom,page-mode = <0>; qcom,rx_head_buf_size = <1540>; qcom,wan_port_id_mask = <0x10>; qcom,mdio_supported; qcom,phy_mdio_addr = <4>; qcom,poll_required = <1>; qcom,forced_speed = <1000>; qcom,forced_duplex = <1>; interrupts = <0 65 1>, <0 66 1>, <0 67 1>, <0 68 1>, <0 69 1>, <0 70 1>, <0 71 1>, <0 72 1>, <0 73 1>, <0 74 1>, <0 75 1>, <0 76 1>, <0 77 1>, <0 78 1>, <0 79 1>, <0 80 1>, <0 240 1>, <0 241 1>, <0 242 1>, <0 243 1>, <0 244 1>, <0 245 1>, <0 246 1>, <0 247 1>, <0 248 1>, <0 249 1>, <0 250 1>, <0 251 1>, <0 252 1>, <0 253 1>, <0 254 1>, <0 255 1>; gmac0 { local-mac-address = [000000000000]; }; gmac1 { local-mac-address = [000000000000]; }; }; ess-switch@c000000 { compatible = "qcom,ess-switch"; reg = <0xc000000 0x80000>; /* 512KB */ switch_access_mode = "local bus"; resets = <&gcc ESS_RESET>, <&gcc ESS_MAC1_CLK_DIS>, \ <&gcc ESS_MAC2_CLK_DIS>, <&gcc ESS_MAC3_CLK_DIS>, \ <&gcc ESS_MAC4_CLK_DIS>, <&gcc ESS_MAC5_CLK_DIS>; reset-names = "ess_rst","ess_mac1_clk_dis", \ "ess_mac2_clk_dis","ess_mac3_clk_dis", \ "ess_mac4_clk_dis", "ess_mac5_clk_dis"; clocks = <&gcc GCC_ESS_CLK>; clock-names = "ess_clk"; switch_cpu_bmp = <0x1>; /* cpu port bitmap */ switch_lan_bmp = <0x1e>; /* lan port bitmap */ switch_wan_bmp = <0x20>; /* wan port bitmap */ }; ess-psgmii@98000 { compatible = "qcom,ess-psgmii"; reg = <0x98000 0x800>; /* 2k */ psgmii_access_mode = "local bus"; resets = <&gcc ESS_PSGMII_ARES>; reset-names = "psgmii_rst"; }; mdio@90000 { #address-cells = <1>; #size-cells = <1>; compatible = "qcom,ipq40xx-mdio"; reg = <0x90000 0x64>; phy0: ethernet-phy@0 { reg = <0>; }; phy1: ethernet-phy@1 { reg = <1>; }; phy2: ethernet-phy@2 { reg = <2>; }; phy3: ethernet-phy@3 { reg = <3>; }; phy4: ethernet-phy@4 { reg = <4>; }; };
source files
/net/ethernet/eth.c
/drivers/net/ethernet/qcom/essedma/
/drivers/net/phy/mdio-ipq40xx.c
CONFIG_RFS_ACCEL
It macro controls the Accelerated RFS(Receive Flow Steering). And the kernel documents related it are:
And other complementary techniques in the Linux networking stack to increase parallelism and improve performance for multi-processor systems.
- RFS: Receive Flow Steering
- RSS: Receive Side Scaling
- RPS: Receive Packet Steering
Receive Flow Steering
While RPS steers packets solely based on hash, and thus generally provides good load distribution, it does not take into account application locality. This is accomplished by Receive Flow Steering (RFS). The goal of RFS is to increase datacache hitrate by steering kernel processing of packets to the CPU where the application thread consuming the packet is running. RFS relies on the same RPS mechanisms to enqueue packets onto the backlog of another CPU and to wake up that CPU.
Accelerated RFS
Accelerated RFS is to RFS what RSS is to RPS: a hardware-accelerated load balancing mechanism that uses soft state to steer flows based on where the application thread consuming the packets of each flow is running. Accelerated RFS should perform better than RFS since packets are sent directly to a CPU local to the thread consuming the data. The target CPU will either be the same CPU where the application runs, or at least a CPU which is local to the application thread’s CPU in the cache hierarchy.
init module
ret = platform_driver_register(&edma_axi_driver);
static struct platform_driver edma_axi_driver = { .driver = { .name = edma_axi_driver_name, .owner = THIS_MODULE, .of_match_table = edma_of_mtable, }, .probe = edma_axi_probe, .remove = edma_axi_remove, .shutdown = edma_axi_shutdown, };
Eth driver probe
#define EDMA_WAN 0 #define EDMA_LAN 1 由此就觉得了,net device ethx 哪一个是所谓的LAN(eth1),那一个是所谓的WAN(eth0) /* VLAN tag */ #define EDMA_LAN_DEFAULT_VLAN 1 #define EDMA_WAN_DEFAULT_VLAN 2
- create
net_device
for wan and lan port - 16 tx queues, 4 tx queues per core (4 cores)
- 4 rx queues, 1 rx queue per core
- alloc tx queues, tx rings
- alloc rx queues, rx rings
- fill adapter and netdev structure
- Register the netdevice
- register net sysctl table
- Disable all 16 Tx and 8 rx irqs
- Reset
- populate
per_core_info
, do anapi_Add
, request 16 TX irqs, 8 RX irqs, do a napi enable - Configure misc things
- Enable All 16 tx and 8 rx irq mask
- init statistics timer (
edma_statistics_timer
)
net device ops
static const struct net_device_ops edma_axi_netdev_ops = { .ndo_open = edma_open, .ndo_stop = edma_close, .ndo_start_xmit = edma_xmit, .ndo_set_mac_address = edma_set_mac_addr, .ndo_select_queue = edma_select_xps_queue, #ifdef CONFIG_RFS_ACCEL .ndo_rx_flow_steer = edma_rx_flow_steer, .ndo_register_rfs_filter = edma_register_rfs_filter, .ndo_get_default_vlan_tag = edma_get_default_vlan_tag, #endif .ndo_get_stats = edma_get_stats, .ndo_change_mtu = edma_change_mtu, };
control table
// struct ctl_table_header *edma_ctl_table_hdr; // edma_cinfo->edma_ctl_table_hdr = register_net_sysctl(&init_net, "net/edma", edma_table); static struct ctl_table edma_table[] = { { .procname = "default_lan_tag", .data = &edma_default_ltag, .maxlen = sizeof(int), .mode = 0644, .proc_handler = edma_change_default_lan_vlan }, { .procname = "default_wan_tag", .data = &edma_default_wtag, .maxlen = sizeof(int), .mode = 0644, .proc_handler = edma_change_default_wan_vlan }, { .procname = "weight_assigned_to_queues", .data = &edma_weight_assigned_to_q, .maxlen = sizeof(int), .mode = 0644, .proc_handler = edma_weight_assigned_to_queues }, { .procname = "queue_to_virtual_queue_map", .data = &edma_queue_to_virtual_q, .maxlen = sizeof(int), .mode = 0644, .proc_handler = edma_queue_to_virtual_queue_map }, { .procname = "enable_stp_rstp", .data = &edma_enable_rstp, .maxlen = sizeof(int), .mode = 0644, .proc_handler = edma_enable_stp_rstp }, { .procname = "athr_hdr_eth_type", .data = &edma_athr_hdr_eth_type, .maxlen = sizeof(int), .mode = 0644, .proc_handler = edma_ath_hdr_eth_type }, {} };
Eth IRQ
- 16 queues of eth tx
- 4 queues of eth rx
root@OpenWrt:/# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 20: 36300197 208695 2407496 78552 GIC 20 arch_timer 35: 0 0 0 0 GIC 35 watchdog bark 97: 0 0 0 0 GIC 97 edma_eth_tx0 98: 0 0 0 0 GIC 98 edma_eth_tx1 99: 0 0 738 0 GIC 99 edma_eth_tx2 100: 0 0 0 0 GIC 100 edma_eth_tx3 101: 0 0 0 0 GIC 101 edma_eth_tx4 102: 0 0 0 0 GIC 102 edma_eth_tx5 103: 0 0 0 797 GIC 103 edma_eth_tx6 104: 0 0 0 6 GIC 104 edma_eth_tx7 105: 0 0 0 0 GIC 105 edma_eth_tx8 106: 0 0 0 0 GIC 106 edma_eth_tx9 107: 73 0 0 0 GIC 107 edma_eth_tx10 108: 0 0 0 0 GIC 108 edma_eth_tx11 109: 0 0 0 0 GIC 109 edma_eth_tx12 110: 0 0 0 0 GIC 110 edma_eth_tx13 111: 0 9 0 0 GIC 111 edma_eth_tx14 112: 0 0 0 0 GIC 112 edma_eth_tx15 127: 52330 0 0 0 GIC 127 78b5000.spi 129: 24 0 0 0 GIC 129 i2c-msm-v2-irq 133: 0 0 0 0 GIC 133 sps 139: 1148 0 0 0 GIC 139 msm_serial_hsl0 155: 0 0 0 0 GIC 155 mmc0 164: 0 0 0 0 GIC 164 xhci-hcd:usb1 168: 29 0 0 0 GIC 168 xhci-hcd:usb3 173: 0 0 0 0 GIC 173 int_msi 174: 2103 48023 0 0 GIC 174 wifi2 183: 0 0 0 0 GIC 183 int_pls_link_down 200: 3367 0 2982303 0 GIC 200 wifi0 201: 3244 0 0 24028 GIC 201 wifi1 239: 0 0 0 0 GIC 239 sps 270: 15220 0 0 0 GIC 270 sps 272: 23697 0 0 0 GIC 272 edma_eth_rx0 274: 0 5607 0 0 GIC 274 edma_eth_rx2 276: 0 0 5424 0 GIC 276 edma_eth_rx4 278: 0 0 0 5911 GIC 278 edma_eth_rx6 306: 0 0 0 0 msmgpio 18 wps 310: 0 0 0 0 msmgpio 22 7824900.sdhci cd IPI0: 0 0 0 0 CPU wakeup interrupts IPI1: 0 0 0 0 Timer broadcast interrupts IPI2: 63422 52002 19887 46408 Rescheduling interrupts IPI3: 5 7 7 5 Function call interrupts IPI4: 4 2 4 2 Single function call interrupts IPI5: 0 0 0 0 CPU stop interrupts IPI6: 12161 14746 7912 6997 IRQ work interrupts IPI7: 0 0 0 0 completion interrupts Err: 0