linux内核下，rapidio网络系统建立的过程(枚举 和 发现)

1 简介

        假设现在有2台相同的机器，这2台机器都带有TSI721芯片，它们之间通过TSI721的rapidio接口相连。

        A机器在安装rapidio.ko时指定模块参数“hdid=-1”；

        B机器在安装rapidio.ko时指定模块参数“hdid=0”；        

2 基础知识

2.1 建立RapidIO system的方式

There are several basic ways of exploring and initializing a RapidIO system.

The simplest method is to somehow define the power-up state of the system components
such that all devices have adequate knowledge of the rest of the system to
communicate as needed. This is frequently accomplished by shifting initialization
information into all of the devices in the machine at boot time from serial ROMs or
similar devices. This method is most applicable for relatively static systems and
systems where boot-up time is important.

A second method, having processors explore and configure the system at boot time, requires more time but is much more flexible in order to support relatively fast changing plug-and-play or hot-swap systems.

This document describes a simple form of this second method. 

        《RapidIO TM Interconnect Specification, Part 7: System and Device Inter-operability Specification》(Rev. 2.1, 08/2009), PAGE13

2.2 Host Base Device ID Lock CSR(Configuration Space Offset 0x68)

The host base device ID lock CSR contains the base device ID value for the
processing element in the system that is responsible for initializing this processing
element. The Host_base_deviceID field is a write-once/reset-able field which
provides a lock function. Once the Host_base_deviceID field is written, all
subsequent writes to the field are ignored, except in the case that the value written
matches the value contained in the field. In this case, the register is re-initialized to
0xFFFF. After writing the Host_base_deviceID field a processing element must then
read the Host Base Device ID Lock CSR to verify that it owns the lock before
attempting to initialize this processing element.

《RapidIO ™ Interconnect Specification Part 3: Common Transport Specification》(Rev. 2.1, 08/2009), PAGE25

3 hdid(rapidio.ko的模块参数)、Master Enable 位、“发现” 和 “枚举”

3.1 Master Enable位

        在 rapidio 通用寄存器中，有一个 RapidIO Port General Control CSR 寄存器，这个寄存器中的 Master Enable 位用来限制 rapidio 设备是否有权利主动发起 rapidio 通讯。

        在 rapidio 网络初始化时，其中部分 rapidio 设备的 Master Enable 位会被写为 1，然后只有这些设备才可以主动发起通讯，也就是只有这个设备才能发起枚举的操作。

手册上相关信息如下：

If the Master Enable bit is clear the agent device is not allowed to issue requests and is only able to respond to received requests.

the base device ID value for all devices except the Host and boot ROM device after reset is applied is 0xFF, the Host has it’s Master
Enable and Discovered bits set, and the agent devices have their Master Enable and Discovered bits cleared.

《RapidIO TM Interconnect Specification, Part 7: System and Device Inter-operability Specification》(Rev. 2.1, 08/2009), PAGE15,PAGE16

3.2 hdid和Master Enable位的关系

在tsi721驱动初始化时，会根据hdid来决定那些TSI721的Master Enable位被写1。如下：

3.3 总结

在安装rapidio.ko时指定模块参数“hdid>=0”时，对应的TSI721的Master Enable 位被写为1，这个TSI721才能主动发起通讯，才能进行“枚举”操作。

在安装rapidio.ko时指定模块参数“hdid<0”时，只能执行“发现”操作。

4 触发“枚举” 和 “发现”后的驱动流程

触发操作：echo -1 > /sys/bus/rapidio/scan

触发后的驱动流程

5 “发现”的流程： rio_disc_mport()

        A 机器会等待 B 机器“枚举”完成，等待时间是 CONFIG_RAPIDIO_DISC_TIMEOUT 秒。
        A 机器等待期间会不断读取自己的 RapidIO Port General Control CSR 寄存器中“Discovered”位，如果“Discovered” 位是 1，表示枚举完成。
        B 机器枚举完成后会调用 rio_clear_locks()函数通过 rapidio 的 maintenance write 将 A 机器的 RapidIO Port General Control CSR 寄存器中的“Discovered” 和 “Master Enable”位写 1。

注意：“发现”流程要先于“枚举”流程运行

When automatic start of enumeration/discovery is used a user has to ensure that all discovering endpoints are started before the
enumerating endpoint and are waiting for enumeration to be completed.

《Documentation/driver-api/rapidio/rapidio.rst(kernel-5.4.18)》

6 “ 枚举”的流程： rio_enum_mport() 和 rio_enum_peer()

6.1 rio_enum_mport()

6.2 执行具体枚举工作的函数：rio_enum_peer()