GEM5 Ruby Garnet kernel 源代码解读： 2/38

#简介
这一篇是gem5 Ruby Garnet kernel的源代码解读。之前写了不少怎么用GEM5以及关于configuration的代码，都是gem5 user层面的代码解读，这一系列会有38个文件，这一篇读一共6个文件 。 两个文件 是python文件，2个是network.cc/hh两个是router.cc/hh。

目录

以后再补充这个系列其他文章。

GarnetNetwork.py

默认的GarnetNetwork

class GarnetNetwork(RubyNetwork):
    type = "GarnetNetwork"
    cxx_header = "mem/ruby/network/garnet/GarnetNetwork.hh"
    cxx_class = "gem5::ruby::garnet::GarnetNetwork"

    num_rows = Param.Int(0, "number of rows if 2D (mesh/torus/..) topology")
    ni_flit_size = Param.UInt32(16, "network interface flit size in bytes")
    vcs_per_vnet = Param.UInt32(4, "virtual channels per virtual network")
    buffers_per_data_vc = Param.UInt32(4, "buffers per data virtual channel")
    buffers_per_ctrl_vc = Param.UInt32(1, "buffers per ctrl virtual channel")
    routing_algorithm = Param.Int(0, "0: Weight-based Table, 1: XY, 2: Custom")
    enable_fault_model = Param.Bool(False, "enable network fault model")
    fault_model = Param.FaultModel(NULL, "network fault model")
    garnet_deadlock_threshold = Param.UInt32(
        50000, "network-level deadlock threshold"
    )

由RubyNetwork继承而来，其中NI flit size是16 byte，也就是16*8=128bits， GEM5 GARNET DVFS论文阅读 Part 1：Thread Voting DVFS for Manycore NoCs 这篇里用的NoC设置也是128bits，但是TVD文章用的 6个 VC per VNet，默认的Garnet是4个。depth则用的都是4个 flit per VC。

文件里还有两个python class，都只是传递class GarnetNetwork(RubyNetwork):的参数，略过。

class GarnetNetworkInterface(ClockedObject):
class GarnetRouter(BasicRouter):

Garnet link.py

有五个class，

class NetworkLink(ClockedObject):
class CreditLink(NetworkLink):
class NetworkBridge(CreditLink):
class GarnetIntLink(BasicIntLink):
class GarnetExtLink(BasicExtLink):

相当于近乎从0手写了一个 NetworkBridge，然后沿用改写了GarnetIntLink和 GarnetExtLink。他们都有对应的5个cc和5个hh文件。

Garnetwork.hh Garnetwork.cc

Garnetwork.hh 和 Garnetwork.cc 一起读，namespace里，garnet一共有6个类 NetworkInterface;Router; NetworkLink;
NetworkBridge; CreditLink; GarnetNetwork。

namespace gem5
{
namespace ruby
{
class FaultModel;
class NetDest;
namespace garnet
{
class NetworkInterface;
class Router;
class NetworkLink;
class NetworkBridge;
class CreditLink;
class GarnetNetwork : public Network
{

Garnetwork.hh

Garnetwork.hh 代码解读：先是一个和kernel运行无关的值，但是表明了当前2023年11月的时候，GEM5里的 garnet版本是3.0。
还有一个注释，就是说configuration是在Garnetwork.hh 源代码之外做的，

提供配置信息

const char *garnetVersion = "3.0";
  // Configuration (set externally)

还有一些给外界提供网络信息的查询函数，分别返回行数和列数。
返回网络的配置，比如在一个node内的：NiFlitSize(默认是16Bytes=128bits)，BuffersPerDataVC（默认4个flits，这是data的），BuffersPerCtrlVC（默认是一个，这是控制命令的vc），还有返回routing_algorithm。
还有返回网路配置信息，在node之间的：比如返回VNET 类型，返回这个router的id和router的总数。

 // for 2D topology
    int getNumRows() const { return m_num_rows; }
    int getNumCols() { return m_num_cols; }
    // for network
    uint32_t getNiFlitSize() const { return m_ni_flit_size; }
    uint32_t getBuffersPerDataVC() { return m_buffers_per_data_vc; }
    uint32_t getBuffersPerCtrlVC() { return m_buffers_per_ctrl_vc; }
    int getRoutingAlgorithm() const { return m_routing_algorithm; }
    // Internal configuration
    bool isVNetOrdered(int vnet) const { return m_ordered[vnet]; }
    VNET_type
    get_vnet_type(int vnet)
    {
        return m_vnet_type[vnet];
    }
    int getNumRouters();
    int get_router_id(int ni, int vnet);

对noc网络进行配置

makeExtOutLink是从switch进Node，相当于从router出去，通过NI 给到L1 cache或者cores(取决于caches怎么设置，一般是连到私有的L1上)
void makeExtInLink是Node 进Switch，相当与注入了NoC。
makeInternalLink则是router之间的互联。

  // Methods used by Topology to setup the network
    void makeExtOutLink(SwitchID src, NodeID dest, BasicLink* link,
                     std::vector<NetDest>& routing_table_entry);
    void makeExtInLink(NodeID src, SwitchID dest, BasicLink* link,
                    std::vector<NetDest>& routing_table_entry);
    void makeInternalLink(SwitchID src, SwitchID dest, BasicLink* link,
                          std::vector<NetDest>& routing_table_entry,
                          PortDirection src_outport_dirn,
                          PortDirection dest_inport_dirn);

供统计的变量

挺多的。大多只是个变量声明罢了。

Garnetwork.cc

核心代码分为两类:配置好router之间和router与node的互联，以及统计信息。

统计信息先跳过，准备放在分析结果的部分再来看。我们先看互联的代码。

初始化可连接的组件

如下文代码，在 GarnetNetwork::GarnetNetwork(const Params &p)

Network§ 中 新建了很多NI 和router。 但是没有连起来。

GarnetNetwork::GarnetNetwork(const Params &p)
    : Network(p)
 // record the routers
    for (std::vector<BasicRouter*>::const_iterator i =  p.routers.begin();
         i != p.routers.end(); ++i) {
        Router* router = safe_cast<Router*>(*i);
        m_routers.push_back(router);

        // initialize the router's network pointers
        router->init_net_ptr(this);
    }

    // record the network interfaces
    for (std::vector<ClockedObject*>::const_iterator i = p.netifs.begin();
         i != p.netifs.end(); ++i) {
        NetworkInterface *ni = safe_cast<NetworkInterface *>(*i);
        m_nis.push_back(ni);
        ni->init_net_ptr(this);
    }

在用户的config中完成了garnet的链接，以gem5/configs/network/Network.py为例子,在python文件中，它遍历了每一个link并且连起来。

def init_network(options, network, InterfaceClass):

    if options.network == "garnet":
        network.num_rows = options.mesh_rows
        network.vcs_per_vnet = options.vcs_per_vnet
        network.ni_flit_size = options.link_width_bits / 8
        network.routing_algorithm = options.routing_algorithm
        network.garnet_deadlock_threshold = options.garnet_deadlock_threshold

        # Create Bridges and connect them to the corresponding links
        for intLink in network.int_links:
            intLink.src_net_bridge = NetworkBridge(
                link=intLink.network_link,
                vtype="OBJECT_LINK",
                width=intLink.src_node.width,
            )
            intLink.src_cred_bridge = NetworkBridge(
                link=intLink.credit_link,
                vtype="LINK_OBJECT",
                width=intLink.src_node.width,
            )
            intLink.dst_net_bridge = NetworkBridge(
                link=intLink.network_link,
                vtype="LINK_OBJECT",
                width=intLink.dst_node.width,
            )
            intLink.dst_cred_bridge = NetworkBridge(
                link=intLink.credit_link,
                vtype="OBJECT_LINK",
                width=intLink.dst_node.width,
            )

或者在gem5/configs/topologies/Mesh_XY.py中，下文是一部分，对每一行的左边router的east port，连上右边相连的router的west inport。

可以看出来，每个router其实只主动连接四次（下文的weight是因为：# XY routing is enforced (using link weights)）：
#East output to West input links (weight = 1)
#West output to East input links (weight = 1)
#North output to South input links (weight = 2)
#South output to North input links (weight = 2)

 # East output to West input links (weight = 1)
        for row in range(num_rows):
            for col in range(num_columns):
                if col + 1 < num_columns:
                    east_out = col + (row * num_columns)
                    west_in = (col + 1) + (row * num_columns)
                    int_links.append(
                        IntLink(
                            link_id=link_count,
                            src_node=routers[east_out],
                            dst_node=routers[west_in],
                            src_outport="East",
                            dst_inport="West",
                            latency=link_latency,
                            weight=1,
                        )
                    )
                    link_count += 1 

GarnetNetwork::makeExtOutLink 和 makeExtInLink

这俩几乎一样，只是方向不一样，取makeExtOutLink来看。
代码如下：

void
GarnetNetwork::makeExtOutLink(SwitchID src, NodeID global_dest,
                              BasicLink* link,
                              std::vector<NetDest>& routing_table_entry)
{
    NodeID local_dest = getLocalNodeID(global_dest);
    assert(local_dest < m_nodes);
    assert(src < m_routers.size());
    assert(m_routers[src] != NULL);

    GarnetExtLink* garnet_link = safe_cast<GarnetExtLink*>(link);

    // GarnetExtLink is bi-directional
    NetworkLink* net_link = garnet_link->m_network_links[LinkDirection_Out];
    net_link->setType(EXT_OUT_);
    CreditLink* credit_link = garnet_link->m_credit_links[LinkDirection_Out];

    m_networklinks.push_back(net_link);
    m_creditlinks.push_back(credit_link);

    PortDirection src_outport_dirn = "Local";

    m_max_vcs_per_vnet = std::max(m_max_vcs_per_vnet,
                             m_routers[src]->get_vc_per_vnet());

    /*
     * We check if a bridge was enabled at any end of the link.
     * The bridge is enabled if either of clock domain
     * crossing (CDC) or Serializer-Deserializer(SerDes) unit is
     * enabled for the link at each end. The bridge encapsulates
     * the functionality for both CDC and SerDes and is a Consumer
     * object similiar to a NetworkLink.
     *
     * If a bridge was enabled we connect the NI and Routers to
     * bridge before connecting the link. Example, if an external
     * bridge is enabled, we would connect:
     * NI<---NetworkBridge<---GarnetExtLink<----Router
     */
    if (garnet_link->extBridgeEn) {
        DPRINTF(RubyNetwork, "Enable external bridge for %s\n",
            garnet_link->name());
        NetworkBridge *n_bridge = garnet_link->extNetBridge[LinkDirection_Out];
        m_nis[local_dest]->
            addInPort(n_bridge, garnet_link->extCredBridge[LinkDirection_Out]);
        m_networkbridges.push_back(n_bridge);
    } else {
        m_nis[local_dest]->addInPort(net_link, credit_link);
    }

    if (garnet_link->intBridgeEn) {
        DPRINTF(RubyNetwork, "Enable internal bridge for %s\n",
            garnet_link->name());
        NetworkBridge *n_bridge = garnet_link->intNetBridge[LinkDirection_Out];
        m_routers[src]->
            addOutPort(src_outport_dirn,
                       n_bridge,
                       routing_table_entry, link->m_weight,
                       garnet_link->intCredBridge[LinkDirection_Out],
                       m_routers[src]->get_vc_per_vnet());
        m_networkbridges.push_back(n_bridge);
    } else {
        m_routers[src]->
            addOutPort(src_outport_dirn, net_link,
                       routing_table_entry,
                       link->m_weight, credit_link,
                       m_routers[src]->get_vc_per_vnet());
    }
}

先是遇到了type的问题 NodeID local_dest = getLocalNodeID(global_dest);什么是NodeID？
这是通过#include “mem/ruby/network/Network.hh”，在ruby的#include "mem/ruby/common/TypeDefines.hh"有定义：

其实就是int罢了，但是相互之间有了名字上的区分。

NodeID local_dest = getLocalNodeID(global_dest);

getLocalNodeID这个函数一样在mem/ruby/network/Network.hh里，返回的globalToLocalMap的一个值：

globalToLocalMap则是两个int 。之前我以为做的是一个转换，比如8x8的mesh，最左上角的id是0,坐标就是（0，0）,最右下角的id是63，那么坐标是（7,7）
其实实际上只是将id转化一下而已：可能nodeid是0,local的router是63，都是一个单独的int而不是一个值转化为一对。

然后代码调用了 garnet_link，并且存了一些link进m_networklinks和 m_creditlinks。

GarnetExtLink* garnet_link = safe_cast<GarnetExtLink*>(link);

    // GarnetExtLink is bi-directional
    NetworkLink* net_link = garnet_link->m_network_links[LinkDirection_In];
    net_link->setType(EXT_IN_);
    CreditLink* credit_link = garnet_link->m_credit_links[LinkDirection_In];

    m_networklinks.push_back(net_link);
    m_creditlinks.push_back(credit_link);

再之后是判断是否link的一端有bridge，有bridge就有一些额外的操作，具体的我们放在NetworkBridge.cc的部分说。

GarnetNetwork::makeInternalLink

代码如下：

void
GarnetNetwork::makeInternalLink(SwitchID src, SwitchID dest, BasicLink* link,
                                std::vector<NetDest>& routing_table_entry,
                                PortDirection src_outport_dirn,
                                PortDirection dst_inport_dirn)
{
    GarnetIntLink* garnet_link = safe_cast<GarnetIntLink*>(link);

    // GarnetIntLink is unidirectional
    NetworkLink* net_link = garnet_link->m_network_link;
    net_link->setType(INT_);
    CreditLink* credit_link = garnet_link->m_credit_link;

    m_networklinks.push_back(net_link);
    m_creditlinks.push_back(credit_link);

    m_max_vcs_per_vnet = std::max(m_max_vcs_per_vnet,
                             std::max(m_routers[dest]->get_vc_per_vnet(),
                             m_routers[src]->get_vc_per_vnet()));

    /*
     * We check if a bridge was enabled at any end of the link.
     * The bridge is enabled if either of clock domain
     * crossing (CDC) or Serializer-Deserializer(SerDes) unit is
     * enabled for the link at each end. The bridge encapsulates
     * the functionality for both CDC and SerDes and is a Consumer
     * object similiar to a NetworkLink.
     *
     * If a bridge was enabled we connect the NI and Routers to
     * bridge before connecting the link. Example, if a source
     * bridge is enabled, we would connect:
     * Router--->NetworkBridge--->GarnetIntLink---->Router
     */
    if (garnet_link->dstBridgeEn) {
        DPRINTF(RubyNetwork, "Enable destination bridge for %s\n",
            garnet_link->name());
        NetworkBridge *n_bridge = garnet_link->dstNetBridge;
        m_routers[dest]->addInPort(dst_inport_dirn, n_bridge,
                                   garnet_link->dstCredBridge);
        m_networkbridges.push_back(n_bridge);
    } else {
        m_routers[dest]->addInPort(dst_inport_dirn, net_link, credit_link);
    }

    if (garnet_link->srcBridgeEn) {
        DPRINTF(RubyNetwork, "Enable source bridge for %s\n",
            garnet_link->name());
        NetworkBridge *n_bridge = garnet_link->srcNetBridge;
        m_routers[src]->
            addOutPort(src_outport_dirn, n_bridge,
                       routing_table_entry,
                       link->m_weight, garnet_link->srcCredBridge,
                       m_routers[dest]->get_vc_per_vnet());
        m_networkbridges.push_back(n_bridge);
    } else {
        m_routers[src]->addOutPort(src_outport_dirn, net_link,
                        routing_table_entry,
                        link->m_weight, credit_link,
                        m_routers[dest]->get_vc_per_vnet());
    }
}

一样是建一些link，存起来，有bridge的话做一些额外操作。

router.cc router.hh

在之前我们大概想明白了，garnetnetwork.cc就是把一堆node连起来，那么其中的一个node长什么样呢？我们看router.cc和

router.hh

先是声明inport和outport，其中有direction和link。

 void addInPort(PortDirection inport_dirn, NetworkLink *link,
                   CreditLink *credit_link);
    void addOutPort(PortDirection outport_dirn, NetworkLink *link,
                    std::vector<NetDest>& routing_table_entry,
                    int link_weight, CreditLink *credit_link,
                    uint32_t consumerVcs);

其中会涉及InputUnit.cc OutputUnit.cc ，这里面InputUnit会用到flitBuffer creditQueue; OutputUnit会用到flitBuffer outBuffer;

还有一个计算路由信息的部分

int route_compute(RouteInfo route, int inport, PortDirection direction);
//cpp中：
return routingUnit.outportCompute(route, inport, inport_dirn);

可以看到细节是routingunit做的，router.hh和router.cc只是调用。
同时，还有很多经典的router的部分实现例如SA，crossbar 也在RoutingUnit.cc SwitchAllocator.cc CrossbarSwitch.cc 中。

RoutingUnit routingUnit;
SwitchAllocator switchAllocator;
CrossbarSwitch crossbarSwitch;

画个图就是这样：