pcitools in OpenSolaris, a good pci debug tool

pci_tools (i86pc/io/pci/pci_tools.c) in Solaris kernel define some function level infrastructure for the kernel code to achieve the following goal:

pci device register operation
=============================
pci_tools support configure space, memory, ROM and io space for both pci and pci express device. Function pcitool_dev_reg_ops() dispatch the operations to the following functions.

1. access the pci express extended configure space (above 256 bytes till 4k), the protype of this function is

static int pcitool_pciex_cfg_access(dev_info_t *dip, pcitool_reg_t *prg,
    boolean_t write_flag)

where pcitool_reg_t is a common struct describes the access attribute of the access. It is definded as

/*
 * Data stucture to read and write to pci device registers.
 * This is the argument to the following ioctls:
 *      PCITOOL_DEVICE_SET/GET_REG
 *      PCITOOL_NEXUS_SET/GET_REG
 */
typedef struct pcitool_reg {
        uint16_t        user_version;   /* Userland program version - to krnl */
        uint16_t        drvr_version;   /* Driver version - from kernel */
        uint8_t         bus_no;         /* pci bus - to kernel */
        uint8_t         dev_no;         /* pci dev - to kernel */
        uint8_t         func_no;        /* pci function - to kernel */
        uint8_t         barnum;         /* bank (DEVCTL_NEXUS_SET/GET_REG) or */
                                        /*   BAR from pcitools_bar_t */
                                        /*   (DEVCTL_DEVICE_SET/GET_REG) */
                                        /*   to kernel */
        uint64_t        offset;         /* to kernel */
        uint32_t        acc_attr;       /* access attributes - to kernel */
        uint32_t        padding1;       /* 8-byte align next uint64_t for X86 */
        uint64_t        data;           /* to/from kernel, 64-bit alignment */
        uint32_t        status;         /* from kernel */
        uint32_t        padding2;       /* 8-byte align next uint64_t for X86 */
        uint64_t        phys_addr;      /* from kernel, 64-bit alignment */
} pcitool_reg_t;

2. access the pci configure space header (below 256 bytes), the protype of this function is

static int pcitool_cfg_access(dev_info_t *dip, pcitool_reg_t *prg,
    boolean_t write_flag);

3. access io space, the protype of this function is

static int pcitool_io_access(dev_info_t *dip, pcitool_reg_t *prg,
    boolean_t write_flag);

4. access memory mapped space, the protype of this function is

static int pcitool_mem_access(dev_info_t *dip, pcitool_reg_t *prg,
     uint64_t virt_addr, boolean_t write_flag);

5. map and unmap memory mapped address space of a pci device, the functions are defined as

static uint64_t pcitool_map(uint64_t phys_addr, size_t size, size_t *num_pages);
static void pcitool_unmap(uint64_t virt_addr, size_t num_pages);

interrupt CPU binding requests and queries
==========================================

This functionality is handled by pcitool_intr_admn(), which also dispatch the request to the following functions.

Note: This functionality is under construction at least in build 90. So we leave it empty here.

initialize of pci_tools
=======================

Kernel component which wants to utilize pci_tools should call pcitool_init() first.

  96 pcitool_init(dev_info_t *dip, boolean_t is_pciex)
  97 {
  98         int instance = ddi_get_instance(dip);
  99
 100         /* Create pcitool nodes for register access and interrupt routing. */
 101
 102         if (ddi_create_minor_node(dip, PCI_MINOR_REG, S_IFCHR,
 103             PCIHP_AP_MINOR_NUM(instance, PCI_TOOL_REG_MINOR_NUM),
 104             DDI_NT_REGACC, 0) != DDI_SUCCESS) {
 105                 return (DDI_FAILURE);
 106         }
 107
 108         if (ddi_create_minor_node(dip, PCI_MINOR_INTR, S_IFCHR,
 109             PCIHP_AP_MINOR_NUM(instance, PCI_TOOL_INTR_MINOR_NUM),
 110             DDI_NT_INTRCTL, 0) != DDI_SUCCESS) {
 111                 ddi_remove_minor_node(dip, PCI_MINOR_REG);
 112                 return (DDI_FAILURE);
 113         }
 114
 115         if (is_pciex)
 116                 max_cfg_size = PCIE_CONF_HDR_SIZE;
 117
 118         return (DDI_SUCCESS);
 119 }

This function create two minor node for the pass-in dip, one for reg access and another for intr access. max_cfg_size if determined by whether it is a native pci express device (lagcy pci express device has no extened configure space).

pci_tools interface to user-land
================================

pcitool_init() is called during attachment of pci and pci express root nexus. In i86pc/io/pciex/npe.c,

        /* Second arg: initialize for pci_express root nexus */
        if (pcitool_init(devi, B_TRUE) != DDI_SUCCESS) {
                (void) pcihp_uninit(devi);
                ddi_soft_state_free(npe_statep, instance);
                return (DDI_FAILURE);
        }

user land application can issue ioctl() against the pci or pci express root nexus for pci_tool operations. In i86pc/io/pci/pci_common.c,

 857         switch (PCIHP_AP_MINOR_NUM_TO_PCI_DEVNUM(minor)) {
 858         case PCI_TOOL_REG_MINOR_NUM:
 859
 860                 switch (cmd) {
 861                 case PCITOOL_DEVICE_SET_REG:
 862                 case PCITOOL_DEVICE_GET_REG:
 863
 864                         /* Require full privileges. */
 865                         if (secpolicy_kmdb(credp))
 866                                 rv = EPERM;
 867                         else
 868                                 rv = pcitool_dev_reg_ops(dip, (void *)arg,
 869                                     cmd, mode);
 870                         break;
 871
 872                 case PCITOOL_NEXUS_SET_REG:
 873                 case PCITOOL_NEXUS_GET_REG:
 874
 875                         /* Require full privileges. */
 876                         if (secpolicy_kmdb(credp))
 877                                 rv = EPERM;
 878                         else
 879                                 rv = pcitool_bus_reg_ops(dip, (void *)arg,
 880                                     cmd, mode);
 881                         break;
 882                 }
 883                 break;
 884
 885         case PCI_TOOL_INTR_MINOR_NUM:
 886
 887                 switch (cmd) {
 888                 case PCITOOL_DEVICE_SET_INTR:
 889
 890                         /* Require PRIV_SYS_RES_CONFIG, same as psradm */
 891                         if (secpolicy_ponline(credp)) {
 892                                 rv = EPERM;
 893                                 break;
 894                         }
 895
 896                 /*FALLTHRU*/
 897                 /* These require no special privileges. */
 898                 case PCITOOL_DEVICE_GET_INTR:
 899                 case PCITOOL_SYSTEM_INTR_INFO:
 900                         rv = pcitool_intr_admn(dip, (void *)arg, cmd, mode);
 901                         break;
 902                 }
 903                 break;