Inside OpenSolaris: Introduction to Solaris Drivers

Inside OpenSolaris: Introduction to Solaris Drivers

April 1, 2005

This article explains basic driver programming. We assume some familiarity with device drivers and devices and in general discuss Solaris 10 and Linux 2.6. Most of the article should hold for earlier releases of either system.

Introduction

This section describes how devices are visible to users on Solaris, that is, where the device files reside on Solaris and what their organization. It also gives an overview of Solaris and Linux device drivers, listing routines and data structures needed to implement device drivers in both systems along with a brief description. A table lists similarities and differences between routines and data structures.

Both Solaris and Linux device drivers serve the same purpose, to provide a means for the system, as well as users on the system, to control and access devices. For the most part, user access to devices is via standard I/O system calls: open(2), close(2), read(2), write(2), ioctl(2), poll(2), mmap(2), readv(2), writev(2). These system calls result in calls to device driver routines. In addition, the operating system itself calls driver routines for handling page faults and the various bus architectures that may exist on a system. Various system administrative related system calls may also make calls to device driver routines.

System Architecture and the Device Tree

Solaris organizes its devices in a "device tree". Each node on the tree represents a device. The device tree shows the geographical location of devices on the system. In other words, it shows the directory tree in Solaris which corresponds to the location of the the system. Users and programs can examine it via the /devices directory tree and via the prtconf(1) command. Following is an example, with some output truncated and various devices omitted.

bash-2.05b# ls -R /devices
/devices:                       <---  the "root" of the device tree
isa               pci@0,0           pseudo:devctl     xsvc
objmgr            pci@0,0:devctl    scsi_vhci         xsvc:xsvc
options           pseudo            scsi_vhci:devctl

/devices/isa:                   <--- the system has an isa bus
asy@1,2f8       asy@1,2f8:b,cu  i8042@1,60      lp@1,3bc:ecpp0  <--- devices on the isa bus
asy@1,2f8:b     fdc@1,3f0       lp@1,3bc

/devices/isa/fdc@1,3f0:         <--- a floppy disk controller on the isa bus
fd@0,0        fd@0,0:a,raw  fd@0,0:b,raw  fd@0,0:c,raw  <--- floppy disk drives/slices
fd@0,0:a      fd@0,0:b      fd@0,0:c

/devices/isa/i8042@1,60:        <--- an i8042 controller
keyboard@0         keyboard@0:kb8042  mouse@1            mouse@1:l

/devices/pci@0,0:       <--- pci-host bus bridge
pci-ide@7,1            pci8086,7112@7,2       pci8086,7112@7,2:hubd
pci1014,130@2          pci8086,7112@7,2:1     pci8086,7191@1
pci1014,130@2,1        pci8086,7112@7,2:2     pci8086,7191@1:devctl

/devices/pci@0,0/pci-ide@7,1:   <--- pci-ide bridge
ide@0          ide@0:control  ide@1          ide@1:control

/devices/pci@0,0/pci-ide@7,1/ide@0:     <--- 1 of 2 ide controllers
cmdk@0,0        cmdk@0,0:e      cmdk@0,0:i,raw  cmdk@0,0:n      cmdk@0,0:r,raw  <--- disks/slices
cmdk@0,0:a      cmdk@0,0:e,raw  cmdk@0,0:j      cmdk@0,0:n,raw  cmdk@0,0:s
cmdk@0,0:a,raw  cmdk@0,0:f      cmdk@0,0:j,raw  cmdk@0,0:o      cmdk@0,0:s,raw
cmdk@0,0:b      cmdk@0,0:f,raw  cmdk@0,0:k      cmdk@0,0:o,raw  cmdk@0,0:t
cmdk@0,0:b,raw  cmdk@0,0:g      cmdk@0,0:k,raw  cmdk@0,0:p      cmdk@0,0:t,raw
cmdk@0,0:c      cmdk@0,0:g,raw  cmdk@0,0:l      cmdk@0,0:p,raw  cmdk@0,0:u
cmdk@0,0:c,raw  cmdk@0,0:h      cmdk@0,0:l,raw  cmdk@0,0:q      cmdk@0,0:u,raw
cmdk@0,0:d      cmdk@0,0:h,raw  cmdk@0,0:m      cmdk@0,0:q,raw
cmdk@0,0:d,raw  cmdk@0,0:i      cmdk@0,0:m,raw  cmdk@0,0:r

/devices/pci@0,0/pci-ide@7,1/ide@1:  <--- second ide controller
sd@0,0        sd@0,0:e      sd@0,0:i,raw  sd@0,0:n      sd@0,0:r,raw
sd@0,0:a      sd@0,0:e,raw  sd@0,0:j      sd@0,0:n,raw  sd@0,0:s
sd@0,0:a,raw  sd@0,0:f      sd@0,0:j,raw  sd@0,0:o      sd@0,0:s,raw
sd@0,0:b      sd@0,0:f,raw  sd@0,0:k      sd@0,0:o,raw  sd@0,0:t
sd@0,0:b,raw  sd@0,0:g      sd@0,0:k,raw  sd@0,0:p      sd@0,0:t,raw
sd@0,0:c      sd@0,0:g,raw  sd@0,0:l      sd@0,0:p,raw  sd@0,0:u
sd@0,0:c,raw  sd@0,0:h      sd@0,0:l,raw  sd@0,0:q      sd@0,0:u,raw
sd@0,0:d      sd@0,0:h,raw  sd@0,0:m      sd@0,0:q,raw
sd@0,0:d,raw  sd@0,0:i      sd@0,0:m,raw  sd@0,0:r

/devices/pseudo:  <--- pseudo device nexus node
arp@0                  pm@0                   ptsl@0:ttyp4  <--- pseudo devices
arp@0:arp              pm@0:pm                ptsl@0:ttyp5
bl@0                   poll@0                 ptsl@0:ttyp6
        <output truncated>
bash-2.05b# 

In the example ls listing above, the name of a device, for instance, /devices/pci@0,0/pci-ide@7,1 shows that there is a PCI controller on the system bus (or motherboard), and on the PCI bus is a PCI-IDE bridge. The device above the given device on the tree chooses the name of the device (preceding the '@' sign). The device above also chooses the portion of the name following the @ until the ':' and typically shows the slot number and an offset on the bus on which the device resides. The device driver itself chooses the portion following the ':'. Linux uses the same names for devices regardless of their location (and regardless of the hardware). For instance, Linux disks are /dev/hda, /dev/hdb, etc., regardless of whether the disk is IDE or SCSI. On Solaris, IDE disks will be cmdk and SCSI (or ATAPI-based) CD-ROM drives show up as sd. There are files under /dev as in other Unixes, but these files are symbolic links to the "real" devices under /devices.

NOTE: Both linux and Solaris 10 use a devfs device file system to manage devices. This means that the device drivers themselves create and remove nodes. On Solaris, drivers do this by calling ddi_create_minor_node(9f) and ddi_remove_minor_node(9f) in their attach(9f) and detach(9f) routines respectively. The corresponding routines in Linux are devfs_mk_dir(), devfs_mk_cdev(), and devfs_mk_bdev(), called from the driver's init function and devfs_remove() called from the driver's cleanup function. Note also that a Linux driver does not have to use devfs, in which case it creates device files by calling mknod() within the driver or register_chrdev(), (or register_netdev() for network devices, or other related routine), or by hand. Solaris drivers always use ddi_create_minor_node(9f) even in Solaris versions before Solaris 10 which did not have devfs, and regardless of the type of device (disk, network, pseudo, etc.).

Here is prtconf(1) output, again truncated.

bash-2.05b# prtconf
System Configuration:  Sun Microsystems  i86pc
Memory size: 512 Megabytes
System Peripherals (Software Nodes):

i86pc
    scsi_vhci, instance #0
    +boot (driver not attached)  <--- "driver not attached" explained below
        memory (driver not attached)
    aliases (driver not attached)
    chosen (driver not attached)
    i86pc-memory (driver not attached)
    i86pc-mmu (driver not attached)
    openprom (driver not attached)
    options, instance #0
    packages (driver not attached)
    delayed-writes (driver not attached)
    itu-props (driver not attached)
    isa, instance #0            <--- an isa bus on the system board
        motherboard (driver not attached)
        i8042, instance #0
            mouse, instance #0
            keyboard, instance #0
        fdc, instance #0
            fd, instance #0
        lp, instance #0
        asy, instance #0
    pci, instance #0            <--- a pci bus on the system board
        pci8086,7190 (driver not attached)
        pci8086,7191, instance #0
            display, instance #0
        pci1014,130, instance #0
        pci1014,130, instance #1
        pci10b7,6356 (driver not attached)
        pci10b7,6159 (driver not attached)
        pci1014,153 (driver not attached)
        pci8086,7110 (driver not attached)
        pci-ide, instance #0    <--- pci-ide bridge
            ide, instance #0
                cmdk, instance #0
            ide, instance #1
                sd, instance #0
        pci8086,7112, instance #0
        pci8086,7113 (driver not attached)
    used-resources (driver not attached)
    pseudo, instance #0
    xsvc, instance #0
    objmgr, instance #0
    cpus (driver not attached)
        cpu, instance #0 (driver not attached)
bash-2.05b# 

"driver not attached" means either there is no driver, the device is not being used and the driver is not loaded, or the hardware is configured but physically does not exist. Use prtconf -D to find the driver (if one exists) for each device.

It's possible to classify drivers on Solaris as "nexus" drivers and "leaf node" drivers. Those devices which have other devices attached to them (typically via a bus) use nexus drivers. Examples include bus controllers, such as PCI-host bus bridges, and PCI-PCI bridges. Leaf node drivers are for devices that do not have additional attached devices. These include pseudo devices, network drivers, graphics cards, and storage devices. Most devices on a system are leaf node devices.

Leaf node device drivers are either character devices and/or block devices. A block device is any device that has addressable, re-usable data -- disks, for example. Any device can be implemented as a character device. A disk driver is both a character and a block device. Some nexus devices have a character device implementation as well as a nexus driver implementation.

Types of Devices/Drivers

Some bus drivers allow ioctl(2) calls, which necessitates leaf driver structures and routines. SCSI host bus adapters (devices that convert between some bus on the system; PCI, for instance), and a SCSI bus are a form of nexus node that use a framework developed for handling SCSI hba's called SCSA (Sun Common SCSI Architecture). The following sections briefly describe some of the different types of devices and device driver frameworks used to support the different types of devices.

Host Bus Adapters (SCSI, IDE, etc.)

The Sun Common SCSI Architecture, or SCSA, allows the development of target drivers (such as disk, tape, scanner, etc.) without knowledge of the HBA controller that manages the SCSI bus. Likewise, HBA drivers do not need to know anything about the targets that are on the SCSI bus.

The following is a high level view of how SCSA-compliant target and HBA drivers work together. All routines starting with scsi_ are part of the SCSA. NOTE: This is not a complete description. Refer to Writing Device Drivers for more detail.

User reads/writes a file;

  foo_strategy(struct buf *bp)  /* target driver strategy routine */
     scsi_init_pkt(...);  /* calls hba to alloc and init a SCSI packet */
                          /* see scsi_init_pkt(9F) and scsi_pkt(9s) */
     makecom_g5(...);  /* build the scsi command descriptor block, see makecom(9f) */
     place completion callback routine pointer into scsi_pkt;

     scsi_transport(...);  /* hand over built scsi packet to HBA for transport on the bus */

            bar_tran_start(...)  /* hba routine to handle scsi pkts */

                send command to target on SCSI bus via HBA;

            bar_intr(...)  /* interrupt handler following completion/timeout */
                clean up;
                call target callback function via scsi_pkt;
       foo_completion(...)  /* target callback function */
         biodone(bp);  /* wakeup waiting readers/writers */
         scsi_free_pkt(...);

Network Drivers

Solaris uses STREAMS messages to pass packets between different components of the network protocol stack. The STREAMS Programming Guide has a detailed discussion of STREAMS. Note that Solaris 10 has reworked TCP/IP so that TCP and IP are not separate STREAMS modules. This was done mostly to address performance issues. See the description of "fireengine" on Solaris Express for more details.

On Solaris, a network driver must be STREAMS cognizant and must adhere to the DLPI specification. DLPI (Data Link Provider Interface), is a specification for STREAMS data link (network controller) drivers. Because STREAMS and DLPI add a lot of complexity to writing a network driver, Sun developed the Generic Lan Driver (gld) to hide DLPI and most of STREAMS from network drivers.

With gld, you register a set of callbacks with GLD in your driver's attach function. When gld needs your driver to do something, it calls one of your registered callbacks. A full Chapter 18 of Writing Device Drivers gives a full example of a gld-based driver.

USB Drivers

Solaris has a framework analogous to Linux's framework (api) for developing USB drivers. See the USB Skeleton Device Driver for an example. Also, the Writing Device Drivers guide has a description of the USB framework in Chapter 19.

Tools

This section describes how to build and install a Solaris driver and debugging tools.

Compilation and Installation

To compile your driver, use Sun's C compiler (though I have had some success with gcc). THe steps are:

# cc -xarch=v9 -D_KERNEL -O -c foo.c  <-- -xarch=v9 needed for SPARC
# cc -xarch=v9 -D_KERNEL -O -c bar.c  <-- compile each .c file for your driver
# ld -r -o foobar foo.o bar.o  <-- create a relocatable object, foobar
                               <-- dependencies on other kernel modules
                               <-- can be specified with -Ndrv/module
                               <-- for example, ld -Nmisc/scsi ...
                               <-- for a scsa-compliant driver

To install your driver, copy the relocatable module to the appropriate directory (typically /usr/kernel/drv, /kernel/drv, or /kernel/`uname -m`/kernel/drv. On SPARC, the driver goes in the sparcv9 subdirectory. If your driver has a driver.conf file, it goes in the drv directory (not the sparcv9 directory). After copying the driver to the appropriate location, use devfsadm(1M) or add_drv(1M).

# cp foobar /usr/kernel/drv/sparcv9/foobar  <-- no sparcv9 for x86
# cp foobar.conf /usr/kernel/drv/foobar.conf
#
# add_drv foobar
#                  <--  any output here is probably an error!
# 

After installing the driver, you should find device files for it in the /devices tree. NOTE: If you run into problems, be sure to rem_drv foobar before adding it again. Also note that devfsadm -v foobar can give better diagnostics when things don't work.

Debugging

Solaris has four basic debugging tools that come with the system:

• mdb, the Modular Debugger. This is useful in examining panic dumps as well as the running system. It also works with applications. The syntax of mdb is basically the same as ed, the old Unix line text editor. The Modular Debugger Guide gives a complete description and various usage examples.
• kmdb allows you to set breakpoints and single step within the kernel on a live system. Invoke it via mdb -K. The syntax and features are very similar to mdb. It's very useful for analyzing hangs (provided you can reproduce them under kmdb). You may load kmdb via mdb -k, or have it loaded on boot by boot kmdb. To enter into kmdb once it loads, use stop-A on a Sun console or ctl-alt-d on an x86 console, or use tip(1) or telnet into a terminal server with the console set up to use a serial port. Be careful with ctl-alt-d on x86, as you must not be in a graphics mode, so no windowing system (JDS, Gnome, KDE, CDE, etc.). If you enter kmdb on such a system, you will not be able to see what is going on. It's better to reboot and use command line login, or, even better, set up a serial console and run kmdb from a window on another machine. Chapter 21 of Writing Device Drivers gives examples of using mdb and kmdb for debugging.
• dtrace, the Dynamic Tracing Tool. This tool allows you to trace entry and return from function calls within the kernel dynamically. It also does much more. Solaris Dynamic Tracing Guide is a ~275 page manual describing dtrace.
• cmn_err(9f) is the equivalent to Linux's printk().

Conclusion

Now that you know how Solaris manages its drivers and have some familiarity with the available tools for using, compiling, and debugging drivers, perhaps you'd like to write drivers. Fear not; the next article in this series will explain some of the kernel interfaces and conventions for Solaris driver programming.