OpenSolaris project Brussels wite paper

The Driver Configuration Framework component of the Brussels project became available in SXCE build 83 and is included in the OpenSolaris^TM release. The Framework component provides users the following benefits when configuring drivers that use the GLDv3 framework:

• A single user interface, dladm, is used to configure network driver properties.

• A uniform syntax is used regardless of the properties: dladm subcommand properties data-link.

• The dladm interface is applied to both public and private properties of the driver.

• Using the tool on a specific driver does not disrupt other network connections.

This article elaborates on the administrative features provided by the Brussels project.

Introduction and Problem Description

Prior to the Brussels implementation, configuring data–link parameters on a machine running the Solaris^TM Operating System (Solaris OS) was needlessly difficult. Commands and interfaces were cryptic and inconsistent across the drivers, and the process often required service interruption to all connected networks, regardless of whether the connecting link was actually being impacted by the change.

An example of the deficiencies in these administrative methods is the refining of interface maximum transmission unit (MTU) that is typically done to enable support for the Jumbo Frame MTU. The complications are evident as described in an inquiry in the Sun Managers Mailing List.

Overview of NIC Driver Properties

NIC driver properties that configurable by using the dladm command fall into one of two categories:

• Public properties that can be applied to any driver of the given media type such as link speed, autonegotiation for Ethernet, or the MTU size that can be applied to all data-link drivers.

• Private properties that are particular to a certain subset of drivers for a given media type. These properties can be specific to that subset because they are closely related either to the hardware that is associated with the driver or to the details of the driver implementation itself, such as debugging-related tunables.

Properties of a NIC driver are typically set with default values. However, certain networking scenarios might require you to change specific property settings of a NIC. These property settings can be either public or private properties. For example, a NIC might be communicating with an old switch that does not properly perform autonegotiation. Or, a switch might have been configured to support Jumbo frames. Or, driver specific properties that regulate packet transmission or packet receiving might need to be modified for the given driver. With the implementation of the Brussels project, these settings can now be reset by a single administrative tool, dladm.

dladm Subcommands to Administer NIC Properties

For NIC drivers that have been converted to the GLDv3 framework, properties are configured by using the dladm command. This command enables you to configure the properties dynamically without causing any network disruption on other NICs of similar types. The values that you set are stored in a dladm repository and persist even after you reboot the system or unplumb the interface. Therefore, use dladm as the preferred command to configure NICs, instead of the ndd command.

To administer NIC drivers, you use the following dladm subcommands:

• dladm show-linkprop displays the properties that are associated with the data link.

• dladm set-linkprop sets values for specified data-link properties.

• dladm reset-linkprop restores property settings to the default values.

• dladm show-ether displays Ethernet parameter settings of a data link.

For more information about these commands, see the dladm(1M) man page.

---

Note - Customizing NIC properties by using the dladm command is supported only in network drivers that have been converted to the GLDv3 framework, such as the following:

• bge – supported in SXCE build 83

• nge – supported in SXCE build 86

• e1000g – supported in SXCE build 88

• nxge – supported in SXCE build 88

Work continues to make other drivers become supported in the Brussels implementation. To confirm whether your specific driver supports this feature, refer to the driver's man page.

---

The following section provides procedures to set certain NIC driver properties. The selected properties are public and common to all NIC drivers. A separate section describes driver specific properties and also provides procedures to configure selected private properties of the e1000g driver..

Procedures and Examples

The following section provides procedures with examples that show how to configure public drivers by using the dladm command.

How to Enable Support for Jumbo Frames

Enabling support for Jumbo frames in a network setup is a common task for most network scenarios. Support for Jumbo frames requires increasing the size of a data link's maximum transmission unit (MTU). The following procedure includes the use of customized names to identify data links, a feature that is introduced by the Clearview project. This Clearview overview document introduces the concept of customized link names.

1. On the system that has the link whose MTU you want to modify, assume the System Administrator role.

   The System Administrator role includes the Network Management profile. To create the role and assign the role to a user, see Chapter 9, Using Role-Based Access Control (Tasks) in System Administration Guide: Security Services.

2. Identify the specific Ethernet device whose MTU size you need to reset, display the links in the system.

   # dladm show-phys

   You might need to perform this step if your network configuration uses customized names for data links. With customized names, data links are no longer necessarily identified by their hardware-based names. For example, the Ethernet device is bge0. However, the data link over the device is renamed net0. Therefore, you would need to configure the MTU size of net0. Refer to the documentation in the Clearview project page in OpenSolaris for examples of configuration tasks on data links that use customized names.

3. (Optional) Display the data link's current MTU size and other properties.
   • To display a specific property of a data link, use the following syntax:

     dladm show-linkprop -p property data-link

     This command displays the settings of the property that you specify.

   • To display several selected properties of the data link, use the following syntax:

     # dladm show-link data-link

     This command displays data-link information, including MTU size.

   ---

   Note - See How to Display Data Link in the Project Clearview documentation page for additional examples of the use of the dladm show-link syntax to display data-link information.

   ---

4. Unplumb the interface that is configured over the data link.

   # ifconfig interface unplumb

5. Change the value of the link's MTU size to 9000, the value for Jumbo frames.

   # dladm set-linkprop -p mtu=9000 data-link

6. Plumb the IP interface over the link.

   # ifconfig interface plumb IP-address up

   For additional options that you can use with the ifconfig command, see the ifconfig(1M) man page.

7. (Optional) Verify that the interface uses the new MTU size by using one of the command syntaxes in Step 3.

   # dladm show-linkprop -p mtu data-link

8. (Optional) Display the link's current Ethernet settings.

   # dladm show-ether data-link

Example 1 Enabling Support for Jumbo Frames

The following example of enabling support for Jumbo frames builds on the following scenario:

• The system has two bge NICS: bge0 and bge1.

• The device bge0 is used as a primary interface, while the device bge1 is used for test purposes.

• You want to enable support for Jumbo frames on bge1, while you retain the default MTU size of the primary interface.

• The network configuration uses customized names for data links. The link name of bge0 is net0. The link name of bge1 is web1.

# dladm show-phys
LINK       MEDIA        STATE     SPEED     DUPLEX     DEVICE
net0       ether        up        100Mb     full       bge0
itops1     ether        up        100Mb     full       qfe3
web1       ether        up        100Mb     full       bge1

# dladm show-linkprop -p mtu web1
LINK     PROPERTY     VALUE     DEFAULT     POSSIBLE
web1     mtu          1500      1500        --

# ifconfig web1 unplumb
# dladm set-linkprop -p mtu=9000 web1
# ifconfig web1 plumb 10.10.1.2/24 up

# dladm show-link web1
LINK     CLASS     MTU      STATE     OVER
web1     phys      9000     up        --

Notice that the MTU value is now 9000. In this example, the dladm command enabled you to change web1's MTU size directly. The previous method would have required you to unplumb net0 as well, which would have unnecessarily disrupted the primary interface's operations.

How to Change Link Speed Parameters

Most network setups consist of a combination of systems with varying speed capabilities. For example, the advertised speed between an older system and a newer system might need to be changed to a lower setting to allow communication. By default, all the speed and duplex capabilities of a NIC card are advertised. This procedure shows how to turn off the gigabit capabilities and advertise only the megabit capabilities.

1. On the system that has the NIC whose properties you want to modify, assume the System Administrator role.
2. (Optional) Display the current status of the property you want to modify.

   # dladm show-linkprop -p property data-link

3. To advertise lower speed capabilities, turn off the higher speed capabilities to prevent them from being advertised.

   # dladm set-linkprop -p property=value1 data-link

Example 2 Disabling Advertisement of a NIC's Gigabit Capabilities

This example shows how you can prevent the link web1 from advertising gigabit capabilities.

# dladm show-linkprop -p adv_1000fdx_cap web1
LINK     PROPERTY             VALUE     DEFAULT     POSSIBLE
web1     adv_1000fdx_cap      1         --          1,0

# dladm show-linkprop -p adv_1000hdx_cap web1
LINK     PROPERTY             VALUE     DEFAULT     POSSIBLE
web1     adv_1000hdx_cap      1         --          1,0

The properties that advertise the link's gigabit capabilities are adv_1000fdx_cap and adv_1000hdx_cap. To disable these properties from being advertised, you would type the following commands:

# dladm set-linkprop -p adv_1000fdx_cap=0 web1
# dladm set-linkprop -p adv_1000hdx_cap=0 web1

Listing the Ethernet parameter settings would display the following output:

# dladm show-ether web1
LINK     PTYPE       STATE    AUTO  SPEED-DUPLEX             PAUSE
web1     current     up       yes   1G-f                     both

How to Obtain Status Information About NIC Properties

You can obtain information about the NIC driver's properties by displaying either the Ethernet parameter settings or the link properties.

1. On the system that has the NIC whose properties you want to modify, assume the System Administrator role.
2. To obtain information about the Ethernet parameter settings, use the following command:

   # dladm show-ether [-x] data-link

   where the -x option includes additional parameter information about the link. Without the -x option, only the current parameter settings are displayed.

3. To obtain information about all the properties of the link, use the following command:

   # dladm show-linkprop data-link

Example 3 Displaying Ethernet Parameter Settings

This example display an extended list of parameter information about a specified link.

# dladm show-ether -x web1
LINK     PTYPE       STATE    AUTO  SPEED-DUPLEX             PAUSE
web1     current     up       yes   1G-f                     both
--       capable     --       yes   1G-fh,100M-fh,10M-fh     both
--       adv         --       yes   100M-fh,10M-fh           both
--       peeradv     --       yes   100M-f,10M-f             both

With the -x option, the command also displays the built-in capabilities of the specified link, as well as the capabilities that are currently advertised between the host and the link partner. The following information is displayed:

• For the Ethernet device's current state, the link is up and functioning at 1 gigabits per second at full duplex. Its autonegotiation capability is enabled and has bidirectional flow control, in which both host and link partner can send and receive pause frames.

• Regardless of the current setting, the capabilities of the Ethernet device are listed. The negotiation type can be set to automatic, the device can support speeds of 1 gigabits per second, 100 megabits per second, and 10 megabits per second, at both full and half duplex. Likewise, pause frames can be received or sent in both directions between host and link partner.

• The following capabilities of web1 are advertised: autonegotiation, speed-duplex, and flow control of pause frames.

• Similarly, the link partner advertises the following capabilities: autonegotiation, speed-duplex, and flow control of pause frames.

Example 4 Displaying Link Properties

This example shows how to list all the properties of a link. If you want to display only a specific property, you use the -p option with the specific property that you want to monitor.

# dladm show-linkprop web1
LINK     PROPERTY             VALUE       DEFAULT     POSSIBLE
web1     speed                1000        --          -- 
web1     autopush             --          --          -- 
web1     zone                 --          --          -- 
web1     duplex               half        --          half,full 
web1     state                unknown     up          up,down 
web1     adv_autoneg_cap      1           1           1,0 
web1     mtu                  1500        1500        -- 
web1     flowctrl             no          bi          no,tx,rx,bi 
web1     adv_1000fdx_cap      1           1           1,0 
web1     en_1000fdx_cap       1           1           1,0 
web1     adv_1000hdx_cap      1           1           1,0 
web1     en_1000hdx_cap       1           1           1,0 
web1     adv_100fdx_cap       0           0           1,0 
web1     en_100fdx_cap        0           0           1,0 
web1     adv_100hdx_cap       0           0           1,0 
web1     en_100hdx_cap        0           0           1,0 
web1     adv_10fdx_cap        0           0           1,0 
web1     en_10fdx_cap         0           0           1,0 
web1     adv_10hdx_cap        0           0           1,0 
web1     en_10hdx_cap         0           0           1,0

The settings for the speed and duplex capabilities of the link are manually configured on the enabled-speed properties which are labeled en_*_cap. For example, en_1000fdx_cap is the property for the gigabit full-duplex capability, and en_100hdx_cap is the property for the 100 megabits half-duplex capability. The settings of these enabled speed properties are advertised between the host and its link partner by corresponding advertised speed properties, which are labeled adv_*_cap such as adv_1000fdx_cap and adv_100hdx_cap. Normally, the settings of a given enabled speed property and the corresponding advertised property are identical. However, if a NIC supports some advanced features such as Power Management, those features might set limits on the bits that are actually advertised between the host and its link partner. For example, with Power Management, the values of the adv_*_cap properties might only be a subset of the values of the en_*_cap properties. For more details about the enabled and advertised speed properties, see the dladm(1M) man page.

Configuring Properties Specific to the e1000g Driver

Brussels support for the e1000g driver is available in SXCE build 88. The driver is widely used on many Sun platforms to support the Intel^® PRO/1000 Gigabit NICs, such as the Sun Fire^TM X4200 server, the Sun Netra^TM X4450 server, and the Sun Fire T2000 server.

Previously, the e1000g driver, like many other NIC drivers, was configured by using the ndd command and by defining property settings in the driver's e1000g.conf configuration file. This driver is now converted to the GLDv3 framework. Thus, the administration of its configuration can be uniformly performed just as other similarly converted drivers with the dladm command.

The same command can be used to adjust parameters that are specific to the e1000g driver. For example, you might need to configure the driver to use Direct Memory Access (DMA) binding for transmission instead of the bcopy mode. Likewise, you might need to reset certain interrupt parameters to improve the performance of the driver. See the procedures that show how to configure these parameters.

---

Note - The nxge driver has become Brussels-supported in SXCE build 88. The driver, which is widely used on many platforms, also has nxge-specific properties. The following procedures about configuring private properties apply to the e1000g driver. However, the general procedures can also apply to the nxge driver.

---

Procedures and Example

This section provides procedures and an example that show how to configure selected private properties of the e1000g driver.

How to Set the e1000g Driver to Use Direct Memory Access Binding

Bulk traffic, such as file transfers, normally involves negotiation of large packets across the network. In such cases, you can obtain better performance from the e1000g driver by configuring it to automatically use DMA binding.

In the following procedure, a threshold is defined for packet fragment sizes. If a fragment size surpasses the threshold, then DMA binding is used for transmitting. If a fragment size is within the threshold, then bcopy mode is used, where the fragment data is copied to the preallocated transmit buffer. For more details about this threshold and other tunables, see the e1000 tunables list in OpenSolaris.

To set the threshold, perform the following steps:

1. On the system that has the NIC whose properties you want to modify, assume the System Administrator role.
2. Set the appropriate value for the _tx_bcopy_threshold property.

   # dladm set-linkprop -p _tx_bcopy_threshold=value e1000g-data-link

   For this property, the valid values for the threshold range from 60 through 2048.

   ---

   Note - As with configuring public properties, the interface must also be unplumbed before private property settings can be modified.

   ---

3. (Optional) Verify the new threshold value.

   # dladm show-linkprop -p _tx_bcopy_threshold e1000g-data-link

How to Manually Set the Interrupt Rate

Parameters that regulate the rate at which interrupts are delivered by the e1000g driver also affect network and system performance. Typically network packets are delivered to the upper layer of the stack by generating an interrupt for every packet. In turn the interrupt rate, by default, is automatically adjusted by the GLD layer in the kernel. However, this mode might not be desirable in all network traffic conditions. For a discussion of this issue, refer to this document (http://www.stanford.edu/class/cs240/readings/mogul.pdf) that was presented at the USENIX technical conference in 1996. Thus, in certain circumstances, setting the interrupt rate manually becomes necessary to obtain better performance.

To define the interrupt rate, you set the following parameters:

• _intr_throttling_rate determines the delay between interrupt assertions regardless of network traffic conditions.

• _intr_adaptive determines whether automatic tuning of the interrupt throttling rate is enabled. By default, this parameter is enabled.

1. On the system that has the NIC whose driver properties you want to modify, assume the System Administrator role.
2. If necessary, identify the device whose driver property you want to modify.

   # dladm show-phys

3. Disable automatic tuning of the interrupt throttling rate.

   # dladm set-linkprop -p _intr_adaptive=0 e1000g-data-link

   ---

   Note - When automatic tuning of the interrupt throttling rate is enabled, then any value that is set for the parameter _intr_throttling_rate is ignored.

   ---

4. Unplumb the network interface.
5. Set the value for the minimum inter interrupt level.

   # dladm set-linkprop -p _intr_throttling_rate=value e1000g-data-link

   ---

   Note - The default value of the _intr_throttling_rate parameter is 550 on SPARC^® based systems and 260 on x86 based systems. Setting the minimum inter-interrupt level to 0 disables the interrupt throttling logic.

   ---

6. Plumb the interface and configure an IP address for the interface.
7. (Optional) Display the threshold's new settings.

Example 5 Configuring for DMA Binding and Setting the Interrupt Throttling Rate

This example uses an x86 based system with an e1000g NIC. The driver is configured with a threshold setting toggle between using DMA binding or the bcopy mode for transmitting packets. The setting for the interrupt throttling rate is also modified. Further, the e1000g data link has been renamed with a customized name. Therefore, the configuration is performed on the data link by referring to the customized name, public0.

# dladm show-phys
LINK       MEDIA        STATE     SPEED     DUPLEX     DEVICE
public0    ether        up        100Mb     full       e1000g0

# dladm show-linkprop -p _tx_bcopy_threshold public0
LINK        PROPERTY                VALUE     DEFAULT     POSSIBLE
public0     _tx_bcopy_threshold     512       512         --

# dladm show-linkprop -p _intr-throttling_rate
LINK        PROPERTY                  VALUE     DEFAULT     POSSIBLE
public0     _intr-throttling_rate     260       260         --

# ifconfig public0 unplumb
# dladm set-linkprop -p _tx_bcopy_threshold=1024 public0
# dladm set-linkprop -p _intr_adaptive=0 public0
# dladm set-linkprop -p _intr-throttling_rate=1024 public0
# ifconfig public0 plumb 10.10.1.2/24 up

# dladm show-linkprop -p _tx_bocopy_threshold=1024 public0
LINK        PROPERTY                VALUE     DEFAULT     POSSIBLE
public0     _tx_bcopy_threshold     1024      512         --

# dladm show-linkprop -p _intr_adaptive public0
LINK        PROPERTY           VALUE     DEFAULT     POSSIBLE
public0     _intr-adaptive     0         1           --

# dladm show-linkprop -p _intr-throttling_rate
LINK        PROPERTY                  VALUE     DEFAULT     POSSIBLE
public0     _intr-throttling_rate     1024      260         --

Ongoing Work

The Brussels ndd compat component will provide additional functionality in the Framework for legacy support of the ndd(1m) commands so that drivers will not have to include complex and undocumented interfaces to support the ndd command.

Also, the Brussels Persistence component will add additional functionality to allow the property settings to be automatically applied across reboot and driver restart.

Future articles will describe the details of the Framework, as well as the methods used to provide ndd compatibility and persistent property settings.

Work also continues to provide a GUI to administer data links. A prototype of the GUI is available at the OpenSolaris project page. The prototype is an interactive GUI for demonstration purposes. Please send feedback to the Brussels discussion list.

Access to the Brussels-supported command-line interfaces is now available in the OpenSolaris 2008.05 release.

Changes subject to http://opensolaris.org/os/project/brussels/Documentation/wp/