Introduction to Receive Side Scaling

Non-RSS Receive Processing

http://msdn.microsoft.com/en-us/library/windows/hardware/ff568798(v=vs.85).aspx 

Miniport drivers that do not support RSS handle receive processing as described in this topic.

The following figure illustrates non-RSS receive processing.

In the figure, the dashed paths represent an alternate path for the send and receive processing. Because the system controls the scaling, the processing doesn't always occur on the CPU that provides the best performance. Connections are processed on the same CPU over successive interrupts only by chance.

The following process repeats for each non-RSS interrupt cycle:

1. The NIC uses DMA to fill a buffer with received data and interrupts the system.

   The miniport driver allocated the receive buffers in shared memory during initialization.

2. The NIC can continue to fill additional receive buffers at any time in this interrupt cycle. However, the NIC does not interrupt again until the miniport driver enables interrupts.

   The received buffers that the system handles in one interrupt cycle can be associated with many different network connections.

3. NDIS calls the miniport driver's MiniportInterrupt function (ISR) on a system-determined CPU.

   Ideally, the ISR should go to the least busy CPU. However, in some systems, the system assigns the ISR to an available CPU or to a CPU that is associated with the NIC.

4. The ISR disables the interrupts and requests NDIS to queue a deferred procedure call (DPC) to process the received data.

5. NDIS calls the MiniportInterruptDPC function (the DPC) on the current CPU.

6. The DPC builds receive descriptors for all of the received buffers and indicates the data up the driver stack. For more information, see Receiving Network Data.

   There can be many buffers for many different connections and there is potentially a lot of processing to complete. The received data associated with subsequent interrupt cycles can be processed on other CPUs. The send processing for a given network connection can also run on a different CPU.

7. The DPC enables the interrupts. This interrupt cycle is complete and the process starts again.

=====================================================================================

Introduction to Receive Side Scaling

http://msdn.microsoft.com/en-us/library/windows/hardware/ff556942(v=vs.85).aspx

----------------------------------------------------------------------------------------------------------------------

Receive side scaling (RSS) is a network driver technology that enables the efficient distribution of network receive processing across multiple CPUs in multiprocessor systems.

Note  Because hyper-threaded CPUs on the same core processor share the same execution engine, the effect is not the same as having multiple core processors. For this reason, RSS does not use hyper-threaded processors.

To process received data efficiently, a miniport driver's receive interrupt service function schedules a deferred procedure call (DPC). Without RSS, a typical DPC indicates all received data within the DPC call. Therefore, all of the receive processing that is associated with the interrupt runs on the CPU where the receive interrupt occurs. For an overview of non-RSS receive processing, see Non-RSS Receive Processing.

With RSS, the NIC and miniport driver provide the ability to schedule receive DPCs on other processors. Also, the RSS design ensures that the processing that is associated with a given connection stays on an assigned CPU. The NIC implements a hash function and the resulting hash value provides the means to select a CPU.

The following figure illustrates the RSS mechanism for determining a CPU.

A NIC uses a hashing function to compute a hash value over a defined area (hash type) within the received network data. The defined area can be noncontiguous.

A number of least significant bits (LSBs) of the hash value are used to index an indirection table. The values in the indirection table are used to assign the received data to a CPU. For more detailed information about the indirection table, see RSS Configuration.

With message signaled interrupt (MSI) support, a NIC can also interrupt the associated CPU. For more information about NDIS support for MSIs, see NDIS MSI-X.

RSS can improve network system performance by reducing:

• Processing delays by distributing receive processing from a NIC across multiple CPUs.

  This helps to ensure that no CPU is heavily loaded while another CPU is idle.

• Spin lock overhead by increasing the probability that software algorithms that share data execute on the same CPU.

  Spin lock overhead occurs, for example, when a function executing on CPU0 possesses a spin lock on data that a function running on CPU1 must access. CPU1 spins (waits) until CPU0 releases the lock.

• Reloading of caches and other resources by increasing the probability that software algorithms that share data execute on the same CPU.

  Such reloading occurs, for example, when a function that is executing and accessing shared data on CPU0, executes on CPU1 in a subsequent interrupt.

To achieve these performance improvements in a secure environment, RSS provides the following mechanisms:

• Distributed processing

  RSS distributes the processing of receive indications from a given NIC in DPCs to multiple CPUs.

• In-order processing

  RSS preserves the order of delivery of received data packets. For each network connection, RSS processes receive indications on an associated CPU. For more information about RSS receive processing, see Indicating RSS Receive Data.

• Dynamic load balancing

  RSS provides a means to rebalance the network processing load between CPUs as host system load varies. To rebalance the load, overlying drivers can change the indirection table. For more information about specifying indirection tables, hash types, and hashing functions, see RSS Configuration.

• Send-side scaling

  RSS enables driver stacks to process send and receive-side data for a given connection on the same CPU. Typically, an overlying driver (for example, TCP) sends part of a data block and waits for an acknowledgment before sending the balance of the data. The acknowledgment then triggers subsequent send requests. The RSS indirection table identifies a particular CPU for the receive data processing. By default, the send processing runs on the same CPU if it is triggered by the receive acknowledgment. A driver can also specify the CPU (for example, if a timer is used).

• Secure hash

  RSS includes a signature that provides added security. This signature protects the system from malicious remote hosts that might attempt to force the system into an unbalanced state.

• MSI-X support

  RSS, with support for MSI-X, runs the interrupt service routine (ISR) on the same CPU that later executes the DPC. This reduces spin lock overhead and reloading of caches.

The following figure illustrates the levels of hardware support for RSS.

There are three possible levels of hardware support for RSS:

Hash calculation with a single queue

The NIC calculates the hash value and the miniport driver assigns received packets to queues that are associated with CPUs. For more information, see RSS with a Single Hardware Receive Queue.

Hash calculation with multiple receive queues

The NIC assigns the received data buffers to queues that are associated with CPUs. For more information, see RSS with Hardware Queuing.

Message Signaled Interrupts (MSIs)

The NIC interrupts the CPU that should handle the received packets. For more information, see RSS with Message Signaled Interrupts.

The NIC always passes on the 32-bit hash value.