IOMMU on the SUN SPARC system

IOMMU on the SUN SPARC system

 

This article is intended to describe the IOMMU in Sun Blade workstations and Sun Fire servers. A new system architecture has been introduced to take advantage of the UltraSPARC III processor.

1. UltraSPARC III System Architecture  

  2．Host PCI Bridge

 

3. PCI IOMMU

The I/O Memory Management Unit (IOMMU) performs virtual-to-physical address translation during DVMA cycles initiated by PCI masters. PCI master devices provide a 32-bit virtual address at the beginning of a DVMA transfer. The IOMMU translates the virtual address into a 32-bit physical address plus an identifier for cacheable vs. non-cacheable address space. There is a separate PCI IOMMU in each PCI leaf, which is responsible for translating virtual addresses only from devices controlled by that PCI leaf. The IOMMU is leveraged from the previous design with mostly minor changes (41-bit to 43-bit physical address) and one major change, which is the addition of the context flush capability. The IOMMU consists of a 16-entry, fully associative Translation Lookaside Buffer (TLB) implemented in hardware to cache recently used translations and a Translation Storage Buffer (TSB), which is a software managed data structure (translation table) in memory. It contains one-level mapping information for the virtual pages. IOMMU Hardware performs TSB lookup when the translation cannot be found in the TLB. A TSB entry is called a Translation Table Entry (TTE) and takes 8 bytes. Supported TSB table sizes include 1nK entries where n = 1 through 8 and is typically set to 8 Kbyte. The IOMMU supports two different page sizes, 8 Kbyte and 64 Kbyte. Mixed page sizes can be used in the system, but the TSB table lookup only assumes the smaller page size of 8 Kbyte. No overlapping of pages is allowed. This gives support for DVMA address space of 8 Mbyte to 1 Gbyte for an 8-Kbyte page and 64 Kbyte to 2 Gbyte for a 64-Kbyte page (DVMA space = page size * TSB table size). A DVMA space larger than 2 Gbyte is not supported, which means that 64-Kbyte and 128-Kbyte TSB sizes are not supported with a 64-Kbyte page size. Software must set up TSB before it allows translation to start. The TSB can be accessed in the IOMMU control register. The PCI IOMMU can operate in three different modes: translation, bypass, and pass-through. Its operating mode is determined by the values in IOMMU Control Register, the PCI addressing mode used (32 bits vs. 64 bits), and values in the PCI virtual address. Translation is initiated by the PBM block by providing a 32-bit virtual address. The IOMMU hardware performs a TLB lookup first. If the lookup results in TLB hit, the IOMMU returns a 43-bit physical address to the PBM block. If a TLB miss happens, the IOMMU waits for the PBM to request a TSB lookup. If the TSB locates a valid mapping for the virtual page, information in the TSB entry will be loaded into TLB and translation continues. If the TSB lookup results in a miss, an error will be returned to the PBM. The translation of a virtual address to a physical address is illustrated in Figure 4-3.

 

 

The implementation of the PCI IOMMU allows PCI devices to have their own MMU and bypass the IOMMU that is in the HPB. A PCI device operating in bypass mode has direct access to the entire physical space of the Fireplane interconnect bus. Pass-through mode allows access to the 2 Gbyte of memory address space. A DVMA access in pass-through mode will always be cacheable to space.