vDPA kernel fmk: vDPA bus for abstracting HW

目的

• dive into the technical details of the architecture 
• use cases for kernel vDPA fmk with containers and VMs
• Review design and implementation
  • vDPA bus
  • vDPA bus driver
  • vDPA device driver

vDPA devices

• datapath complies with virtio specifications
• vendor specific control path
• physically HW or SW emulated
• vDPA HW devices type：
  • PF (Physical Function): Physical Function.
  • VF (Virtual Function): SR-IOV VF
  • VDEV (Virtual Device): virtual device composed by host OS utilizing ADIs (ADI - Assignable Device Interfaces)
  • SF (Sub Function): Vendor specific interface to slice PF to multiple subfunctions
  • Could be a non PCIe hardware
• vDPA devices type （how and where DMA translation）
  • Platform specific DMA translation：platform IOMMU
  • Device specific DMA translation：IOMMU on-chip cooperate with platform IOMMU （two stage DMA translation）
• device agnostic framework
  • hide differences and complexity of vDPA type/IOMMU type
  • present generic virtio/vhost device

 

vDPA fmk 概览

•  combining vDPA fmk and vhost/virtio subsystems
• kernel virtio drv or userspace vhost drv think controlling a vhost or virtio device
• in practice a vDPA device

vDPA bus

• vDPA fmk implements SW bus
  • several types of devices and drivers can be attached
• vDPA bus abstracts common attributes of devices 
• allows bus drivers to connect vDPA devices with other kernel subsystems
• bus defines operations  (vdpa_config_ops)
  • virtio specific operations: controlling the virtio device
    • Set and get device configuration space
    • Set and get virtqueue state
    • Virtio feature negotiation
  • Interrupt operations: register callbacks for virtqueue, config interrupt or IRQ forward
  • Doorbell operations：register callbacks when a specific virtqueue is kicked or provide an operation method report doorbell location and map to userspace directly
  • Migration operations
    • method to set and get the device or virtqueue state 
    • method to allow bus driver to collect dirty pages modified by HW
  • DMA mapping operations
    • set vendor own DMA mapping
    • optional and only required for device specific DMA translations
• bus driver (vdpa_config_ops) support kinds of vDPA bus driver and vDPA device
• vDPA bus driver use all or only part of operations
• Abstraction of vDPA bus and vDPA bus operations,
  • hide difference and complexity of HW
  • vDPA bus driver and vDPA device cooperate with unified bus operations without knowing the details of each other

vDPA device driver

• communicate directly with the vDPA device through a vendor specific method
• presenting a vDPA device to vDPA bus
  • implement the common vDPA bus operations
• from HW view, vDPA device driver is a driver
• from vDPA bus view, vDPA device driver is a vDPA device

• vDPA device drivers need to accomplish
• vDPA device probing/removing
  • requires a vendor or platform specific procedure to probe/remove vDPA HW on HW bus
  • needs to register/unregister vDPA device to vDPA bus
• vDPA device management
  • cooperate with different management modules (such as PF driver) to support life cycle management for vDPA device
• Interrupt processing
  • vDPA device driver need to allocate and process HW interrupt in platform or vendor specific way
  • vDPA device driver report interrupt used by each virtqueue to vDPA bus.
  • vDPA bus layer propagate interrupt to KVM for direct injection to guest
• Device abstraction
  • Abstract hardware as a vDPA device and implement vdpa_config_ops
  • vDPA device drivers perform mediation between the vDPA bus operations and vendor/platform specific access functions provided by the hardware/platform 
• Perform DMA mapping
  • For devices to have own DMA translation unit, vDPA device driver needs to accept the DMA mapping request from vDPA bus and translate those mappings into a data structure that could be used by vendor specific DMA translation units.
  • device driver provide DMA device to vDPA bus that is used for performing DMA/IOMMU related operations such as IOMMU probing and populating DMA mappings
• vDPA device driver serves as a mediator between the vDPA bus operations and vendor specific device access methods.
• vDPA device driver communicates with vDPA bus drivers through vdpa bus operations
• vDPA device driver communicates with hardware using vendor specific access methods.
• Provisioning and management tasks are done by cooperating with the management module.
• From vDPA bus drivers view, only a common vDPA device abstraction which uses the vDPA bus operations to control different types of vDPA devices.

vDPA bus drivers

• connect the vDPA bus to the vhost and virtio subsystems sitting on top of the vDPA kernel framework.
• two types of vDPA bus drivers
  • vhost-vDPA bus driver
    • connects vDPA bus to vhost subsystem and presents a vhost char device to the userspace
    • useful for cases when the datapath is expected to bypass the kernel completely
    • Userspace drivers can control the vDPA device via vhost ioctls as if a vhost device.
    • A typical use case is for performing direct I/O to userspace (or VM). 
  • virtio-vDPA bus driver
    • bridges vDPA bus to virtio bus and from there to a virtio interface
    • vDPA device behaves as a virtio device so it can be used by various kernel subsystems such as networking, block, crypto etc.
    • Applications that do not use vhost userspace APIs can keep using userspace APIs that are provided by kernel networking, block and other subsystems
• vDPA bus drive

 

• supporting different vDPA bus drivers in the vDPA kernel fmk, vDPA gains flexibility of vhost and virtio
• Most kernel subsystems can be used with vDPA devices
• new vDPA bus drivers can be developed to support features derived from new platforms and hardware
• sysfs based device and driver management interface is provided for userspace management tools
  • used for naming the vDPA device and selecting the driver that needs to be bound to the vDPA device

Management API

• basic device/driver management interface via sysfs 
• vDPA kernel fmk does not enforce a unified management API for operations
• Leaves this to the vendor specific vDPA device driver to implement

Summary

• introduced concept of the vDPA bus
• discussed design considerations and implementation of the vDPA device driver 
• provided overview of vhost-vDPA and virtio-vDPA bus drivers

参考文档

vDPA kernel framework part 1: vDPA bus for abstracting hardware