1 PCI Express I/O Virtualization Howto 2 Copyright (C) 2009 Intel Corporation 3 Yu Zhao <yu.zhao@intel.com> 4 5 Update: November 2012 6 -- sysfs-based SRIOV enable-/disable-ment 7 Donald Dutile <ddutile@redhat.com> 8 91. Overview 10 111.1 What is SR-IOV 12 13Single Root I/O Virtualization (SR-IOV) is a PCI Express Extended 14capability which makes one physical device appear as multiple virtual 15devices. The physical device is referred to as Physical Function (PF) 16while the virtual devices are referred to as Virtual Functions (VF). 17Allocation of the VF can be dynamically controlled by the PF via 18registers encapsulated in the capability. By default, this feature is 19not enabled and the PF behaves as traditional PCIe device. Once it's 20turned on, each VF's PCI configuration space can be accessed by its own 21Bus, Device and Function Number (Routing ID). And each VF also has PCI 22Memory Space, which is used to map its register set. VF device driver 23operates on the register set so it can be functional and appear as a 24real existing PCI device. 25 262. User Guide 27 282.1 How can I enable SR-IOV capability 29 30Multiple methods are available for SR-IOV enablement. 31In the first method, the device driver (PF driver) will control the 32enabling and disabling of the capability via API provided by SR-IOV core. 33If the hardware has SR-IOV capability, loading its PF driver would 34enable it and all VFs associated with the PF. Some PF drivers require 35a module parameter to be set to determine the number of VFs to enable. 36In the second method, a write to the sysfs file sriov_numvfs will 37enable and disable the VFs associated with a PCIe PF. This method 38enables per-PF, VF enable/disable values versus the first method, 39which applies to all PFs of the same device. Additionally, the 40PCI SRIOV core support ensures that enable/disable operations are 41valid to reduce duplication in multiple drivers for the same 42checks, e.g., check numvfs == 0 if enabling VFs, ensure 43numvfs <= totalvfs. 44The second method is the recommended method for new/future VF devices. 45 462.2 How can I use the Virtual Functions 47 48The VF is treated as hot-plugged PCI devices in the kernel, so they 49should be able to work in the same way as real PCI devices. The VF 50requires device driver that is same as a normal PCI device's. 51 523. Developer Guide 53 543.1 SR-IOV API 55 56To enable SR-IOV capability: 57(a) For the first method, in the driver: 58 int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn); 59 'nr_virtfn' is number of VFs to be enabled. 60(b) For the second method, from sysfs: 61 echo 'nr_virtfn' > \ 62 /sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_numvfs 63 64To disable SR-IOV capability: 65(a) For the first method, in the driver: 66 void pci_disable_sriov(struct pci_dev *dev); 67(b) For the second method, from sysfs: 68 echo 0 > \ 69 /sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_numvfs 70 713.2 Usage example 72 73Following piece of code illustrates the usage of the SR-IOV API. 74 75static int dev_probe(struct pci_dev *dev, const struct pci_device_id *id) 76{ 77 pci_enable_sriov(dev, NR_VIRTFN); 78 79 ... 80 81 return 0; 82} 83 84static void dev_remove(struct pci_dev *dev) 85{ 86 pci_disable_sriov(dev); 87 88 ... 89} 90 91static int dev_suspend(struct pci_dev *dev, pm_message_t state) 92{ 93 ... 94 95 return 0; 96} 97 98static int dev_resume(struct pci_dev *dev) 99{ 100 ... 101 102 return 0; 103} 104 105static void dev_shutdown(struct pci_dev *dev) 106{ 107 ... 108} 109 110static int dev_sriov_configure(struct pci_dev *dev, int numvfs) 111{ 112 if (numvfs > 0) { 113 ... 114 pci_enable_sriov(dev, numvfs); 115 ... 116 return numvfs; 117 } 118 if (numvfs == 0) { 119 .... 120 pci_disable_sriov(dev); 121 ... 122 return 0; 123 } 124} 125 126static struct pci_driver dev_driver = { 127 .name = "SR-IOV Physical Function driver", 128 .id_table = dev_id_table, 129 .probe = dev_probe, 130 .remove = dev_remove, 131 .suspend = dev_suspend, 132 .resume = dev_resume, 133 .shutdown = dev_shutdown, 134 .sriov_configure = dev_sriov_configure, 135}; 136