[[PageOutline]] {{{ #!html

實作七：Xen PCI Passthrough 操作手冊

}}} ---- = 【前言】 = * '''''小叮嚀__''''' * 為了使 Xen patched Kerenl 能支援 PCI Passthrough, 請務必確認以下Xen 相關的核心配置. {{{ CONFIG_XEN_PCIDEV_FRONTEND=y CONFIG_XEN_PCIDEV_BACKEND=y CONFIG_XEN_PCIDEV_BACKEND_PASS is not set CONFIG_XEN_PCIDEV_BACKEND_VPCI=y CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set }}} * '''''小技巧__''''' * Binding at Boot-Time. * 直接將顯示卡所佔的 PCI_Address 在開機時就隱藏起來,使得一開始便能直接 bind 到該 PCI_Address. * 透過 lspci 指令來查詢顯示卡所佔的實體 PCI Address. {{{ # 台中機器 /boot/vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda1 ro console=tty0 pciback.permissive pciback.hide=(0000:01:00.0) }}} {{{ # 新竹機器 /boot/vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda1 ro console=tty0 pciback.permissive pciback.hide=(0000:08:00.0) }}} * 如果透過 lspci 指令查出顯示卡為 unknown 或是其他不明資訊, 請先更新 PCI ID database 試看看. {{{ $ update-pciids }}} ---- = 【Step 0: 流程】 = * 本範例將說明如何使用及設定 Xen PCI Passthrough {{{ #!graphviz digraph finite_state_machine { rankdir=LR; ranksep=0.3; size="11,8"; node[shape=box,width=3.0]; "1.先確認主機上的顯示卡相關資訊及規格" -> "2. 建立一台虛擬機器供測試用" -> "3.如何設定 PCI Frontend ?"; "1.先確認主機上的顯示卡相關資訊及規格" -> "2. 建立一台虛擬機器供測試用" -> "4.如何設定 PCI Backend ?"; "5.如何設定 Xen 硬體定址存取 ?" -> "6.啟動並登入虛擬機器" -> "7.檢查虛擬機器有無成功分配到顯示卡的資源"; } }}} ---- = 【Step 1: 首先查看主機上的顯卡資訊】 = {{{ $ lspci -vv | more }}} {{{ # 台中機器 01:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 9800 GT] (rev a2) Subsystem: ASUSTeK Computer Inc. Device 82a0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: nvidia Kernel modules: nvidia, nvidiafb }}} {{{ # 新竹機器 08:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 9800 GT] (rev a2) (prog-if 00 [VGA controller]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: nvidia Kernel modules: nvidiafb, nvidia }}} ---- = 【Step 2: 產生一台實驗用虛擬機器】 = * 設定你想要怎樣規格的虛擬機器. {{{ $ sudo gedit /etc/xen-tools/xen-tools.conf }}} {{{ #!sh 45 dir = /home 128 size = 2Gb # Disk image size. 129 memory = 128Mb # Memory size 130 swap = 128Mb # Swap size 133 dist = lenny # Default distribution to install. 134 image = sparse # Specify sparse vs. full disk images. 163 gateway = 192.168.168.254 164 netmask = 255.255.255.0 165 broadcast = 192.168.168.255 214 mirror = http://free.nchc.org.tw/debian/ 257 serial_device = hvc0 #default }}} {{{ $ sudo xen-create-image --hostname=xg01 --ip=192.168.168.X --mac=00:16:00:00:00:XX --force }}} ---- = 【Step 3: PCI Frontend Configuration 設定你的 DomU 】 = {{{ $ sudo gedit /home/domains/xg01.cfg }}} {{{ #!sh 9 kernel = '/boot/vmlinuz-2.6.26-2-xen-amd64' 10 ramdisk = '/boot/initrd.img-2.6.26-2-xen-amd64' 11 memory = '128' 12 vcpus = '1' 14 # 配置你的 PCIE 顯示卡 15 pci = ['08:00.0'] 19 root = '/dev/sda2 ro' 20 disk = [ 'file:/home/domains/xg01/disk.img,sda2,w', 'file:/home/domains/xg01/swap.img,sda1,w', ] 29 name = 'xg01' # # Networking # 34 vif = [ 'ip=192.168.168.X ,mac=00:16:00:00:00:XX' ] # # Behaviour # 39 on_poweroff = 'destroy' 40 on_reboot = 'restart' 41 on_crash = 'restart' }}} ---- = 【Step 4: PCI Backend Configuration 設定你的 Dom0 】 = * 切換身份為 Super user. {{{ $ sudo su - }}} * 隱藏 dom0 的PCI 匯流排的位址, 並交由 pciback 模組來控制該匯流排存取權. {{{ # echo -n "0000:08:00.0" > /sys/bus/pci/drivers/nvidia/unbind }}} * 將 dev_ids 轉到 pciback 成為新的 binding slot. {{{ # echo -n "0000:08:00.0" > /sys/bus/pci/drivers/pciback/new_slot # echo -n "0000:08:00.0" > /sys/bus/pci/drivers/pciback/bind }}} {{{ # cat /sys/bus/pci/drivers/pciback/slots }}} {{{ #!sh 0000:08:00.0 }}} * 警告: 請確認您的顯示卡資源沒有被其 Kernel module 佔有及控制. {{{ $ ls -al /sys/bus/pci/devices/0000:08:00.0/ | grep driver }}} {{{ #!sh driver -> ../../../../bus/pci/drivers/nvidia ---> 此連結應該已經不存在. NVIDIA module 已無該裝置控制權. driver -> ../../../../bus/pci/drivers/pciback ---> 控制權應該已經交賦予 pciback module. }}} ---- = 【Step 5: 硬體定址存取設定】 = == Permissive Flag == * 說明: 透過 "lspci" 指令來查詢您顯示卡的 PCI address. {{{ # su clouder $ sudo gedit /etc/xen/xend-pci-permissive.sxp }}} {{{ /*** 台中機器 ***/ (unconstrained_dev_ids #('0123:4567:89AB:CDEF') ('0000:01:00.0') ) }}} {{{ /*** 新竹機器 ***/ (unconstrained_dev_ids #('0123:4567:89AB:CDEF') ('0000:08:00.0') ) }}} == User-space Quirks == {{{ $ sudo gedit /etc/xen/xend-pci-quirks.sxp }}} * 說明: 請到 PCI_DB 網站: http://www.pcidatabase.com/ 來查詢您顯示卡的 vendor_id & device_id. {{{ #!sh (pci_ids # Entries are formated as follows: # :[::] ('10de:0605' # NVIDIA 9800GT ) ) }}} ---- = 【Step 6: 啟動並登入你的虛擬機器 DomU 】 = * 說明: 先用 root 免密碼登入. * @ Dom0 {{{ $ sudo xm create -c xg01.cfg }}} ---- = 【Step 7: 檢查 DomU 的 PCI Passthrough 有無成功】 = * 先用 root 免密碼登入 xg01, 然後新增使用者 clouder. {{{ # adduser clouder }}} {{{ #!sh Adding user `clouder' ... Adding new group `clouder' (1000) ... Adding new user `clouder' (1000) with group `clouder' ... Creating home directory `/home/clouder' ... Copying files from `/etc/skel' ... Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully Changing the user information for clouder Enter the new value, or press ENTER for the default Full Name []: Room Number []: Work Phone []: Home Phone []: Other []: Is the information correct? [Y/n] y }}} * 先登出 "root" 並且改用使用者 "clouder" 登入,並查看顯卡資訊有無正常顯示. {{{ $ exit $ sudo apt-get update $ sudo apt-get install wget pciutils udev $ lspci }}} {{{ #!sh 00:00.0 VGA compatible controller: nVidia Corporation Unknown device 0605 (rev a2) }}} * 更新 PCI ID Database.(當您發現系統無法正常偵測到你的顯示卡時) {{{ $ sudo update-pciids $ lspci }}} {{{ #!sh 00:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 9800 GT] (rev a2) }}} * @DomU 查看顯卡資源有無順利分配到 DomU. {{{ $ dmesg | grep pci }}} {{{ #!sh [ 0.745795] pcifront pci-0: Installing PCI frontend [ 0.745872] pcifront pci-0: Creating PCI Frontend Bus 0000:00 [ 0.848291] pci 0000:00:00.0: Boot video device }}} * @Dom0 查看顯卡資源有無順利分配到 DomU. {{{ $ dmesg | grep pci }}} {{{ # 台中機器 [ 0.718704] ACPI: bus type pci registered [ 0.902598] pci 0000:00:1f.0: quirk: region 0400-047f claimed by ICH6 ACPI/GPIO/TCO [ 0.902695] pci 0000:00:1f.0: quirk: region 0480-04bf claimed by ICH6 GPIO [ 0.925960] pnp 00:01: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c [ 0.928118] pnp 00:0a: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c [ 0.928580] pnp 00:0b: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c [ 0.929060] pnp 00:0c: PNP0c01: calling quirk_system_pci_resources+0x0/0x15c [ 1.061488] pci 0000:01:00.0: Boot video device [ 1.061674] Allocate Port Service[0000:00:01.0:pcie00] [ 1.065543] Allocate Port Service[0000:00:01.0:pcie03] [ 476.851360] pciback 0000:08:00.0: seizing device [ 560.262910] pciback: vpci: 0000:08:00.0: assign to virtual slot 0 }}} {{{ # 新竹機器 [ 0.655891] ACPI: bus type pci registered [ 0.673706] pci 0000:00:1f.0: quirk: region f800-f87f claimed by ICH6 ACPI/GPIO/TCO [ 0.673706] pci 0000:00:1f.0: quirk: region fa00-fa3f claimed by ICH6 GPIO [ 0.706664] pnp 00:0c: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c [ 0.706664] pnp 00:0d: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c [ 0.706664] pnp 00:0e: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c [ 0.707107] pnp 00:0f: PNP0c01: calling quirk_system_pci_resources+0x0/0x15c [ 0.836266] pci 0000:08:00.0: Boot video device [ 0.836266] Allocate Port Service[0000:00:06.0:pcie00] [ 0.836266] Allocate Port Service[0000:00:06.0:pcie03] [ 0.836266] Allocate Port Service[0000:00:1c.0:pcie00] [ 0.836266] Allocate Port Service[0000:00:1c.0:pcie02] [ 0.836266] Allocate Port Service[0000:00:1c.0:pcie03] [ 0.836266] Allocate Port Service[0000:00:1c.4:pcie00] [ 0.836266] Allocate Port Service[0000:00:1c.4:pcie02] [ 0.836266] Allocate Port Service[0000:00:1c.4:pcie03] [ 0.836266] Allocate Port Service[0000:00:1c.5:pcie00] [ 0.840261] Allocate Port Service[0000:00:1c.5:pcie02] [ 0.840261] Allocate Port Service[0000:00:1c.5:pcie03] [ 1891.425509] pciback 0000:08:00.0: seizing device [ 2133.589319] pciback: vpci: 0000:08:00.0: assign to virtual slot 0 }}} ---- = 【討論 & 休息】 = [[BR]] [[BR]]