[[PageOutline]] = Xen GPU Cluster = == Hardware == ||Machine|| Dell !OptiPlex 755 ||Node|| 9 nodes ||CPU|| Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz ||Memory|| 6GB/node ||Storage|| 160GB/node ||Video Card|| NVIDIA !GeForce 9800GT 1GB/node == Software == ||OS!#1|| Ubuntu 8.10 with Kernel: 2.6.28 x86_64 (non-xen-patched kernel) ||OS!#2|| Ubuntu 8.10 with Kernel: 2.6.22-9 x86_64 (Xen-3.3.1+Lustre patched kernel) [[BR]] == System Architecture == [[Image(Architecture.jpg)]] = Part 1 Build essential environment = == 1.1 - Basic Environment == # NVIDIA CUDA driver # [[BR]] rock@cloud:~/nvidia/cuda$ wget !http://developer.download.nvidia.com/compute/cuda/2_1/drivers/NVIDIA-Linux-x86_64-180.22-pkg2.run [[BR]] # NVIDIA CUDA toolkit # [[BR]] rock@cloud:~/nvidia/cuda$ wget !http://developer.download.nvidia.com/compute/cuda/2_1/toolkit/cudatoolkit_2.1_linux64_ubuntu8.04.run [[BR]] # NVIDIA CUDA SDK # [[BR]] rock@cloud:~/nvidia/cuda$ wget !http://developer.download.nvidia.com/compute/cuda/2_1/SDK/cuda-sdk-linux-2.10.1215.2015-3233425.run [[BR]] {{{ ~$ sudo apt-get install autoconf automake build-essential gcc make libtool initramfs-tools libxi6 libxi-dev libxmu6 libxmu-dev linux-kernel-devel linux-headers-2.6.27-11-server xserver-xorg-core xserver-xorg-dev }}} rock@cloud:~$ sudo ln -sf /usr/src/linux-2.6.22 /usr/src/linux [[BR]] rock@cloud:~/nvidia/cuda$ sudo sh NVIDIA-Linux-x86_64-180.22-pkg2.run [[BR]] rock@cloud:~/nvidia/cuda$ sudo sh cudatoolkit_2.1_linux64_ubuntu8.04.run [[BR]] {{{ Enter install path (default /usr/local/cuda, '/cuda' will be appended): /usr/local/cuda }}} # Note: {{{ * Please make sure your PATH includes /usr/local/cuda/bin * Please make sure your LD_LIBRARY_PATH includes /usr/local/cuda/lib * or add /usr/local/cuda/lib to /etc/ld.so.conf and run ldconfig as root * Please read the release notes in /usr/local/cuda/doc/ * To uninstall CUDA, delete /usr/local/cuda * Installation Complete }}} rock@cloud:~/nvidia/cuda$ sudo sh cuda-sdk-linux-2.10.1215.2015-3233425.run [[BR]] # Note: {{{ {{{ Enter install path (default /usr/local/cuda, '/cuda' will be appended): /usr/local/NVIDIA_CUDA_SDK }}} }}} {{{ Configuring SDK Makefile (/usr/local/NVIDIA_CUDA_SDK/common/common.mk)... * Please make sure your PATH includes /usr/local/cuda/bin * Please make sure your LD_LIBRARY_PATH includes /usr/local/cuda/lib * To uninstall the NVIDIA CUDA SDK, please delete /usr/local/NVIDIA_CUDA_SDK }}} rock@cloud:~$ sudo vim /etc/profile [[BR]] {{{ Add: export PATH=$PATH:/usr/local/cuda/bin }}} rock@cloud:~$ source /etc/profile [[BR]] rock@cloud:~$ sudo vim /etc/ld.so.conf [[BR]] {{{ Add: /usr/local/cuda/lib }}} rock@cloud:~$ sudo ldconfig [[BR]] == 1.2 NVIDIA Driver !HowTo !OnNoneXenKernel == # Rock said that the unknown identification of the VGA device might be the "pciids" problem. [[BR]] Sol1: [[BR]] rock@cloud:~$ sudo update-pciids [[BR]] Sol2: [[BR]] rock@cloud:~$ wget !http://pciids.sourceforge.net/v2.2/pci.ids [[BR]] rock@cloud:~$ sudo cp pci.ids /usr/share/misc/ [[BR]] rock@cloud:~$ sudo lspci -v -v [[BR]] {{{ 01:00.0 VGA compatible controller: nVidia Corporation GeForce 9800 GT (rev a2) Subsystem: ASUSTeK Computer Inc. Device 82a0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [128] Power Budgeting Capabilities: [600] Vendor Specific Information Kernel driver in use: nvidia Kernel modules: nvidia, nvidiafb }}} rock@cloud:~$ less /var/log/Xorg.0.log | grep nVidia [[BR]] {{{ (--) PCI:*(0@1:0:0) nVidia Corporation GeForce 9800 GT rev 162, Mem @ 0xfd000000/16777216, 0xd0000000/268435456, 0xfa000000/33554432, I/O @ 0x0000dc80/128, BIOS @ 0x????????/131072 }}} rock@cloud:~$ sudo vim /etc/X11/xorg.conf [[BR]] {{{ # Allocate the BusID for the VGA Device Section "Device" Identifier "Device0" Driver "nvidia" BusID "PCI:1:0:0" VendorName "NVIDIA Corporation" BoardName "GeForce 9800 GT" Option "RenderAccel" "True" Option "UseEdidDpi" "False" EndSection }}} rider@cloud:~$ export DISPLAY=:0 [[BR]] rider@cloud:~$ glxinfo -display :0 [[BR]] {{{ #It seems that the 3D accerlation works fine without any trouble. name of display: :0.0 display: :0 screen: 0 direct rendering: Yes server glx vendor string: NVIDIA Corporation server glx version string: 1.4 server glx extensions: GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, GLX_EXT_texture_from_pixmap, GLX_ARB_create_context, GLX_ARB_multisample, GLX_NV_float_buffer, GLX_ARB_fbconfig_float, GLX_EXT_framebuffer_sRGB client glx vendor string: NVIDIA Corporation client glx version string: 1.4 client glx extensions: GLX_ARB_get_proc_address, GLX_ARB_multisample, GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_EXT_import_context, GLX_SGI_video_sync, GLX_NV_swap_group, GLX_NV_video_out, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGI_swap_control, GLX_ARB_create_context, GLX_NV_float_buffer, GLX_ARB_fbconfig_float, GLX_EXT_fbconfig_packed_float, GLX_EXT_texture_from_pixmap, GLX_EXT_framebuffer_sRGB, GLX_NV_present_video, GLX_NV_multisample_coverage GLX version: 1.3 GLX extensions: GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, GLX_EXT_texture_from_pixmap, GLX_ARB_create_context, GLX_ARB_multisample, GLX_NV_float_buffer, GLX_ARB_fbconfig_float, GLX_EXT_framebuffer_sRGB, GLX_ARB_get_proc_address OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: GeForce 9800 GT/PCI/SSE2 OpenGL version string: 3.0.0 NVIDIA 180.44 OpenGL shading language version string: 1.30 NVIDIA via Cg compiler }}} == 1.3 NVIDIA Driver !HowTo !OnXenKernel == ''In this case,we pick the driver Ver.180.22/180.44 x86_64 for Xen_Lustre kernel.''[[BR]] # Test1- Success [[BR]] rider@cloud:~/nvidia/driver$ export IGNORE_XEN_PRESENCE=1 [[BR]] rider@cloud:~/nvidia/driver$ export SYSSRC=/lib/modules/2.6.22.9/source [[BR]] rider@cloud:~/nvidia/driver$ export SYSOUT=/lib/modules/2.6.22.9/build [[BR]] rider@cloud:~/nvidia/driver$ sudo IGNORE_XEN_PRESENCE=1 CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM" ./NVIDIA-Linux-x86_64-180.44-pkg2.run --kernel-source-path=/usr/src/linux/ [[BR]] rider@cloud:~$ sudo modprobe -l | grep nv {{{ /lib/modules/2.6.22.9/kernel/drivers/video/nvidia.ko }}} rider@cloud:~/nvidia/driver$ sudo modprobe nvidia [[BR]] rider@cloud:~/nvidia/driver$ dmesg | grep NVIDIA [[BR]] {{{ NV Driver: 180.22 NVRM: loading NVIDIA UNIX x86_64 Kernel Module 180.22 Tue Jan 6 09:15:58 PST 2009 NV Driver: 180.44 NVRM: loading NVIDIA UNIX x86_64 Kernel Module 180.44 Tue Mar 24 05:46:32 PST 2009 }}} # Test2- Testing failure [[BR]] rider@cloud:~/nvidia/driver$ export IGNORE_XEN_PRESENCE=1 [[BR]] rider@cloud:~/nvidia/driver$ export SYSSRC=/lib/modules/2.6.22.9/source [[BR]] rider@cloud:~/nvidia/driver$ export SYSOUT=/lib/modules/2.6.22.9/build [[BR]] rider@cloud:~/nvidia/driver$ ./NVIDIA-Linux-x86_64-180.22-pkg2.run --extract-only [[BR]] rider@cloud:~/nvidia/driver$ cd ./NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/ [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM" make SYSSRC=/lib/modules/2.6.22.9/source SYSOUT=/lib/modules/2.6.22.9/build module [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ mkdir -p /lib/modules/2.6.22.9/extra [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo cp nvidia.ko /lib/modules/2.6.22.9/extra/ [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo depmod -a [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo modprobe nvidia [[BR]] {{{ errMsg: nvidia: Unknown symbol __phys_addr }}} PS: Test1 & Test2 Modified File [[BR]] {{{ #Kernel Source (Test1) /usr/src/linux/include/asm/smp.h /usr/src/linux/include/xen/interface/memory.h #NVIDIA Source (Test2) NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv.c NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-vm.c NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/conftest.sh NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/Makefile.kbuild NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-linux.h NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/os-interface.c NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-linux.h_old NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/conftest.sh_old }}} == 1.4 NVIDIA GPU !StatusCheck == rock@cloud:~$ sudo nvidia-xconfig -query-gpu-info [[BR]] {{{ # GPU Status check Number of GPUs: 1 GPU #0: Name : GeForce 9800 GT PCI BusID : PCI:1:0:0 Number of Display Devices: 1 Display Device 0 (CRT-0): EDID Name : ViewSonic VA721 Minimum HorizSync : 30.000 kHz Maximum HorizSync : 82.000 kHz Minimum VertRefresh : 50 Hz Maximum VertRefresh : 85 Hz Maximum PixelClock : 140.000 MHz Maximum Width : 1280 pixels Maximum Height : 1024 pixels Preferred Width : 1280 pixels Preferred Height : 1024 pixels Preferred VertRefresh : 60 Hz Physical Width : 340 mm Physical Height : 270 mm }}} rock@cloud:~$ sudo nvidia-smi [[BR]] {{{ Gpus found in probe: Found Gpuid 0x1000 Attaching all probed Gpus...OK Getting unit information...OK Getting all static information.. }}} = Part 2 Xen PCI Express configuration !HowTo = == 2.1 Xen_Kernel_config == {{{ CONFIG_XEN_PCIDEV_FRONTEND=y CONFIG_XEN_PCIDEV_BACKEND=y CONFIG_XEN_PCIDEV_BACKEND_PASS is not set CONFIG_XEN_PCIDEV_BACKEND_VPCI=y CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set }}} == 2.2 DEV_IDs confirmation == rider@cloud:~$ sudo lspci -vvn [[BR]] {{{ 01:00.0 0300: 10de:0605 (rev a2) ---> vendorID + DEV_IDs Subsystem: 1043:82a0 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- /sys/bus/pci/drivers/nvidia/unbind [[BR]] # Give the dev_ids to pciback, and give it a new slot then bind.[[BR]] root@cloud:~$ echo -n "0000:01:00.0" > /sys/bus/pci/drivers/pciback/new_slot [[BR]] root@cloud:~$ echo -n "0000:01:00.0" > /sys/bus/pci/drivers/pciback/bind [[BR]] # You can use an initialization script to invoke the PCIE device at startup. [[BR]] root@cloud:~$ cat /sys/bus/pci/drivers/pciback/slots [[BR]] {{{ 0000:01:00.0 }}} # Caution: Make sure that the device is not controlled by any driver: there should be no driver symlink for nvidia. [[BR]] {{{ PATH: /sys/bus/pci/devices/0000:01:00.0/ driver -> ../../../../bus/pci/drivers/nvidia ---> This symlink shouldn't exist. }}} === 2.3.3 Permissive Flag === rider@cloud:~$ sudo vim /etc/xen/xend-pci-permissive.sxp [[BR]] {{{ (unconstrained_dev_ids #('0123:4567:89AB:CDEF') ('0000:01:00.0') ) }}} === 2.3.4 User-space Quirks === rider@cloud:~$ sudo vim /etc/xen/xend-pci-quirks.sxp [[BR]] {{{ (pci_ids # Entries are formated as follows: # :[::] ('10de:0605' # NVIDIA 9800GT ) ) }}} === 2.3.5 PCI Frontend Configuration === rider@cloud:~$ sudo vim /etc/xen/vm01.cfg [[BR]] {{{ # We create a new virtual machine named "vm01",and pci_ids configuration example is as below. # In this case, we take the "PCI Express" deviceID for example. pci =['01:00.0'] }}} {{{ 01:00.0 --> PCI Express 00:01.0 --> PCI bridge: Intel Corporation 82Q35 Express PCI Express Root Port 00:1d.0 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller 00:1d.1 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller 00:1d.2 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller 00:1d.7 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller }}} rider@cloud:/etc/xen$ sudo xm create vm01.cfg[[BR]] rider@cloud:/etc/xen$ dmesg | grep pciback [[BR]] {{{ pciback 0000:01:00.0: seizing device pciback: vpci: 0000:01:00.0: assign to virtual slot 0 }}} rider@cloud:~$ sudo xm console vm01 [[BR]] vm01:~# dmesg | grep pci [[BR]] {{{ pcifront pci-0: Installing PCI frontend pcifront pci-0: Creating PCI Frontend Bus 0000:00 }}} = Part 3 Running CUDA on Xen !HowTo = == 3.1 Create a virtual machine for CUDA == rider@cloud:~$ sudo vim /etc/xen-tools/xen-tools.conf [[BR]] {{{ dir = /home install-method = debootstrap size = 6Gb # Disk image size. memory = 512Mb # Memory size swap = 128Mb # Swap size fs = ext3 # use the EXT3 filesystem for the disk image. dist = hardy # Default distribution to install. ---> For CUDA Support (Ubuntu 8.0.4) image = sparse # Specify sparse vs. full disk images. gateway = 140.XXX.XXX.XXX netmask = 255.255.255.0 broadcast = 140.XXX.XXX.XXX kernel = /boot/vmlinuz-`uname -r` initrd = /boot/initrd.img-`uname -r` mirror = http://gb.archive.ubuntu.com/ubuntu/ ext3_options = noatime,nodiratime,errors=remount-ro ext2_options = noatime,nodiratime,errors=remount-ro xfs_options = defaults reiser_options = defaults }}} rider@cloud:~$ sudo xen-create-image --hostname cuda --ip 140.XXX.XXX.XXX [[BR]] == 3.2 Running CUDA Example on !VirtualMachine == Step1: [[BR]] # !VirtualMachine startup [[BR]] rider@cloud:~$ sudo xm create cuda.cfg [[BR]] Step2: [[BR]] # Remote login [[BR]] rider@350Z:~$ ssh 140.xxx.xxx.xxx [[BR]] # Local login [[BR]] rider@cloud:~$ sudo xm console cuda [[BR]] Step3: [[BR]] # NVIDIA CUDA toolkit & sdk installation. Reference: Chapter1: 1.1 - Basic Environment [[BR]] Step4: [[BR]] # Build your own cuda project or cuda example running test. Reference: Chapter3: 3.3 - Running CUDA Example on Xen [[BR]] # Example:Device Bandwidth [[BR]] rider@cuda:/usr/local/NVIDIA_CUDA_SDK/bin/linux/release$ sudo ./bandwidthTest [[BR]] {{{ (Running on Xen_VirtualMachine) Device 0: "GeForce 9800 GT" Quick Mode Host to Device Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 31999998.0 Quick Mode Device to Host Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 320000000.0 Quick Mode Device to Device Bandwidth . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 640000000.0 &&&& Test PASSED }}} # Example:Device Query [[BR]] rider@cuda:/usr/local/NVIDIA_CUDA_SDK/bin/linux/release$ sudo ./deviceQuery [[BR]] {{{ (Running on Xen_VirtualMachine) Device 0: "GeForce 9800 GT" Major revision number: 0 Minor revision number: 0 Total amount of global memory: 6385920 bytes Number of multiprocessors: 11007 Number of cores: 88056 Total amount of constant memory: 6385872 bytes Total amount of shared memory per block: 3236702400 bytes Total number of registers available per block: 6385904 Warp size: 0 Maximum number of threads per block: 0 Maximum sizes of each dimension of a block: 0 x 6385808 x 0 Maximum sizes of each dimension of a grid: 0 x 0 x 2 Maximum memory pitch: 3234490924 bytes Texture alignment: 3236702608 bytes Clock rate: 0.00 GHz Concurrent copy and execution: Yes }}} == 3.3 Running CUDA Example on Xen == '''Important:'''[[BR]] In this case, We have to use "gcc-4.1" & "g++-4.1" instead of "gcc-4.3" to avoid getting stdio error.[[BR]] {{{ /usr/include/bits/stdio2.h(...): error: identifier "__builtin_va_arg_pack" is undefined }}} Example !HowTo : [[BR]] rider@cloud:~/opt/NVIDIA_CUDA_SDK$ sudo make [[BR]] rider@cloud:~/opt/NVIDIA_CUDA_SDK$ cd ./bin/linux/release/ [[BR]] rider@cloud:~/opt/NVIDIA_CUDA_SDK/bin/linux/release$ ./bandwidthTest [[BR]] rider@cloud:~/opt/NVIDIA_CUDA_SDK/bin/linux/release$ ./deviceQuery [[BR]] ### Demo Example ### {{{ (Running on Xen + Lustre Kernel) Running on...... device 0:GeForce 9800 GT Quick Mode Host to Device Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 1574.6 Quick Mode Device to Host Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 1187.9 Quick Mode Device to Device Bandwidth . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 41442.7 &&&& Test PASSED Press ENTER to exit... }}} {{{ (Running on Xen + Lustre Kernel) There is 1 device supporting CUDA Device 0: "GeForce 9800 GT" Major revision number: 1 Minor revision number: 1 Total amount of global memory: 1073414144 bytes Number of multiprocessors: 14 Number of cores: 112 Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 8192 Warp size: 32 Maximum number of threads per block: 512 Maximum sizes of each dimension of a block: 512 x 512 x 64 Maximum sizes of each dimension of a grid: 65535 x 65535 x 1 Maximum memory pitch: 262144 bytes Texture alignment: 256 bytes Clock rate: 1.51 GHz Concurrent copy and execution: Yes }}} = Part 4 Running 3D-Applications on Xen by using Virtualizing OpenGL = == 4.1 Build VNC for Dom0 remote-desktop == # Try to set the locale environment variables as you met the locale problem. [[BR]] # This section provides the solution for those who wants to surf the Dom0 desktop. [[BR]] # Actually, you can use tightvnc or turbovnc instead of gdm. [[BR]] rider@cloud:~$ sudo vim /etc/profile [[BR]] {{{ # Locale export LANGUAGE="en_US.UTF-8" export LC_ALL="en_US.UTF-8" export LANG="en_US.UTF-8" }}} rider@cloud:~$ source /etc/profile [[BR]] rider@cloud:~$ sudo dpkg-reconfigure locales [[BR]] rider@cloud:~$ sudo apt-get install autoconf automake build-essential gcc htop iptraf make libtool gdm sun-java6-jdk xdebconfigurator xfonts-100dpi xfonts-75dpi xfs xfonts-base xutils-dev tightvncserver ubuntu-desktop[[BR]] rider@cloud:~$ sudo vim /etc/gdm/gdm.conf [[BR]] {{{ DisallowTCP=false }}} {{{ # VNC Server [server-VNC] name=VNC server command=/usr/bin/Xvnc -geometry 1024x768 -depth 24 flexible=true }}} {{{ rider@cloud:~$ sudo perl -pi.bak -e 's/^0=Standard/0=VNC/g' /etc/gdm/gdm.conf }}} rider@cloud:~$ sudo /etc/init.d/gdm restart [[BR]] rider@cloud:~$ netstat -tunlp [[BR]] {{{ Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN - ---> Xvnc port tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN 3877/Xtightvnc tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:6001 0.0.0.0:* LISTEN 3877/Xtightvnc tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN - tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 ::1:631 :::* LISTEN - udp 0 0 0.0.0.0:32769 0.0.0.0:* - udp 0 0 0.0.0.0:5353 0.0.0.0:* - }}} === 4.1.1 Kernel Module === rider@cloud:~$ lsmod (Make sure that the "nvidia" module is loaded, and you can also try to load the nvidiafb.) [[BR]] {{{ Module Size Used by bridge 55080 0 llc 10544 1 bridge iptable_filter 6784 0 ip_tables 20824 1 iptable_filter x_tables 18952 1 ip_tables loop 19076 0 nvidia 8111512 36 i2c_core 24960 1 nvidia e1000 120000 0 e1000e 131500 0 fuse 43312 3 }}} == 4.2 Build VNC for DomU remote-desktop == # Try to set the locale environment variables as you met the locale problem. [[BR]] # This section provides the solution for those who wants to surf the Dom0 desktop. [[BR]] # Actually, you can use tightvnc or turbovnc instead of gdm. [[BR]] rider@nv:~$ sudo vim /etc/profile [[BR]] {{{ # Locale export LANGUAGE="en_US.UTF-8" export LC_ALL="en_US.UTF-8" export LANG="en_US.UTF-8" }}} rider@nv:~$ source /etc/profile [[BR]] rider@nv:~$ sudo dpkg-reconfigure locales [[BR]] rider@nv:~$ sudo apt-get install autoconf automake build-essential gcc htop iptraf make libtool gdm sun-java6-jdk xdebconfigurator xfonts-100dpi xfonts-75dpi xfs xfonts-base xutils-dev tightvncserver ubuntu-desktop[[BR]] rider@nv:~$ sudo vim /etc/gdm/gdm.conf [[BR]] {{{ DisallowTCP=false }}} {{{ # VNC Server [server-VNC] name=VNC server command=/usr/bin/Xvnc -geometry 1024x768 -depth 24 flexible=true }}} {{{ rider@nv:~$ sudo perl -pi.bak -e 's/^0=Standard/0=VNC/g' /etc/gdm/gdm.conf }}} rider@nv:~$ sudo /etc/init.d/gdm restart [[BR]] == 4.3 Build VMGL !HowTo == === 4.3.1 VMGL installation on Dom0 !HowTo === rider@cloud:~/vmgl$ sudo apt-get install build-essential mesa-common-dev libglu1-mesa-dev mesa-utils libxmu-headers libxmu6 libxmu-dev zlib1g-dev libjpeg62 libjpeg62-dev xutils-dev libxaw-headers libxaw7 libxaw7-dev libxext6 libxext-dev rxvt lwm xauth xvfb xfonts-100dpi xfonts-75dpi culmus xfonts-scalable xfonts-base [[BR]] rider@cloud:~/vmgl$ wget !http://www.cs.toronto.edu/~andreslc/software/vmgl-0.1.tar.bz2 [[BR]] rider@cloud:~/vmgl$ tar jxvf vmgl-0.1.tar.bz2 [[BR]] rider@cloud:~/vmgl$ cd ./vmgl.hg/tightvnc/ [[BR]] rider@cloud:~/vmgl/vmgl.hg/tightvnc$ patch -p0 < ../../tightvnc-1.2.9-amd64support.patch [[BR]] rider@cloud:~/vmgl$ cd .. [[BR]] rider@cloud:~/vmgl$ sudo ln -sf /usr/bin/make /usr/bin/gmake (if necessary) [[BR]] rider@cloud:~/vmgl/vmgl.hg$ make -j 4 [[BR]] rider@cloud:~/vmgl/vmgl.hg$ sudo make install-host [[BR]] rider@cloud:~$ xauth [[BR]] {{{ # Set the authority for remote guest. Using authority file /home/rider/.Xauthority xauth> add guest/unix:10 MIT-MAGIC-COOKIE-1 ec0ffd387888b9749d55f88031505888 xauth> add guest/unix:1 MIT-MAGIC-COOKIE-1 6824789b4ce0ac5743aeb57fd3ef8f5b xauth> exit }}} === 4.3.2 VMGL installation on DomU or Guest OS !HowTo === rider@guest:~/vmgl$ sudo apt-get install build-essential mesa-common-dev libglu1-mesa-dev mesa-utils libxmu-headers libxmu6 libxmu-dev zlib1g-dev libjpeg62 libjpeg62-dev xutils-dev libxaw-headers libxaw7 libxaw7-dev libxext6 libxext-dev rxvt lwm xauth xvfb xfonts-100dpi xfonts-75dpi culmus xfonts-scalable xfonts-base [[BR]] {{{ # Desgin for lightweight basic windows support xfonts-base xfonts-100dpi xfonts-75dpi -> fonts support rxvt -> x-terminal-emulator for VNC lwm -> x-window-manager for VNC }}} rider@guest:~/vmgl$ wget !http://www.cs.toronto.edu/~andreslc/software/vmgl-0.1.tar.bz2 [[BR]] rider@guest:~/vmgl$ tar jxvf vmgl-0.1.tar.bz2 [[BR]] rider@guest:~/vmgl$ cd ./vmgl.hg/tightvnc/ [[BR]] rider@guest:~/vmgl/vmgl.hg/tightvnc$ patch -p0 < ../../tightvnc-1.2.9-amd64support.patch [[BR]] rider@guest:~/vmgl$ cd .. [[BR]] rider@guest:~/vmgl$ sudo ln -sf /usr/bin/make /usr/bin/gmake (if necessary) [[BR]] rider@guest:~/vmgl/vmgl.hg$ sudo mkdir -p /usr/lib/xorg/modules/extensions (if necessary) [[BR]] rider@guest:~/vmgl/vmgl.hg$ make [[BR]] rider@guest:~/vmgl/vmgl.hg$ sudo make install-guest [[BR]] {{{ ## Fix the rgb_Path problem. rider@guest:~/vmgl/vmgl.hg$ sudo mkdir -p /usr/X11R6/lib/X11 rider@guest:~/vmgl/vmgl.hg$ sudo ln -sf /etc/X11/rgb.txt /usr/X11R6/lib/X11/rgb }}} == 4.4 VMGL !UsageNote == === 4.4.1 VMGL on Dom0 (host) === # Start the host VMGL stub-daemon. [[BR]] rider@cloud:~$ export DISPLAY=:0 [[BR]] rider@cloud:~$ stub-daemon [[BR]] rider@cloud:~$ netstat -tunlp [[BR]] {{{ Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:8002 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:7000 0.0.0.0:* LISTEN 29082/stub-daemon ---> VMGL stub-daemon udp 0 0 0.0.0.0:32769 0.0.0.0:* - udp 0 0 0.0.0.0:5353 0.0.0.0:* - }}} '''Using X forwarding''' [[BR]] rider@cloud:~$ ssh -X guest [[BR]] rider@guest:~$ glxinfo [[BR]] {{{ name of display: localhost:10.0 display: localhost:10 screen: 0 direct rendering: Yes server glx vendor string: VMGL server glx version string: 1.2 VMGL server glx extensions: client glx vendor string: VMGL client glx version string: 1.2 VMGL client glx extensions: GLX version: 1.3 GLX extensions: OpenGL extensions: GL_ARB_depth_texture, GL_ARB_fragment_program, GL_ARB_multisample, GL_ARB_multitexture, GL_ARB_occlusion_query, GL_ARB_point_parameters, GL_ARB_point_sprite, GL_ARB_shadow, GL_ARB_texture_border_clamp, GL_ARB_texture_compression, GL_ARB_texture_cube_map, GL_ARB_texture_env_add, GL_ARB_texture_env_combine, GL_EXT_texture_env_combine, GL_ARB_texture_env_dot3, GL_EXT_texture_env_dot3, GL_ARB_texture_mirrored_repeat, GL_ARB_texture_non_power_of_two, GL_ARB_transpose_matrix, GL_ARB_vertex_buffer_object, GL_ARB_vertex_program, GL_ARB_window_pos, GL_EXT_blend_color, GL_EXT_blend_minmax, GL_EXT_blend_func_separate, GL_EXT_blend_subtract, GL_EXT_texture_env_add, GL_EXT_fog_coord, GL_EXT_multi_draw_arrays, GL_EXT_secondary_color, GL_EXT_shadow_funcs, GL_EXT_stencil_wrap, GL_EXT_texture_cube_map, GL_EXT_texture_edge_clamp, GL_EXT_texture_filter_anisotropic, GL_EXT_texture_lod_bias, GL_EXT_texture_object, GL_EXT_texture3D, GL_EXT_bgra, GL_IBM_rasterpos_clip, GL_NV_fog_distance, GL_NV_fragment_program, GL_NV_register_combiners, GL_NV_register_combiners2, GL_NV_texgen_reflection, GL_NV_texture_rectangle, GL_NV_vertex_program, GL_NV_vertex_program1_1, GL_NV_vertex_program2, GL_SGIS_generate_mipmap, GL_CR_state_parameter, GL_CR_cursor_position, GL_CR_bounding_box, GL_CR_print_string, GL_CR_tilesort_info, GL_CR_synchronization, GL_CR_head_spu_name, GL_CR_performance_info, GL_CR_window_size, GL_CR_tile_info, GL_CR_saveframe, GL_CR_readback_barrier_size, GL_CR_server_id_sharing, GL_CR_server_matrix GLX_ARB_multisample OpenGL vendor string: H. Andres Lagar-Cavilla OpenGL renderer string: VMGL OpenGL version string: 1.5 VMGL 1.9 0x52 24 dc 0 32 0 r y . 8 8 8 0 4 24 8 16 16 16 16 2 1 None 0x53 24 dc 0 32 0 r y . 8 8 8 8 4 24 8 16 16 16 16 2 1 None 0x54 24 dc 0 32 0 r y . 8 8 8 0 4 24 8 16 16 16 16 4 1 None 0x55 24 dc 0 32 0 r y . 8 8 8 8 4 24 8 16 16 16 16 4 1 None 0x56 24 dc 0 32 0 r . . 8 8 8 0 4 24 8 16 16 16 16 2 1 None 0x57 24 dc 0 32 0 r . . 8 8 8 8 4 24 8 16 16 16 16 2 1 None 0x58 24 dc 0 32 0 r . . 8 8 8 0 4 24 8 16 16 16 16 4 1 None 0x59 24 dc 0 32 0 r . . 8 8 8 8 4 24 8 16 16 16 16 4 1 None 0x23 32 tc 0 32 0 r y . 8 8 8 0 4 24 8 16 16 16 16 0 0 None 0x5a 32 tc 0 32 0 r y . 8 8 8 8 4 24 8 16 16 16 16 0 0 None 0x5b 32 tc 0 32 0 r . . 8 8 8 0 4 24 8 16 16 16 16 0 0 None 0x5c 32 tc 0 32 0 r . . 8 8 8 8 4 24 8 16 16 16 16 0 0 None 0x5d 32 tc 0 32 0 r y . 8 8 8 0 4 24 0 16 16 16 16 0 0 None 0x5e 32 tc 0 32 0 r y . 8 8 8 8 4 24 0 16 16 16 16 0 0 None 0x5f 32 tc 0 32 0 r . . 8 8 8 0 4 24 0 16 16 16 16 0 0 None 0x60 32 tc 0 32 0 r . . 8 8 8 8 4 24 0 16 16 16 16 0 0 None 0x61 32 tc 0 32 0 r y . 8 8 8 0 4 0 0 16 16 16 16 0 0 None 0x62 32 tc 0 32 0 r y . 8 8 8 8 4 0 0 16 16 16 16 0 0 None 0x63 32 tc 0 32 0 r . . 8 8 8 0 4 0 0 16 16 16 16 0 0 None 0x64 32 tc 0 32 0 r . . 8 8 8 8 4 0 0 16 16 16 16 0 0 None 0x65 32 tc 0 32 0 r y . 8 8 8 0 4 24 0 16 16 16 16 2 1 None 0x66 32 tc 0 32 0 r y . 8 8 8 8 4 24 0 16 16 16 16 2 1 None 0x67 32 tc 0 32 0 r y . 8 8 8 0 4 24 0 16 16 16 16 4 1 None 0x68 32 tc 0 32 0 r y . 8 8 8 8 4 24 0 16 16 16 16 4 1 None 0x69 32 tc 0 32 0 r . . 8 8 8 0 4 24 0 16 16 16 16 2 1 None 0x6a 32 tc 0 32 0 r . . 8 8 8 8 4 24 0 16 16 16 16 2 1 None 0x6b 32 tc 0 32 0 r . . 8 8 8 0 4 24 0 16 16 16 16 4 1 None 0x6c 32 tc 0 32 0 r . . 8 8 8 8 4 24 0 16 16 16 16 4 1 None 0x6d 32 tc 0 32 0 r y . 8 8 8 0 4 24 8 16 16 16 16 2 1 None 0x6e 32 tc 0 32 0 r y . 8 8 8 8 4 24 8 16 16 16 16 2 1 None 0x6f 32 tc 0 32 0 r y . 8 8 8 0 4 24 8 16 16 16 16 4 1 None 0x70 32 tc 0 32 0 r y . 8 8 8 8 4 24 8 16 16 16 16 4 1 None 0x71 32 tc 0 32 0 r . . 8 8 8 0 4 24 8 16 16 16 16 2 1 None 0x72 32 tc 0 32 0 r . . 8 8 8 8 4 24 8 16 16 16 16 2 1 None 0x73 32 tc 0 32 0 r . . 8 8 8 0 4 24 8 16 16 16 16 4 1 None 0x74 32 tc 0 32 0 r . . 8 8 8 8 4 24 8 16 16 16 16 4 1 None }}} rider@guest:~$ glxgears [[BR]] {{{ 47819 frames in 5.0 seconds = 9563.678 FPS 46064 frames in 5.0 seconds = 9212.566 FPS 44584 frames in 5.0 seconds = 8916.581 FPS 44256 frames in 5.0 seconds = 8850.974 FPS 44688 frames in 5.0 seconds = 8937.528 FPS }}} === 4.4.2 VMGL on DomU or Guest OS === # Set the VMGL environment for VMGL guest. [[BR]] rider@guest:~$ sudo ln -sf /usr/share/fonts/X11/ /usr/X11R6/lib/X11/fonts (if necessary - fix font path) [[BR]] {{{ # FontPath: /usr/X11R6/lib/X11/fonts or /usr/share/fonts/X11 }}} rider@guest:~$ less /usr/X11R6/lib/X11/rgb.txt (if necessary - rgb path confirmation) [[BR]] rider@guest:~$ sudo vim /etc/profile [[BR]] {{{ GLSTUB=Cloud_IP(host_IP):7000 LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/vmgl LD_PRELOAD=/usr/local/lib/vmgl/libGL.so export GLSTUB LD_LIBRARY_PATH LD_PRELOAD }}} rider@guest:~$ source /etc/profile [[BR]] '''Using X forwarding''' [[BR]] rider@guest:~$ sudo vim /etc/ssh/sshd_config [[BR]] {{{ X11Forwarding yes }}} rider@guest:~$ sudo vim /etc/ssh/ssh_config [[BR]] {{{ ForwardX11 yes ForwardX11Trusted yes }}} rider@guest:~$ sudo /etc/init.d/ssh restart [[BR]] rider@guest:~$ vncserver -geometry 1024x768 -depth 24 :1 [[BR]] rider@guest:~$ vncserver -geometry 1024x768 -depth 24 :2 [[BR]] rider@guest:~$ netstat -tunlp [[BR]] {{{ (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN 10652/Xtightvnc ---> guest:1 tcp 0 0 0.0.0.0:5902 0.0.0.0:* LISTEN 10630/Xtightvnc tcp 0 0 0.0.0.0:6001 0.0.0.0:* LISTEN 10652/Xtightvnc ---> guest:2 tcp 0 0 0.0.0.0:6002 0.0.0.0:* LISTEN 10630/Xtightvnc tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN - tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 ::1:6010 :::* LISTEN - }}} === 4.4.3 Using VNCViewer === # All of the client users can run their 3D-apps with "Direct rendering" between Multi-VNC. [[BR]] rider@PC:~$ vncviewer guest:1 [[BR]] {{{ Run Your apps via command in rxvt }}} rider@NB:~$ vncviewer guest:2 [[BR]] {{{ Run Your apps via command in rxvt }}} == 4.5 Customize youre VM desktop == === 4.5.1 Enlightenment Installation === # You can download it from !WebSite: http://www.enlightenment.org/p.php?p=download&l=en [[BR]] rider@cloud:~$ sudo mkdir /opt/enlightenment [[BR]] rider@cloud:~/enlightenment/e16-0.16.8.15$ sudo apt-get install libimlib2 libimlib2-dev [[BR]] rider@cloud:~/enlightenment$ tar zxvf e16-0.16.8.15.tar.gz [[BR]] rider@cloud:~/enlightenment/e16-0.16.8.15$ ./configure --prefix=/opt/enlightenment --with-x --enable-modules [[BR]] rider@cloud:~/enlightenment/e16-0.16.8.15$ make -j 4 [[BR]] rider@cloud:~/enlightenment/e16-0.16.8.15$ sudo make install [[BR]] = Part 5 Comparing with Xen passthrough and VMGL = || || Xen Passthrough || VMGL || || method || 直接 (Xen支援) || 間接 (使用VMGL library) || || 資源提供 || 一張顯卡能給一台VM使用 (會 lock 顯卡資源) || 可以給多台 VM 使用 (效能隨著VM數目遞減) || '''Reference & Support:''' [[BR]] 1.NVIDIA CUDA: http://www.nvidia.com/object/cuda_home.html [[BR]] 2.openSUSE NVIDIA + Xen: http://en.opensuse.org/Use_Nvidia_driver_with_Xen [[BR]] 3.NVIDIA GPUs DEV_IDs: http://www.laptopvideo2go.com/forum/index.php?showtopic=7664 [[BR]] 4.pci_ids db: http://www.pcidatabase.com/ [[BR]] 5.Xen: assigning PCI devices to a domain: http://www.bestgrid.org/index.php/Xen:_assigning_PCI_devices_to_a_domain [[BR]] 6.Xen PCI Passthrough: http://www.wlug.org.nz/XenPciPassthrough [[BR]] 7.Xen Users' Manual v3.0: http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/ [[BR]] 8.VT-d in Xen: http://wiki.xensource.com/xenwiki/VTdHowTo [[BR]] 9.VMGL support: https://code.fluendo.com/elisa/trac/wiki [[BR]] 10.Latest NVIDIA driver: ftp://download.nvidia.com/XFree86/Linux-x86_64/ [[BR]]