= Xen GPU cluster = == Hardware == ||Machine|| Dell !OptiPlex 755 ||Node|| 9 nodes ||CPU|| Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz ||Memory|| 6GB/node ||Storage|| 160GB/node ||Video Card|| NVIDIA !GeForce 9800GT 1GB/node == Software == ||OS!#1|| Ubuntu 8.10 with Kernel: 2.6.27-11-server x86_64 ||OS!#2|| Ununtu 8.10 with Kernel: 2.6.22-9 x86_64 (Xen+Lustre patched kernel) [[BR]] = Part 1 Build essential environment = == 1.1 - Basic Environment == # NVIDIA CUDA driver # [[BR]] rock@cloud:~/nvidia/cuda$ wget http://developer.download.nvidia.com/compute/cuda/2_1/drivers/NVIDIA-Linux-x86_64-180.22-pkg2.run [[BR]] # NVIDIA CUDA toolkit # [[BR]] rock@cloud:~/nvidia/cuda$ wget http://developer.download.nvidia.com/compute/cuda/2_1/toolkit/cudatoolkit_2.1_linux64_ubuntu8.04.run [[BR]] # NVIDIA CUDA SDK # [[BR]] rock@cloud:~/nvidia/cuda$ wget http://developer.download.nvidia.com/compute/cuda/2_1/SDK/cuda-sdk-linux-2.10.1215.2015-3233425.run [[BR]] rock@cloud:~$ sudo apt-get install autoconf automake build-essential gcc make libtool initramfs-tools libxi6 libxi-dev libxmu6 libxmu-dev linux-kernel-devel linux-headers-2.6.27-11-server xserver-xorg-core xserver-xorg-dev [[BR]] rock@cloud:~$ sudo ln -sf /usr/src/linux-2.6.22 /usr/src/linux [[BR]] rock@cloud:~/nvidia/cuda$ sudo sh NVIDIA-Linux-x86_64-180.22-pkg2.run [[BR]] rock@cloud:~/nvidia/cuda$ sudo sh cudatoolkit_2.1_linux64_ubuntu8.04.run [[BR]] {{{ Enter install path (default /usr/local/cuda, '/cuda' will be appended): /usr/local/cuda }}} # Note: {{{ * Please make sure your PATH includes /usr/local/cuda/bin * Please make sure your LD_LIBRARY_PATH includes /usr/local/cuda/lib * or add /usr/local/cuda/lib to /etc/ld.so.conf and run ldconfig as root * Please read the release notes in /usr/local/cuda/doc/ * To uninstall CUDA, delete /usr/local/cuda * Installation Complete }}} rock@cloud:~/nvidia/cuda$ sudo sh cuda-sdk-linux-2.10.1215.2015-3233425.run [[BR]] # Note: {{{ {{{ Enter install path (default /usr/local/cuda, '/cuda' will be appended): /usr/local/NVIDIA_CUDA_SDK }}} }}} {{{ Configuring SDK Makefile (/usr/local/NVIDIA_CUDA_SDK/common/common.mk)... * Please make sure your PATH includes /usr/local/cuda/bin * Please make sure your LD_LIBRARY_PATH includes /usr/local/cuda/lib * To uninstall the NVIDIA CUDA SDK, please delete /usr/local/NVIDIA_CUDA_SDK }}} rock@cloud:~$ sudo vim /etc/profile [[BR]] {{{ Add: export PATH=$PATH:/usr/local/cuda/bin }}} rock@cloud:~$ source /etc/profile [[BR]] rock@cloud:~$ sudo vim /etc/ld.so.conf [[BR]] {{{ Add: include /usr/local/cuda/lib }}} rock@cloud:~$ sudo ldconfig [[BR]] {{{ /usr/local/cuda/lib: libcublasemu.so.2 -> libcublasemu.so.2.1 libcufftemu.so.2 -> libcufftemu.so.2.1 libcublas.so.2 -> libcublas.so.2.1 libcudart.so.2 -> libcudart.so.2.1 libcufft.so.2 -> libcufft.so.2.1 }}} == 1.2 NVIDIA Driver !HowTo !OnNoneXenKernel == # Rock said that the unknown identification of the VGA device might be the "pciids" problem. [[BR]] Sol1: [[BR]] rock@cloud:~$ sudo update-pciids [[BR]] Sol2: [[BR]] rock@cloud:~$ wget http://pciids.sourceforge.net/v2.2/pci.ids [[BR]] rock@cloud:~$ sudo cp pci.ids /usr/share/misc/ [[BR]] rock@cloud:~$ sudo lspci -v -v [[BR]] {{{ 01:00.0 VGA compatible controller: nVidia Corporation GeForce 9800 GT (rev a2) Subsystem: ASUSTeK Computer Inc. Device 82a0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [128] Power Budgeting Capabilities: [600] Vendor Specific Information Kernel driver in use: nvidia Kernel modules: nvidia, nvidiafb }}} rock@cloud:~$ less /var/log/Xorg.0.log | grep nVidia [[BR]] {{{ (--) PCI:*(0@1:0:0) nVidia Corporation GeForce 9800 GT rev 162, Mem @ 0xfd000000/16777216, 0xd0000000/268435456, 0xfa000000/33554432, I/O @ 0x0000dc80/128, BIOS @ 0x????????/131072 }}} rock@cloud:~$ less /usr/share/misc/pci.ids | grep 9800 [[BR]] {{{ 0601 GeForce 9800 GT 512 0604 GeForce 9800 GX2 0605 GeForce 9800 GT 0612 GeForce 9800 GTX 0613 GeForce 9800 GTX+ 0614 GeForce 9800 GT 0617 GeForce 9800M GTX 10de GeForce 9800M GTX }}} rock@cloud:~$ sudo Xorg -scanpci [[BR]] {{{ Probing for PCI devices (Bus:Device:Function) (0:0:0) unknown card (0x1028/0x0211) using a Intel Corporation DRAM Controller (0:1:0) Intel Corporation PCI Express Root Port (0:3:0) unknown card (0x1028/0x0211) using a Intel Corporation MEI Controller (0:3:2) unknown card (0x1028/0x0211) using a Intel Corporation PT IDER Controller (0:3:3) unknown card (0x1028/0x0211) using a Intel Corporation Serial KT Controller (0:25:0) unknown card (0x1028/0x0211) using a Intel Corporation 82566DM-2 Gigabit Network Connection (0:26:0) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (0:26:1) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (0:26:7) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (0:27:0) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) HD Audio Controller (0:28:0) Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (0:29:0) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (0:29:1) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (0:29:2) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (0:29:7) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (0:30:0) Intel Corporation 82801 PCI Bridge (0:31:0) Intel Corporation LPC Interface Controller (0:31:2) unknown card (0x1028/0x0211) using a Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port SATA IDE Controller (0:31:3) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) SMBus Controller (0:31:5) unknown card (0x1028/0x0211) using a Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE Controller ---> (1:0:0) unknown card (0x1043/0x82a0) using an unknown chip (DeviceId 0x0605) from nVidia Corporation }}} rock@cloud:~$ sudo /etc/X11/xorg.conf [[BR]] {{{ # Allocate the BusID for the VGA Device Section "Device" Identifier "Device0" Driver "nvidia" BusID "PCI:1:0:0" VendorName "NVIDIA Corporation" BoardName "GeForce 9800 GT" Option "RenderAccel" "True" Option "UseEdidDpi" "False" EndSection }}} rock@cloud:~$ sudo glxinfo -display :0 {{{ #It seems that the 3D accerlation works fine without any trouble. name of display: :0.0 display: :0 screen: 0 direct rendering: Yes server glx vendor string: NVIDIA Corporation server glx version string: 1.4 OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: GeForce 9800 GT/PCI/SSE2 OpenGL version string: 3.0.0 NVIDIA 180.29 OpenGL shading language version string: 1.30 NVIDIA via Cg compiler }}} == 1.3 NVIDIA Driver !HowTo !OnXenKernel == ''In this case,we pick the driver Ver.180.22 x86_64 for Xen_Lustre kernel.''[[BR]] # Test1- Success [[BR]] rider@cloud:~/nvidia/driver$ export IGNORE_XEN_PRESENCE=1 [[BR]] rider@cloud:~/nvidia/driver$ export SYSSRC=/lib/modules/2.6.22.9/source [[BR]] rider@cloud:~/nvidia/driver$ export SYSOUT=/lib/modules/2.6.22.9/build [[BR]] rider@cloud:~/nvidia/driver$ sudo IGNORE_XEN_PRESENCE=1 ./NVIDIA-Linux-x86_64-180.22-pkg2.run --x-module-path=/usr/lib/xorg/modules/ --kernel-source-path=/usr/src/linux/ [[BR]] rider@cloud:~$ sudo modprobe -l | grep nv {{{ /lib/modules/2.6.22.9/kernel/drivers/video/nvidia.ko }}} rider@cloud:~/nvidia/driver$ sudo modprobe nvidia [[BR]] rider@cloud:~/nvidia/driver$ dmesg [[BR]] {{{ NVRM: loading NVIDIA UNIX x86_64 Kernel Module 180.22 Tue Jan 6 09:15:58 PST 2009 }}} # Test2- Testing [[BR]] rider@cloud:~/nvidia/driver$ export IGNORE_XEN_PRESENCE=1 [[BR]] rider@cloud:~/nvidia/driver$ export SYSSRC=/lib/modules/2.6.22.9/source [[BR]] rider@cloud:~/nvidia/driver$ export SYSOUT=/lib/modules/2.6.22.9/build [[BR]] rider@cloud:~/nvidia/driver$ ./NVIDIA-Linux-x86_64-180.22-pkg2.run --extract-only [[BR]] rider@cloud:~/nvidia/driver$ cd ./NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/ [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM" make SYSSRC=/lib/modules/2.6.22.9/source SYSOUT=/lib/modules/2.6.22.9/build module [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ mkdir -p /lib/modules/2.6.22.9/extra [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo cp nvidia.ko /lib/modules/2.6.22.9/extra/ [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo depmod -a [[BR]] rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo modprobe nvidia [[BR]] {{{ errMsg: nvidia: Unknown symbol __phys_addr }}} PS: Test1 & Test2 Modified File [[BR]] {{{ #Kernel Source (Test1) /usr/src/linux/include/asm/smp.h /usr/src/linux/include/xen/interface/memory.h #NVIDIA Source (Test2) NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv.c NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-vm.c NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/conftest.sh NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/Makefile.kbuild NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-linux.h NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/os-interface.c NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-linux.h_old NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/conftest.sh_old }}} == 1.4 NVIDIA GPU !StatusCheck == rock@cloud:~$ sudo nvidia-xconfig -query-gpu-info [[BR]] {{{ # GPU Status check Number of GPUs: 1 GPU #0: Name : GeForce 9800 GT PCI BusID : PCI:1:0:0 Number of Display Devices: 1 Display Device 0 (CRT-0): EDID Name : ViewSonic VA721 Minimum HorizSync : 30.000 kHz Maximum HorizSync : 82.000 kHz Minimum VertRefresh : 50 Hz Maximum VertRefresh : 85 Hz Maximum PixelClock : 140.000 MHz Maximum Width : 1280 pixels Maximum Height : 1024 pixels Preferred Width : 1280 pixels Preferred Height : 1024 pixels Preferred VertRefresh : 60 Hz Physical Width : 340 mm Physical Height : 270 mm }}} rock@cloud:~$ sudo nvidia-smi [[BR]] {{{ Gpus found in probe: Found Gpuid 0x1000 Attaching all probed Gpus...OK Getting unit information...OK Getting all static information.. }}} = Part 2 Xen PCI(Express) configuration !HowTo = == 2.1 DEV_IDs confirmation == rider@cloud:~$ lspci -vvn [[BR]] {{{ 01:00.0 0300: 10de:0605 (rev a2) ---> DEV_IDs Subsystem: 1043:82a0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: nvidia Kernel modules: nvidia }}} rider@cloud:~$ sudo vim /etc/xen/xend-pci-permissive.sxp [[BR]] {{{ (unconstrained_dev_ids #('0123:4567:89AB:CDEF') ('0000:01:00.0' '0000:00:01.0' '0000:00:1d.0' '0000:00:1d.1' '0000:00:1d.2' '0000:00:1d.7') ) }}} rider@cloud:~$ sudo vim /etc/xen/vm01.cfg [[BR]] {{{ # We add a new virtual machine named "vm01". pci = ['01:00.0', '00:01.0', '00:1d.0', '00:1d.1', '00:1d.2', '00:1d.7'] }}} {{{ 01:00.0 --> PCI Express 00:01.0 --> PCI bridge: Intel Corporation 82Q35 Express PCI Express Root Port 00:1d.0 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller 00:1d.1 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller 00:1d.2 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller 00:1d.7 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller }}} = Part 3 CUDA !HowTo = == 3.1 NVIDIA CUDA Example == For example: [[BR]] Choose an example from /opt/NVIDIA_CUDA_SDK/projects [[BR]] rock@cloud:~$ cd /opt/NVIDIA_CUDA_SDK/projects/bandwidthTest/ [[BR]] rock@cloud:~/opt/NVIDIA_CUDA_SDK/projects/bandwidthTest$ sudo make [[BR]] rock@cloud:~/opt/NVIDIA_CUDA_SDK/projects/bandwidthTest$ cd ../../bin/linux/release/ [[BR]] rock@cloud:~/opt/NVIDIA_CUDA_SDK/bin/linux/release$ ./bandwidthTest [[BR]] {{{ Running on...... device 0:GeForce 9800 GT Quick Mode Host to Device Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 1574.6 Quick Mode Device to Host Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 1187.9 Quick Mode Device to Device Bandwidth . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 41442.7 &&&& Test PASSED Press ENTER to exit... }}}