= Xen GPU Cluster =

== Hardware ==
||Machine|| Dell !OptiPlex 755
||Node|| 9 nodes
||CPU|| Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
||Memory|| 6GB/node
||Storage|| 160GB/node
||Video Card|| NVIDIA !GeForce 9800GT 1GB/node

== Software ==
||OS!#1|| Ubuntu 8.10 with Kernel: 2.6.28 x86_64 (non-xen-patched kernel)
||OS!#2|| Ubuntu 8.10 with Kernel: 2.6.22-9 x86_64 (Xen-3.3.1+Lustre patched kernel)
[[BR]]

= Part 1 Build essential environment =

== 1.1 - Basic Environment ==

# NVIDIA CUDA driver # [[BR]]
rock@cloud:~/nvidia/cuda$ wget http://developer.download.nvidia.com/compute/cuda/2_1/drivers/NVIDIA-Linux-x86_64-180.22-pkg2.run [[BR]]

# NVIDIA CUDA toolkit # [[BR]]
rock@cloud:~/nvidia/cuda$ wget http://developer.download.nvidia.com/compute/cuda/2_1/toolkit/cudatoolkit_2.1_linux64_ubuntu8.04.run [[BR]]

# NVIDIA CUDA SDK # [[BR]]
rock@cloud:~/nvidia/cuda$ wget http://developer.download.nvidia.com/compute/cuda/2_1/SDK/cuda-sdk-linux-2.10.1215.2015-3233425.run [[BR]]
rock@cloud:~$ sudo apt-get install autoconf automake build-essential gcc make libtool initramfs-tools libxi6 libxi-dev libxmu6 libxmu-dev linux-kernel-devel linux-headers-2.6.27-11-server xserver-xorg-core xserver-xorg-dev [[BR]]
rock@cloud:~$ sudo ln -sf /usr/src/linux-2.6.22 /usr/src/linux [[BR]]
rock@cloud:~/nvidia/cuda$ sudo sh NVIDIA-Linux-x86_64-180.22-pkg2.run [[BR]]
rock@cloud:~/nvidia/cuda$ sudo sh cudatoolkit_2.1_linux64_ubuntu8.04.run [[BR]]
{{{
Enter install path (default /usr/local/cuda, '/cuda' will be appended): /usr/local/cuda
}}}

# Note:
{{{
* Please make sure your PATH includes /usr/local/cuda/bin
* Please make sure your LD_LIBRARY_PATH includes /usr/local/cuda/lib
*   or add /usr/local/cuda/lib to /etc/ld.so.conf and run ldconfig as root

* Please read the release notes in /usr/local/cuda/doc/

* To uninstall CUDA, delete /usr/local/cuda
* Installation Complete
}}}

rock@cloud:~/nvidia/cuda$ sudo sh cuda-sdk-linux-2.10.1215.2015-3233425.run [[BR]]

# Note:
{{{
{{{
Enter install path (default /usr/local/cuda, '/cuda' will be appended): /usr/local/NVIDIA_CUDA_SDK
}}}

}}}
{{{
Configuring SDK Makefile (/usr/local/NVIDIA_CUDA_SDK/common/common.mk)...

* Please make sure your PATH includes /usr/local/cuda/bin
* Please make sure your LD_LIBRARY_PATH includes /usr/local/cuda/lib

* To uninstall the NVIDIA CUDA SDK, please delete /usr/local/NVIDIA_CUDA_SDK
}}}

rock@cloud:~$ sudo vim /etc/profile [[BR]]
{{{
Add:
export PATH=$PATH:/usr/local/cuda/bin
}}}
rock@cloud:~$ source /etc/profile [[BR]]
rock@cloud:~$ sudo vim /etc/ld.so.conf [[BR]]
{{{
Add:
/usr/local/cuda/lib
}}}
rock@cloud:~$ sudo ldconfig [[BR]]

== 1.2 NVIDIA Driver !HowTo !OnNoneXenKernel ==

# Rock said that the unknown identification of the VGA device might be the "pciids" problem. [[BR]]
Sol1: [[BR]]
rock@cloud:~$ sudo update-pciids <older version> [[BR]]
Sol2: [[BR]]
rock@cloud:~$ wget http://pciids.sourceforge.net/v2.2/pci.ids <latest version> [[BR]]
rock@cloud:~$ sudo cp pci.ids /usr/share/misc/ [[BR]]
rock@cloud:~$ sudo lspci -v -v [[BR]]

{{{
01:00.0 VGA compatible controller: nVidia Corporation GeForce 9800 GT (rev a2)
	Subsystem: ASUSTeK Computer Inc. Device 82a0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M]
	Region 5: I/O ports at dc80 [size=128]
	[virtual] Expansion ROM at fea00000 [disabled] [size=128K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <1us
			ClockPM- Suprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100] Virtual Channel <?>
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [600] Vendor Specific Information <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidia, nvidiafb
}}}

rock@cloud:~$ less /var/log/Xorg.0.log | grep nVidia [[BR]]
{{{
(--) PCI:*(0@1:0:0) nVidia Corporation GeForce 9800 GT rev 162, Mem @ 0xfd000000/16777216, 0xd0000000/268435456, 0xfa000000/33554432, I/O @ 0x0000dc80/128, BIOS @ 0x????????/131072
}}}

rock@cloud:~$ sudo vim /etc/X11/xorg.conf [[BR]]
{{{
# Allocate the BusID for the VGA Device
Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    BusID          "PCI:1:0:0"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce 9800 GT"
    Option         "RenderAccel" "True"
    Option         "UseEdidDpi" "False"
EndSection
}}}

rider@cloud:~$ export DISPLAY=:0 [[BR]]
rider@cloud:~$ sudo glxinfo -display :0 [[BR]]
{{{
#It seems that the 3D accerlation works fine without any trouble.
name of display: :0.0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, 
    GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, 
    GLX_EXT_texture_from_pixmap, GLX_ARB_create_context, GLX_ARB_multisample, 
    GLX_NV_float_buffer, GLX_ARB_fbconfig_float, GLX_EXT_framebuffer_sRGB
client glx vendor string: NVIDIA Corporation
client glx version string: 1.4
client glx extensions:
    GLX_ARB_get_proc_address, GLX_ARB_multisample, GLX_EXT_visual_info, 
    GLX_EXT_visual_rating, GLX_EXT_import_context, GLX_SGI_video_sync, 
    GLX_NV_swap_group, GLX_NV_video_out, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, 
    GLX_SGI_swap_control, GLX_ARB_create_context, GLX_NV_float_buffer, 
    GLX_ARB_fbconfig_float, GLX_EXT_fbconfig_packed_float, 
    GLX_EXT_texture_from_pixmap, GLX_EXT_framebuffer_sRGB, 
    GLX_NV_present_video, GLX_NV_multisample_coverage
GLX version: 1.3
GLX extensions:
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, 
    GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, 
    GLX_EXT_texture_from_pixmap, GLX_ARB_create_context, GLX_ARB_multisample, 
    GLX_NV_float_buffer, GLX_ARB_fbconfig_float, GLX_EXT_framebuffer_sRGB, 
    GLX_ARB_get_proc_address
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce 9800 GT/PCI/SSE2
OpenGL version string: 3.0.0 NVIDIA 180.44
OpenGL shading language version string: 1.30 NVIDIA via Cg compiler
}}}

== 1.3 NVIDIA Driver !HowTo !OnXenKernel ==

''In this case,we pick the driver Ver.180.22/180.44 x86_64 for Xen_Lustre kernel.''[[BR]]
# Test1- Success [[BR]]
rider@cloud:~/nvidia/driver$ export IGNORE_XEN_PRESENCE=1 [[BR]]
rider@cloud:~/nvidia/driver$ export SYSSRC=/lib/modules/2.6.22.9/source [[BR]]
rider@cloud:~/nvidia/driver$ export SYSOUT=/lib/modules/2.6.22.9/build [[BR]]
rider@cloud:~/nvidia/driver$ sudo IGNORE_XEN_PRESENCE=1 CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM" ./NVIDIA-Linux-x86_64-180.44-pkg2.run --kernel-source-path=/usr/src/linux/ [[BR]]
rider@cloud:~$ sudo modprobe -l | grep nv
{{{
/lib/modules/2.6.22.9/kernel/drivers/video/nvidia.ko
}}}
rider@cloud:~/nvidia/driver$ sudo modprobe nvidia [[BR]]
rider@cloud:~/nvidia/driver$ dmesg | grep NVIDIA [[BR]]
{{{
NV Driver: 180.22
NVRM: loading NVIDIA UNIX x86_64 Kernel Module  180.22  Tue Jan  6 09:15:58 PST 2009

NV Driver: 180.44
NVRM: loading NVIDIA UNIX x86_64 Kernel Module  180.44  Tue Mar 24 05:46:32 PST 2009
}}}

# Test2- Testing failure [[BR]]
rider@cloud:~/nvidia/driver$ export IGNORE_XEN_PRESENCE=1 [[BR]]
rider@cloud:~/nvidia/driver$ export SYSSRC=/lib/modules/2.6.22.9/source [[BR]]
rider@cloud:~/nvidia/driver$ export SYSOUT=/lib/modules/2.6.22.9/build [[BR]]
rider@cloud:~/nvidia/driver$ ./NVIDIA-Linux-x86_64-180.22-pkg2.run --extract-only [[BR]]
rider@cloud:~/nvidia/driver$ cd ./NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/ [[BR]]
rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM" make SYSSRC=/lib/modules/2.6.22.9/source SYSOUT=/lib/modules/2.6.22.9/build module [[BR]]
rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ mkdir -p /lib/modules/2.6.22.9/extra [[BR]]
rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo cp nvidia.ko /lib/modules/2.6.22.9/extra/ [[BR]]
rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo depmod -a [[BR]]
rider@cloud:~/nvidia/driver/NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv$ sudo modprobe nvidia [[BR]]
{{{
errMsg: nvidia: Unknown symbol __phys_addr
}}}

PS: Test1 & Test2 Modified File [[BR]]
{{{
#Kernel Source (Test1)
/usr/src/linux/include/asm/smp.h
/usr/src/linux/include/xen/interface/memory.h

#NVIDIA Source (Test2)
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv.c
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-vm.c
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/conftest.sh
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/Makefile.kbuild
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-linux.h
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/os-interface.c
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/nv-linux.h_old
NVIDIA-Linux-x86_64-180.22-pkg2/usr/src/nv/conftest.sh_old
}}}

== 1.4 NVIDIA GPU !StatusCheck ==

rock@cloud:~$ sudo nvidia-xconfig -query-gpu-info [[BR]]
{{{
# GPU Status check
Number of GPUs: 1

GPU #0:
  Name      : GeForce 9800 GT
  PCI BusID : PCI:1:0:0

  Number of Display Devices: 1

  Display Device 0 (CRT-0):
     EDID Name             : ViewSonic VA721
     Minimum HorizSync     : 30.000 kHz
     Maximum HorizSync     : 82.000 kHz
     Minimum VertRefresh   : 50 Hz
     Maximum VertRefresh   : 85 Hz
     Maximum PixelClock    : 140.000 MHz
     Maximum Width         : 1280 pixels
     Maximum Height        : 1024 pixels
     Preferred Width       : 1280 pixels
     Preferred Height      : 1024 pixels
     Preferred VertRefresh : 60 Hz
     Physical Width        : 340 mm
     Physical Height       : 270 mm
}}}
rock@cloud:~$ sudo nvidia-smi [[BR]]
{{{
Gpus found in probe:
Found Gpuid 0x1000
Attaching all probed Gpus...OK
Getting unit information...OK
Getting all static information..
}}}

= Part 2 Xen PCI Express configuration !HowTo =

== 2.1 Xen_Kernel_config ==

{{{
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
}}}

== 2.2 DEV_IDs confirmation ==

rider@cloud:~$ sudo lspci -vvn [[BR]]
{{{
01:00.0 0300: 10de:0605 (rev a2) ---> ---> vendorID + DEV_IDs
	Subsystem: 1043:82a0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fd000000 (32-bit, non-prefetchable) [disabled] [size=16M]
	Region 1: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M]
	Region 3: Memory at fa000000 (64-bit, non-prefetchable) [disabled] [size=32M]
	Region 5: I/O ports at dc80 [disabled] [size=128]
	Expansion ROM at fea00000 [disabled] [size=128K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <1us
			ClockPM- Suprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: nvidia
	Kernel modules: nvidia, nvidiafb
}}}

== 2.3 PCI Backend Configuration ==

=== 2.3.1 Binding at Boot ===

# Important: You'll need to unload the NVIDIA driver before setting up the pci-backend. [[BR]] 
rider@cloud:~$ sudo modprobe -r nvidia [[BR]]
rider@cloud:~$ sudo vim /boot/grub/menu.lst [[BR]]
{{{
module  /boot/vmlinuz-2.6.22.9 root=UUID=d3fa560e-7071-46d8-a168-036f40960c7b ro console=tty0 pciback.hide=(0000:01:00.0)
}}}

=== 2.3.2 Late Binding ===

rider@cloud:~$ sudo modprobe -r nvidia [[BR]]
rider@cloud:~$ sudo vim /etc/modprobe.d/blacklist [[BR]]
{{{
# nvidia driver autoload-disabled
blacklist nvidia                                                                                                        
}}}
rider@cloud:~$ sudo su - [[BR]]
# Hide the device from dom0 so pciback can take control. [[BR]]
root@cloud:~$ echo -n "0000:01:00.0" > /sys/bus/pci/drivers/nvidia/unbind [[BR]]

# Give the dev_ids to pciback, and give it a new slot then bind.[[BR]]
root@cloud:~$ echo -n "0000:01:00.0" > /sys/bus/pci/drivers/pciback/new_slot [[BR]]
root@cloud:~$ echo -n "0000:01:00.0" > /sys/bus/pci/drivers/pciback/bind [[BR]]

# You can use an initialization script to invoke the PCIE device at startup. [[BR]]

root@cloud:~$ cat /sys/bus/pci/drivers/pciback/slots [[BR]]
{{{ 
0000:01:00.0
}}}

# Caution: Make sure that the device is not controlled by any driver: there should be no driver symlink for nvidia. [[BR]] 
{{{
PATH: /sys/bus/pci/devices/0000:01:00.0/
driver -> ../../../../bus/pci/drivers/nvidia ---> This symlink shouldn't exist.
}}}

=== 2.3.3 Permissive Flag ===

rider@cloud:~$ sudo vim /etc/xen/xend-pci-permissive.sxp [[BR]]
{{{
(unconstrained_dev_ids
     #('0123:4567:89AB:CDEF')
     ('0000:01:00.0')
)
}}}

=== 2.3.4 User-space Quirks ===

rider@cloud:~$ sudo vim /etc/xen/xend-pci-quirks.sxp [[BR]]
{{{
(pci_ids
   # Entries are formated as follows:  
   #     <vendor>:<device>[:<subvendor>:<subdevice>]

   ('10de:0605'   # NVIDIA 9800GT
   )
)
}}}

=== 2.3.5 PCI Frontend Configuration ===

rider@cloud:~$ sudo vim /etc/xen/vm01.cfg [[BR]]
{{{
# We create a new virtual machine named "vm01",and pci_ids configuration example is as below.
# In this case, we take the "PCI Express" deviceID for example.
pci =['01:00.0']
}}}

{{{
01:00.0 --> PCI Express
00:01.0 --> PCI bridge: Intel Corporation 82Q35 Express PCI Express Root Port
00:1d.0 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
00:1d.1 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
00:1d.2 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
00:1d.7 --> USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
}}}

rider@cloud:/etc/xen$ sudo xm create vm01.cfg[[BR]]
rider@cloud:/etc/xen$ dmesg | grep pciback [[BR]]
{{{
pciback 0000:01:00.0: seizing device
pciback: vpci: 0000:01:00.0: assign to virtual slot 0
}}}

rider@cloud:~$ sudo xm console vm01 [[BR]]
vm01:~# dmesg | grep pci [[BR]]
{{{
pcifront pci-0: Installing PCI frontend
pcifront pci-0: Creating PCI Frontend Bus 0000:00
}}}

= Part 3 Running CUDA on Xen !HowTo =

== 3.1 Create a virtual machine for CUDA ==

rider@cloud:~$ sudo vim /etc/xen-tools/xen-tools.conf [[BR]]
{{{
dir = /home
install-method = debootstrap
size   = 6Gb      # Disk image size.
memory = 512Mb    # Memory size
swap   = 128Mb    # Swap size
fs     = ext3     # use the EXT3 filesystem for the disk image.
dist   = hardy    # Default distribution to install. ---> For CUDA Support (Ubuntu 8.0.4)
image  = sparse   # Specify sparse vs. full disk images.
gateway   = 140.XXX.XXX.XXX
netmask   = 255.255.255.0
broadcast = 140.XXX.XXX.XXX
kernel      = /boot/vmlinuz-`uname -r`
initrd      = /boot/initrd.img-`uname -r`
mirror = http://gb.archive.ubuntu.com/ubuntu/
ext3_options   = noatime,nodiratime,errors=remount-ro
ext2_options   = noatime,nodiratime,errors=remount-ro
xfs_options    = defaults
reiser_options = defaults
}}}
rider@cloud:~$ sudo xen-create-image --hostname cuda --ip 140.XXX.XXX.XXX [[BR]]

== 3.2 Running CUDA Example on !VirtualMachine ==

Step1: [[BR]]
# !VirtualMachine startup [[BR]]
rider@cloud:~$ sudo xm create cuda.cfg [[BR]]

Step2: [[BR]]
# Remote login [[BR]]
rider@350Z:~$ ssh 140.xxx.xxx.xxx [[BR]]
# Local login [[BR]]
rider@cloud:~$ sudo xm console cuda [[BR]]

Step3: [[BR]]
# NVIDIA CUDA toolkit & sdk installation. Reference: Chapter1: 1.1 - Basic Environment [[BR]]

Step4: [[BR]]
# Build your own cuda project or cuda example running test. Reference: Chapter3: 3.3 - Running CUDA Example on Xen [[BR]]

# Example:Device Bandwidth [[BR]]
rider@cuda:/usr/local/NVIDIA_CUDA_SDK/bin/linux/release$ sudo ./bandwidthTest  [[BR]]

{{{
(Running on Xen_VirtualMachine)
Device 0: "GeForce 9800 GT"
Quick Mode
Host to Device Bandwidth for Pageable memory
.
Transfer Size (Bytes)	Bandwidth(MB/s)
 33554432		31999998.0

Quick Mode
Device to Host Bandwidth for Pageable memory
.
Transfer Size (Bytes)	Bandwidth(MB/s)
 33554432		320000000.0

Quick Mode
Device to Device Bandwidth
.
Transfer Size (Bytes)	Bandwidth(MB/s)
 33554432		640000000.0

&&&& Test PASSED
}}}


# Example:Device Query [[BR]]
rider@cuda:/usr/local/NVIDIA_CUDA_SDK/bin/linux/release$ sudo ./deviceQuery [[BR]]

{{{
(Running on Xen_VirtualMachine)
Device 0: "GeForce 9800 GT"
  Major revision number:                         0
  Minor revision number:                         0
  Total amount of global memory:                 6385920 bytes
  Number of multiprocessors:                     11007
  Number of cores:                               88056
  Total amount of constant memory:               6385872 bytes
  Total amount of shared memory per block:       3236702400 bytes
  Total number of registers available per block: 6385904
  Warp size:                                     0
  Maximum number of threads per block:           0
  Maximum sizes of each dimension of a block:    0 x 6385808 x 0
  Maximum sizes of each dimension of a grid:     0 x 0 x 2
  Maximum memory pitch:                          3234490924 bytes
  Texture alignment:                             3236702608 bytes
  Clock rate:                                    0.00 GHz
  Concurrent copy and execution:                 Yes
}}}

== 3.3 Running CUDA Example on Xen ==

'''Important:'''[[BR]]
In this case, We have to use "gcc-4.1" & "g++-4.1" instead of "gcc-4.3" to avoid getting stdio error.[[BR]]
{{{
/usr/include/bits/stdio2.h(...): error: identifier "__builtin_va_arg_pack" is undefined
}}}

Example !HowTo : [[BR]]
rider@cloud:~/opt/NVIDIA_CUDA_SDK$ sudo make [[BR]]
rider@cloud:~/opt/NVIDIA_CUDA_SDK$ cd ./bin/linux/release/ [[BR]]
rider@cloud:~/opt/NVIDIA_CUDA_SDK/bin/linux/release$ ./bandwidthTest [[BR]]
rider@cloud:~/opt/NVIDIA_CUDA_SDK/bin/linux/release$ ./deviceQuery [[BR]]

### Demo Example ###
{{{
(Running on Xen + Lustre Kernel)
Running on......
      device 0:GeForce 9800 GT
Quick Mode
Host to Device Bandwidth for Pageable memory
.
Transfer Size (Bytes)	Bandwidth(MB/s)
 33554432		1574.6

Quick Mode
Device to Host Bandwidth for Pageable memory
.
Transfer Size (Bytes)	Bandwidth(MB/s)
 33554432		1187.9

Quick Mode
Device to Device Bandwidth
.
Transfer Size (Bytes)	Bandwidth(MB/s)
 33554432		41442.7

&&&& Test PASSED

Press ENTER to exit...
}}}
{{{
(Running on Xen + Lustre Kernel)
There is 1 device supporting CUDA

Device 0: "GeForce 9800 GT"
  Major revision number:                         1
  Minor revision number:                         1
  Total amount of global memory:                 1073414144 bytes
  Number of multiprocessors:                     14
  Number of cores:                               112
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       16384 bytes
  Total number of registers available per block: 8192
  Warp size:                                     32
  Maximum number of threads per block:           512
  Maximum sizes of each dimension of a block:    512 x 512 x 64
  Maximum sizes of each dimension of a grid:     65535 x 65535 x 1
  Maximum memory pitch:                          262144 bytes
  Texture alignment:                             256 bytes
  Clock rate:                                    1.51 GHz
  Concurrent copy and execution:                 Yes
}}}

= Part 4 Running 3D-Applications on Xen by using Virtualizing OpenGL =

== 4.1 Build VNC for Dom0 remote-desktop ==

# Try to set the locale environment variables as you met the locale problem. [[BR]]
# This section provides the solution for those who wants to surf the Dom0 desktop. [[BR]]
# Actually, you can use tightvnc or turbovnc instead of gdm. [[BR]]

rider@cloud:~$ sudo vim /etc/profile [[BR]]
{{{
# Locale
export LANGUAGE="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LANG="en_US.UTF-8"
}}}
rider@cloud:~$ source /etc/profile [[BR]]
rider@cloud:~$ sudo dpkg-reconfigure locales [[BR]]
rider@cloud:~$ sudo apt-get install autoconf automake build-essential gcc htop iptraf make libtool gdm sun-java6-jdk xdebconfigurator xfonts-100dpi xfonts-75dpi xfs xfonts-base xutils-dev tightvncserver ubuntu-desktop[[BR]]
rider@cloud:~$ sudo vim /etc/gdm/gdm.conf [[BR]]
{{{
DisallowTCP=false
}}}
{{{
# VNC Server
[server-VNC]
name=VNC server
command=/usr/bin/Xvnc -geometry 1024x768 -depth 24
flexible=true
}}}
{{{
rider@cloud:~$ sudo perl -pi.bak -e 's/^0=Standard/0=VNC/g' /etc/gdm/gdm.conf
}}}
rider@cloud:~$ sudo /etc/init.d/gdm restart [[BR]]
rider@cloud:~$ netstat -tunlp [[BR]]
{{{
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN      -                 ---> Xvnc port
tcp        0      0 0.0.0.0:5901            0.0.0.0:*               LISTEN      3877/Xtightvnc  
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:6001            0.0.0.0:*               LISTEN      3877/Xtightvnc  
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -               
tcp6       0      0 :::22                   :::*                    LISTEN      -               
tcp6       0      0 ::1:631                 :::*                    LISTEN      -               
udp        0      0 0.0.0.0:32769           0.0.0.0:*                           -               
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           -               
}}}

=== 4.1.1 Kernel Module ===

rider@cloud:~$ lsmod (Make sure that the "nvidia" module is loaded, and you can also try to load the nvidiafb.) [[BR]]
{{{
Module                  Size  Used by
bridge                 55080  0 
llc                    10544  1 bridge
iptable_filter          6784  0 
ip_tables              20824  1 iptable_filter
x_tables               18952  1 ip_tables
loop                   19076  0 
nvidia               8111512  36 
i2c_core               24960  1 nvidia
e1000                 120000  0 
e1000e                131500  0 
fuse                   43312  3 
}}}

== 4.2 Build VNC for DomU remote-desktop ==

# Try to set the locale environment variables as you met the locale problem. [[BR]]
# This section provides the solution for those who wants to surf the Dom0 desktop. [[BR]]
# Actually, you can use tightvnc or turbovnc instead of gdm. [[BR]]

rider@nv:~$ sudo vim /etc/profile [[BR]]
{{{
# Locale
export LANGUAGE="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LANG="en_US.UTF-8"
}}}
rider@nv:~$ source /etc/profile [[BR]]
rider@nv:~$ sudo dpkg-reconfigure locales [[BR]]
rider@nv:~$ sudo apt-get install autoconf automake build-essential gcc htop iptraf make libtool gdm sun-java6-jdk xdebconfigurator xfonts-100dpi xfonts-75dpi xfs xfonts-base xutils-dev tightvncserver ubuntu-desktop[[BR]]
rider@nv:~$ sudo vim /etc/gdm/gdm.conf [[BR]]
{{{
DisallowTCP=false
}}}
{{{
# VNC Server
[server-VNC]
name=VNC server
command=/usr/bin/Xvnc -geometry 1024x768 -depth 24
flexible=true
}}}
{{{
rider@nv:~$ sudo perl -pi.bak -e 's/^0=Standard/0=VNC/g' /etc/gdm/gdm.conf
}}}
rider@nv:~$ sudo /etc/init.d/gdm restart [[BR]]

== 4.3 Build VMGL !HowTo ==

=== 4.3.1 VMGL installation on Dom0 !HowTo ===

rider@cloud:~/vmgl$ sudo apt-get install build-essential mesa-common-dev libglu1-mesa-dev mesa-utils libxmu-headers libxmu6 libxmu-dev zlib1g-dev libjpeg62 libjpeg62-dev xutils-dev libxaw-headers libxaw7 libxaw7-dev libxext6 libxext-dev rxvt lwm xauth xvfb xfonts-100dpi xfonts-75dpi culmus xfonts-scalable xfonts-base [[BR]]
rider@cloud:~/vmgl$ wget http://www.cs.toronto.edu/~andreslc/software/vmgl-0.1.tar.bz2 [[BR]]
rider@cloud:~/vmgl$ tar jxvf vmgl-0.1.tar.bz2 [[BR]]
rider@cloud:~/vmgl$ cd ./vmgl.hg/tightvnc/ [[BR]]
rider@cloud:~/vmgl/vmgl.hg/tightvnc$ patch -p0 < ../../tightvnc-1.2.9-amd64support.patch [[BR]]
rider@cloud:~/vmgl$ cd .. [[BR]]
rider@cloud:~/vmgl$ sudo ln -sf /usr/bin/make /usr/bin/gmake (if necessary) [[BR]] 
rider@cloud:~/vmgl/vmgl.hg$ make -j 4 [[BR]]
rider@cloud:~/vmgl/vmgl.hg$ sudo make install-host [[BR]]
rider@cloud:~$ xauth [[BR]]
{{{
# Set the authority for remote guest.

Using authority file /home/rider/.Xauthority
xauth> add guest/unix:10  MIT-MAGIC-COOKIE-1  ec0ffd387888b9749d55f88031505888
xauth> add guest/unix:1  MIT-MAGIC-COOKIE-1  6824789b4ce0ac5743aeb57fd3ef8f5b
xauth> exit
}}} 

=== 4.3.2 VMGL installation on DomU or Guest OS !HowTo ===

rider@guest:~/vmgl$ sudo apt-get install build-essential mesa-common-dev libglu1-mesa-dev mesa-utils libxmu-headers libxmu6 libxmu-dev zlib1g-dev libjpeg62 libjpeg62-dev xutils-dev libxaw-headers libxaw7 libxaw7-dev libxext6 libxext-dev rxvt lwm xauth xvfb xfonts-100dpi xfonts-75dpi culmus xfonts-scalable xfonts-base [[BR]]
{{{
# Desgin for lightweight basic windows support

xfonts-base xfonts-100dpi xfonts-75dpi -> fonts support
rxvt -> x-terminal-emulator for VNC
lwm  -> x-window-manager for VNC
}}}

rider@guest:~/vmgl$ wget http://www.cs.toronto.edu/~andreslc/software/vmgl-0.1.tar.bz2 [[BR]]
rider@guest:~/vmgl$ tar jxvf vmgl-0.1.tar.bz2 [[BR]]
rider@guest:~/vmgl$ cd ./vmgl.hg/tightvnc/ [[BR]]
rider@guest:~/vmgl/vmgl.hg/tightvnc$ patch -p0 < ../../tightvnc-1.2.9-amd64support.patch [[BR]]
rider@guest:~/vmgl$ cd .. [[BR]]
rider@guest:~/vmgl$ sudo ln -sf /usr/bin/make /usr/bin/gmake (if necessary) [[BR]]
rider@guest:~/vmgl/vmgl.hg$ sudo mkdir -p /usr/lib/xorg (if necessary) [[BR]]
rider@guest:~/vmgl/vmgl.hg$ sudo mkdir -p /usr/lib/xorg/modules (if necessary) [[BR]]
rider@guest:~/vmgl/vmgl.hg$ sudo mkdir -p /usr/lib/xorg/modules/extensions (if necessary) [[BR]]
rider@guest:~/vmgl/vmgl.hg$ make [[BR]]
rider@guest:~/vmgl/vmgl.hg$ sudo make install-guest [[BR]]

{{{
## Fix the rgb_Path problem.
rider@guest:~/vmgl/vmgl.hg$ sudo mkdir -p /usr/X11R6/lib
rider@guest:~/vmgl/vmgl.hg$ sudo mkdir /usr/X11R6/lib/X11
rider@guest:~/vmgl/vmgl.hg$ sudo ln -sf /etc/X11/rgb.txt /usr/X11R6/lib/X11/rgb
}}}

== 4.4 VMGL !UsageNote ==

=== 4.4.1 VMGL on Dom0 (host) ===

# Start the host VMGL stub-daemon. [[BR]]
rider@cloud:~$ export DISPLAY=:0 [[BR]]
rider@cloud:~$ stub-daemon [[BR]]
rider@cloud:~$ netstat -tunlp [[BR]]

{{{
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8002            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:7000            0.0.0.0:*               LISTEN      29082/stub-daemon ---> VMGL stub-daemon
udp        0      0 0.0.0.0:32769           0.0.0.0:*                           -               
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           -               
}}}

'''Using X forwarding''' [[BR]]
rider@cloud:~$ ssh -X guest [[BR]]
rider@guest:~$ glxinfo [[BR]]
{{{
name of display: localhost:10.0
display: localhost:10  screen: 0
direct rendering: Yes
server glx vendor string: VMGL
server glx version string: 1.2 VMGL
server glx extensions:
client glx vendor string: VMGL
client glx version string: 1.2 VMGL
client glx extensions:
GLX version: 1.3
GLX extensions:
OpenGL extensions:
    GL_ARB_depth_texture, GL_ARB_fragment_program, GL_ARB_multisample,
    GL_ARB_multitexture, GL_ARB_occlusion_query, GL_ARB_point_parameters,
    GL_ARB_point_sprite, GL_ARB_shadow, GL_ARB_texture_border_clamp,
    GL_ARB_texture_compression, GL_ARB_texture_cube_map,
    GL_ARB_texture_env_add, GL_ARB_texture_env_combine,
    GL_EXT_texture_env_combine, GL_ARB_texture_env_dot3,
    GL_EXT_texture_env_dot3, GL_ARB_texture_mirrored_repeat,
    GL_ARB_texture_non_power_of_two, GL_ARB_transpose_matrix,
    GL_ARB_vertex_buffer_object, GL_ARB_vertex_program, GL_ARB_window_pos,
    GL_EXT_blend_color, GL_EXT_blend_minmax, GL_EXT_blend_func_separate,
    GL_EXT_blend_subtract, GL_EXT_texture_env_add, GL_EXT_fog_coord,
    GL_EXT_multi_draw_arrays, GL_EXT_secondary_color, GL_EXT_shadow_funcs,
    GL_EXT_stencil_wrap, GL_EXT_texture_cube_map, GL_EXT_texture_edge_clamp,
    GL_EXT_texture_filter_anisotropic, GL_EXT_texture_lod_bias,
    GL_EXT_texture_object, GL_EXT_texture3D, GL_EXT_bgra,
    GL_IBM_rasterpos_clip, GL_NV_fog_distance, GL_NV_fragment_program,
    GL_NV_register_combiners, GL_NV_register_combiners2,
    GL_NV_texgen_reflection, GL_NV_texture_rectangle, GL_NV_vertex_program,
    GL_NV_vertex_program1_1, GL_NV_vertex_program2, GL_SGIS_generate_mipmap,
    GL_CR_state_parameter, GL_CR_cursor_position, GL_CR_bounding_box,
    GL_CR_print_string, GL_CR_tilesort_info, GL_CR_synchronization,
    GL_CR_head_spu_name, GL_CR_performance_info, GL_CR_window_size,
    GL_CR_tile_info, GL_CR_saveframe, GL_CR_readback_barrier_size,
    GL_CR_server_id_sharing, GL_CR_server_matrix
    GLX_ARB_multisample
OpenGL vendor string: H. Andres Lagar-Cavilla
OpenGL renderer string: VMGL
OpenGL version string: 1.5 VMGL 1.9
0x52 24 dc  0 32  0 r  y  .  8  8  8  0  4 24  8 16 16 16 16  2 1 None
0x53 24 dc  0 32  0 r  y  .  8  8  8  8  4 24  8 16 16 16 16  2 1 None
0x54 24 dc  0 32  0 r  y  .  8  8  8  0  4 24  8 16 16 16 16  4 1 None
0x55 24 dc  0 32  0 r  y  .  8  8  8  8  4 24  8 16 16 16 16  4 1 None
0x56 24 dc  0 32  0 r  .  .  8  8  8  0  4 24  8 16 16 16 16  2 1 None
0x57 24 dc  0 32  0 r  .  .  8  8  8  8  4 24  8 16 16 16 16  2 1 None
0x58 24 dc  0 32  0 r  .  .  8  8  8  0  4 24  8 16 16 16 16  4 1 None
0x59 24 dc  0 32  0 r  .  .  8  8  8  8  4 24  8 16 16 16 16  4 1 None
0x23 32 tc  0 32  0 r  y  .  8  8  8  0  4 24  8 16 16 16 16  0 0 None
0x5a 32 tc  0 32  0 r  y  .  8  8  8  8  4 24  8 16 16 16 16  0 0 None
0x5b 32 tc  0 32  0 r  .  .  8  8  8  0  4 24  8 16 16 16 16  0 0 None
0x5c 32 tc  0 32  0 r  .  .  8  8  8  8  4 24  8 16 16 16 16  0 0 None
0x5d 32 tc  0 32  0 r  y  .  8  8  8  0  4 24  0 16 16 16 16  0 0 None
0x5e 32 tc  0 32  0 r  y  .  8  8  8  8  4 24  0 16 16 16 16  0 0 None
0x5f 32 tc  0 32  0 r  .  .  8  8  8  0  4 24  0 16 16 16 16  0 0 None
0x60 32 tc  0 32  0 r  .  .  8  8  8  8  4 24  0 16 16 16 16  0 0 None
0x61 32 tc  0 32  0 r  y  .  8  8  8  0  4  0  0 16 16 16 16  0 0 None
0x62 32 tc  0 32  0 r  y  .  8  8  8  8  4  0  0 16 16 16 16  0 0 None
0x63 32 tc  0 32  0 r  .  .  8  8  8  0  4  0  0 16 16 16 16  0 0 None
0x64 32 tc  0 32  0 r  .  .  8  8  8  8  4  0  0 16 16 16 16  0 0 None
0x65 32 tc  0 32  0 r  y  .  8  8  8  0  4 24  0 16 16 16 16  2 1 None
0x66 32 tc  0 32  0 r  y  .  8  8  8  8  4 24  0 16 16 16 16  2 1 None
0x67 32 tc  0 32  0 r  y  .  8  8  8  0  4 24  0 16 16 16 16  4 1 None
0x68 32 tc  0 32  0 r  y  .  8  8  8  8  4 24  0 16 16 16 16  4 1 None
0x69 32 tc  0 32  0 r  .  .  8  8  8  0  4 24  0 16 16 16 16  2 1 None
0x6a 32 tc  0 32  0 r  .  .  8  8  8  8  4 24  0 16 16 16 16  2 1 None
0x6b 32 tc  0 32  0 r  .  .  8  8  8  0  4 24  0 16 16 16 16  4 1 None
0x6c 32 tc  0 32  0 r  .  .  8  8  8  8  4 24  0 16 16 16 16  4 1 None
0x6d 32 tc  0 32  0 r  y  .  8  8  8  0  4 24  8 16 16 16 16  2 1 None
0x6e 32 tc  0 32  0 r  y  .  8  8  8  8  4 24  8 16 16 16 16  2 1 None
0x6f 32 tc  0 32  0 r  y  .  8  8  8  0  4 24  8 16 16 16 16  4 1 None
0x70 32 tc  0 32  0 r  y  .  8  8  8  8  4 24  8 16 16 16 16  4 1 None
0x71 32 tc  0 32  0 r  .  .  8  8  8  0  4 24  8 16 16 16 16  2 1 None
0x72 32 tc  0 32  0 r  .  .  8  8  8  8  4 24  8 16 16 16 16  2 1 None
0x73 32 tc  0 32  0 r  .  .  8  8  8  0  4 24  8 16 16 16 16  4 1 None
0x74 32 tc  0 32  0 r  .  .  8  8  8  8  4 24  8 16 16 16 16  4 1 None
}}}

rider@guest:~$ glxgears [[BR]]
{{{
47819 frames in 5.0 seconds = 9563.678 FPS
46064 frames in 5.0 seconds = 9212.566 FPS
44584 frames in 5.0 seconds = 8916.581 FPS
44256 frames in 5.0 seconds = 8850.974 FPS
44688 frames in 5.0 seconds = 8937.528 FPS
}}}

=== 4.4.2 VMGL on DomU or Guest OS ===

# Set the VMGL environment for VMGL guest. [[BR]]
rider@guest:~$ sudo ln -sf /usr/share/fonts/X11/ /usr/X11R6/lib/X11/fonts (if necessary - fix font path) [[BR]]
{{{
# FontPath:

/usr/X11R6/lib/X11/fonts
or
/usr/share/fonts/X11
}}}
rider@guest:~$ less /usr/X11R6/lib/X11/rgb.txt (if necessary - rgb path confirmation) [[BR]]
rider@guest:~$ sudo vim /etc/profile [[BR]]
{{{
GLSTUB=Cloud_IP(host_IP):7000
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/vmgl
LD_PRELOAD=/usr/local/lib/vmgl/libGL.so
export GLSTUB LD_LIBRARY_PATH LD_PRELOAD
}}}
rider@guest:~$ source /etc/profile [[BR]]

'''Using X forwarding''' [[BR]]
rider@guest:~$ sudo vim /etc/ssh/sshd_config [[BR]]
{{{
X11Forwarding yes
}}}
rider@guest:~$ sudo vim /etc/ssh/ssh_config [[BR]]
{{{
    ForwardX11 yes
    ForwardX11Trusted yes
}}}
rider@guest:~$ sudo /etc/init.d/ssh restart [[BR]]
rider@guest:~$ vncserver -geometry 1024x768 -depth 24 :1 [[BR]]
rider@guest:~$ netstat -tunlp [[BR]]
{{{
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5901            0.0.0.0:*               LISTEN      10652/Xtightvnc --> display :1
tcp        0      0 0.0.0.0:6001            0.0.0.0:*               LISTEN      10652/Xtightvnc
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 ::1:6010                :::*                    LISTEN      -
}}}

== 4.5 Course Demo: QUAKE 3 !OnLine ==

QUAKE 3: http://necromanthus.com/Games/ShockWave/quake3.html [[BR]]

== 4.6 Suspending & Resuming between DomU ==



Reference & Support: [[BR]]
1.NVIDIA CUDA: http://www.nvidia.com/object/cuda_home.html [[BR]]
2.openSUSE NVIDIA + Xen: http://en.opensuse.org/Use_Nvidia_driver_with_Xen [[BR]]
3.NVIDIA GPUs DEV_IDs: http://www.laptopvideo2go.com/forum/index.php?showtopic=7664 [[BR]]
4.pci_ids db: http://www.pcidatabase.com/ [[BR]]
5.Xen: assigning PCI devices to a domain: http://www.bestgrid.org/index.php/Xen:_assigning_PCI_devices_to_a_domain [[BR]]
6.Xen PCI Passthrough: http://www.wlug.org.nz/XenPciPassthrough [[BR]]
7.Xen Users' Manual v3.0: http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/ [[BR]]
8.VT-d in Xen: http://wiki.xensource.com/xenwiki/VTdHowTo [[BR]]
9.VMGL support: https://code.fluendo.com/elisa/trac/wiki [[BR]]
10.Latest NVIDIA driver: ftp://download.nvidia.com/XFree86/Linux-x86_64/ [[BR]]