[[PageOutline]] = Build Lustre Storage Cluster for 3D Fly platform = == 0.Introduction == Some domain-applications usually need to computing with huge data. It usually meet a bottleneck of performance, the main problem is I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance.[[BR]] Abiut Lustre, you can refer to Lustre web site http://wiki.lustre.org/ == 1.Machine information & Software == * Hardware ||Node ||8 nodes (1 server , 7 client provide disks)|| ||CPU ||Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)|| ||Memory ||2GB DDR2 667 (each node)|| ||Disk ||All nodes: 320G * 7 = 2.24T|| ||NIC ||Intel Corporation 82566DM Gigabit Network Connection|| ||Switch ||D-link 24 port GE switch|| * Software ||OS||OpenSuSE 10.2|| ||DRBL||1.9.0-35|| ||Amira||Testing Version|| ||Kernel||2.6.18|| * Architecture [[Image(fly_arch.png)]] == 2.Config OpenSuSE Environment == * Install packages for Lustre * Install packages for DRBL == 3.Install Lustre == * Build kernel {{{ $ sudo su $ cd /usr/src $ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2 $ tar jxvf linux-2.6.18.tar.bz2 $ cd 2.6.18 $ cp /boot/config-xxx .config $ make oldconfig ; make menuconfig $ make -j 5 $ make modules $ make modules_install $ make install $ mkinitrd $ vim /boot/grub/menu.list }}} * recompiler NIC driver {{{ wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw tar e1000-7.6.15.4.tar.gz cd e1000-7.6.15.4/src make install }}} * Bonding NIC (bond0 -> eth1 & eth2) {{{ $ cd /etc/sysconfig/network $ vim ifcfg-eth1 $ vim ifeth-eth2 $ vim ifcfg-bond0 $ vim /etc/rc.d/after.local ifenslave bond0 eth1 eth2 modprobe drm modprobe ldiskfs modprobe lnet networks=tcp $ vim /etc/module/modules.conf.local alias bond0 bonding alias eth0 e1000 alias eth1 e1000 alias eth2 e1000 options bond0 mode=6 miimon=100 }}} * Build Lustre Modules {{{ $ configure --with-linux=/usr/src/2.6.18 $ make install $ depmod -a $ vim /etc/rc.d/after.local }}} == 4.Build DRBL Environment == {{{ $ zypper install drbl $ /opt/drbl/sbin/drblsrv -i $ /opt/drbl/sbin/drblpush -i = Check nodes booting setup = /tftpboot/nodes/192.168.1.X/etc/rc.d/after.local /tftpboot/nodes/192.168.1.X/etc/module/modules.conf.local /tftfpood/node_root/etc/sysconfig/network/ }}} == 5.Build Lustre Storage Cluster == * Setup autologin {{{ = fly = mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1 mount -t lustre /dev/sdb1 /home/mdt = fly01 ~ fly07 = mkfs.lustre --fsname=flyfs --ost --mgsnode=fly@tcp0 /dev/sdb mount -t lustre /dev/sdb /home/ost = fly = mkdir /home/flyfs mount -t lustre fly@tcp0:/flyfs /home/flyfs }}} == 6.Install TurboJPEG == rock@fly:~/turbojpeg> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm [[BR]] rock@fly:~/turbojpeg> sudo rpm -ivh turbojpeg-1.10.3.x86_64.rpm [[BR]] rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg [[BR]] rock@fly:~/VirtualGL> switchtjpeg --help [[BR]] {{{ USAGE: /usr/bin/switchtjpeg < ipp | libjpeg > ipp = Use the accelerated version of TurboJPEG, which embeds functions from the Intel(R) Integrated Performance Primitives libjpeg = Use the unaccelerated version of TurboJPEG, which is GPL-compatible }}} == 7.Install VirtualGL == rock@fly:~/VirtualGL> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm [[BR]] rock@fly:~/VirtualGL> sudo rpm -ivh VirtualGL-2.1.x86_64.rpm [[BR]] rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL (Check) [[BR]] rock@fly:~/VirtualGL> vglrun --help ### VGL Server Configuration [[BR]] rock@fly:~> sudo /opt/VirtualGL/bin/vglserver_config [[BR]] {{{ 1) Configure server for use with VirtualGL in GLX mode 2) Unconfigure server for use with VirtualGL in GLX mode X) Exit Choose: 1 Restrict local X server access to vglusers group (recommended)? [Y/n] n Restrict framebuffer device access to vglusers group (recommended)? [Y/n] n Disable XTEST extension (recommended)? [Y/n] Y ... Granting write permission to /dev/nvidia* for all users ... ... Adding xhost +localhost to /etc/X11/xdm/Xsetup script ... ... Adding xhost +localhost to /etc/opt/gnome/gdm/Init/Default script ... ... Disabling XTEST extension in /etc/X11/xdm/Xservers ... ... Setting default run level to 5 (enabling graphical login prompt) ... Done. You must restart the display manager for the changes to take effect. 1) Configure server for use with VirtualGL in GLX mode 2) Unconfigure server for use with VirtualGL in GLX mode X) Exit Choose: X }}} rock@fly:~> sudo /etc/init.d/xdm restart [[BR]] == 8.Install TurboVNC == rock@fly:~/turbovnc> wget http://downloads.sourceforge.net/virtualgl/turbovnc-0.4.i386.rpm?modtime=1205803835&big_mirror=0 [[BR]] rock@fly:~/turbovnc> sudo rpm -ivh turbovnc-0.4.i386.rpm [[BR]] rock@fly:~/turbovnc> rpm -qs turbovnc (Check) [[BR]] rock@fly:~> sudo zypper install xorg-x11-fonts xorg-x11-fonts-core xorg-x11-fonts-devel [[BR]] rock@fly:~> sudo zypper install xorg-x11-devel xorg-x11-driver-input xorg-x11-driver-video [[BR]] rock@fly:~> sudo zypper install libdrm libdrm-devel Mesa Mesa-devel [[BR]] rock@fly:~> sudo zypper remove beagle beagle-evolution beagle-firefox beagle-gui beagle-index tomboy [[BR]] rock@fly:/usr/X11R6/lib> sudo /etc/init.d/xfs start [[BR]] rock@fly:~> cd /usr/X11R6/lib [[BR]] rock@fly:/usr/X11R6/lib> sudo mkdir X11 [[BR]] rock@fly:~> sudo cp rgb.txt /usr/X11R6/lib/X11/ [[BR]] rock@fly:~> sudo vim /etc/X11/xorg.conf [[BR]] {{{ Section "Module" Load "dbe" Load "type1" Load "freetype" Load "extmod" Load "glx" Load "v4l" Load "drm" Load "dri" }}} {{{ Section "DRI" Group "video" Mode 0666 EndSection }}} rock@fly:~> sudo vim /etc/X11/fs/config [[BR]] {{{ #no-listen = tcp }}} rock@fly:~> /opt/TurboVNC/bin/vncserver -geometry 1024x768 :1 [[BR]] New 'X' desktop is fly:1 Creating default startup script /home/rock/.vnc/xstartup Starting applications specified in /home/rock/.vnc/xstartup Log file is /home/rock/.vnc/fly:1.log rock@fly:~> ps aux | grep X {{{ root 4216 0.0 0.9 75012 19760 tty7 Ss+ Jul03 0:04 /usr/bin/X :0 -audit 0 -br -auth /var/lib/gdm/:0.Xauth vt7 rock 4811 0.0 0.0 46628 988 ? Ss Jul03 0:00 /usr/bin/ssh-agent /bin/bash /etc/X11/xinit/xinitrc rock 20758 1.1 0.6 16760 12900 pts/9 S 00:56 0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop X -httpd /opt/TurboVNC/bin/../vnc/classes -auth /home/rock/.Xauthority -dontdisconnect -geometry 1024x768 -depth 24 -rfbwait 120000 -rfbauth /home/rock/.vnc/passwd -rfbport 5901 -fp unix/:7100 -deferupdate 1 rock 22008 0.0 0.0 5004 808 pts/9 S+ 00:56 0:00 grep X }}} === 8.1 Using TurboVNC with vncviewer === @ Client {{{ < By Port > rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server::{5900+n} OR < By Display > rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server:{m} }}} Example: If the TurboVNC server is running on X display fly:1, then point your web browser to: [[BR]] {{{ < By Port > rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server::{5900+n} OR < By Display > rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server:1 }}} === 8.2 Using TurboVNC in a Web Browser === @ Client To use the Java TurboVNC viewer, point your web browser to: [[BR]] http://Fly_Server:5800+n [[BR]] Example: If the TurboVNC server is running on X display fly:1, then point your web browser to: [[BR]] http://Fly_Server:5801 [[BR]] {{{ ### TurboVNC Initialization Bug fixed for ALSA Library ### rock@fly:~/alsa> wget http://gd.tuwien.ac.at/opsys/linux/alsa/lib/alsa-lib-1.0.13.tar.bz2 rock@fly:~/alsa> tar jxvf alsa-lib-1.0.13.tar.bz2 rock@fly:~/alsa> cd alsa-lib-1.0.13 rock@fly:~/alsa/alsa-lib-1.0.13> ./configure --enable-static rock@fly:~/alsa/alsa-lib-1.0.13> make -j 4 rock@fly:~/alsa/alsa-lib-1.0.13> sudo make install openSUSE(XDM)-> Computer -> Control Center -> System -> Sound -> Choose ALSA for each option. }}} == 9.Install Amira == Necessary Library: libg2c.so.0 [[BR]] Download libg2c33(Fortran Compiler Runtime Library) for openSUSE 10.2 [[BR]] rock@fly:~/amira_source/download_lib> wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/algraf/openSUSE_10.2/x86_64/libg2c33-3.3.3-43.3.x86_64.rpm [[BR]] rock@fly:~/amira_source/download_lib> sudo rpm -i libg2c33-3.3.3-43.3.x86_64.rpm [[BR]] rock@fly:~/amira_source> chmod a+x Amira411-LinuxAMD64.bin [[BR]] rock@fly:~/amira_source> ./Amira411-LinuxAMD64.bin --console (console mode)[[BR]] Installation succeed !!! [[BR]] Installation path: /home/rock/Amira4.1.1 [[BR]] rock@fly:~/Amira4.1.1/bin> ./start --console (console mode)[[BR]] Finished ! It works... === 9.1 Amira Service === rock@fly:~> cd ./Amira4.1.1/bin/ [[BR]] rock@fly:~/Amira4.1.1/bin> sudo chmod a+x serviceSetup.sh [[BR]] rock@fly:~/Amira4.1.1/bin> ./serviceSetup.sh [[BR]] rock@fly:~/Amira4.1.1/bin> sudo /etc/init.d/xinetd restart [[BR]] rock@fly:~/Amira4.1.1/bin> sudo netstat -tnlp {{{ Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:32802 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:616 0.0.0.0:* LISTEN 3831/ypbind tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 18763/xinetd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3589/portmap tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 16769/X tcp 0 0 127.0.0.1:2544 0.0.0.0:* LISTEN 3867/zmd tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 3827/perl tcp 0 0 0.0.0.0:17234 0.0.0.0:* LISTEN 18763/xinetd ---> Amira daemon tcp 0 0 0.0.0.0:756 0.0.0.0:* LISTEN 3974/rpc.mountd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 4089/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 4596/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 4167/master tcp 0 0 0.0.0.0:603 0.0.0.0:* LISTEN 3817/ypserv tcp 0 0 0.0.0.0:607 0.0.0.0:* LISTEN 3822/rpc.ypxfrd }}} rock@fly:~/Amira4.1.1/bin> less /etc/xinetd.d/amira [[BR]] {{{ # Amira's Service service amira { type = UNLISTED socket_type = STREAM protocol = TCP wait = no user = root server = /home/rock/Amira4.1.1/bin/amiradaemon server_args = -daemonExePath /home/rock/Amira4.1.1/bin/start port = 17234 } }}} == 10.NVIDIA GeForce 7300 GT Installation == Change to root before following steps. [[BR]] fly:/home/rock/nvidia # wget http://tw.download.nvidia.com/XFree86/Linux-x86_64/173.14.09/NVIDIA-Linux-x86_64-173.14.09-pkg2.run [[BR]] fly:/home/rock/nvidia # sh NVIDIA-Linux-x86_64-173.14.09-pkg2.run [[BR]] Then follow extra steps to continue: Setup , kernel module , configuration ..etc [[BR]] Finished!!! Now, Check it out ~ fly:/home/rock/nvidia # modprobe -l | grep nv [[BR]] {{{ /lib/modules/2.6.18-default/kernel/drivers/scsi/sata_nv.ko /lib/modules/2.6.18-default/kernel/drivers/video/nvidia/nvidiafb.ko /lib/modules/2.6.18-default/kernel/drivers/video/nvidia.ko }}} We only need nvidia.ko, and it'll actually loaded after installation. [[BR]] fly:/home/rock/nvidia # lsmod | grep nv {{{ nvidia 8132208 24 i2c_core 41472 2 nvidia,i2c_i801 }}} fly:/home/rock # glxinfo | more {{{ direct rendering: Yes server glx vendor string: NVIDIA Corporation server glx version string: 1.4 server glx extensions: GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, GLX_EXT_texture_from_pixmap, GLX_ARB_multisample, GLX_NV_float_buffer, GLX_ARB_fbconfig_float client glx vendor string: NVIDIA Corporation client glx version string: 1.4 OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: GeForce 7300 GT/PCI/SSE2 OpenGL version string: 2.1.2 NVIDIA 173.14.09 }}} fly:/home/rock/nvidia # vim /etc/X11/xorg.conf [[BR]] {{{ Section "Module" Load "dbe" Load "v4l" Load "freetype" Load "type1" Load "extmod" Load "glx" Load "vnc" # Load "dri" EndSection Section "Device" BoardName "GeForce 7300GT" Driver "nvidia" Identifier "Device[0]" # Option "alwaysshared" # Option "usevnc" "yes" # Option "httpdir" "/usr/share/vnc/classes" # Option "httpport" "5800" # Option "rfbauth" "/root/.vnc/passwd" Screen 0 VendorName "NVIDIA Corporation" EndSection # Section "DRI" # Group "video" # Mode 0666 # EndSection }}} fly:/home/rock/nvidia # less /var/log/Xorg.0.log [[BR]] {{{ (--) PCI:*(1:0:0) nVidia Corporation G70 [GeForce 7300 GT] rev 161, Mem @ 0xfd000000/24, 0xe0000000/ 28, 0xfc000000/24, I/O @ 0xcc00/7, BIOS @ 0xfe8e0000/17 (II) Module glx: vendor="NVIDIA Corporation" compiled for 4.0.2, module version = 1.0.0 Module class: X.Org Server Extension (II) NVIDIA GLX Module 173.14.09 Thu Jun 5 00:05:57 PDT 2008 (II) Loading extension GLX (II) LoadModule: "vnc" (II) Loading /usr/lib64/xorg/modules/extensions//libvnc.so (II) Module vnc: vendor="RealVNC Ltd" compiled for 4.3.99.902, module version = 1.0.0 Module class: X.Org Server Extension ABI class: X.Org Server Extension, version 0.3 (II) Loading extension VNC (II) LoadModule: "nvidia" (II) Loading /usr/lib64/xorg/modules/drivers//nvidia_drv.so (II) Module nvidia: vendor="NVIDIA Corporation" compiled for 4.0.2, module version = 1.0.0 Module class: X.Org Video Driver (II) NVIDIA dlloader X Driver 173.14.09 Wed Jun 4 23:48:23 PDT 2008 (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs (II) Primary Device is: PCI 01:00:0 (--) Assigning device section with no busID to primary device (--) Chipset NVIDIA GPU found (**) NVIDIA(0): Depth 16, (--) framebuffer bpp 16 (==) NVIDIA(0): RGB weight 565 (==) NVIDIA(0): Default visual is TrueColor (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0) (**) NVIDIA(0): Enabling RENDER acceleration (II) NVIDIA(0): NVIDIA GPU GeForce 7300 GT (G73) at PCI:1:0:0 (GPU-0) (--) NVIDIA(0): Memory: 262144 kBytes (--) NVIDIA(0): VideoBIOS: 05.73.22.51.00 (II) NVIDIA(0): Detected PCI Express Link width: 16X (--) NVIDIA(0): Interlaced video modes are supported on this GPU (--) NVIDIA(0): Connected display device(s) on GeForce 7300 GT at PCI:1:0:0: (--) NVIDIA(0): Philips 190B (CRT-0) (--) NVIDIA(0): Philips 190B (CRT-0): 400.0 MHz maximum pixel clock (II) NVIDIA(0): Assigned Display Device: CRT-0 (II) NVIDIA(0): Validated modes: (II) NVIDIA(0): "1024x768" (II) NVIDIA(0): "1024x600" (II) NVIDIA(0): "800x600" (II) NVIDIA(0): "768x576" (II) NVIDIA(0): "640x480" (II) NVIDIA(0): Virtual screen size determined to be 1024 x 768 (--) NVIDIA(0): DPI set to (68, 65); computed from "UseEdidDpi" X config (--) NVIDIA(0): option (==) NVIDIA(0): Disabling 32-bit ARGB GLX visuals. }}} fly:/home/rock/nvidia # /etc/init.d/xdm restart [[BR]] === 10.1 NVIDIA 7300GT Performance === rock@fly:~> glxgears [[BR]] {{{ 37672 frames in 5.0 seconds = 7534.325 FPS 38320 frames in 5.0 seconds = 7663.902 FPS 38317 frames in 5.0 seconds = 7663.308 FPS 38320 frames in 5.0 seconds = 7663.936 FPS 38329 frames in 5.0 seconds = 7665.636 FPS 38316 frames in 5.0 seconds = 7663.152 FPS 38307 frames in 5.0 seconds = 7661.244 FPS 38321 frames in 5.0 seconds = 7664.146 FPS 38308 frames in 5.0 seconds = 7661.435 FPS 38310 frames in 5.0 seconds = 7661.956 FPS 38323 frames in 5.0 seconds = 7664.566 FPS 38311 frames in 5.0 seconds = 7662.112 FPS 38306 frames in 5.0 seconds = 7661.194 FPS 38318 frames in 5.0 seconds = 7663.445 FPS 38320 frames in 5.0 seconds = 7663.884 FPS 38310 frames in 5.0 seconds = 7661.989 FPS 38325 frames in 5.0 seconds = 7664.809 FPS 38317 frames in 5.0 seconds = 7663.257 FPS 38308 frames in 5.0 seconds = 7661.587 FPS 38318 frames in 5.0 seconds = 7663.428 FPS 38305 frames in 5.0 seconds = 7660.958 FPS 38317 frames in 5.0 seconds = 7663.250 FPS 38327 frames in 5.0 seconds = 7665.265 FPS 38313 frames in 5.0 seconds = 7662.488 FPS 38311 frames in 5.0 seconds = 7662.028 FPS 38317 frames in 5.0 seconds = 7663.277 FPS 38313 frames in 5.0 seconds = 7662.439 FPS 38314 frames in 5.0 seconds = 7662.786 FPS 38325 frames in 5.0 seconds = 7664.986 FPS 38311 frames in 5.0 seconds = 7662.035 FPS 38304 frames in 5.0 seconds = 7660.689 FPS 38333 frames in 5.0 seconds = 7666.563 FPS 38309 frames in 5.0 seconds = 7661.684 FPS 38311 frames in 5.0 seconds = 7662.048 FPS 38330 frames in 5.0 seconds = 7665.816 FPS 38316 frames in 5.0 seconds = 7663.065 FPS 38306 frames in 5.0 seconds = 7661.012 FPS 38022 frames in 5.0 seconds = 7604.208 FPS 38030 frames in 5.0 seconds = 7605.954 FPS 38020 frames in 5.0 seconds = 7603.871 FPS 38027 frames in 5.0 seconds = 7605.272 FPS 38023 frames in 5.0 seconds = 7604.425 FPS 38028 frames in 5.0 seconds = 7605.428 FPS 38032 frames in 5.0 seconds = 7606.298 FPS 38008 frames in 5.0 seconds = 7601.419 FPS 38032 frames in 5.0 seconds = 7606.386 FPS 38034 frames in 5.0 seconds = 7606.672 FPS 38029 frames in 5.0 seconds = 7605.690 FPS }}} == 11.Lustre Storage Cluster for 3D Fly platform Performance (I/O Benchmark without Ethernet Bonding) == == Testbed Information == * Server ||CPU ||Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz || ||Memory ||2GB DDR2 667|| ||Disk ||sda:160G sdb:320G|| ||Graphic || NVIDIA GeForce 7300GT 256MB DDR2 X1 PCI-E 16X ||NIC ||Intel Corporation 82566DM Gigabit Network Connection|| ||Switch ||D-link 24 port GE switch|| [[BR]] * Clients ||Node number||7 nodes|| ||CPU ||Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)|| ||Memory ||2GB DDR2 667 (each node)|| ||Disk ||sda:160G sdb:320G|| ||NIC ||Intel Corporation 82566DM Gigabit Network Connection|| ||Switch ||D-link 24 port GE switch|| [[BR]] * Software Information ||OS||openSUSE 10.3 Linux with kernel 2.6.22 x86_64 || ||Cluster Environment||DRBL|| ||3D Modol tools||IvIsee|| [[BR]] == < '''Test Cases''' > == '''Case 1''': 1 remote client access without Lustre [[BR]] '''Case 2''': 1 remote client access with Lustre [[BR]] '''Case 3''': 3 remote clients access without Lustre [[BR]] '''Case 4''': 3 remote clients access with Lustre [[BR]] '''Case 5''': 5 remote clients access without Lustre [[BR]] '''Case 6''': 5 remote clients access with Lustre [[BR]] [[BR]] == Reference == * Lustre, http://wiki.lustre.org/ * Amira, http://www.amiravis.com/ * NVIDIA, http://zh_tw.opensuse.org/NVIDIA * Glxgears, http://wiki.cchtml.com/index.php/Glxgears_is_not_a_Benchmark