[[PageOutline]] = Build Lustre Storage Cluster for 3D Fly platform = == 0.Introduction == Some domain-applications usually need to computing huge data. It usually meet a bottleneck problem of performance. It will lead I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance. You can refer to Lustre web site http://wiki.lustre.org/index.php?title=Main_Page == 1.Machine information & Software == * Hardware ||Node ||8 nodes (1 server , 7 client provide disks)|| ||CPU ||Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)|| ||Memory ||2GB DDR2 667 (each node)|| ||Disk ||All nodes: 320G * 7 = 2.24T|| ||NIC ||Intel Corporation 82566DM Gigabit Network Connection|| ||Switch ||D-link 24 port GE switch|| * Software ||OS||OpenSuSE 10.2|| ||DRBL||1.9.0-35|| ||Amira||Testing Version|| ||Kernel||2.6.18|| * Architecture [[Attachement:fly_arch.png]] == 2.Config OpenSuSE == * Install packages for Lustre * Install packages for DRBL == 3.Install Lustre == * Build kernel {{{ $ sudo su $ cd /usr/src $ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2 $ tar jxvf linux-2.6.18.tar.bz2 $ cd 2.6.18 $ cp /boot/config-xxx .config $ make oldconfig ; make menuconfig $ make -j 5 $ make modules $ make modules_install $ make install $ mkinitrd $ vim /boot/grub/menu.list }}} * recompiler NIC driver {{{ wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw tar e1000-7.6.15.4.tar.gz cd e1000-7.6.15.4/src make install }}} * Bonding NIC (bond0 -> eth1 & eth2) {{{ $ cd /etc/sysconfig/network $ vim ifcfg-eth1 $ vim ifeth-eth2 $ vim ifcfg-bond0 $ vim /etc/rc.d/after.local $ vim /etc/module/modules.conf.local $ vim /etc/rc.d/after.local }}} * Build Lustre Modules {{{ $ configure --with-linux=/usr/src/2.6.18 $ make install $ depmod -a $ vim /etc/rc.d/after.local }}} == 4.Build DRBL Environment == {{{ $ check node_root/etc/rc.d/module.conf.local check nodes/192.168.1.X/etc/rc.d/module.conf.local rm /tftfpood/node_root/sysconf/network/if }}} == 5.Build Lustre Storage Cluster == * setup autologin {{{ = fly = mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1 mount -t lustre /dev/sdb1 /home/mdt = fly01 ~ fly07 = mkfs.lustre --fsname=flyfs --ost --mgsnode=fly@tcp0 /dev/sdb mount -t lustre /dev/sdb /home/ost = fly = mkdir /home/flyfs mount -t lustre fly@tcp0:/flyfs /home/flyfs }}} == 6.Install TurboJPEG == rock@fly:~/turbojpeg> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm [[BR]] rock@fly:~/turbojpeg> sudo rpm -i turbojpeg-1.10.3.x86_64.rpm [[BR]] rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg [[BR]] {{{ normal /usr/bin/switchtjpeg normal /usr/include/turbojpeg.h normal /usr/lib/libturbojpeg-ipp.so normal /usr/lib/libturbojpeg-libjpeg.so normal /usr/lib/libturbojpeg.so normal /usr/lib64/libturbojpeg-ipp.so normal /usr/lib64/libturbojpeg-libjpeg.so normal /usr/lib64/libturbojpeg.so normal /usr/share/doc/turbojpeg-1.10.3 normal /usr/share/doc/turbojpeg-1.10.3/LGPL.txt normal /usr/share/doc/turbojpeg-1.10.3/LICENSE.txt normal /usr/share/doc/turbojpeg-1.10.3/README.txt }}} rock@fly:~/VirtualGL> switchtjpeg --help [[BR]] {{{ USAGE: /usr/bin/switchtjpeg < ipp | libjpeg > ipp = Use the accelerated version of TurboJPEG, which embeds functions from the Intel(R) Integrated Performance Primitives libjpeg = Use the unaccelerated version of TurboJPEG, which is GPL-compatible }}} == 7.Install VirtualGL == rock@fly:~/VirtualGL> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm [[BR]] rock@fly:~/VirtualGL> sudo rpm -i VirtualGL-2.1.x86_64.rpm [[BR]] rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL {{{ normal /opt/VirtualGL normal /opt/VirtualGL/bin normal /opt/VirtualGL/bin/cpustat normal /opt/VirtualGL/bin/glxinfo normal /opt/VirtualGL/bin/glxspheres normal /opt/VirtualGL/bin/glxspheres64 normal /opt/VirtualGL/bin/nettest normal /opt/VirtualGL/bin/tcbench normal /opt/VirtualGL/bin/vglclient normal /opt/VirtualGL/bin/vglconfig normal /opt/VirtualGL/bin/vglconfig64 normal /opt/VirtualGL/bin/vglconnect normal /opt/VirtualGL/bin/vglgenkey normal /opt/VirtualGL/bin/vgllogin normal /opt/VirtualGL/bin/vglrun normal /opt/VirtualGL/bin/vglserver_config normal /opt/VirtualGL/doc normal /opt/VirtualGL/fakelib normal /opt/VirtualGL/fakelib/64 normal /opt/VirtualGL/fakelib/64/libGL.so normal /opt/VirtualGL/fakelib/libGL.so normal /opt/VirtualGL/lib normal /opt/VirtualGL/lib64 normal /usr/bin/vglclient normal /usr/bin/vglconfig normal /usr/bin/vglconfig64 normal /usr/bin/vglconnect normal /usr/bin/vglgenkey normal /usr/bin/vgllogin normal /usr/bin/vglrun normal /usr/bin/vglserver_config normal /usr/lib/libdlfaker.so normal /usr/lib/libgefaker.so normal /usr/lib/librrfaker.so normal /usr/lib64/libdlfaker.so normal /usr/lib64/libgefaker.so normal /usr/lib64/librrfaker.so normal /usr/share/doc/VirtualGL-2.1 normal /usr/share/doc/VirtualGL-2.1/411.gif normal /usr/share/doc/VirtualGL-2.1/422.gif normal /usr/share/doc/VirtualGL-2.1/444.gif normal /usr/share/doc/VirtualGL-2.1/ChangeLog.txt normal /usr/share/doc/VirtualGL-2.1/LGPL.txt normal /usr/share/doc/VirtualGL-2.1/LICENSE-OpenSSL.txt normal /usr/share/doc/VirtualGL-2.1/LICENSE.txt normal /usr/share/doc/VirtualGL-2.1/chromium-displaywall.png normal /usr/share/doc/VirtualGL-2.1/chromium-sortfirst.png normal /usr/share/doc/VirtualGL-2.1/chromium-sortlast.png normal /usr/share/doc/VirtualGL-2.1/configdialog.gif normal /usr/share/doc/VirtualGL-2.1/exceed1.png normal /usr/share/doc/VirtualGL-2.1/exceed2.png normal /usr/share/doc/VirtualGL-2.1/exceed3.png normal /usr/share/doc/VirtualGL-2.1/exceed6.png normal /usr/share/doc/VirtualGL-2.1/index.html normal /usr/share/doc/VirtualGL-2.1/somerights20.png normal /usr/share/doc/VirtualGL-2.1/sshtunnel.png normal /usr/share/doc/VirtualGL-2.1/sunray.png normal /usr/share/doc/VirtualGL-2.1/turbovnc1.png normal /usr/share/doc/VirtualGL-2.1/turbovnc2.png normal /usr/share/doc/VirtualGL-2.1/turbovnc3.png normal /usr/share/doc/VirtualGL-2.1/turbovnc4.png normal /usr/share/doc/VirtualGL-2.1/vgltransport.png normal /usr/share/doc/VirtualGL-2.1/vgltransportglp.png normal /usr/share/doc/VirtualGL-2.1/vgltransportservernetwork.png normal /usr/share/doc/VirtualGL-2.1/virtualgl.css normal /usr/share/doc/VirtualGL-2.1/x11transport.png normal /usr/share/doc/VirtualGL-2.1/x11transportglp.png }}} rock@fly:~/VirtualGL> vglrun --help {{{ Usage: vglrun [options] [--] [OpenGL app arguments] VirtualGL options -c : proxy = Send 3D images uncompressed using X11 transport jpeg = Compress 3D images using JPEG/send using VGL transport rgb = Encode 3D images as RGB/send using VGL transport sr = Compress 3D images using DPCM/send using Sun Ray transport srrgb = Encode 3D images as RGB/send using Sun Ray transport -cl : = The hostname or IP address of the VirtualGL client (or Sun Ray server) -dl : Force apps that dlopen() libGL to load VirtualGL instead -d : = the X Display or GLP device to use for 3D rendering -fps : Limit client/server frame rate to frames/sec +/-g : Enable/disable gamma correction -gamma : Set gamma correction factor to (see docs) -ge : Fool application into thinking that LD_PRELOAD is unset -np : Use CPUs to perform image compression (VGL transport only) -p

: TCP port to use when connecting to vglclient (VGL transport only) +/-pr : Enable/disable performance profiling output -q : JPEG compression quality [1 <= <= 100] +/-s : Enable/disable SSL encryption of VGL image transport -samp : JPEG chrominance subsampling factor [ = gray, 1x, 2x, 4x, 8x, or 16x] +/-sp : Turn on/off frame spoiling -st : left = Send only left eye buffer right = Send only right eye buffer quad = Use quad-buffered stereo if available, otherwise use red/cyan (anaglyphic) stereo [default] rc = Always use red/cyan (anaglyphic) stereo +/-sync : Enable/disable strict 2D/3D synchronization +/-tr : Enable/disable function call tracing (generates a lot of output) +/-v : Enable/disable verbose VirtualGL messages -64 : (Solaris only) Preload VirtualGL only into 64-bit apps -32 : (Solaris only) Preload VirtualGL only into 32-bit apps }}} == 8.Install TurboVNC == == 9.Install Amira == == 10.Test Amira == == Reference ==