[[PageOutline]] = Build Lustre Storage Cluster for 3D Fly platform = == 0.Introduction == Some domain-applications usually need to computing with huge data. It usually meet a bottleneck of performance, the main problem is I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance.[[BR]] Abiut Lustre, you can refer to Lustre web site http://wiki.lustre.org/ == 1.Machine information & Software == * Hardware ||Node ||8 nodes (1 server , 7 client provide disks)|| ||CPU ||Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)|| ||Memory ||2GB DDR2 667 (each node)|| ||Disk ||All nodes: 320G * 7 = 2.24T|| ||NIC ||Intel Corporation 82566DM Gigabit Network Connection|| ||Switch ||D-link 24 port GE switch|| * Software ||OS||OpenSuSE 10.2|| ||DRBL||1.9.0-35|| ||Amira||Testing Version|| ||Kernel||2.6.18|| * Architecture [[Image(fly_arch.png)]] == 2.Config OpenSuSE Environment == * Install packages for Lustre * Install packages for DRBL == 3.Install Lustre == * Build kernel {{{ $ sudo su $ cd /usr/src $ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2 $ tar jxvf linux-2.6.18.tar.bz2 $ cd 2.6.18 $ cp /boot/config-xxx .config $ make oldconfig ; make menuconfig $ make -j 5 $ make modules $ make modules_install $ make install $ mkinitrd $ vim /boot/grub/menu.list }}} * recompiler NIC driver {{{ wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw tar e1000-7.6.15.4.tar.gz cd e1000-7.6.15.4/src make install }}} * Bonding NIC (bond0 -> eth1 & eth2) {{{ $ cd /etc/sysconfig/network $ vim ifcfg-eth1 $ vim ifeth-eth2 $ vim ifcfg-bond0 $ vim /etc/rc.d/after.local ifenslave bond0 eth1 eth2 modprobe drm modprobe ldiskfs modprobe lnet networks=tcp $ vim /etc/module/modules.conf.local alias bond0 bonding alias eth0 e1000 alias eth1 e1000 alias eth2 e1000 options bond0 mode=6 miimon=100 }}} * Build Lustre Modules {{{ $ configure --with-linux=/usr/src/2.6.18 $ make install $ depmod -a $ vim /etc/rc.d/after.local }}} == 4.Build DRBL Environment == {{{ $ zypper install drbl $ /opt/drbl/sbin/drblsrv -i $ /opt/drbl/sbin/drblpush -i = Check nodes booting setup = /tftpboot/nodes/192.168.1.X/etc/rc.d/after.local /tftpboot/nodes/192.168.1.X/etc/module/modules.conf.local /tftfpood/node_root/etc/sysconfig/network/ }}} == 5.Build Lustre Storage Cluster == * Setup autologin {{{ = fly = mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1 mount -t lustre /dev/sdb1 /home/mdt = fly01 ~ fly07 = mkfs.lustre --fsname=flyfs --ost --mgsnode=fly@tcp0 /dev/sdb mount -t lustre /dev/sdb /home/ost = fly = mkdir /home/flyfs mount -t lustre fly@tcp0:/flyfs /home/flyfs }}} == 6.Install TurboJPEG == rock@fly:~/turbojpeg> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm [[BR]] rock@fly:~/turbojpeg> sudo rpm -i turbojpeg-1.10.3.x86_64.rpm [[BR]] rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg [[BR]] {{{ normal /usr/bin/switchtjpeg normal /usr/include/turbojpeg.h normal /usr/lib/libturbojpeg-ipp.so normal /usr/lib/libturbojpeg-libjpeg.so normal /usr/lib/libturbojpeg.so normal /usr/lib64/libturbojpeg-ipp.so normal /usr/lib64/libturbojpeg-libjpeg.so normal /usr/lib64/libturbojpeg.so normal /usr/share/doc/turbojpeg-1.10.3 normal /usr/share/doc/turbojpeg-1.10.3/LGPL.txt normal /usr/share/doc/turbojpeg-1.10.3/LICENSE.txt normal /usr/share/doc/turbojpeg-1.10.3/README.txt }}} rock@fly:~/VirtualGL> switchtjpeg --help [[BR]] {{{ USAGE: /usr/bin/switchtjpeg < ipp | libjpeg > ipp = Use the accelerated version of TurboJPEG, which embeds functions from the Intel(R) Integrated Performance Primitives libjpeg = Use the unaccelerated version of TurboJPEG, which is GPL-compatible }}} == 7.Install VirtualGL == rock@fly:~/VirtualGL> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm [[BR]] rock@fly:~/VirtualGL> sudo rpm -i VirtualGL-2.1.x86_64.rpm [[BR]] rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL {{{ normal /opt/VirtualGL normal /opt/VirtualGL/bin normal /opt/VirtualGL/bin/cpustat normal /opt/VirtualGL/bin/glxinfo normal /opt/VirtualGL/bin/glxspheres normal /opt/VirtualGL/bin/glxspheres64 normal /opt/VirtualGL/bin/nettest normal /opt/VirtualGL/bin/tcbench normal /opt/VirtualGL/bin/vglclient normal /opt/VirtualGL/bin/vglconfig normal /opt/VirtualGL/bin/vglconfig64 normal /opt/VirtualGL/bin/vglconnect normal /opt/VirtualGL/bin/vglgenkey normal /opt/VirtualGL/bin/vgllogin normal /opt/VirtualGL/bin/vglrun normal /opt/VirtualGL/bin/vglserver_config normal /opt/VirtualGL/doc normal /opt/VirtualGL/fakelib normal /opt/VirtualGL/fakelib/64 normal /opt/VirtualGL/fakelib/64/libGL.so normal /opt/VirtualGL/fakelib/libGL.so normal /opt/VirtualGL/lib normal /opt/VirtualGL/lib64 normal /usr/bin/vglclient normal /usr/bin/vglconfig normal /usr/bin/vglconfig64 normal /usr/bin/vglconnect normal /usr/bin/vglgenkey normal /usr/bin/vgllogin normal /usr/bin/vglrun normal /usr/bin/vglserver_config normal /usr/lib/libdlfaker.so normal /usr/lib/libgefaker.so normal /usr/lib/librrfaker.so normal /usr/lib64/libdlfaker.so normal /usr/lib64/libgefaker.so normal /usr/lib64/librrfaker.so normal /usr/share/doc/VirtualGL-2.1 normal /usr/share/doc/VirtualGL-2.1/411.gif normal /usr/share/doc/VirtualGL-2.1/422.gif normal /usr/share/doc/VirtualGL-2.1/444.gif normal /usr/share/doc/VirtualGL-2.1/ChangeLog.txt normal /usr/share/doc/VirtualGL-2.1/LGPL.txt normal /usr/share/doc/VirtualGL-2.1/LICENSE-OpenSSL.txt normal /usr/share/doc/VirtualGL-2.1/LICENSE.txt normal /usr/share/doc/VirtualGL-2.1/chromium-displaywall.png normal /usr/share/doc/VirtualGL-2.1/chromium-sortfirst.png normal /usr/share/doc/VirtualGL-2.1/chromium-sortlast.png normal /usr/share/doc/VirtualGL-2.1/configdialog.gif normal /usr/share/doc/VirtualGL-2.1/exceed1.png normal /usr/share/doc/VirtualGL-2.1/exceed2.png normal /usr/share/doc/VirtualGL-2.1/exceed3.png normal /usr/share/doc/VirtualGL-2.1/exceed6.png normal /usr/share/doc/VirtualGL-2.1/index.html normal /usr/share/doc/VirtualGL-2.1/somerights20.png normal /usr/share/doc/VirtualGL-2.1/sshtunnel.png normal /usr/share/doc/VirtualGL-2.1/sunray.png normal /usr/share/doc/VirtualGL-2.1/turbovnc1.png normal /usr/share/doc/VirtualGL-2.1/turbovnc2.png normal /usr/share/doc/VirtualGL-2.1/turbovnc3.png normal /usr/share/doc/VirtualGL-2.1/turbovnc4.png normal /usr/share/doc/VirtualGL-2.1/vgltransport.png normal /usr/share/doc/VirtualGL-2.1/vgltransportglp.png normal /usr/share/doc/VirtualGL-2.1/vgltransportservernetwork.png normal /usr/share/doc/VirtualGL-2.1/virtualgl.css normal /usr/share/doc/VirtualGL-2.1/x11transport.png normal /usr/share/doc/VirtualGL-2.1/x11transportglp.png }}} rock@fly:~/VirtualGL> vglrun --help {{{ Usage: vglrun [options] [--] [OpenGL app arguments] VirtualGL options -c : proxy = Send 3D images uncompressed using X11 transport jpeg = Compress 3D images using JPEG/send using VGL transport rgb = Encode 3D images as RGB/send using VGL transport sr = Compress 3D images using DPCM/send using Sun Ray transport srrgb = Encode 3D images as RGB/send using Sun Ray transport -cl : = The hostname or IP address of the VirtualGL client (or Sun Ray server) -dl : Force apps that dlopen() libGL to load VirtualGL instead -d : = the X Display or GLP device to use for 3D rendering -fps : Limit client/server frame rate to frames/sec +/-g : Enable/disable gamma correction -gamma : Set gamma correction factor to (see docs) -ge : Fool application into thinking that LD_PRELOAD is unset -np : Use CPUs to perform image compression (VGL transport only) -p

: TCP port to use when connecting to vglclient (VGL transport only) +/-pr : Enable/disable performance profiling output -q : JPEG compression quality [1 <= <= 100] +/-s : Enable/disable SSL encryption of VGL image transport -samp : JPEG chrominance subsampling factor [ = gray, 1x, 2x, 4x, 8x, or 16x] +/-sp : Turn on/off frame spoiling -st : left = Send only left eye buffer right = Send only right eye buffer quad = Use quad-buffered stereo if available, otherwise use red/cyan (anaglyphic) stereo [default] rc = Always use red/cyan (anaglyphic) stereo +/-sync : Enable/disable strict 2D/3D synchronization +/-tr : Enable/disable function call tracing (generates a lot of output) +/-v : Enable/disable verbose VirtualGL messages -64 : (Solaris only) Preload VirtualGL only into 64-bit apps -32 : (Solaris only) Preload VirtualGL only into 32-bit apps }}} == 8.Install TurboVNC == rock@fly:~/turbovnc> wget http://downloads.sourceforge.net/virtualgl/turbovnc-0.4.i386.rpm?modtime=1205803835&big_mirror=0 [[BR]] rock@fly:~/turbovnc> sudo rpm -i turbovnc-0.4.i386.rpm [[BR]] rock@fly:~/turbovnc> rpm -qs turbovnc {{{ normal /etc/X11/applnk/Applications/tvncviewer.desktop normal /etc/init.d/tvncserver normal /etc/sysconfig/tvncservers normal /opt/TurboVNC normal /opt/TurboVNC/bin normal /opt/TurboVNC/bin/Xvnc normal /opt/TurboVNC/bin/vncconnect normal /opt/TurboVNC/bin/vncpasswd normal /opt/TurboVNC/bin/vncserver normal /opt/TurboVNC/bin/vncviewer normal /opt/TurboVNC/man normal /opt/TurboVNC/man/man1 normal /opt/TurboVNC/man/man1/Xvnc.1 normal /opt/TurboVNC/man/man1/vncconnect.1 normal /opt/TurboVNC/man/man1/vncpasswd.1 normal /opt/TurboVNC/man/man1/vncserver.1 normal /opt/TurboVNC/man/man1/vncviewer.1 normal /opt/TurboVNC/vnc normal /opt/TurboVNC/vnc/classes normal /opt/TurboVNC/vnc/classes/AuthPanel.class normal /opt/TurboVNC/vnc/classes/ButtonPanel.class normal /opt/TurboVNC/vnc/classes/CapabilityInfo.class normal /opt/TurboVNC/vnc/classes/CapsContainer.class normal /opt/TurboVNC/vnc/classes/ClipboardFrame.class normal /opt/TurboVNC/vnc/classes/DesCipher.class normal /opt/TurboVNC/vnc/classes/InStream.class normal /opt/TurboVNC/vnc/classes/MemInStream.class normal /opt/TurboVNC/vnc/classes/OptionsFrame.class normal /opt/TurboVNC/vnc/classes/RFBUpdateEvent.class normal /opt/TurboVNC/vnc/classes/RecordingFrame.class normal /opt/TurboVNC/vnc/classes/ReloginPanel.class normal /opt/TurboVNC/vnc/classes/RfbProto.class normal /opt/TurboVNC/vnc/classes/SessionRecorder.class normal /opt/TurboVNC/vnc/classes/SocketFactory.class normal /opt/TurboVNC/vnc/classes/VncCanvas.class normal /opt/TurboVNC/vnc/classes/VncCanvas2.class normal /opt/TurboVNC/vnc/classes/VncViewer.class normal /opt/TurboVNC/vnc/classes/VncViewer.jar normal /opt/TurboVNC/vnc/classes/ZlibInStream.class normal /opt/TurboVNC/vnc/classes/index.vnc normal /usr/share/doc/turbovnc-0.4 normal /usr/share/doc/turbovnc-0.4/LICENCE.TXT normal /usr/share/doc/turbovnc-0.4/TurboVNC-ChangeLog.txt normal /usr/share/doc/turbovnc-0.4/WhatsNew }}} rock@fly:~> sudo zypper install xorg-x11-fonts xorg-x11-fonts-core xorg-x11-fonts-devel [[BR]] rock@fly:~> sudo zypper install xorg-x11-devel xorg-x11-driver-input xorg-x11-driver-video [[BR]] rock@fly:~> sudo zypper install libdrm libdrm-devel Mesa Mesa-devel [[BR]] rock@fly:~> sudo zypper remove tomboy [[BR]] rock@fly:/usr/X11R6/lib> sudo /etc/init.d/xfs start [[BR]] rock@fly:~> cd /usr/X11R6/lib [[BR]] rock@fly:/usr/X11R6/lib> sudo mkdir X11 [[BR]] rock@fly:~> sudo cp rgb.txt /usr/X11R6/lib/X11/ [[BR]] rock@fly:~> sudo vim /etc/X11/xorg.conf [[BR]] {{{ Section "Module" Load "dbe" Load "type1" Load "freetype" Load "extmod" Load "glx" Load "v4l" Load "drm" Load "dri" }}} {{{ Section "DRI" Group "video" Mode 0666 EndSection }}} rock@fly:~> sudo vim /etc/X11/fs/config [[BR]] {{{ #no-listen = tcp }}} rock@fly:~> /opt/TurboVNC/bin/vncserver -geometry 1280x1024 :1 [[BR]] New 'X' desktop is fly:1 Creating default startup script /home/rock/.vnc/xstartup Starting applications specified in /home/rock/.vnc/xstartup Log file is /home/rock/.vnc/fly:1.log rock@fly:~> ps aux | grep X {{{ root 4216 0.0 0.9 75012 19760 tty7 Ss+ Jul03 0:04 /usr/bin/X :0 -audit 0 -br -auth /var/lib/gdm/:0.Xauth vt7 rock 4811 0.0 0.0 46628 988 ? Ss Jul03 0:00 /usr/bin/ssh-agent /bin/bash /etc/X11/xinit/xinitrc rock 20758 1.1 0.6 16760 12900 pts/9 S 00:56 0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop X -httpd /opt/TurboVNC/bin/../vnc/classes -auth /home/rock/.Xauthority -dontdisconnect -geometry 1240x900 -depth 24 -rfbwait 120000 -rfbauth /home/rock/.vnc/passwd -rfbport 5901 -fp unix/:7100 -deferupdate 1 rock 22008 0.0 0.0 5004 808 pts/9 S+ 00:56 0:00 grep X }}} ### TurboVNC Initialization Bug fixed for ALSA Library ### [[BR]] rock@fly:~/alsa> wget http://gd.tuwien.ac.at/opsys/linux/alsa/lib/alsa-lib-1.0.13.tar.bz2 [[BR]] rock@fly:~/alsa> tar jxvf alsa-lib-1.0.13.tar.bz2 [[BR]] rock@fly:~/alsa> cd alsa-lib-1.0.13 [[BR]] rock@fly:~/alsa/alsa-lib-1.0.13> ./configure --enable-static [[BR]] rock@fly:~/alsa/alsa-lib-1.0.13> make -j 4 [[BR]] rock@fly:~/alsa/alsa-lib-1.0.13> sudo make install [[BR]] openSUSE(GUI)-> Computer -> Control Center -> System -> Sound -> Choose ALSA for each option. [[BR]] == 9.Install Amira == Necessary Library: libg2c.so.0 [[BR]] Download libg2c33(Fortran Compiler Runtime Library) for openSUSE 10.2 [[BR]] rock@fly:~/amira_source/download_lib> wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/algraf/openSUSE_10.2/x86_64/libg2c33-3.3.3-43.3.x86_64.rpm [[BR]] rock@fly:~/amira_source/download_lib> sudo rpm -i libg2c33-3.3.3-43.3.x86_64.rpm [[BR]] rock@fly:~/amira_source> chmod a+x Amira411-LinuxAMD64.bin [[BR]] rock@fly:~/amira_source> ./Amira411-LinuxAMD64.bin --console (console mode)[[BR]] Installation succeed !!! [[BR]] Installation path: /home/rock/Amira4.1.1 [[BR]] rock@fly:~/Amira4.1.1/bin> ./start --console (console mode)[[BR]] Finished ! It works... == 10.Test Amira == == Reference == * Lustre, http://wiki.lustre.org/ * Amira, http://www.amiravis.com/