Build Lustre Storage Cluster for 3D Fly platform

0.Introduction

Some domain-applications usually need to computing huge data. It usually meet a bottleneck problem of performance. It will lead I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance. You can refer to Lustre web site ​http://wiki.lustre.org/index.php?title=Main_Page

1.Machine information & Software

• Hardware

Node	8 nodes (1 server , 7 client provide disks)
CPU	Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory	2GB DDR2 667 (each node)
Disk	All nodes: 320G * 7 = 2.24T
NIC	Intel Corporation 82566DM Gigabit Network Connection
Switch	D-link 24 port GE switch

• Software

OS	OpenSuSE 10.2
DRBL	1.9.0-35
Amira	Testing Version
Kernel	2.6.18

• Architecture

[[Attachement:fly_arch.png]]

2.Config OpenSuSE

• Install packages for Lustre
• Install packages for DRBL

3.Install Lustre

• Build kernel

  $ sudo su
  $ cd /usr/src
  $ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
  $ tar jxvf linux-2.6.18.tar.bz2
  $ cd 2.6.18
  $ cp /boot/config-xxx .config
  $ make oldconfig ; make menuconfig
  $ make -j 5
  $ make modules
  $ make modules_install
  $ make install
  $ mkinitrd
  $ vim /boot/grub/menu.list
  

• recompiler NIC driver

  wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw
  tar e1000-7.6.15.4.tar.gz
  cd e1000-7.6.15.4/src
  make install
  

• Bonding NIC (bond0 -> eth1 & eth2)

  $ cd /etc/sysconfig/network
  $ vim ifcfg-eth1 
  $ vim ifeth-eth2
  $ vim ifcfg-bond0
  $ vim /etc/rc.d/after.local
  $ vim /etc/module/modules.conf.local
  $ vim /etc/rc.d/after.local
  
  

• Build Lustre Modules

  $ configure --with-linux=/usr/src/2.6.18
  $ make install
  $ depmod -a
  $ vim /etc/rc.d/after.local
  

4.Build DRBL Environment

$ 
check node_root/etc/rc.d/module.conf.local
check nodes/192.168.1.X/etc/rc.d/module.conf.local
rm /tftfpood/node_root/sysconf/network/if

5.Build Lustre Storage Cluster

• setup autologin

  = fly =
  mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1
  mount -t lustre /dev/sdb1 /home/mdt
  = fly01 ~ fly07 =
  mkfs.lustre --fsname=flyfs --ost  --mgsnode=fly@tcp0 /dev/sdb
  mount -t lustre /dev/sdb /home/ost
  = fly =
  mkdir /home/flyfs
  mount -t lustre fly@tcp0:/flyfs /home/flyfs
  
  

6.Install TurboJPEG

rock@fly:~/turbojpeg> wget ​ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -i turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg

normal        /usr/bin/switchtjpeg
normal        /usr/include/turbojpeg.h
normal        /usr/lib/libturbojpeg-ipp.so
normal        /usr/lib/libturbojpeg-libjpeg.so
normal        /usr/lib/libturbojpeg.so
normal        /usr/lib64/libturbojpeg-ipp.so
normal        /usr/lib64/libturbojpeg-libjpeg.so
normal        /usr/lib64/libturbojpeg.so
normal        /usr/share/doc/turbojpeg-1.10.3
normal        /usr/share/doc/turbojpeg-1.10.3/LGPL.txt
normal        /usr/share/doc/turbojpeg-1.10.3/LICENSE.txt
normal        /usr/share/doc/turbojpeg-1.10.3/README.txt

rock@fly:~/VirtualGL> switchtjpeg --help

USAGE: /usr/bin/switchtjpeg < ipp | libjpeg >

ipp = Use the accelerated version of TurboJPEG, which embeds functions from
      the Intel(R) Integrated Performance Primitives
libjpeg = Use the unaccelerated version of TurboJPEG, which is
          GPL-compatible

7.Install VirtualGL

rock@fly:~/VirtualGL> wget ​ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -i VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL

normal        /opt/VirtualGL
normal        /opt/VirtualGL/bin
normal        /opt/VirtualGL/bin/cpustat
normal        /opt/VirtualGL/bin/glxinfo
normal        /opt/VirtualGL/bin/glxspheres
normal        /opt/VirtualGL/bin/glxspheres64
normal        /opt/VirtualGL/bin/nettest
normal        /opt/VirtualGL/bin/tcbench
normal        /opt/VirtualGL/bin/vglclient
normal        /opt/VirtualGL/bin/vglconfig
normal        /opt/VirtualGL/bin/vglconfig64
normal        /opt/VirtualGL/bin/vglconnect
normal        /opt/VirtualGL/bin/vglgenkey
normal        /opt/VirtualGL/bin/vgllogin
normal        /opt/VirtualGL/bin/vglrun
normal        /opt/VirtualGL/bin/vglserver_config
normal        /opt/VirtualGL/doc
normal        /opt/VirtualGL/fakelib
normal        /opt/VirtualGL/fakelib/64
normal        /opt/VirtualGL/fakelib/64/libGL.so
normal        /opt/VirtualGL/fakelib/libGL.so
normal        /opt/VirtualGL/lib
normal        /opt/VirtualGL/lib64
normal        /usr/bin/vglclient
normal        /usr/bin/vglconfig
normal        /usr/bin/vglconfig64
normal        /usr/bin/vglconnect
normal        /usr/bin/vglgenkey
normal        /usr/bin/vgllogin
normal        /usr/bin/vglrun
normal        /usr/bin/vglserver_config
normal        /usr/lib/libdlfaker.so
normal        /usr/lib/libgefaker.so
normal        /usr/lib/librrfaker.so
normal        /usr/lib64/libdlfaker.so
normal        /usr/lib64/libgefaker.so
normal        /usr/lib64/librrfaker.so
normal        /usr/share/doc/VirtualGL-2.1
normal        /usr/share/doc/VirtualGL-2.1/411.gif
normal        /usr/share/doc/VirtualGL-2.1/422.gif
normal        /usr/share/doc/VirtualGL-2.1/444.gif
normal        /usr/share/doc/VirtualGL-2.1/ChangeLog.txt
normal        /usr/share/doc/VirtualGL-2.1/LGPL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE-OpenSSL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE.txt
normal        /usr/share/doc/VirtualGL-2.1/chromium-displaywall.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortfirst.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortlast.png
normal        /usr/share/doc/VirtualGL-2.1/configdialog.gif
normal        /usr/share/doc/VirtualGL-2.1/exceed1.png
normal        /usr/share/doc/VirtualGL-2.1/exceed2.png
normal        /usr/share/doc/VirtualGL-2.1/exceed3.png
normal        /usr/share/doc/VirtualGL-2.1/exceed6.png
normal        /usr/share/doc/VirtualGL-2.1/index.html
normal        /usr/share/doc/VirtualGL-2.1/somerights20.png
normal        /usr/share/doc/VirtualGL-2.1/sshtunnel.png
normal        /usr/share/doc/VirtualGL-2.1/sunray.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc1.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc2.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc3.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc4.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransport.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportglp.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportservernetwork.png
normal        /usr/share/doc/VirtualGL-2.1/virtualgl.css
normal        /usr/share/doc/VirtualGL-2.1/x11transport.png
normal        /usr/share/doc/VirtualGL-2.1/x11transportglp.png

rock@fly:~/VirtualGL> vglrun --help

Usage: vglrun [options] [--] <OpenGL app> [OpenGL app arguments]

VirtualGL options
-c <c>    : proxy = Send 3D images uncompressed using X11 transport
            jpeg = Compress 3D images using JPEG/send using VGL transport
            rgb = Encode 3D images as RGB/send using VGL transport
            sr = Compress 3D images using DPCM/send using Sun Ray transport
            srrgb = Encode 3D images as RGB/send using Sun Ray transport
-cl <c>   : <c> = The hostname or IP address of the VirtualGL client (or
                  Sun Ray server)
-dl       : Force apps that dlopen() libGL to load VirtualGL instead
-d <d>    : <d> = the X Display or GLP device to use for 3D rendering
-fps <f>  : Limit client/server frame rate to <f> frames/sec
+/-g      : Enable/disable gamma correction
-gamma <g>: Set gamma correction factor to <g> (see docs)
-ge       : Fool application into thinking that LD_PRELOAD is unset
-np <n>   : Use <n> CPUs to perform image compression (VGL transport only)
-p <p>    : TCP port to use when connecting to vglclient (VGL transport only)
+/-pr     : Enable/disable performance profiling output
-q <q>    : JPEG compression quality [1 <= <q> <= 100]
+/-s      : Enable/disable SSL encryption of VGL image transport
-samp <s> : JPEG chrominance subsampling factor
            [<s> = gray, 1x, 2x, 4x, 8x, or 16x]
+/-sp     : Turn on/off frame spoiling
-st <s>   : left = Send only left eye buffer
            right = Send only right eye buffer
            quad = Use quad-buffered stereo if available, otherwise use
                   red/cyan (anaglyphic) stereo [default]
            rc = Always use red/cyan (anaglyphic) stereo
+/-sync   : Enable/disable strict 2D/3D synchronization
+/-tr     : Enable/disable function call tracing (generates a lot of output)
+/-v      : Enable/disable verbose VirtualGL messages
-64       : (Solaris only) Preload VirtualGL only into 64-bit apps
-32       : (Solaris only) Preload VirtualGL only into 32-bit apps

8.Install TurboVNC

9.Install Amira

10.Test Amira

Reference