wiki:3D_Lustre

Version 10 (modified by rider, 16 years ago) (diff)

--

  1. Build Lustre Storage Cluster for 3D Fly platform
    1. 0.Introduction
    2. 1.Machine information & Software
    3. 2.Config OpenSuSE
    4. 3.Install Lustre
    5. 4.Build DRBL Environment
    6. 5.Build Lustre Storage Cluster
    7. 6.Install TurboJPEG
    8. 7.Install VirtualGL
    9. 8.Install TurboVNC
    10. 9.Install Amira
    11. 10.Test Amira
    12. Reference

Build Lustre Storage Cluster for 3D Fly platform

0.Introduction

Some domain-applications usually need to computing huge data. It usually meet a bottleneck problem of performance. It will lead I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance. You can refer to Lustre web site ​http://wiki.lustre.org/index.php?title=Main_Page

1.Machine information & Software

  • Hardware
Node 8 nodes (1 server , 7 client provide disks)
CPU Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory 2GB DDR2 667 (each node)
Disk All nodes: 320G * 7 = 2.24T
NIC Intel Corporation 82566DM Gigabit Network Connection
Switch D-link 24 port GE switch
  • Software
OSOpenSuSE 10.2
DRBL1.9.0-35
AmiraTesting Version
Kernel2.6.18
  • Architecture

3D fly architecture

2.Config OpenSuSE

  • Install packages for Lustre
  • Install packages for DRBL

3.Install Lustre

  • Build kernel 
    $ sudo su
$ cd /usr/src
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
$ tar jxvf linux-2.6.18.tar.bz2
$ cd 2.6.18
$ cp /boot/config-xxx .config
$ make oldconfig ; make menuconfig
$ make -j 5
$ make modules
$ make modules_install
$ make install
$ mkinitrd
$ vim /boot/grub/menu.list
  • recompiler NIC driver 
    wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw
tar e1000-7.6.15.4.tar.gz
cd e1000-7.6.15.4/src
make install
  • Bonding NIC (bond0 -> eth1 & eth2) 
    $ cd /etc/sysconfig/network
$ vim ifcfg-eth1 
$ vim ifeth-eth2
$ vim ifcfg-bond0
$ vim /etc/rc.d/after.local
$ vim /etc/module/modules.conf.local
$ vim /etc/rc.d/after.local
  • Build Lustre Modules 
    $ configure --with-linux=/usr/src/2.6.18
$ make install
$ depmod -a
$ vim /etc/rc.d/after.local

4.Build DRBL Environment

$ 
check node_root/etc/rc.d/module.conf.local
check nodes/192.168.1.X/etc/rc.d/module.conf.local
rm /tftfpood/node_root/sysconf/network/if

5.Build Lustre Storage Cluster

  • setup autologin 
    = fly =
mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1
mount -t lustre /dev/sdb1 /home/mdt
= fly01 ~ fly07 =
mkfs.lustre --fsname=flyfs --ost  --mgsnode=fly@tcp0 /dev/sdb
mount -t lustre /dev/sdb /home/ost
= fly =
mkdir /home/flyfs
mount -t lustre fly@tcp0:/flyfs /home/flyfs

6.Install TurboJPEG

rock@fly:~/turbojpeg> wget ​ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -i turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg 

normal        /usr/bin/switchtjpeg
normal        /usr/include/turbojpeg.h
normal        /usr/lib/libturbojpeg-ipp.so
normal        /usr/lib/libturbojpeg-libjpeg.so
normal        /usr/lib/libturbojpeg.so
normal        /usr/lib64/libturbojpeg-ipp.so
normal        /usr/lib64/libturbojpeg-libjpeg.so
normal        /usr/lib64/libturbojpeg.so
normal        /usr/share/doc/turbojpeg-1.10.3
normal        /usr/share/doc/turbojpeg-1.10.3/LGPL.txt
normal        /usr/share/doc/turbojpeg-1.10.3/LICENSE.txt
normal        /usr/share/doc/turbojpeg-1.10.3/README.txt

rock@fly:~/VirtualGL> switchtjpeg --help 

USAGE: /usr/bin/switchtjpeg < ipp | libjpeg >

ipp = Use the accelerated version of TurboJPEG, which embeds functions from
      the Intel(R) Integrated Performance Primitives
libjpeg = Use the unaccelerated version of TurboJPEG, which is
          GPL-compatible

7.Install VirtualGL

rock@fly:~/VirtualGL> wget ​ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -i VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL 

normal        /opt/VirtualGL
normal        /opt/VirtualGL/bin
normal        /opt/VirtualGL/bin/cpustat
normal        /opt/VirtualGL/bin/glxinfo
normal        /opt/VirtualGL/bin/glxspheres
normal        /opt/VirtualGL/bin/glxspheres64
normal        /opt/VirtualGL/bin/nettest
normal        /opt/VirtualGL/bin/tcbench
normal        /opt/VirtualGL/bin/vglclient
normal        /opt/VirtualGL/bin/vglconfig
normal        /opt/VirtualGL/bin/vglconfig64
normal        /opt/VirtualGL/bin/vglconnect
normal        /opt/VirtualGL/bin/vglgenkey
normal        /opt/VirtualGL/bin/vgllogin
normal        /opt/VirtualGL/bin/vglrun
normal        /opt/VirtualGL/bin/vglserver_config
normal        /opt/VirtualGL/doc
normal        /opt/VirtualGL/fakelib
normal        /opt/VirtualGL/fakelib/64
normal        /opt/VirtualGL/fakelib/64/libGL.so
normal        /opt/VirtualGL/fakelib/libGL.so
normal        /opt/VirtualGL/lib
normal        /opt/VirtualGL/lib64
normal        /usr/bin/vglclient
normal        /usr/bin/vglconfig
normal        /usr/bin/vglconfig64
normal        /usr/bin/vglconnect
normal        /usr/bin/vglgenkey
normal        /usr/bin/vgllogin
normal        /usr/bin/vglrun
normal        /usr/bin/vglserver_config
normal        /usr/lib/libdlfaker.so
normal        /usr/lib/libgefaker.so
normal        /usr/lib/librrfaker.so
normal        /usr/lib64/libdlfaker.so
normal        /usr/lib64/libgefaker.so
normal        /usr/lib64/librrfaker.so
normal        /usr/share/doc/VirtualGL-2.1
normal        /usr/share/doc/VirtualGL-2.1/411.gif
normal        /usr/share/doc/VirtualGL-2.1/422.gif
normal        /usr/share/doc/VirtualGL-2.1/444.gif
normal        /usr/share/doc/VirtualGL-2.1/ChangeLog.txt
normal        /usr/share/doc/VirtualGL-2.1/LGPL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE-OpenSSL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE.txt
normal        /usr/share/doc/VirtualGL-2.1/chromium-displaywall.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortfirst.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortlast.png
normal        /usr/share/doc/VirtualGL-2.1/configdialog.gif
normal        /usr/share/doc/VirtualGL-2.1/exceed1.png
normal        /usr/share/doc/VirtualGL-2.1/exceed2.png
normal        /usr/share/doc/VirtualGL-2.1/exceed3.png
normal        /usr/share/doc/VirtualGL-2.1/exceed6.png
normal        /usr/share/doc/VirtualGL-2.1/index.html
normal        /usr/share/doc/VirtualGL-2.1/somerights20.png
normal        /usr/share/doc/VirtualGL-2.1/sshtunnel.png
normal        /usr/share/doc/VirtualGL-2.1/sunray.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc1.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc2.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc3.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc4.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransport.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportglp.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportservernetwork.png
normal        /usr/share/doc/VirtualGL-2.1/virtualgl.css
normal        /usr/share/doc/VirtualGL-2.1/x11transport.png
normal        /usr/share/doc/VirtualGL-2.1/x11transportglp.png

rock@fly:~/VirtualGL> vglrun --help 

Usage: vglrun [options] [--] <OpenGL app> [OpenGL app arguments]

VirtualGL options
-c <c>    : proxy = Send 3D images uncompressed using X11 transport
            jpeg = Compress 3D images using JPEG/send using VGL transport
            rgb = Encode 3D images as RGB/send using VGL transport
            sr = Compress 3D images using DPCM/send using Sun Ray transport
            srrgb = Encode 3D images as RGB/send using Sun Ray transport
-cl <c>   : <c> = The hostname or IP address of the VirtualGL client (or
                  Sun Ray server)
-dl       : Force apps that dlopen() libGL to load VirtualGL instead
-d <d>    : <d> = the X Display or GLP device to use for 3D rendering
-fps <f>  : Limit client/server frame rate to <f> frames/sec
+/-g      : Enable/disable gamma correction
-gamma <g>: Set gamma correction factor to <g> (see docs)
-ge       : Fool application into thinking that LD_PRELOAD is unset
-np <n>   : Use <n> CPUs to perform image compression (VGL transport only)
-p <p>    : TCP port to use when connecting to vglclient (VGL transport only)
+/-pr     : Enable/disable performance profiling output
-q <q>    : JPEG compression quality [1 <= <q> <= 100]
+/-s      : Enable/disable SSL encryption of VGL image transport
-samp <s> : JPEG chrominance subsampling factor
            [<s> = gray, 1x, 2x, 4x, 8x, or 16x]
+/-sp     : Turn on/off frame spoiling
-st <s>   : left = Send only left eye buffer
            right = Send only right eye buffer
            quad = Use quad-buffered stereo if available, otherwise use
                   red/cyan (anaglyphic) stereo [default]
            rc = Always use red/cyan (anaglyphic) stereo
+/-sync   : Enable/disable strict 2D/3D synchronization
+/-tr     : Enable/disable function call tracing (generates a lot of output)
+/-v      : Enable/disable verbose VirtualGL messages
-64       : (Solaris only) Preload VirtualGL only into 64-bit apps
-32       : (Solaris only) Preload VirtualGL only into 32-bit apps

8.Install TurboVNC

rock@fly:~/turbovnc> wget ​http://downloads.sourceforge.net/virtualgl/turbovnc-0.4.i386.rpm?modtime=1205803835&big_mirror=0
rock@fly:~/turbovnc> sudo rpm -i turbovnc-0.4.i386.rpm
rock@fly:~/turbovnc> rpm -qs turbovnc 

normal        /etc/X11/applnk/Applications/tvncviewer.desktop
normal        /etc/init.d/tvncserver
normal        /etc/sysconfig/tvncservers
normal        /opt/TurboVNC
normal        /opt/TurboVNC/bin
normal        /opt/TurboVNC/bin/Xvnc
normal        /opt/TurboVNC/bin/vncconnect
normal        /opt/TurboVNC/bin/vncpasswd
normal        /opt/TurboVNC/bin/vncserver
normal        /opt/TurboVNC/bin/vncviewer
normal        /opt/TurboVNC/man
normal        /opt/TurboVNC/man/man1
normal        /opt/TurboVNC/man/man1/Xvnc.1
normal        /opt/TurboVNC/man/man1/vncconnect.1
normal        /opt/TurboVNC/man/man1/vncpasswd.1
normal        /opt/TurboVNC/man/man1/vncserver.1
normal        /opt/TurboVNC/man/man1/vncviewer.1
normal        /opt/TurboVNC/vnc
normal        /opt/TurboVNC/vnc/classes
normal        /opt/TurboVNC/vnc/classes/AuthPanel.class
normal        /opt/TurboVNC/vnc/classes/ButtonPanel.class
normal        /opt/TurboVNC/vnc/classes/CapabilityInfo.class
normal        /opt/TurboVNC/vnc/classes/CapsContainer.class
normal        /opt/TurboVNC/vnc/classes/ClipboardFrame.class
normal        /opt/TurboVNC/vnc/classes/DesCipher.class
normal        /opt/TurboVNC/vnc/classes/InStream.class
normal        /opt/TurboVNC/vnc/classes/MemInStream.class
normal        /opt/TurboVNC/vnc/classes/OptionsFrame.class
normal        /opt/TurboVNC/vnc/classes/RFBUpdateEvent.class
normal        /opt/TurboVNC/vnc/classes/RecordingFrame.class
normal        /opt/TurboVNC/vnc/classes/ReloginPanel.class
normal        /opt/TurboVNC/vnc/classes/RfbProto.class
normal        /opt/TurboVNC/vnc/classes/SessionRecorder.class
normal        /opt/TurboVNC/vnc/classes/SocketFactory.class
normal        /opt/TurboVNC/vnc/classes/VncCanvas.class
normal        /opt/TurboVNC/vnc/classes/VncCanvas2.class
normal        /opt/TurboVNC/vnc/classes/VncViewer.class
normal        /opt/TurboVNC/vnc/classes/VncViewer.jar
normal        /opt/TurboVNC/vnc/classes/ZlibInStream.class
normal        /opt/TurboVNC/vnc/classes/index.vnc
normal        /usr/share/doc/turbovnc-0.4
normal        /usr/share/doc/turbovnc-0.4/LICENCE.TXT
normal        /usr/share/doc/turbovnc-0.4/TurboVNC-ChangeLog.txt
normal        /usr/share/doc/turbovnc-0.4/WhatsNew

rock@fly:~> sudo zypper install xorg-x11-fonts xorg-x11-fonts-core xorg-x11-fonts-devel
rock@fly:~> sudo zypper install xorg-x11-devel xorg-x11-driver-input xorg-x11-driver-video
rock@fly:/usr/X11R6/lib> sudo /etc/init.d/xfs start
rock@fly:~> cd /usr/X11R6/lib
rock@fly:/usr/X11R6/lib> sudo mkdir X11
rock@fly:~> sudo cp rgb.txt /usr/X11R6/lib/X11/
rock@fly:~> sudo vim /etc/X11/xorg.conf 

Section "Module"
  Load         "dbe"
  Load         "type1"
  Load         "freetype"
  Load         "extmod"
  Load         "glx"
  Load         "v4l"
  Load         "drm"
  Load         "dri"

Section "DRI"
    Group      "video"
    Mode       0666
EndSection

rock@fly:~> sudo vim /etc/X11/fs/config 

#no-listen = tcp

rock@fly:~> /opt/TurboVNC/bin/vncserver

New 'X' desktop is fly:1

Creating default startup script /home/rock/.vnc/xstartup Starting applications specified in /home/rock/.vnc/xstartup Log file is /home/rock/.vnc/fly:1.log

rock@fly:~> ps aux | grep X 

root      4216  0.0  0.9  75012 19760 tty7     Ss+  Jul03   0:04 /usr/bin/X :0 -audit 0 -br -auth /var/lib/gdm/:0.Xauth vt7
rock      4811  0.0  0.0  46628   988 ?        Ss   Jul03   0:00 /usr/bin/ssh-agent /bin/bash /etc/X11/xinit/xinitrc
rock     20758  1.1  0.6  16760 12900 pts/9    S    00:56   0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop X -httpd /opt/TurboVNC/bin/../vnc/classes -auth /home/rock/.Xauthority -dontdisconnect -geometry 1240x900 -depth 24 -rfbwait 120000 -rfbauth /home/rock/.vnc/passwd -rfbport 5901 -fp unix/:7100 -deferupdate 1
rock     22008  0.0  0.0   5004   808 pts/9    S+   00:56   0:00 grep X

9.Install Amira

10.Test Amira

Reference

Attachments (4)

Download all attachments as: .zip