wiki:3D_Lustre

Version 17 (modified by rider, 16 years ago) (diff)

--

  1. Build Lustre Storage Cluster for 3D Fly platform
    1. 0.Introduction
    2. 1.Machine information & Software
    3. 2.Config OpenSuSE Environment
    4. 3.Install Lustre
    5. 4.Build DRBL Environment
    6. 5.Build Lustre Storage Cluster
    7. 6.Install TurboJPEG
    8. 7.Install VirtualGL
    9. 8.Install TurboVNC
    10. 9.Install Amira
    11. 10.NVIDIA GeForce? 7300 GT Installation
    12. Reference

Build Lustre Storage Cluster for 3D Fly platform

0.Introduction

Some domain-applications usually need to computing with huge data. It usually meet a bottleneck of performance, the main problem is I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance.
Abiut Lustre, you can refer to Lustre web site ​http://wiki.lustre.org/

1.Machine information & Software

  • Hardware
Node 8 nodes (1 server , 7 client provide disks)
CPU Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory 2GB DDR2 667 (each node)
Disk All nodes: 320G * 7 = 2.24T
NIC Intel Corporation 82566DM Gigabit Network Connection
Switch D-link 24 port GE switch
  • Software
OSOpenSuSE 10.2
DRBL1.9.0-35
AmiraTesting Version
Kernel2.6.18
  • Architecture

3D fly architecture

2.Config OpenSuSE Environment

  • Install packages for Lustre
  • Install packages for DRBL

3.Install Lustre

  • Build kernel 
    $ sudo su
$ cd /usr/src
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
$ tar jxvf linux-2.6.18.tar.bz2
$ cd 2.6.18
$ cp /boot/config-xxx .config
$ make oldconfig ; make menuconfig
$ make -j 5
$ make modules
$ make modules_install
$ make install
$ mkinitrd
$ vim /boot/grub/menu.list
  • recompiler NIC driver 
    wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw
tar e1000-7.6.15.4.tar.gz
cd e1000-7.6.15.4/src
make install
  • Bonding NIC (bond0 -> eth1 & eth2) 
    $ cd /etc/sysconfig/network
$ vim ifcfg-eth1 
$ vim ifeth-eth2
$ vim ifcfg-bond0
$ vim /etc/rc.d/after.local
ifenslave bond0 eth1 eth2
modprobe drm
modprobe ldiskfs
modprobe lnet networks=tcp

$ vim /etc/module/modules.conf.local
alias bond0 bonding
alias eth0 e1000
alias eth1 e1000
alias eth2 e1000
options bond0 mode=6 miimon=100
  • Build Lustre Modules 
    $ configure --with-linux=/usr/src/2.6.18
$ make install
$ depmod -a
$ vim /etc/rc.d/after.local

4.Build DRBL Environment

$ zypper install drbl
$ /opt/drbl/sbin/drblsrv -i 
$ /opt/drbl/sbin/drblpush -i
= Check nodes booting setup =
/tftpboot/nodes/192.168.1.X/etc/rc.d/after.local
/tftpboot/nodes/192.168.1.X/etc/module/modules.conf.local
/tftfpood/node_root/etc/sysconfig/network/

5.Build Lustre Storage Cluster

  • Setup autologin 
    = fly =
mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1
mount -t lustre /dev/sdb1 /home/mdt
= fly01 ~ fly07 =
mkfs.lustre --fsname=flyfs --ost  --mgsnode=fly@tcp0 /dev/sdb
mount -t lustre /dev/sdb /home/ost
= fly =
mkdir /home/flyfs
mount -t lustre fly@tcp0:/flyfs /home/flyfs

6.Install TurboJPEG

rock@fly:~/turbojpeg> wget ​ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -i turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg 

normal        /usr/bin/switchtjpeg
normal        /usr/include/turbojpeg.h
normal        /usr/lib/libturbojpeg-ipp.so
normal        /usr/lib/libturbojpeg-libjpeg.so
normal        /usr/lib/libturbojpeg.so
normal        /usr/lib64/libturbojpeg-ipp.so
normal        /usr/lib64/libturbojpeg-libjpeg.so
normal        /usr/lib64/libturbojpeg.so
normal        /usr/share/doc/turbojpeg-1.10.3
normal        /usr/share/doc/turbojpeg-1.10.3/LGPL.txt
normal        /usr/share/doc/turbojpeg-1.10.3/LICENSE.txt
normal        /usr/share/doc/turbojpeg-1.10.3/README.txt

rock@fly:~/VirtualGL> switchtjpeg --help 

USAGE: /usr/bin/switchtjpeg < ipp | libjpeg >

ipp = Use the accelerated version of TurboJPEG, which embeds functions from
      the Intel(R) Integrated Performance Primitives
libjpeg = Use the unaccelerated version of TurboJPEG, which is
          GPL-compatible

7.Install VirtualGL

rock@fly:~/VirtualGL> wget ​ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -i VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL 

normal        /opt/VirtualGL
normal        /opt/VirtualGL/bin
normal        /opt/VirtualGL/bin/cpustat
normal        /opt/VirtualGL/bin/glxinfo
normal        /opt/VirtualGL/bin/glxspheres
normal        /opt/VirtualGL/bin/glxspheres64
normal        /opt/VirtualGL/bin/nettest
normal        /opt/VirtualGL/bin/tcbench
normal        /opt/VirtualGL/bin/vglclient
normal        /opt/VirtualGL/bin/vglconfig
normal        /opt/VirtualGL/bin/vglconfig64
normal        /opt/VirtualGL/bin/vglconnect
normal        /opt/VirtualGL/bin/vglgenkey
normal        /opt/VirtualGL/bin/vgllogin
normal        /opt/VirtualGL/bin/vglrun
normal        /opt/VirtualGL/bin/vglserver_config
normal        /opt/VirtualGL/doc
normal        /opt/VirtualGL/fakelib
normal        /opt/VirtualGL/fakelib/64
normal        /opt/VirtualGL/fakelib/64/libGL.so
normal        /opt/VirtualGL/fakelib/libGL.so
normal        /opt/VirtualGL/lib
normal        /opt/VirtualGL/lib64
normal        /usr/bin/vglclient
normal        /usr/bin/vglconfig
normal        /usr/bin/vglconfig64
normal        /usr/bin/vglconnect
normal        /usr/bin/vglgenkey
normal        /usr/bin/vgllogin
normal        /usr/bin/vglrun
normal        /usr/bin/vglserver_config
normal        /usr/lib/libdlfaker.so
normal        /usr/lib/libgefaker.so
normal        /usr/lib/librrfaker.so
normal        /usr/lib64/libdlfaker.so
normal        /usr/lib64/libgefaker.so
normal        /usr/lib64/librrfaker.so
normal        /usr/share/doc/VirtualGL-2.1
normal        /usr/share/doc/VirtualGL-2.1/411.gif
normal        /usr/share/doc/VirtualGL-2.1/422.gif
normal        /usr/share/doc/VirtualGL-2.1/444.gif
normal        /usr/share/doc/VirtualGL-2.1/ChangeLog.txt
normal        /usr/share/doc/VirtualGL-2.1/LGPL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE-OpenSSL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE.txt
normal        /usr/share/doc/VirtualGL-2.1/chromium-displaywall.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortfirst.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortlast.png
normal        /usr/share/doc/VirtualGL-2.1/configdialog.gif
normal        /usr/share/doc/VirtualGL-2.1/exceed1.png
normal        /usr/share/doc/VirtualGL-2.1/exceed2.png
normal        /usr/share/doc/VirtualGL-2.1/exceed3.png
normal        /usr/share/doc/VirtualGL-2.1/exceed6.png
normal        /usr/share/doc/VirtualGL-2.1/index.html
normal        /usr/share/doc/VirtualGL-2.1/somerights20.png
normal        /usr/share/doc/VirtualGL-2.1/sshtunnel.png
normal        /usr/share/doc/VirtualGL-2.1/sunray.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc1.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc2.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc3.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc4.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransport.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportglp.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportservernetwork.png
normal        /usr/share/doc/VirtualGL-2.1/virtualgl.css
normal        /usr/share/doc/VirtualGL-2.1/x11transport.png
normal        /usr/share/doc/VirtualGL-2.1/x11transportglp.png

rock@fly:~/VirtualGL> vglrun --help 

Usage: vglrun [options] [--] <OpenGL app> [OpenGL app arguments]

VirtualGL options
-c <c>    : proxy = Send 3D images uncompressed using X11 transport
            jpeg = Compress 3D images using JPEG/send using VGL transport
            rgb = Encode 3D images as RGB/send using VGL transport
            sr = Compress 3D images using DPCM/send using Sun Ray transport
            srrgb = Encode 3D images as RGB/send using Sun Ray transport
-cl <c>   : <c> = The hostname or IP address of the VirtualGL client (or
                  Sun Ray server)
-dl       : Force apps that dlopen() libGL to load VirtualGL instead
-d <d>    : <d> = the X Display or GLP device to use for 3D rendering
-fps <f>  : Limit client/server frame rate to <f> frames/sec
+/-g      : Enable/disable gamma correction
-gamma <g>: Set gamma correction factor to <g> (see docs)
-ge       : Fool application into thinking that LD_PRELOAD is unset
-np <n>   : Use <n> CPUs to perform image compression (VGL transport only)
-p <p>    : TCP port to use when connecting to vglclient (VGL transport only)
+/-pr     : Enable/disable performance profiling output
-q <q>    : JPEG compression quality [1 <= <q> <= 100]
+/-s      : Enable/disable SSL encryption of VGL image transport
-samp <s> : JPEG chrominance subsampling factor
            [<s> = gray, 1x, 2x, 4x, 8x, or 16x]
+/-sp     : Turn on/off frame spoiling
-st <s>   : left = Send only left eye buffer
            right = Send only right eye buffer
            quad = Use quad-buffered stereo if available, otherwise use
                   red/cyan (anaglyphic) stereo [default]
            rc = Always use red/cyan (anaglyphic) stereo
+/-sync   : Enable/disable strict 2D/3D synchronization
+/-tr     : Enable/disable function call tracing (generates a lot of output)
+/-v      : Enable/disable verbose VirtualGL messages
-64       : (Solaris only) Preload VirtualGL only into 64-bit apps
-32       : (Solaris only) Preload VirtualGL only into 32-bit apps

8.Install TurboVNC

rock@fly:~/turbovnc> wget ​http://downloads.sourceforge.net/virtualgl/turbovnc-0.4.i386.rpm?modtime=1205803835&big_mirror=0
rock@fly:~/turbovnc> sudo rpm -i turbovnc-0.4.i386.rpm
rock@fly:~/turbovnc> rpm -qs turbovnc 

normal        /etc/X11/applnk/Applications/tvncviewer.desktop
normal        /etc/init.d/tvncserver
normal        /etc/sysconfig/tvncservers
normal        /opt/TurboVNC
normal        /opt/TurboVNC/bin
normal        /opt/TurboVNC/bin/Xvnc
normal        /opt/TurboVNC/bin/vncconnect
normal        /opt/TurboVNC/bin/vncpasswd
normal        /opt/TurboVNC/bin/vncserver
normal        /opt/TurboVNC/bin/vncviewer
normal        /opt/TurboVNC/man
normal        /opt/TurboVNC/man/man1
normal        /opt/TurboVNC/man/man1/Xvnc.1
normal        /opt/TurboVNC/man/man1/vncconnect.1
normal        /opt/TurboVNC/man/man1/vncpasswd.1
normal        /opt/TurboVNC/man/man1/vncserver.1
normal        /opt/TurboVNC/man/man1/vncviewer.1
normal        /opt/TurboVNC/vnc
normal        /opt/TurboVNC/vnc/classes
normal        /opt/TurboVNC/vnc/classes/AuthPanel.class
normal        /opt/TurboVNC/vnc/classes/ButtonPanel.class
normal        /opt/TurboVNC/vnc/classes/CapabilityInfo.class
normal        /opt/TurboVNC/vnc/classes/CapsContainer.class
normal        /opt/TurboVNC/vnc/classes/ClipboardFrame.class
normal        /opt/TurboVNC/vnc/classes/DesCipher.class
normal        /opt/TurboVNC/vnc/classes/InStream.class
normal        /opt/TurboVNC/vnc/classes/MemInStream.class
normal        /opt/TurboVNC/vnc/classes/OptionsFrame.class
normal        /opt/TurboVNC/vnc/classes/RFBUpdateEvent.class
normal        /opt/TurboVNC/vnc/classes/RecordingFrame.class
normal        /opt/TurboVNC/vnc/classes/ReloginPanel.class
normal        /opt/TurboVNC/vnc/classes/RfbProto.class
normal        /opt/TurboVNC/vnc/classes/SessionRecorder.class
normal        /opt/TurboVNC/vnc/classes/SocketFactory.class
normal        /opt/TurboVNC/vnc/classes/VncCanvas.class
normal        /opt/TurboVNC/vnc/classes/VncCanvas2.class
normal        /opt/TurboVNC/vnc/classes/VncViewer.class
normal        /opt/TurboVNC/vnc/classes/VncViewer.jar
normal        /opt/TurboVNC/vnc/classes/ZlibInStream.class
normal        /opt/TurboVNC/vnc/classes/index.vnc
normal        /usr/share/doc/turbovnc-0.4
normal        /usr/share/doc/turbovnc-0.4/LICENCE.TXT
normal        /usr/share/doc/turbovnc-0.4/TurboVNC-ChangeLog.txt
normal        /usr/share/doc/turbovnc-0.4/WhatsNew

rock@fly:~> sudo zypper install xorg-x11-fonts xorg-x11-fonts-core xorg-x11-fonts-devel
rock@fly:~> sudo zypper install xorg-x11-devel xorg-x11-driver-input xorg-x11-driver-video
rock@fly:~> sudo zypper install libdrm libdrm-devel Mesa Mesa-devel
rock@fly:~> sudo zypper remove beagle beagle-evolution beagle-firefox beagle-gui beagle-index tomboy
rock@fly:/usr/X11R6/lib> sudo /etc/init.d/xfs start

rock@fly:~> cd /usr/X11R6/lib
rock@fly:/usr/X11R6/lib> sudo mkdir X11
rock@fly:~> sudo cp rgb.txt /usr/X11R6/lib/X11/
rock@fly:~> sudo vim /etc/X11/xorg.conf 

Section "Module"
  Load         "dbe"
  Load         "type1"
  Load         "freetype"
  Load         "extmod"
  Load         "glx"
  Load         "v4l"
  Load         "drm"
  Load         "dri"

Section "DRI"
    Group      "video"
    Mode       0666
EndSection

rock@fly:~> sudo vim /etc/X11/fs/config 

#no-listen = tcp

rock@fly:~> /opt/TurboVNC/bin/vncserver -geometry 1280x1024 :1

New 'X' desktop is fly:1

Creating default startup script /home/rock/.vnc/xstartup Starting applications specified in /home/rock/.vnc/xstartup Log file is /home/rock/.vnc/fly:1.log

rock@fly:~> ps aux | grep X 

root      4216  0.0  0.9  75012 19760 tty7     Ss+  Jul03   0:04 /usr/bin/X :0 -audit 0 -br -auth /var/lib/gdm/:0.Xauth vt7
rock      4811  0.0  0.0  46628   988 ?        Ss   Jul03   0:00 /usr/bin/ssh-agent /bin/bash /etc/X11/xinit/xinitrc
rock     20758  1.1  0.6  16760 12900 pts/9    S    00:56   0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop X -httpd /opt/TurboVNC/bin/../vnc/classes -auth /home/rock/.Xauthority -dontdisconnect -geometry 1240x900 -depth 24 -rfbwait 120000 -rfbauth /home/rock/.vnc/passwd -rfbport 5901 -fp unix/:7100 -deferupdate 1
rock     22008  0.0  0.0   5004   808 pts/9    S+   00:56   0:00 grep X

### TurboVNC Initialization Bug fixed for ALSA Library ###

rock@fly:~/alsa> wget ​http://gd.tuwien.ac.at/opsys/linux/alsa/lib/alsa-lib-1.0.13.tar.bz2
rock@fly:~/alsa> tar jxvf alsa-lib-1.0.13.tar.bz2
rock@fly:~/alsa> cd alsa-lib-1.0.13
rock@fly:~/alsa/alsa-lib-1.0.13> ./configure --enable-static
rock@fly:~/alsa/alsa-lib-1.0.13> make -j 4
rock@fly:~/alsa/alsa-lib-1.0.13> sudo make install

openSUSE(GUI)-> Computer -> Control Center -> System -> Sound -> Choose ALSA for each option.

9.Install Amira

Necessary Library: libg2c.so.0
Download libg2c33(Fortran Compiler Runtime Library) for openSUSE 10.2

rock@fly:~/amira_source/download_lib> wget ​ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/algraf/openSUSE_10.2/x86_64/libg2c33-3.3.3-43.3.x86_64.rpm
rock@fly:~/amira_source/download_lib> sudo rpm -i libg2c33-3.3.3-43.3.x86_64.rpm

rock@fly:~/amira_source> chmod a+x Amira411-LinuxAMD64.bin
rock@fly:~/amira_source> ./Amira411-LinuxAMD64.bin --console (console mode)
Installation succeed !!!
Installation path: /home/rock/Amira4.1.1
rock@fly:~/Amira4.1.1/bin> ./start --console (console mode)

Finished ! It works...

10.NVIDIA GeForce? 7300 GT Installation

Changee to root Before following steps.

fly:/home/rock/nvidia # wget ​http://tw.download.nvidia.com/XFree86/Linux-x86_64/173.14.09/NVIDIA-Linux-x86_64-173.14.09-pkg2.run

fly:/home/rock/nvidia # sh NVIDIA-Linux-x86_64-173.14.09-pkg2.run

Then follow extra steps to continue: Setup , kernel module , configuration ..etc

Finished!!! Now, Check it out ~

fly:/home/rock/nvidia # modprobe -l | grep nv 

/lib/modules/2.6.18-default/kernel/drivers/scsi/sata_nv.ko
/lib/modules/2.6.18-default/kernel/drivers/video/nvidia/nvidiafb.ko
/lib/modules/2.6.18-default/kernel/drivers/video/nvidia.ko

We only need nvidia.ko, and it'll actually loaded after installation.

fly:/home/rock/nvidia # lsmod | grep nv 

nvidia               8132208  24 
i2c_core               41472  2 nvidia,i2c_i801

fly:/home/rock/nvidia # vim /etc/X11/xorg.conf 

Section "Module"
  Load         "dbe"
  Load         "v4l"
  Load         "freetype"
  Load         "type1"
  Load         "extmod"
  Load         "glx"
  Load         "vnc"
EndSection

fly:/home/rock/nvidia # less /var/log/Xorg.0.log 

(--) PCI:*(1:0:0) nVidia Corporation G70 [GeForce 7300 GT] rev 161, Mem @ 0xfd000000/24, 0xe0000000/
28, 0xfc000000/24, I/O @ 0xcc00/7, BIOS @ 0xfe8e0000/17

(II) Module glx: vendor="NVIDIA Corporation"
        compiled for 4.0.2, module version = 1.0.0
        Module class: X.Org Server Extension
(II) NVIDIA GLX Module  173.14.09  Thu Jun  5 00:05:57 PDT 2008
(II) Loading extension GLX
(II) LoadModule: "vnc"
(II) Loading /usr/lib64/xorg/modules/extensions//libvnc.so
(II) Module vnc: vendor="RealVNC Ltd"
        compiled for 4.3.99.902, module version = 1.0.0
        Module class: X.Org Server Extension
        ABI class: X.Org Server Extension, version 0.3
(II) Loading extension VNC
(II) LoadModule: "nvidia"
(II) Loading /usr/lib64/xorg/modules/drivers//nvidia_drv.so
(II) Module nvidia: vendor="NVIDIA Corporation"
        compiled for 4.0.2, module version = 1.0.0
        Module class: X.Org Video Driver

(II) NVIDIA dlloader X Driver  173.14.09  Wed Jun  4 23:48:23 PDT 2008
(II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
(II) Primary Device is: PCI 01:00:0
(--) Assigning device section with no busID to primary device
(--) Chipset NVIDIA GPU found

(**) NVIDIA(0): Depth 16, (--) framebuffer bpp 16
(==) NVIDIA(0): RGB weight 565
(==) NVIDIA(0): Default visual is TrueColor
(==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
(**) NVIDIA(0): Enabling RENDER acceleration
(II) NVIDIA(0): NVIDIA GPU GeForce 7300 GT (G73) at PCI:1:0:0 (GPU-0)
(--) NVIDIA(0): Memory: 262144 kBytes
(--) NVIDIA(0): VideoBIOS: 05.73.22.51.00
(II) NVIDIA(0): Detected PCI Express Link width: 16X
(--) NVIDIA(0): Interlaced video modes are supported on this GPU
(--) NVIDIA(0): Connected display device(s) on GeForce 7300 GT at PCI:1:0:0:
(--) NVIDIA(0):     Philips 190B (CRT-0)
(--) NVIDIA(0): Philips 190B (CRT-0): 400.0 MHz maximum pixel clock
(II) NVIDIA(0): Assigned Display Device: CRT-0
(II) NVIDIA(0): Validated modes:
(II) NVIDIA(0):     "1024x768"
(II) NVIDIA(0):     "1024x600"
(II) NVIDIA(0):     "800x600"
(II) NVIDIA(0):     "768x576"
(II) NVIDIA(0):     "640x480"
(II) NVIDIA(0): Virtual screen size determined to be 1024 x 768
(--) NVIDIA(0): DPI set to (68, 65); computed from "UseEdidDpi" X config
(--) NVIDIA(0):     option
(==) NVIDIA(0): Disabling 32-bit ARGB GLX visuals.

fly:/home/rock/nvidia # /etc/init.d/xdm restart

Reference

Attachments (4)

Download all attachments as: .zip