wiki:3D_Lustre

Version 20 (modified by rider, 16 years ago) (diff)

--

Build Lustre Storage Cluster for 3D Fly platform

0.Introduction

Some domain-applications usually need to computing with huge data. It usually meet a bottleneck of performance, the main problem is I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance.
Abiut Lustre, you can refer to Lustre web site http://wiki.lustre.org/

1.Machine information & Software

  • Hardware
Node 8 nodes (1 server , 7 client provide disks)
CPU Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory 2GB DDR2 667 (each node)
Disk All nodes: 320G * 7 = 2.24T
NIC Intel Corporation 82566DM Gigabit Network Connection
Switch D-link 24 port GE switch
  • Software
OSOpenSuSE 10.2
DRBL1.9.0-35
AmiraTesting Version
Kernel2.6.18
  • Architecture

3D fly architecture

2.Config OpenSuSE Environment

  • Install packages for Lustre
  • Install packages for DRBL

3.Install Lustre

  • Build kernel
    $ sudo su
    $ cd /usr/src
    $ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
    $ tar jxvf linux-2.6.18.tar.bz2
    $ cd 2.6.18
    $ cp /boot/config-xxx .config
    $ make oldconfig ; make menuconfig
    $ make -j 5
    $ make modules
    $ make modules_install
    $ make install
    $ mkinitrd
    $ vim /boot/grub/menu.list
    
  • recompiler NIC driver
    wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw
    tar e1000-7.6.15.4.tar.gz
    cd e1000-7.6.15.4/src
    make install
    
  • Bonding NIC (bond0 -> eth1 & eth2)
    $ cd /etc/sysconfig/network
    $ vim ifcfg-eth1 
    $ vim ifeth-eth2
    $ vim ifcfg-bond0
    $ vim /etc/rc.d/after.local
    ifenslave bond0 eth1 eth2
    modprobe drm
    modprobe ldiskfs
    modprobe lnet networks=tcp
    
    $ vim /etc/module/modules.conf.local
    alias bond0 bonding
    alias eth0 e1000
    alias eth1 e1000
    alias eth2 e1000
    options bond0 mode=6 miimon=100
    
    
  • Build Lustre Modules
    $ configure --with-linux=/usr/src/2.6.18
    $ make install
    $ depmod -a
    $ vim /etc/rc.d/after.local
    

4.Build DRBL Environment

$ zypper install drbl
$ /opt/drbl/sbin/drblsrv -i 
$ /opt/drbl/sbin/drblpush -i
= Check nodes booting setup =
/tftpboot/nodes/192.168.1.X/etc/rc.d/after.local
/tftpboot/nodes/192.168.1.X/etc/module/modules.conf.local
/tftfpood/node_root/etc/sysconfig/network/

5.Build Lustre Storage Cluster

  • Setup autologin
    = fly =
    mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1
    mount -t lustre /dev/sdb1 /home/mdt
    = fly01 ~ fly07 =
    mkfs.lustre --fsname=flyfs --ost  --mgsnode=fly@tcp0 /dev/sdb
    mount -t lustre /dev/sdb /home/ost
    = fly =
    mkdir /home/flyfs
    mount -t lustre fly@tcp0:/flyfs /home/flyfs
    
    

6.Install TurboJPEG

rock@fly:~/turbojpeg> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -i turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg

normal        /usr/bin/switchtjpeg
normal        /usr/include/turbojpeg.h
normal        /usr/lib/libturbojpeg-ipp.so
normal        /usr/lib/libturbojpeg-libjpeg.so
normal        /usr/lib/libturbojpeg.so
normal        /usr/lib64/libturbojpeg-ipp.so
normal        /usr/lib64/libturbojpeg-libjpeg.so
normal        /usr/lib64/libturbojpeg.so
normal        /usr/share/doc/turbojpeg-1.10.3
normal        /usr/share/doc/turbojpeg-1.10.3/LGPL.txt
normal        /usr/share/doc/turbojpeg-1.10.3/LICENSE.txt
normal        /usr/share/doc/turbojpeg-1.10.3/README.txt

rock@fly:~/VirtualGL> switchtjpeg --help

USAGE: /usr/bin/switchtjpeg < ipp | libjpeg >

ipp = Use the accelerated version of TurboJPEG, which embeds functions from
      the Intel(R) Integrated Performance Primitives
libjpeg = Use the unaccelerated version of TurboJPEG, which is
          GPL-compatible

7.Install VirtualGL

rock@fly:~/VirtualGL> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -i VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL

normal        /opt/VirtualGL
normal        /opt/VirtualGL/bin
normal        /opt/VirtualGL/bin/cpustat
normal        /opt/VirtualGL/bin/glxinfo
normal        /opt/VirtualGL/bin/glxspheres
normal        /opt/VirtualGL/bin/glxspheres64
normal        /opt/VirtualGL/bin/nettest
normal        /opt/VirtualGL/bin/tcbench
normal        /opt/VirtualGL/bin/vglclient
normal        /opt/VirtualGL/bin/vglconfig
normal        /opt/VirtualGL/bin/vglconfig64
normal        /opt/VirtualGL/bin/vglconnect
normal        /opt/VirtualGL/bin/vglgenkey
normal        /opt/VirtualGL/bin/vgllogin
normal        /opt/VirtualGL/bin/vglrun
normal        /opt/VirtualGL/bin/vglserver_config
normal        /opt/VirtualGL/doc
normal        /opt/VirtualGL/fakelib
normal        /opt/VirtualGL/fakelib/64
normal        /opt/VirtualGL/fakelib/64/libGL.so
normal        /opt/VirtualGL/fakelib/libGL.so
normal        /opt/VirtualGL/lib
normal        /opt/VirtualGL/lib64
normal        /usr/bin/vglclient
normal        /usr/bin/vglconfig
normal        /usr/bin/vglconfig64
normal        /usr/bin/vglconnect
normal        /usr/bin/vglgenkey
normal        /usr/bin/vgllogin
normal        /usr/bin/vglrun
normal        /usr/bin/vglserver_config
normal        /usr/lib/libdlfaker.so
normal        /usr/lib/libgefaker.so
normal        /usr/lib/librrfaker.so
normal        /usr/lib64/libdlfaker.so
normal        /usr/lib64/libgefaker.so
normal        /usr/lib64/librrfaker.so
normal        /usr/share/doc/VirtualGL-2.1
normal        /usr/share/doc/VirtualGL-2.1/411.gif
normal        /usr/share/doc/VirtualGL-2.1/422.gif
normal        /usr/share/doc/VirtualGL-2.1/444.gif
normal        /usr/share/doc/VirtualGL-2.1/ChangeLog.txt
normal        /usr/share/doc/VirtualGL-2.1/LGPL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE-OpenSSL.txt
normal        /usr/share/doc/VirtualGL-2.1/LICENSE.txt
normal        /usr/share/doc/VirtualGL-2.1/chromium-displaywall.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortfirst.png
normal        /usr/share/doc/VirtualGL-2.1/chromium-sortlast.png
normal        /usr/share/doc/VirtualGL-2.1/configdialog.gif
normal        /usr/share/doc/VirtualGL-2.1/exceed1.png
normal        /usr/share/doc/VirtualGL-2.1/exceed2.png
normal        /usr/share/doc/VirtualGL-2.1/exceed3.png
normal        /usr/share/doc/VirtualGL-2.1/exceed6.png
normal        /usr/share/doc/VirtualGL-2.1/index.html
normal        /usr/share/doc/VirtualGL-2.1/somerights20.png
normal        /usr/share/doc/VirtualGL-2.1/sshtunnel.png
normal        /usr/share/doc/VirtualGL-2.1/sunray.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc1.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc2.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc3.png
normal        /usr/share/doc/VirtualGL-2.1/turbovnc4.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransport.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportglp.png
normal        /usr/share/doc/VirtualGL-2.1/vgltransportservernetwork.png
normal        /usr/share/doc/VirtualGL-2.1/virtualgl.css
normal        /usr/share/doc/VirtualGL-2.1/x11transport.png
normal        /usr/share/doc/VirtualGL-2.1/x11transportglp.png

rock@fly:~/VirtualGL> vglrun --help

Usage: vglrun [options] [--] <OpenGL app> [OpenGL app arguments]

VirtualGL options
-c <c>    : proxy = Send 3D images uncompressed using X11 transport
            jpeg = Compress 3D images using JPEG/send using VGL transport
            rgb = Encode 3D images as RGB/send using VGL transport
            sr = Compress 3D images using DPCM/send using Sun Ray transport
            srrgb = Encode 3D images as RGB/send using Sun Ray transport
-cl <c>   : <c> = The hostname or IP address of the VirtualGL client (or
                  Sun Ray server)
-dl       : Force apps that dlopen() libGL to load VirtualGL instead
-d <d>    : <d> = the X Display or GLP device to use for 3D rendering
-fps <f>  : Limit client/server frame rate to <f> frames/sec
+/-g      : Enable/disable gamma correction
-gamma <g>: Set gamma correction factor to <g> (see docs)
-ge       : Fool application into thinking that LD_PRELOAD is unset
-np <n>   : Use <n> CPUs to perform image compression (VGL transport only)
-p <p>    : TCP port to use when connecting to vglclient (VGL transport only)
+/-pr     : Enable/disable performance profiling output
-q <q>    : JPEG compression quality [1 <= <q> <= 100]
+/-s      : Enable/disable SSL encryption of VGL image transport
-samp <s> : JPEG chrominance subsampling factor
            [<s> = gray, 1x, 2x, 4x, 8x, or 16x]
+/-sp     : Turn on/off frame spoiling
-st <s>   : left = Send only left eye buffer
            right = Send only right eye buffer
            quad = Use quad-buffered stereo if available, otherwise use
                   red/cyan (anaglyphic) stereo [default]
            rc = Always use red/cyan (anaglyphic) stereo
+/-sync   : Enable/disable strict 2D/3D synchronization
+/-tr     : Enable/disable function call tracing (generates a lot of output)
+/-v      : Enable/disable verbose VirtualGL messages
-64       : (Solaris only) Preload VirtualGL only into 64-bit apps
-32       : (Solaris only) Preload VirtualGL only into 32-bit apps

8.Install TurboVNC

rock@fly:~/turbovnc> wget http://downloads.sourceforge.net/virtualgl/turbovnc-0.4.i386.rpm?modtime=1205803835&big_mirror=0
rock@fly:~/turbovnc> sudo rpm -i turbovnc-0.4.i386.rpm
rock@fly:~/turbovnc> rpm -qs turbovnc

normal        /etc/X11/applnk/Applications/tvncviewer.desktop
normal        /etc/init.d/tvncserver
normal        /etc/sysconfig/tvncservers
normal        /opt/TurboVNC
normal        /opt/TurboVNC/bin
normal        /opt/TurboVNC/bin/Xvnc
normal        /opt/TurboVNC/bin/vncconnect
normal        /opt/TurboVNC/bin/vncpasswd
normal        /opt/TurboVNC/bin/vncserver
normal        /opt/TurboVNC/bin/vncviewer
normal        /opt/TurboVNC/man
normal        /opt/TurboVNC/man/man1
normal        /opt/TurboVNC/man/man1/Xvnc.1
normal        /opt/TurboVNC/man/man1/vncconnect.1
normal        /opt/TurboVNC/man/man1/vncpasswd.1
normal        /opt/TurboVNC/man/man1/vncserver.1
normal        /opt/TurboVNC/man/man1/vncviewer.1
normal        /opt/TurboVNC/vnc
normal        /opt/TurboVNC/vnc/classes
normal        /opt/TurboVNC/vnc/classes/AuthPanel.class
normal        /opt/TurboVNC/vnc/classes/ButtonPanel.class
normal        /opt/TurboVNC/vnc/classes/CapabilityInfo.class
normal        /opt/TurboVNC/vnc/classes/CapsContainer.class
normal        /opt/TurboVNC/vnc/classes/ClipboardFrame.class
normal        /opt/TurboVNC/vnc/classes/DesCipher.class
normal        /opt/TurboVNC/vnc/classes/InStream.class
normal        /opt/TurboVNC/vnc/classes/MemInStream.class
normal        /opt/TurboVNC/vnc/classes/OptionsFrame.class
normal        /opt/TurboVNC/vnc/classes/RFBUpdateEvent.class
normal        /opt/TurboVNC/vnc/classes/RecordingFrame.class
normal        /opt/TurboVNC/vnc/classes/ReloginPanel.class
normal        /opt/TurboVNC/vnc/classes/RfbProto.class
normal        /opt/TurboVNC/vnc/classes/SessionRecorder.class
normal        /opt/TurboVNC/vnc/classes/SocketFactory.class
normal        /opt/TurboVNC/vnc/classes/VncCanvas.class
normal        /opt/TurboVNC/vnc/classes/VncCanvas2.class
normal        /opt/TurboVNC/vnc/classes/VncViewer.class
normal        /opt/TurboVNC/vnc/classes/VncViewer.jar
normal        /opt/TurboVNC/vnc/classes/ZlibInStream.class
normal        /opt/TurboVNC/vnc/classes/index.vnc
normal        /usr/share/doc/turbovnc-0.4
normal        /usr/share/doc/turbovnc-0.4/LICENCE.TXT
normal        /usr/share/doc/turbovnc-0.4/TurboVNC-ChangeLog.txt
normal        /usr/share/doc/turbovnc-0.4/WhatsNew

rock@fly:~> sudo zypper install xorg-x11-fonts xorg-x11-fonts-core xorg-x11-fonts-devel
rock@fly:~> sudo zypper install xorg-x11-devel xorg-x11-driver-input xorg-x11-driver-video
rock@fly:~> sudo zypper install libdrm libdrm-devel Mesa Mesa-devel
rock@fly:~> sudo zypper remove beagle beagle-evolution beagle-firefox beagle-gui beagle-index tomboy
rock@fly:/usr/X11R6/lib> sudo /etc/init.d/xfs start

rock@fly:~> cd /usr/X11R6/lib
rock@fly:/usr/X11R6/lib> sudo mkdir X11
rock@fly:~> sudo cp rgb.txt /usr/X11R6/lib/X11/
rock@fly:~> sudo vim /etc/X11/xorg.conf

Section "Module"
  Load         "dbe"
  Load         "type1"
  Load         "freetype"
  Load         "extmod"
  Load         "glx"
  Load         "v4l"
  Load         "drm"
  Load         "dri"
Section "DRI"
    Group      "video"
    Mode       0666
EndSection

rock@fly:~> sudo vim /etc/X11/fs/config

#no-listen = tcp

rock@fly:~> /opt/TurboVNC/bin/vncserver -geometry 1280x1024 :1

New 'X' desktop is fly:1

Creating default startup script /home/rock/.vnc/xstartup Starting applications specified in /home/rock/.vnc/xstartup Log file is /home/rock/.vnc/fly:1.log

rock@fly:~> ps aux | grep X

root      4216  0.0  0.9  75012 19760 tty7     Ss+  Jul03   0:04 /usr/bin/X :0 -audit 0 -br -auth /var/lib/gdm/:0.Xauth vt7
rock      4811  0.0  0.0  46628   988 ?        Ss   Jul03   0:00 /usr/bin/ssh-agent /bin/bash /etc/X11/xinit/xinitrc
rock     20758  1.1  0.6  16760 12900 pts/9    S    00:56   0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop X -httpd /opt/TurboVNC/bin/../vnc/classes -auth /home/rock/.Xauthority -dontdisconnect -geometry 1240x900 -depth 24 -rfbwait 120000 -rfbauth /home/rock/.vnc/passwd -rfbport 5901 -fp unix/:7100 -deferupdate 1
rock     22008  0.0  0.0   5004   808 pts/9    S+   00:56   0:00 grep X

### TurboVNC Initialization Bug fixed for ALSA Library ###

rock@fly:~/alsa> wget http://gd.tuwien.ac.at/opsys/linux/alsa/lib/alsa-lib-1.0.13.tar.bz2
rock@fly:~/alsa> tar jxvf alsa-lib-1.0.13.tar.bz2
rock@fly:~/alsa> cd alsa-lib-1.0.13
rock@fly:~/alsa/alsa-lib-1.0.13> ./configure --enable-static
rock@fly:~/alsa/alsa-lib-1.0.13> make -j 4
rock@fly:~/alsa/alsa-lib-1.0.13> sudo make install

openSUSE(GUI)-> Computer -> Control Center -> System -> Sound -> Choose ALSA for each option.

9.Install Amira

Necessary Library: libg2c.so.0
Download libg2c33(Fortran Compiler Runtime Library) for openSUSE 10.2

rock@fly:~/amira_source/download_lib> wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/algraf/openSUSE_10.2/x86_64/libg2c33-3.3.3-43.3.x86_64.rpm
rock@fly:~/amira_source/download_lib> sudo rpm -i libg2c33-3.3.3-43.3.x86_64.rpm

rock@fly:~/amira_source> chmod a+x Amira411-LinuxAMD64.bin
rock@fly:~/amira_source> ./Amira411-LinuxAMD64.bin --console (console mode)
Installation succeed !!!
Installation path: /home/rock/Amira4.1.1
rock@fly:~/Amira4.1.1/bin> ./start --console (console mode)

Finished ! It works...

10.NVIDIA GeForce? 7300 GT Installation

Changee to root Before following steps.

fly:/home/rock/nvidia # wget http://tw.download.nvidia.com/XFree86/Linux-x86_64/173.14.09/NVIDIA-Linux-x86_64-173.14.09-pkg2.run

fly:/home/rock/nvidia # sh NVIDIA-Linux-x86_64-173.14.09-pkg2.run

Then follow extra steps to continue: Setup , kernel module , configuration ..etc

Finished!!! Now, Check it out ~

fly:/home/rock/nvidia # modprobe -l | grep nv

/lib/modules/2.6.18-default/kernel/drivers/scsi/sata_nv.ko
/lib/modules/2.6.18-default/kernel/drivers/video/nvidia/nvidiafb.ko
/lib/modules/2.6.18-default/kernel/drivers/video/nvidia.ko

We only need nvidia.ko, and it'll actually loaded after installation.

fly:/home/rock/nvidia # lsmod | grep nv

nvidia               8132208  24 
i2c_core               41472  2 nvidia,i2c_i801

fly:/home/rock # glxinfo | more

direct rendering: Yes
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, 
    GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, 
    GLX_EXT_texture_from_pixmap, GLX_ARB_multisample, GLX_NV_float_buffer, 
    GLX_ARB_fbconfig_float
client glx vendor string: NVIDIA Corporation
client glx version string: 1.4

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce 7300 GT/PCI/SSE2
OpenGL version string: 2.1.2 NVIDIA 173.14.09

fly:/home/rock/nvidia # vim /etc/X11/xorg.conf

Section "Module"
  Load         "dbe"
  Load         "v4l"
  Load         "freetype"
  Load         "type1"
  Load         "extmod"
  Load         "glx"
  Load         "vnc"
# Load         "dri"

EndSection

Section "Device"
  BoardName    "GeForce 7300GT"
  Driver       "nvidia"
  Identifier   "Device[0]"
# Option       "alwaysshared"
# Option       "usevnc" "yes"
# Option       "httpdir" "/usr/share/vnc/classes"
# Option       "httpport" "5800"
# Option       "rfbauth" "/root/.vnc/passwd"
  Screen       0
  VendorName   "NVIDIA Corporation"
EndSection

# Section "DRI"
#    Group      "video"
#    Mode       0666
# EndSection

fly:/home/rock/nvidia # less /var/log/Xorg.0.log

(--) PCI:*(1:0:0) nVidia Corporation G70 [GeForce 7300 GT] rev 161, Mem @ 0xfd000000/24, 0xe0000000/
28, 0xfc000000/24, I/O @ 0xcc00/7, BIOS @ 0xfe8e0000/17

(II) Module glx: vendor="NVIDIA Corporation"
        compiled for 4.0.2, module version = 1.0.0
        Module class: X.Org Server Extension
(II) NVIDIA GLX Module  173.14.09  Thu Jun  5 00:05:57 PDT 2008
(II) Loading extension GLX
(II) LoadModule: "vnc"
(II) Loading /usr/lib64/xorg/modules/extensions//libvnc.so
(II) Module vnc: vendor="RealVNC Ltd"
        compiled for 4.3.99.902, module version = 1.0.0
        Module class: X.Org Server Extension
        ABI class: X.Org Server Extension, version 0.3
(II) Loading extension VNC
(II) LoadModule: "nvidia"
(II) Loading /usr/lib64/xorg/modules/drivers//nvidia_drv.so
(II) Module nvidia: vendor="NVIDIA Corporation"
        compiled for 4.0.2, module version = 1.0.0
        Module class: X.Org Video Driver

(II) NVIDIA dlloader X Driver  173.14.09  Wed Jun  4 23:48:23 PDT 2008
(II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
(II) Primary Device is: PCI 01:00:0
(--) Assigning device section with no busID to primary device
(--) Chipset NVIDIA GPU found

(**) NVIDIA(0): Depth 16, (--) framebuffer bpp 16
(==) NVIDIA(0): RGB weight 565
(==) NVIDIA(0): Default visual is TrueColor
(==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
(**) NVIDIA(0): Enabling RENDER acceleration
(II) NVIDIA(0): NVIDIA GPU GeForce 7300 GT (G73) at PCI:1:0:0 (GPU-0)
(--) NVIDIA(0): Memory: 262144 kBytes
(--) NVIDIA(0): VideoBIOS: 05.73.22.51.00
(II) NVIDIA(0): Detected PCI Express Link width: 16X
(--) NVIDIA(0): Interlaced video modes are supported on this GPU
(--) NVIDIA(0): Connected display device(s) on GeForce 7300 GT at PCI:1:0:0:
(--) NVIDIA(0):     Philips 190B (CRT-0)
(--) NVIDIA(0): Philips 190B (CRT-0): 400.0 MHz maximum pixel clock
(II) NVIDIA(0): Assigned Display Device: CRT-0
(II) NVIDIA(0): Validated modes:
(II) NVIDIA(0):     "1024x768"
(II) NVIDIA(0):     "1024x600"
(II) NVIDIA(0):     "800x600"
(II) NVIDIA(0):     "768x576"
(II) NVIDIA(0):     "640x480"
(II) NVIDIA(0): Virtual screen size determined to be 1024 x 768
(--) NVIDIA(0): DPI set to (68, 65); computed from "UseEdidDpi" X config
(--) NVIDIA(0):     option
(==) NVIDIA(0): Disabling 32-bit ARGB GLX visuals.

fly:/home/rock/nvidia # /etc/init.d/xdm restart

Reference

Attachments (4)

Download all attachments as: .zip