wiki:3D_Lustre

Context Navigation

Version 27 (modified by rider, 16 years ago) (diff)
--

Build Lustre Storage Cluster for 3D Fly platform

Build Lustre Storage Cluster for 3D Fly platform

0.Introduction

Some domain-applications usually need to computing with huge data. It usually meet a bottleneck of performance, the main problem is I/O performance to drag total performance. Thereforce, we use Lustre high performance stroage to improve I/O performance.
Abiut Lustre, you can refer to Lustre web site http://wiki.lustre.org/

1.Machine information & Software

Hardware

Node	8 nodes (1 server , 7 client provide disks)
CPU	Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory	2GB DDR2 667 (each node)
Disk	All nodes: 320G * 7 = 2.24T
NIC	Intel Corporation 82566DM Gigabit Network Connection
Switch	D-link 24 port GE switch

Software

OS	OpenSuSE 10.2
DRBL	1.9.0-35
Amira	Testing Version
Kernel	2.6.18

Architecture

2.Config OpenSuSE Environment

Install packages for Lustre
Install packages for DRBL

3.Install Lustre

Build kernel

$ sudo su
$ cd /usr/src
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
$ tar jxvf linux-2.6.18.tar.bz2
$ cd 2.6.18
$ cp /boot/config-xxx .config
$ make oldconfig ; make menuconfig
$ make -j 5
$ make modules
$ make modules_install
$ make install
$ mkinitrd
$ vim /boot/grub/menu.list

recompiler NIC driver

wget http://trac.nchc.org.tw/grid/attachment/wiki/Problem/e1000-7.6.15.4.tar.gz?format=raw
tar e1000-7.6.15.4.tar.gz
cd e1000-7.6.15.4/src
make install

Bonding NIC (bond0 -> eth1 & eth2)

$ cd /etc/sysconfig/network
$ vim ifcfg-eth1 
$ vim ifeth-eth2
$ vim ifcfg-bond0
$ vim /etc/rc.d/after.local
ifenslave bond0 eth1 eth2
modprobe drm
modprobe ldiskfs
modprobe lnet networks=tcp

$ vim /etc/module/modules.conf.local
alias bond0 bonding
alias eth0 e1000
alias eth1 e1000
alias eth2 e1000
options bond0 mode=6 miimon=100

Build Lustre Modules

$ configure --with-linux=/usr/src/2.6.18
$ make install
$ depmod -a
$ vim /etc/rc.d/after.local

4.Build DRBL Environment

$ zypper install drbl
$ /opt/drbl/sbin/drblsrv -i 
$ /opt/drbl/sbin/drblpush -i
= Check nodes booting setup =
/tftpboot/nodes/192.168.1.X/etc/rc.d/after.local
/tftpboot/nodes/192.168.1.X/etc/module/modules.conf.local
/tftfpood/node_root/etc/sysconfig/network/

5.Build Lustre Storage Cluster

Setup autologin

= fly =
mkfs.lustre --fsname=flyfs --mdt --mgs /dev/sdb1
mount -t lustre /dev/sdb1 /home/mdt
= fly01 ~ fly07 =
mkfs.lustre --fsname=flyfs --ost  --mgsnode=fly@tcp0 /dev/sdb
mount -t lustre /dev/sdb /home/ost
= fly =
mkdir /home/flyfs
mount -t lustre fly@tcp0:/flyfs /home/flyfs

6.Install TurboJPEG

rock@fly:~/turbojpeg> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -ivh turbojpeg-1.10.3.x86_64.rpm
rock@fly:~/turbojpeg> sudo rpm -qs turbojpeg
rock@fly:~/VirtualGL> switchtjpeg --help

USAGE: /usr/bin/switchtjpeg < ipp | libjpeg >

ipp = Use the accelerated version of TurboJPEG, which embeds functions from
      the Intel(R) Integrated Performance Primitives
libjpeg = Use the unaccelerated version of TurboJPEG, which is
          GPL-compatible

7.Install VirtualGL

rock@fly:~/VirtualGL> wget ftp://ftp.pbone.net/mirror/ftp.sourceforge.net/pub/sourceforge/v/vi/virtualgl/VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -ivh VirtualGL-2.1.x86_64.rpm
rock@fly:~/VirtualGL> sudo rpm -qs VirtualGL (Check)
rock@fly:~/VirtualGL> vglrun --help

### VGL Server Configuration

rock@fly:~> sudo /opt/VirtualGL/bin/vglserver_config

1) Configure server for use with VirtualGL in GLX mode
2) Unconfigure server for use with VirtualGL in GLX mode
X) Exit

Choose:
1

Restrict local X server access to vglusers group (recommended)?
[Y/n]
n

Restrict framebuffer device access to vglusers group (recommended)?
[Y/n]
n

Disable XTEST extension (recommended)?
[Y/n]
Y
... Granting write permission to /dev/nvidia* for all users ...
... Adding xhost +localhost to /etc/X11/xdm/Xsetup script ...
... Adding xhost +localhost to /etc/opt/gnome/gdm/Init/Default script ...
... Disabling XTEST extension in /etc/X11/xdm/Xservers ...
... Setting default run level to 5 (enabling graphical login prompt) ...

Done. You must restart the display manager for the changes to take effect.


1) Configure server for use with VirtualGL in GLX mode
2) Unconfigure server for use with VirtualGL in GLX mode
X) Exit

Choose:
X

rock@fly:~> sudo /etc/init.d/xdm restart

8.Install TurboVNC

rock@fly:~/turbovnc> wget http://downloads.sourceforge.net/virtualgl/turbovnc-0.4.i386.rpm?modtime=1205803835&big_mirror=0
rock@fly:~/turbovnc> sudo rpm -ivh turbovnc-0.4.i386.rpm
rock@fly:~/turbovnc> rpm -qs turbovnc (Check)
rock@fly:~> sudo zypper install xorg-x11-fonts xorg-x11-fonts-core xorg-x11-fonts-devel
rock@fly:~> sudo zypper install xorg-x11-devel xorg-x11-driver-input xorg-x11-driver-video
rock@fly:~> sudo zypper install libdrm libdrm-devel Mesa Mesa-devel
rock@fly:~> sudo zypper remove beagle beagle-evolution beagle-firefox beagle-gui beagle-index tomboy
rock@fly:/usr/X11R6/lib> sudo /etc/init.d/xfs start
rock@fly:~> cd /usr/X11R6/lib
rock@fly:/usr/X11R6/lib> sudo mkdir X11
rock@fly:~> sudo cp rgb.txt /usr/X11R6/lib/X11/
rock@fly:~> sudo vim /etc/X11/xorg.conf

Section "Module"
  Load         "dbe"
  Load         "type1"
  Load         "freetype"
  Load         "extmod"
  Load         "glx"
  Load         "v4l"
  Load         "drm"
  Load         "dri"

Section "DRI"
    Group      "video"
    Mode       0666
EndSection

rock@fly:~> sudo vim /etc/X11/fs/config

#no-listen = tcp

rock@fly:~> /opt/TurboVNC/bin/vncserver -geometry 1024x768 :1

New 'X' desktop is fly:1

Creating default startup script /home/rock/.vnc/xstartup Starting applications specified in /home/rock/.vnc/xstartup Log file is /home/rock/.vnc/fly:1.log

rock@fly:~> ps aux | grep X

root      4216  0.0  0.9  75012 19760 tty7     Ss+  Jul03   0:04 /usr/bin/X :0 -audit 0 -br -auth /var/lib/gdm/:0.Xauth vt7
rock      4811  0.0  0.0  46628   988 ?        Ss   Jul03   0:00 /usr/bin/ssh-agent /bin/bash /etc/X11/xinit/xinitrc
rock     20758  1.1  0.6  16760 12900 pts/9    S    00:56   0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop X -httpd /opt/TurboVNC/bin/../vnc/classes -auth /home/rock/.Xauthority -dontdisconnect -geometry 1024x768 -depth 24 -rfbwait 120000 -rfbauth /home/rock/.vnc/passwd -rfbport 5901 -fp unix/:7100 -deferupdate 1
rock     22008  0.0  0.0   5004   808 pts/9    S+   00:56   0:00 grep X

8.1 Using TurboVNC with vncviewer

@ Client

< By Port >
rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server::{5900+n}

                               OR
< By Display >
rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server:{m}

Example: If the TurboVNC server is running on X display fly:1, then point your web browser to:

< By Port >
rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server::{5900+n}

                               OR
< By Display >
rider@350z:~$ /opt/TurboVNC/bin/vncviewer Fly_Server:1

8.2 Using TurboVNC in a Web Browser

@ Client

To use the Java TurboVNC viewer, point your web browser to:

http://Fly_Server:5800+n

Example: If the TurboVNC server is running on X display fly:1, then point your web browser to:

http://Fly_Server:5801

### TurboVNC Initialization Bug fixed for ALSA Library ###

rock@fly:~/alsa> wget http://gd.tuwien.ac.at/opsys/linux/alsa/lib/alsa-lib-1.0.13.tar.bz2
rock@fly:~/alsa> tar jxvf alsa-lib-1.0.13.tar.bz2
rock@fly:~/alsa> cd alsa-lib-1.0.13
rock@fly:~/alsa/alsa-lib-1.0.13> ./configure --enable-static
rock@fly:~/alsa/alsa-lib-1.0.13> make -j 4
rock@fly:~/alsa/alsa-lib-1.0.13> sudo make install

openSUSE(XDM)-> Computer -> Control Center -> System -> Sound -> Choose ALSA for each option.

9.Install Amira

Necessary Library: libg2c.so.0
Download libg2c33(Fortran Compiler Runtime Library) for openSUSE 10.2

rock@fly:~/amira_source/download_lib> wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/algraf/openSUSE_10.2/x86_64/libg2c33-3.3.3-43.3.x86_64.rpm
rock@fly:~/amira_source/download_lib> sudo rpm -i libg2c33-3.3.3-43.3.x86_64.rpm

rock@fly:~/amira_source> chmod a+x Amira411-LinuxAMD64.bin
rock@fly:~/amira_source> ./Amira411-LinuxAMD64.bin --console (console mode)
Installation succeed !!!
Installation path: /home/rock/Amira4.1.1
rock@fly:~/Amira4.1.1/bin> ./start --console (console mode)

Finished ! It works...

9.1 Amira Service

rock@fly:~> cd ./Amira4.1.1/bin/
rock@fly:~/Amira4.1.1/bin> sudo chmod a+x serviceSetup.sh
rock@fly:~/Amira4.1.1/bin> ./serviceSetup.sh
rock@fly:~/Amira4.1.1/bin> sudo /etc/init.d/xinetd restart
rock@fly:~/Amira4.1.1/bin> sudo netstat -tnlp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:32802           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:616             0.0.0.0:*               LISTEN      3831/ypbind         
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN      18763/xinetd        
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      3589/portmap        
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      16769/X             
tcp        0      0 127.0.0.1:2544          0.0.0.0:*               LISTEN      3867/zmd            
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN      3827/perl
           
tcp        0      0 0.0.0.0:17234           0.0.0.0:*               LISTEN      18763/xinetd ---> Amira daemon

tcp        0      0 0.0.0.0:756             0.0.0.0:*               LISTEN      3974/rpc.mountd     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4089/sshd           
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      4596/cupsd          
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      4167/master         
tcp        0      0 0.0.0.0:603             0.0.0.0:*               LISTEN      3817/ypserv         
tcp        0      0 0.0.0.0:607             0.0.0.0:*               LISTEN      3822/rpc.ypxfrd

rock@fly:~/Amira4.1.1/bin> less /etc/xinetd.d/amira

# Amira's Service
service amira
{
        type = UNLISTED
        socket_type = STREAM
        protocol = TCP
        wait = no
        user = root
        server = /home/rock/Amira4.1.1/bin/amiradaemon
        server_args = -daemonExePath /home/rock/Amira4.1.1/bin/start
        port = 17234
}

10.NVIDIA GeForce? 7300 GT Installation

Change to root before following steps.

fly:/home/rock/nvidia # wget http://tw.download.nvidia.com/XFree86/Linux-x86_64/173.14.09/NVIDIA-Linux-x86_64-173.14.09-pkg2.run

fly:/home/rock/nvidia # sh NVIDIA-Linux-x86_64-173.14.09-pkg2.run

Then follow extra steps to continue: Setup , kernel module , configuration ..etc

Finished!!! Now, Check it out ~

fly:/home/rock/nvidia # modprobe -l | grep nv

/lib/modules/2.6.18-default/kernel/drivers/scsi/sata_nv.ko
/lib/modules/2.6.18-default/kernel/drivers/video/nvidia/nvidiafb.ko
/lib/modules/2.6.18-default/kernel/drivers/video/nvidia.ko

We only need nvidia.ko, and it'll actually loaded after installation.

fly:/home/rock/nvidia # lsmod | grep nv

nvidia               8132208  24 
i2c_core               41472  2 nvidia,i2c_i801

fly:/home/rock # glxinfo | more

direct rendering: Yes
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_SGIX_fbconfig, 
    GLX_SGIX_pbuffer, GLX_SGI_video_sync, GLX_SGI_swap_control, 
    GLX_EXT_texture_from_pixmap, GLX_ARB_multisample, GLX_NV_float_buffer, 
    GLX_ARB_fbconfig_float
client glx vendor string: NVIDIA Corporation
client glx version string: 1.4

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce 7300 GT/PCI/SSE2
OpenGL version string: 2.1.2 NVIDIA 173.14.09

fly:/home/rock/nvidia # vim /etc/X11/xorg.conf

Section "Module"
  Load         "dbe"
  Load         "v4l"
  Load         "freetype"
  Load         "type1"
  Load         "extmod"
  Load         "glx"
  Load         "vnc"
# Load         "dri"

EndSection

Section "Device"
  BoardName    "GeForce 7300GT"
  Driver       "nvidia"
  Identifier   "Device[0]"
# Option       "alwaysshared"
# Option       "usevnc" "yes"
# Option       "httpdir" "/usr/share/vnc/classes"
# Option       "httpport" "5800"
# Option       "rfbauth" "/root/.vnc/passwd"
  Screen       0
  VendorName   "NVIDIA Corporation"
EndSection

# Section "DRI"
#    Group      "video"
#    Mode       0666
# EndSection

fly:/home/rock/nvidia # less /var/log/Xorg.0.log

(--) PCI:*(1:0:0) nVidia Corporation G70 [GeForce 7300 GT] rev 161, Mem @ 0xfd000000/24, 0xe0000000/
28, 0xfc000000/24, I/O @ 0xcc00/7, BIOS @ 0xfe8e0000/17

(II) Module glx: vendor="NVIDIA Corporation"
        compiled for 4.0.2, module version = 1.0.0
        Module class: X.Org Server Extension
(II) NVIDIA GLX Module  173.14.09  Thu Jun  5 00:05:57 PDT 2008
(II) Loading extension GLX
(II) LoadModule: "vnc"
(II) Loading /usr/lib64/xorg/modules/extensions//libvnc.so
(II) Module vnc: vendor="RealVNC Ltd"
        compiled for 4.3.99.902, module version = 1.0.0
        Module class: X.Org Server Extension
        ABI class: X.Org Server Extension, version 0.3
(II) Loading extension VNC
(II) LoadModule: "nvidia"
(II) Loading /usr/lib64/xorg/modules/drivers//nvidia_drv.so
(II) Module nvidia: vendor="NVIDIA Corporation"
        compiled for 4.0.2, module version = 1.0.0
        Module class: X.Org Video Driver

(II) NVIDIA dlloader X Driver  173.14.09  Wed Jun  4 23:48:23 PDT 2008
(II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
(II) Primary Device is: PCI 01:00:0
(--) Assigning device section with no busID to primary device
(--) Chipset NVIDIA GPU found

(**) NVIDIA(0): Depth 16, (--) framebuffer bpp 16
(==) NVIDIA(0): RGB weight 565
(==) NVIDIA(0): Default visual is TrueColor
(==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
(**) NVIDIA(0): Enabling RENDER acceleration
(II) NVIDIA(0): NVIDIA GPU GeForce 7300 GT (G73) at PCI:1:0:0 (GPU-0)
(--) NVIDIA(0): Memory: 262144 kBytes
(--) NVIDIA(0): VideoBIOS: 05.73.22.51.00
(II) NVIDIA(0): Detected PCI Express Link width: 16X
(--) NVIDIA(0): Interlaced video modes are supported on this GPU
(--) NVIDIA(0): Connected display device(s) on GeForce 7300 GT at PCI:1:0:0:
(--) NVIDIA(0):     Philips 190B (CRT-0)
(--) NVIDIA(0): Philips 190B (CRT-0): 400.0 MHz maximum pixel clock
(II) NVIDIA(0): Assigned Display Device: CRT-0
(II) NVIDIA(0): Validated modes:
(II) NVIDIA(0):     "1024x768"
(II) NVIDIA(0):     "1024x600"
(II) NVIDIA(0):     "800x600"
(II) NVIDIA(0):     "768x576"
(II) NVIDIA(0):     "640x480"
(II) NVIDIA(0): Virtual screen size determined to be 1024 x 768
(--) NVIDIA(0): DPI set to (68, 65); computed from "UseEdidDpi" X config
(--) NVIDIA(0):     option
(==) NVIDIA(0): Disabling 32-bit ARGB GLX visuals.

fly:/home/rock/nvidia # /etc/init.d/xdm restart

10.1 NVIDIA 7300GT Performance

rock@fly:~> glxgears

37672 frames in 5.0 seconds = 7534.325 FPS
38320 frames in 5.0 seconds = 7663.902 FPS
38317 frames in 5.0 seconds = 7663.308 FPS
38320 frames in 5.0 seconds = 7663.936 FPS
38329 frames in 5.0 seconds = 7665.636 FPS
38316 frames in 5.0 seconds = 7663.152 FPS
38307 frames in 5.0 seconds = 7661.244 FPS
38321 frames in 5.0 seconds = 7664.146 FPS
38308 frames in 5.0 seconds = 7661.435 FPS
38310 frames in 5.0 seconds = 7661.956 FPS

38323 frames in 5.0 seconds = 7664.566 FPS
38311 frames in 5.0 seconds = 7662.112 FPS
38306 frames in 5.0 seconds = 7661.194 FPS
38318 frames in 5.0 seconds = 7663.445 FPS
38320 frames in 5.0 seconds = 7663.884 FPS
38310 frames in 5.0 seconds = 7661.989 FPS
38325 frames in 5.0 seconds = 7664.809 FPS
38317 frames in 5.0 seconds = 7663.257 FPS
38308 frames in 5.0 seconds = 7661.587 FPS
38318 frames in 5.0 seconds = 7663.428 FPS
38305 frames in 5.0 seconds = 7660.958 FPS
38317 frames in 5.0 seconds = 7663.250 FPS
38327 frames in 5.0 seconds = 7665.265 FPS
38313 frames in 5.0 seconds = 7662.488 FPS
38311 frames in 5.0 seconds = 7662.028 FPS
38317 frames in 5.0 seconds = 7663.277 FPS
38313 frames in 5.0 seconds = 7662.439 FPS
38314 frames in 5.0 seconds = 7662.786 FPS
38325 frames in 5.0 seconds = 7664.986 FPS
38311 frames in 5.0 seconds = 7662.035 FPS
38304 frames in 5.0 seconds = 7660.689 FPS
38333 frames in 5.0 seconds = 7666.563 FPS
38309 frames in 5.0 seconds = 7661.684 FPS
38311 frames in 5.0 seconds = 7662.048 FPS
38330 frames in 5.0 seconds = 7665.816 FPS
38316 frames in 5.0 seconds = 7663.065 FPS
38306 frames in 5.0 seconds = 7661.012 FPS
38022 frames in 5.0 seconds = 7604.208 FPS
38030 frames in 5.0 seconds = 7605.954 FPS
38020 frames in 5.0 seconds = 7603.871 FPS
38027 frames in 5.0 seconds = 7605.272 FPS
38023 frames in 5.0 seconds = 7604.425 FPS
38028 frames in 5.0 seconds = 7605.428 FPS
38032 frames in 5.0 seconds = 7606.298 FPS
38008 frames in 5.0 seconds = 7601.419 FPS
38032 frames in 5.0 seconds = 7606.386 FPS
38034 frames in 5.0 seconds = 7606.672 FPS
38029 frames in 5.0 seconds = 7605.690 FPS

11.Lustre Storage Cluster for 3D Fly platform Performance (I/O Benchmark without Ethernet Bonding)

Testbed Information

Server

CPU	Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
Memory	2GB DDR2 667
Disk	sda:160G sdb:320G
Graphic	NVIDIA GeForce? 7300GT 256MB DDR2 X1 PCI-E 16X
NIC	Intel Corporation 82566DM Gigabit Network Connection
Switch	D-link 24 port GE switch

Clients

Node number	7 nodes
CPU	Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory	2GB DDR2 667 (each node)
Disk	sda:160G sdb:320G
NIC	Intel Corporation 82566DM Gigabit Network Connection
Switch	D-link 24 port GE switch

Software Information

OS	openSUSE 10.3 Linux with kernel 2.6.22 x86_64
Cluster Environment	DRBL
3D Modol tools	IvIsee

< Test Cases >

Case 1: 1 remote client access without Lustre
Case 2: 1 remote client access with Lustre
Case 3: 3 remote clients access without Lustre
Case 4: 3 remote clients access with Lustre
Case 5: 5 remote clients access without Lustre
Case 6: 5 remote clients access with Lustre

Reference

Lustre, http://wiki.lustre.org/
Amira, http://www.amiravis.com/
NVIDIA, http://zh_tw.opensuse.org/NVIDIA
Glxgears, http://wiki.cchtml.com/index.php/Glxgears_is_not_a_Benchmark

Attachments (4)

fly_arch.png (53.9 KB) - added by rock 16 years ago. 3D fly architecture
rgb.txt (16.9 KB) - added by rider 16 years ago. RGB Database
glxgears.png (112.2 KB) - added by rider 16 years ago. glxgears test
xorg.conf (5.3 KB) - added by rider 16 years ago. Fly-Server Default X config

Download all attachments as: .zip

Download in other formats:

Plain Text