Build Xen & Lustre enabled kernel

• Our OS: Dbian etch / ubuntu 8.04
• Our Method: We used xen-patched kernel 2.6.18, then used Lustre patched file to patch it. We used xen tar ball to compile with this kernel to build Xen images amd used this kernel to reboot. Finally we used Lustre tar ball to build Lustre modules.

1.Requirement packages

• 1.1 Kernel Packages:

  sudo aptitude install build-essential libncurses5-dev kernel-package
  

• 1.2 XEN Packages:

  $ sudo apt-get install iproute bridge-utils python-twisted gcc binutils make zlib1g-dev python-dev transfig bzip2 \
      screen ssh debootstrap libcurl3-dev libncurses5-dev x-dev libsdl1.2-dev bin86 bcc python-pam patch latex-make \
      qemu qemu-launcher qemuctl graphviz graphviz-dev lvm2 libvncserver-dev libjpeg62-dev libjpeg62 gettext python-xml \
      gawk tetex-extra tetex-base openssl libssl-dev mercurial
  

• 1.3 Lustre Packages:

  $ sudo aptitude install bzip2 debhelper dpatch libsnmp-dev module-assistant quilt libxml2 python-xml
  

2.Download XEN & Lustre

• 2.1 Download XEN

  $ cd /usr/src
  $ sudo wget http://bits.xensource.com/oss-xen/release/3.1.4/xen-3.1.4.tar.gz
  $ sudo tar zxvf xen-3.1.4.tar.gz
  $ sudo hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
  

• 2.2 Download Lustre (​http://www.sun.com/software/products/lustre/get.jsp)

  $ cd /usr/src
  $ tar zxvf lustre-1.6.6.tar.gz
  

3.Make Xen image

$ cd /usr/src/xen-3.2.1
$ sudo make -j 4 world
（SCSI disk I/O must build-in）

• 3.1 Use Lustre's patch file to patch xen kernel

  $ cd /usr/src/xen-3.1.4/linux-2.6.18-xen.hg
  $ sudo ln -s /usr/src/lustre-1.6.6/lustre/kernel_patches/series/2.6.18-vanilla.series series
  $ sudo ln -s /usr/src/lustre-1.6.6/lustre/kernel_patches/patches/ .
  $ quilt -av push
  $ sudo
  

• 3.2 If you want enable some feature for custom kernel

  $ cd /usr/src/xen-3.2.1/build-linux-2.6.18-xen_x86_64/
  $ make menuconfig
  (We make loop and NIC driver to be a modules)
  $ make -j 5
  $ cd ..
  

• 3.3 Install XEN & Lustre patched kernel

$ sudo make ; sudo make modules ; make install ; make modules_install $ sudo mkinitramfs -o /boot/initrd.img-2.6.18.8 2.6.18.8 $ sudo update-grub $ sudo reboot }}}

4.Make Lustre modules

$ cd /usr/src/lustre-1.6.5.1
$ sudo ./configure --with-linux=/usr/src/linux-2.6.18-xen.hg/  --with-linux-obj=/usr/src/xen-3.2.1/build-linux-2.6.18-xen_x86_64/
$ sudo make -j 4
$ sudo make install
$ sduo depmod -a

### Load modules in booting ###
$ sudo vim /etc/modules
loop max_loop=255
ldiskfs
lnet networks=tcp
$ sudo reboot

5.Push DRBL(If you don't push DRBL, you can skip it)

• refer to ​https://trac.nchc.org.tw/grid/wiki/jazz/DRBL_Xen

6.Test Xen & Lustre

### For each client ###
$ sudo vim /etc/modules
loop max_loop=255
ldiskfs
lnet networks=tcp

$ sudo vim /etc/xen/xend-config.sxp
$ sudo xend start
$ sudo xm list

### Startup Lustre ###
#### Steup-> Meta Data & Management Server (virt) ####
$ sudo mkfs.lustre --fsname=testfs --mdt --mgs /dev/sdb1
$ sudo mount -t lustre /dev/sdb1 /home/mdt

#### Setup-> Object Storage Machine (virt01) ####
$ sudo mkfs.lustre --fsname=testfs --ost --mgsnode=virt@tcp0 /dev/sda
$ sudo mount -t lustre /dev/sda /home/ost
$ sudo dmesg
Lustre: flyfs-OST0000: received MDS connection from 140.110.X.X@tcp

#### Setup-> Object Storage Machine (virt02) ####
$ sudo mkfs.lustre --fsname=testfs --ost --mgsnode=virt@tcp0 /dev/sda
$ sudo mount -t lustre /dev/sda /home/ost

#### Setup-> Client (virt03) ####
$ sudo mount -t lustre virt@tcp0:/testfs /home/testfs
$ mount
$ df -h

### Auto mount ###
$ mount -l -t lustre
/dev/sdb1 on /home/mdt type lustre (rw) [flyfs-MDT0000]
/dev/sdb2 on /home/ost type lustre (rw) [flyfs-OST0000]

$ vim /etc/fstab 
LABEL=flyfs-MDT0000 /home/mdt   lustre  defaults,_netdev,noauto 0 0
LABEL=flyfs-OST0000 /home/ost   lustre  defaults,_netdev,noauto 0 0

7.Install xen-tools (​http://www.xen-tools.org/software/xen-tools/)

$ wget http://www.xen-tools.org/software/xen-tools/xen-tools-3.9.tar.gz
$ cd xen-tools-3.9
$ sudo make install
$ sudo vim /etc/xen-tools/xen-tools.conf

Reference