How to deploy kerrighed nodes massively using DRBL

Introduction

People need powerful computing resources to resolve complex problems. A important issue is how to easy build cheap and effective computing resource. We use DRBL and Kerighed to build SSI cluster. It quick to deploy your cluster environment. You just install them in one machine (Server), clients don't install any OS. Client only setup PXE (network boot) in BIOS, when client finish network booting that your cluster is OK!

Software

We use below Software:

• Ubuntu or Debian

Linux Distribution.

• DRBL

​http://drbl.sourceforge.net/

Diskless Remote Boot in Linux (DRBL) provides a diskless or systemless environment for client machines. It works on Debian, Ubuntu, Mandriva, Red Hat, Fedora, CentOS and SuSE. DRBL uses distributed hardware resources and makes it possible for clients to fully access local hardware. It also includes Clonezilla, a partitioning and disk cloning utility similar to Symantec Ghost.

• Kerrighed

​http://www.kerrighed.org/wiki/index.php/Main_Page

Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs. The new version is Kerrighed 2.2.0, and it used for 2.6.20 kernel.

Install Kerrighed

• Install Basic Package

  sudo aptitude install gcc-3.3 automake autoconf libtool initramfs-tools make
  sudo aptitude install kernel-package libncurses5-dev build-essential fakeroot wget bzip2 
  sudo aptitude install xmlto lsb-release 
  sudo aptitude install nfsbooted 
  

• Download Kerighed tar ball

  cd /usr/src
  sudo wget https://gforge.inria.fr/frs/download.php/3791/kerrighed-2.2.1.tar.gz
  sudo wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2 
  
  sudo tar -zxvf kerrighed-2.2.0.tar.gz
  sudo tar -jxvf linux-2.6.20.tar.bz2 
  

• Config and make Kerrighed-enable kernel, modules

  cd kerrighed-2.2.0 
  sudo ./configure --with-kernel=/usr/src/linux-2.6.20 --disable-kernel-config --with-kernel-config=/usr/src/linux-2.6.20/.config 
  sudo make patch 
  
  sudo mv /bin/sh /bin/sh.old
  sudo ln -s /bin/bash /bin/sh
  (Ubuntu default sh is link to dash) 
  
  (1)Choose kernel options
  cd ../linux-2.6.20 
  sudo make defconfig
  sudo make menuconfig
  (0.If your CPU arch is intel, choose i386 
   1.Cancel Preemption, KEYS  and SMP options 
   2.Enable netwok block device, plug and play, NIC Driver and boot from nfs must to build-in
  )
   
  (2)Make
  cd /usr/src/kerrighed-2.2.0
  sudo make kernel 
  sudo make 
  sudo make kernel-install 
  sudo make install  
  

• Make initrd image

  sudo mkinitramfs -o /boot/initrd.img-2.6.20-krg 2.6.20-krg
  sudo vim /boot/grub/menu.lst 
  
  (ex:
  title           Kerrighed 2.6.20-krg
  root            (hd0,0)
  kernel          /boot/vmlinuz-2.6.20-krg root=UUID=8bc93e80-35f9-41db-a349-02e839f2ed7c ro quiet splash
  initrd          /boot/initrd.img-2.6.20-krg
  quiet
  savedefault
  )
  

• Config nodes information

  vim /etc/kerrighed_nodes
  (ex:
  session=7
  nbmin=8
  krg001:0:eth0
  krg002:1:eth0
  krg003:2:eth0
  krg004:3:eth0
  krg005:4:eth0
  krg006:5:eth0
  krg007:6:eth0
  krg008:7:eth0
  )
  

Install DRBL

• Add deb source

  sudo vim /etc/apt/sources.list 
  (add :  deb http://free.nchc.org.tw/drbl-core drbl stable)
  
  wget http://drbl.nchc.org.tw/GPG-KEY-DRBL sudo apt-key add GPG-KEY-DRBL 
  sudo apt-get update
  

• Install DRBL

  Before we install DRBL, we must clear plan our DRBL environment. The below layout is our environment, eth0 used to connect WAN, eth1 used for DRBL internal clients.
  
               NIC     NIC IP                              Clients 
  +-----------------------------+ 
  |     DRBL SERVER         | 
  |                                         | 
  |     +-- [eth0] 140.110.X.X  +- to WAN 
  |                                         | 
  |     +-- [eth1] 192.168.0.1 +- to clients group 1 [ 7 clients, their IP from 192.168.0.2 - 192.168.0.8] 
  |                                         |
  +-----------------------------+ 
  

sudo aptitude install drbl
(DRBL will be installed in directory /opt/drbl )

sudo /opt/drbl/sbin/drblsrv -i
sudo /opt/drbl/sbin/drblpush-offline -s `uname -r` 
(The command used interactive mothod help user to install. It install related packages (nfs, dhcp, tftp......) and create /tftpboot directory. The /tftpboot include:
nbi_img: kenrel , initrd image and grub menu
node_root: server directories copy
nodes: each nodes' individual directories
)

sudo /opt/drbl/sbin/drblpush -i 
(the command will deploy client environment, like client name, DRBL mode, swap ...)

• Setup each node's grub menu

  cd /tftpboot/nbi_im/pxelinux.cfg
  (named rule is IP's hexadecimal  
  ex. 192.168.0.2  ->  C0A80002)
  
  cp default  C0A80002
  vim  C0A80002
  ( add node_id in append line:
  ex.
  label drbl
    MENU DEFAULT
    # MENU HIDE
    MENU LABEL Ubuntu 7.04 Linux (DRBL mode, mostly local resources)
    # MENU PASSWD
    kernel vmlinuz-pxe
    append initrd=initrd-pxe.img devfs=nomount drblthincli=off selinux=0 node_id=0 session_id=9
  )
  

test GPFS

• Running

  If kerrighed module don't auto load when booting:
  sudo /etc/init.d/kerrighed start
  (all node must load, and we can use command dmesg see node message
  ex.
  TIPC: Established link <1.1.1:eth0-1.1.3:eth0> on network plane A
  krg_node_arrival: 2
  )
  
  sudo krgadm cluster start
  (Kerrighed is running on 7 nodes)
  
  top
  (we cane see all clients' CPU and Memory are combined
  ex.
  top - 18:53:16 up 10 min,  2 users,  load average: 0.10, 0.07, 0.04
  Tasks: 221 total,   1 running, 220 sleeping,   0 stopped,   0 zombie
  Cpu0  :  0.7%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
  Cpu1  :  0.7%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
  Cpu2  :  0.7%us,  0.3%sy,  0.0%ni, 98.7%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
  Cpu3  :  0.8%us,  0.4%sy,  0.0%ni, 98.4%id,  0.3%wa,  0.0%hi,  0.1%si,  0.0%st
  Cpu4  :  0.8%us,  0.4%sy,  0.0%ni, 98.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
  Cpu5  :  0.7%us,  0.4%sy,  0.0%ni, 98.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
  Cpu6  :  0.8%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
  Mem:  14530264k total,  1508584k used, 13021680k free,      560k buffers
  Swap:  2650684k total,        0k used,  2650684k free,  1282652k cached
  
    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      1 root      15   0  1580  548  480 S    0  0.0   0:00.30 init.orig
      2 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
      3 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
      4 root      10  -5     0    0    0 S    0  0.0   0:00.00 events/0
  )
  

• Test Kerrighed command

  sudo krgadm nodes status
  (ex.
  [rock@krg002 ~]$ krgadm nodes
    0:1   1:1   2:1   3:1   4:1   5:1   6:1
  )
  
  sudo krgcapset -s
  (ex.
  Permitted Capabilities: 037777777777
          CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY
          CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS
          USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS
          SEE_LOCAL_PROC_STAT
  Effective Capabilities: 01
          CHANGE_KERRIGHED_CAP
  Inheritable Permitted Capabilities: 037777777777
          CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY
          CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS
          USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS
          SEE_LOCAL_PROC_STAT
  Inheritable Effective Capabilities: 01
          CHANGE_KERRIGHED_CAP
  )
  

• Test process migration

  DRBL Server:
  mkdir /home/ker ; chmod 777 /home/ker
  cd /home/ker
  wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz
  tar zxvf linux-2.6.22.tar.gz
  
  Client:
  krgcapset -d +CAN_MIGRATE
  cd /home/linux-2.6.22
  sudo make -j 24 bzImage
  sudo dmesg
  (you can command dmesg to see message of process migration
  ex.
  send_kerrighed_signal: 36647 (Migration Mgr) -> 36885 (bzip2)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 37449 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39064 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39234 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39269 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39325 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39402 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39465 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39543 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39599 (cc1)
  send_kerrighed_signal: 36647 (Migration Mgr) -> 39660 (cc1)
  )
  

Reference

DRBL ​http://drbl.sourceforge.net/
Kerrighed ​http:///www.kerrighed.org/wiki/index.php/Main_Page