wiki:krg_DRBL

  1. How to deploy kerrighed nodes massively using DRBL
    1. 1. Introduction
    2. 2. Software
    3. 3. Install Kerrighed
    4. 4. Install DRBL
    5. 5. Test Kerrighed
    6. 6. Reference

How to deploy kerrighed nodes massively using DRBL


rock, rock@…


1. Introduction

People need powerful computing resources to resolve complex problems. A important issue is how to easy build cheap and effective computing resource. We use DRBL and Kerighed to build SSI cluster. It quick to deploy your cluster environment. You just install them in one machine (Server), clients don't install any OS. Client only setup PXE (network boot) in BIOS, when client finish network booting that your cluster is OK!

2. Software

We use below Software:

  • Ubuntu or Debian
Linux Distribution.
  • DRBL
​http://drbl.sourceforge.net/
Diskless Remote Boot in Linux (DRBL) provides a diskless or systemless environment for client machines. It works on Debian, Ubuntu, Mandriva, Red Hat, Fedora, CentOS and SuSE. DRBL uses distributed hardware resources and makes it possible for clients to fully access local hardware. It also includes Clonezilla, a partitioning and disk cloning utility similar to Symantec Ghost.


  • Kerrighed
​http://www.kerrighed.org/wiki/index.php/Main_Page
Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs. The new version is Kerrighed 2.2.1, and it used for 2.6.20 kernel.


3. Install Kerrighed

  • Install Basic Package 
    $ sudo aptitude install gcc-3.3 automake autoconf libtool initramfs-tools make
$ sudo aptitude install kernel-package libncurses5-dev build-essential fakeroot wget bzip2 
$ sudo aptitude install xmlto lsb-release 
$ sudo aptitude install nfsbooted
  • Download Kerighed tar ball 
    $ cd /usr/src
$ sudo wget https://gforge.inria.fr/frs/download.php/3791/kerrighed-2.2.1.tar.gz
$ sudo wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2 

$ sudo tar -zxvf kerrighed-2.2.1.tar.gz
$ sudo tar -jxvf linux-2.6.20.tar.bz2
  • Config and make Kerrighed-enable kernel, modules 
    $ cd kerrighed-2.2.1 
$ sudo ./configure --with-kernel=/usr/src/linux-2.6.20 --with-kernel-config=/usr/src/linux-2.6.20/.config 
$ sudo make patch 

$ sudo mv /bin/sh /bin/sh.old
$ sudo ln -s /bin/bash /bin/sh
(Ubuntu default sh is link to dash) 

(1)Choose kernel options
$ cd ../linux-2.6.20 
$ sudo make defconfig
$ sudo make menuconfig
(If you want to edit kernel options, note that the following are currently broken with Kerrighed:

    * General setup -> IPC Namespaces
    * Processor type and features -> Preemption Model -> Voluntary Kernel Preemption
    * Processor type and features -> Preemption Model -> Preemptible Kernel
    * Networking -> IrDA (infrared) subsystem support
    * Security options -> Enable access key retention support
    * Networking -> Networking options -> The Ipv6 protocol 

Note: don't forget to add NIC driver and, if you plan to use NFSROOT, include it in the kernel, not as module. 
) 

(2)Make
$ cd /usr/src/kerrighed-2.2.1
$ sudo make kernel 
$ sudo make 
$ sudo make kernel-install 
$ sudo make install
  • Make initrd image 
    $ sudo mkinitramfs -o /boot/initrd.img-2.6.20-krg 2.6.20-krg
$ sudo vim /boot/grub/menu.lst 

(ex:
title           Kerrighed 2.6.20-krg
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.20-krg root=/dev/sda1 ro quiet splash
initrd          /boot/initrd.img-2.6.20-krg
quiet
savedefault
)
  • Config nodes information 
    $ vim /etc/kerrighed_nodes
(ex:
session=7
nbmin=8
krg001:0:eth0
krg002:1:eth0
krg003:2:eth0
krg004:3:eth0
krg005:4:eth0
krg006:5:eth0
krg007:6:eth0
krg008:7:eth0
)

$ reboot
(reboot and choose 2.6.20-krg kernel)


4. Install DRBL

  • Add deb source 
    $ sudo vim /etc/apt/sources.list 
(add :  deb http://free.nchc.org.tw/drbl-core drbl stable)

$ wget http://drbl.nchc.org.tw/GPG-KEY-DRBL sudo apt-key add GPG-KEY-DRBL 
$ sudo apt-get update
  • Install DRBL 
    Before we install DRBL, we must clear plan our DRBL environment. The below layout is our environment, eth0 used to connect WAN, eth1 used for DRBL internal clients.

             NIC     NIC IP                              Clients 
+-----------------------------+ 
|         DRBL SERVER         | 
|                             | 
|     +-- [eth0] 140.110.X.X  +- to WAN 
|                             | 
|     +-- [eth1] 192.168.0.1  +- to clients group 1 [ 7 clients, their IP from 192.168.0.2 - 192.168.0.8] 
|                             |
+-----------------------------+ 

$ sudo aptitude install drbl
(DRBL will be installed in directory /opt/drbl )

$ sudo /opt/drbl/sbin/drblsrv -i
(If DRBL install other kernel version, you can use this command to designate 2.6.20-krg:
$ sudo /opt/drbl/sbin/drblsrv-offline -s `uname -r`
)

$ sudo /opt/drbl/sbin/drblpush -i 
(The command used interactive mothod help user to install. It install related packages (nfs, dhcp, tftp......) and create /tftpboot directory. The /tftpboot include:
nbi_img: kenrel , initrd image and grub menu
node_root: server directories copy
nodes: each nodes' individual directories
)

$ sudo /opt/drbl/sbin/drblpush -i 
(the command will deploy client environment, like client name, DRBL mode, swap ...)
  • Setup each node's grub menu 
    $ cd /tftpboot/nbi_im/pxelinux.cfg
(named rule is IP's hexadecimal   ex. 192.168.0.2  ->  C0A80002)

$ cp default  C0A80002
$ vim  C0A80002
( add node_id in append line:
ex.
label drbl
  MENU DEFAULT
  # MENU HIDE
  MENU LABEL Kerrighed 2.2.1 for kernel 2.6.20-krg
  # MENU PASSWD
  kernel vmlinuz-pxe
  append initrd=initrd-pxe.img devfs=nomount drblthincli=off selinux=0 node_id=0 session_id=7
)


5. Test Kerrighed

  • Running 
    If kerrighed module don't auto load when booting:
$ sudo /etc/init.d/kerrighed start
(all node must load, and we can use command dmesg see node message
ex.
TIPC: Established link <1.1.1:eth0-1.1.3:eth0> on network plane A
krg_node_arrival: 2
)

$ sudo krgadm cluster start
(Kerrighed is running on 7 nodes)

$ top
(we cane see all clients' CPU and Memory are combined
ex.
top - 18:53:16 up 10 min,  2 users,  load average: 0.10, 0.07, 0.04
Tasks: 221 total,   1 running, 220 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.7%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu1  :  0.7%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu2  :  0.7%us,  0.3%sy,  0.0%ni, 98.7%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu3  :  0.8%us,  0.4%sy,  0.0%ni, 98.4%id,  0.3%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu4  :  0.8%us,  0.4%sy,  0.0%ni, 98.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu5  :  0.7%us,  0.4%sy,  0.0%ni, 98.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu6  :  0.8%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  14530264k total,  1508584k used, 13021680k free,      560k buffers
Swap:  2650684k total,        0k used,  2650684k free,  1282652k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      15   0  1580  548  480 S    0  0.0   0:00.30 init.orig
    2 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    3 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
    4 root      10  -5     0    0    0 S    0  0.0   0:00.00 events/0
)
  • Test Kerrighed command 
    $ sudo krgadm nodes status
(ex.
[rock@krg002 ~]$ krgadm nodes
  0:1   1:1   2:1   3:1   4:1   5:1   6:1
)

$ sudo krgcapset -s
(ex.
Permitted Capabilities: 037777777777
        CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY
        CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS
        USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS
        SEE_LOCAL_PROC_STAT
Effective Capabilities: 01
        CHANGE_KERRIGHED_CAP
Inheritable Permitted Capabilities: 037777777777
        CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY
        CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS
        USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS
        SEE_LOCAL_PROC_STAT
Inheritable Effective Capabilities: 01
        CHANGE_KERRIGHED_CAP
)
  • Test process migration 
    DRBL Server:
$ mkdir /home/ker ; chmod 777 /home/ker
$ cd /home/ker
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz
$ tar zxvf linux-2.6.22.tar.gz

Client:
$ krgcapset -d +CAN_MIGRATE
$ cd /home/linux-2.6.22
$ sudo make -j 24 bzImage
$ sudo dmesg
(you can command dmesg to see message of process migration
ex.
send_kerrighed_signal: 36647 (Migration Mgr) -> 36885 (bzip2)
send_kerrighed_signal: 36647 (Migration Mgr) -> 37449 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39064 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39234 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39269 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39325 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39402 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39465 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39543 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39599 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39660 (cc1)
)

6. Reference

DRBL ​http://drbl.sourceforge.net/
Kerrighed ​http:///www.kerrighed.org/wiki/index.php/Main_Page

Last modified 16 years ago Last modified on Mar 12, 2008, 6:26:27 PM

Attachments (2)

Download all attachments as: .zip