= How to deploy kerrighed nodes massively using DRBL = [[BR]] == Introduction == People need powerful computing resources to resolve complex problems. A important issue is how to easy build cheap and effective computing resource. We use DRBL and Kerighed to build SSI cluster. It quick to deploy your cluster environment. You just install them in one machine (Server), clients don't install any OS. Client only setup PXE (network boot) in BIOS, when client finish network booting that your cluster is OK! [[BR]] == Software == We use below Software: * Ubuntu or Debian ||Linux Distribution.|| * DRBL ||Diskless Remote Boot in Linux (DRBL) provides a diskless or systemless environment for client machines. It works on Debian, Ubuntu, Mandriva, Red Hat, Fedora, CentOS and SuSE. DRBL uses distributed hardware resources and makes it possible for clients to fully access local hardware. It also includes Clonezilla, a partitioning and disk cloning utility similar to Symantec Ghost. http://drbl.sourceforge.net/|| [[BR]] * Kerrighed ||Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs. The new version is Kerrighed 2.2.0, and it used for 2.6.20 kernel. http://www.kerrighed.org/wiki/index.php/Main_Page|| [[BR]] == Install Kerrighed == * Install Basic Package {{{ sudo aptitude install gcc-3.3 automake autoconf libtool initramfs-tools make sudo aptitude install kernel-package libncurses5-dev build-essential fakeroot wget bzip2 sudo aptitude install xmlto lsb-release sudo aptitude install nfsbooted }}} * Download Kerighed tar ball {{{ cd /usr/src sudo wget https://gforge.inria.fr/frs/download.php/3791/kerrighed-2.2.1.tar.gz sudo wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2 sudo tar -zxvf kerrighed-2.2.0.tar.gz sudo tar -jxvf linux-2.6.20.tar.bz2 }}} * Config and make Kerrighed-enable kernel, modules {{{ cd kerrighed-2.2.0 sudo ./configure --with-kernel=/usr/src/linux-2.6.20 --disable-kernel-config --with-kernel-config=/usr/src/linux-2.6.20/.config sudo make patch sudo mv /bin/sh /bin/sh.old sudo ln -s /bin/bash /bin/sh (Ubuntu default sh is link to dash) (1)Choose kernel options cd ../linux-2.6.20 sudo make defconfig sudo make menuconfig (0.If your CPU arch is intel, choose i386 1.Cancel Preemption, KEYS and SMP options 2.Enable netwok block device, plug and play, NIC Driver and boot from nfs must to build-in ) (2)Make cd /usr/src/kerrighed-2.2.0 sudo make kernel sudo make sudo make kernel-install sudo make install }}} * Make initrd image {{{ sudo mkinitramfs -o /boot/initrd.img-2.6.20-krg 2.6.20-krg sudo vim /boot/grub/menu.lst (ex: title Kerrighed 2.6.20-krg root (hd0,0) kernel /boot/vmlinuz-2.6.20-krg root=UUID=8bc93e80-35f9-41db-a349-02e839f2ed7c ro quiet splash initrd /boot/initrd.img-2.6.20-krg quiet savedefault ) }}} * Config nodes information {{{ vim /etc/kerrighed_nodes (ex: session=7 nbmin=8 krg001:0:eth0 krg002:1:eth0 krg003:2:eth0 krg004:3:eth0 krg005:4:eth0 krg006:5:eth0 krg007:6:eth0 krg008:7:eth0 ) }}} [[BR]] == Install DRBL == * Add deb source {{{ sudo vim /etc/apt/sources.list (add : deb http://free.nchc.org.tw/drbl-core drbl stable) wget http://drbl.nchc.org.tw/GPG-KEY-DRBL sudo apt-key add GPG-KEY-DRBL sudo apt-get update }}} * Install DRBL {{{ Before we install DRBL, we must clear plan our DRBL environment. The below layout is our environment, eth0 used to connect WAN, eth1 used for DRBL internal clients. NIC NIC IP Clients +-----------------------------+ | DRBL SERVER | | | | +-- [eth0] 140.110.X.X +- to WAN | | | +-- [eth1] 192.168.0.1 +- to clients group 1 [ 7 clients, their IP from 192.168.0.2 - 192.168.0.8] | | +-----------------------------+ }}} {{{ sudo aptitude install drbl (DRBL will be installed in directory /opt/drbl ) sudo /opt/drbl/sbin/drblsrv -i sudo /opt/drbl/sbin/drblpush-offline -s `uname -r` (The command used interactive mothod help user to install. It install related packages (nfs, dhcp, tftp......) and create /tftpboot directory. The /tftpboot include: nbi_img: kenrel , initrd image and grub menu node_root: server directories copy nodes: each nodes' individual directories ) sudo /opt/drbl/sbin/drblpush -i (the command will deploy client environment, like client name, DRBL mode, swap ...) }}} * Setup each node's grub menu {{{ cd /tftpboot/nbi_im/pxelinux.cfg (named rule is IP's hexadecimal ex. 192.168.0.2 -> C0A80002) cp default C0A80002 vim C0A80002 ( add node_id in append line: ex. label drbl MENU DEFAULT # MENU HIDE MENU LABEL Ubuntu 7.04 Linux (DRBL mode, mostly local resources) # MENU PASSWD kernel vmlinuz-pxe append initrd=initrd-pxe.img devfs=nomount drblthincli=off selinux=0 node_id=0 session_id=9 ) }}} [[BR]] == test GPFS == * Running {{{ If kerrighed module don't auto load when booting: sudo /etc/init.d/kerrighed start (all node must load, and we can use command dmesg see node message ex. TIPC: Established link <1.1.1:eth0-1.1.3:eth0> on network plane A krg_node_arrival: 2 ) sudo krgadm cluster start (Kerrighed is running on 7 nodes) top (we cane see all clients' CPU and Memory are combined ex. top - 18:53:16 up 10 min, 2 users, load average: 0.10, 0.07, 0.04 Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie Cpu0 : 0.7%us, 0.3%sy, 0.0%ni, 98.6%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st Cpu1 : 0.7%us, 0.3%sy, 0.0%ni, 98.6%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st Cpu2 : 0.7%us, 0.3%sy, 0.0%ni, 98.7%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st Cpu3 : 0.8%us, 0.4%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.1%si, 0.0%st Cpu4 : 0.8%us, 0.4%sy, 0.0%ni, 98.5%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st Cpu5 : 0.7%us, 0.4%sy, 0.0%ni, 98.5%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st Cpu6 : 0.8%us, 0.3%sy, 0.0%ni, 98.6%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 14530264k total, 1508584k used, 13021680k free, 560k buffers Swap: 2650684k total, 0k used, 2650684k free, 1282652k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 15 0 1580 548 480 S 0 0.0 0:00.30 init.orig 2 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0 3 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0 4 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/0 ) }}} * Test Kerrighed command {{{ sudo krgadm nodes status (ex. [rock@krg002 ~]$ krgadm nodes 0:1 1:1 2:1 3:1 4:1 5:1 6:1 ) sudo krgcapset -s (ex. Permitted Capabilities: 037777777777 CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS SEE_LOCAL_PROC_STAT Effective Capabilities: 01 CHANGE_KERRIGHED_CAP Inheritable Permitted Capabilities: 037777777777 CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS SEE_LOCAL_PROC_STAT Inheritable Effective Capabilities: 01 CHANGE_KERRIGHED_CAP ) }}} * Test process migration {{{ DRBL Server: mkdir /home/ker ; chmod 777 /home/ker cd /home/ker wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz tar zxvf linux-2.6.22.tar.gz Client: krgcapset -d +CAN_MIGRATE cd /home/linux-2.6.22 sudo make -j 24 bzImage sudo dmesg (you can command dmesg to see message of process migration ex. send_kerrighed_signal: 36647 (Migration Mgr) -> 36885 (bzip2) send_kerrighed_signal: 36647 (Migration Mgr) -> 37449 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39064 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39234 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39269 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39325 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39402 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39465 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39543 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39599 (cc1) send_kerrighed_signal: 36647 (Migration Mgr) -> 39660 (cc1) ) }}} == Reference == DRBL http://drbl.sourceforge.net/ [[BR]] Kerrighed http:///www.kerrighed.org/wiki/index.php/Main_Page