wiki:jazz/11-09-27

Version 13 (modified by jazz, 13 years ago) (diff)

--

2011-09-27

  • 為何 rock 要把 bridge.ko 加入 /usr/bin/mkpxeinitrd-net 呢?答案很明顯是跟 brctl 有關!
    root@debian:~# lsmod > before
    root@debian:~# brctl addbr br0
    root@debian:~# lsmod > after
    root@debian:~# diff before after 
    1a2,3
    > bridge                 39662  0 
    > stp                     1440  1 bridge
    
  • rock 把執行 /bin/start_xen_bridge.sh 需要用的指令都補進 initrd 裡:brctl ip egrep awk gawk seq fgrep dirname expr bash
  • 可是他卻漏了 mount umount mount.nfs umount.nfs
  • 至於 netloop.ko 是 xen_brdige.sh 中會用到的,在 KVM 的 case 中並不會用到。
    • /usr/bin/mkpxeinitrd-net

      old new  
      3737# uncompress the compress kernel module. This is special for Mandriva case. It uses compressed kernel module, e.g. pcnet32.ko.gz. Before we used modprobe from busybox, we have to uncompress that. Now the modprobe program is from the OS (See variable include_bin_prog_from_server), so we should allow this type of kernel modules.
      3838use_compressed_kernel_module="yes"
      3939# Some required bin programs to be included in the PXE initrd, which are not provided by busybox or the one provided by busybox does not support the function we want. E.g. sleep (we need "sleep 0.1", while sleep from busybox does not support 0.1 secs).
      40 include_bin_prog_from_server="sleep lspci insmod modprobe rmmod lsmod pkill strings mount umount mount.nfs umount.nfs"
       40# drbl-virt add
       41include_bin_prog_from_server='sleep lspci insmod modprobe rmmod lsmod pkill strings brctl ip egrep awk gawk seq fgrep dirname expr bash'
      4142
      4243# No need to use sudo if we are root
      4344if [ $UID -eq 0 ]; then
       
      303304     cp -a --parents $i $initrd/lib/modules/$kernel_ver/
      304305   done
      305306
       307# drbl-virt add
       308cp -a --parents kernel/net/bridge/bridge.ko $initrd/lib/modules/$kernel_ver/
       309cp -a --parents kernel/drivers/xen/netback/netloop.ko $initrd/lib/modules/$kernel_ver/
      306310   # Deal with firmwares!
      307311   # The following is borrowed from Debian's /usr/share/initramfs-tools/hook-functions
      308312   if [ "$copy_all_firmwares" = "yes" ]; then
  • rock 的 init 切入點蠻特殊的,我原本是把切換成 br0 的程序加在 udhcpc 之前,試圖讓 br0 去取得 IP 位址。可是卻會出現 /sbin/init 第 175 行錯誤的問題。
    150 $echo "Bringing up loopback interface"
    151 ifconfig lo 127.0.0.1 up
    152 route add -net 127.0.0.0 netmask 255.0.0.0 lo
    153 ### 把 /bin/sh 加在這裡
    
  • PXE 開機,進到 shell 後,執行以下指令,然後下 exit 跳出,讓它繼續往下跑:
       brctl addbr br0
       brctl addif br0 eth0
       ifconfig br0 0.0.0.0   ### 在 initrd.img 中要用這種方式才能啟動 br0
       ifconfig eth0 0.0.0.0  
    
  • 就會看到 /sbin/init 第 175 行的錯誤訊息:
  • 實際上 /sbin/init 等於 DRBL Server 的 /tftpboot/node_root/sbin/init ,而第 175 行的錯誤,是因為 161 行只有找 eth 開頭,但拿到 IP 的是 br0
    160 # find my IP address
    161 NETDEVICES="$(cat /proc/net/dev | awk -F: '/eth.:|tr.:/{print $1}')"
    162 for DEVICE in $NETDEVICES; do
    163   IP_tmp="$(ifconfig $DEVICE | grep -A1 $DEVICE | grep -v $DEVICE | grep "inet addr" | sed -e     's/^.*inet addr:\([0-9\.]\+\).*$/\1/')"
    164   if [ -n "$IP_tmp" ]; then
    165     # Got the IP address, stop to get from other port, so break
    166     IP=$IP_tmp
    167     echo "My IP address is $IP ([$DEVICE])."
    168     break
    169   fi
    170 done
    171 
    172 IP_prefix="$(echo $IP | cut -d"." -f1-3)"
    173 if [ -n "$(echo "$NFSSERVER_LIST" | grep -E "$IP_prefix.[0-9]+")" ]; then
    174   for i in $NFSSERVER_LIST; do
    175     if [ "$(echo $i | cut -d"." -f1-3)" = $IP_prefix ]; then
    176       nfsserver=$i
    177       break
    178     fi
    179   done
    180 else
    
  • 因此,有一個變通的方式就是把 br0 更名為 eth*(如 eth9)。簡單測試了一下,確實就可以正常 PXE 開機進入 DRBL Client。
  • 若要維持 br* 的作法,必須修正 /sbin/init 的程式碼。
  • /tftpboot/node_root/sbin/init 是由 /opt/drbl/setup/files/misc/init.drbl 拷貝而來,因此如果要 patch 的話,應該先 patch /opt/drbl/setup/files/misc/init.drbl 這個檔案,如此未來只要跑 drblsrv 或 drblsrv-offline 均會把 br* 加入考慮。
    • /opt/drbl/setup/files/misc/init.drbl

      old new  
      158158create_dev
      159159
      160160# find my IP address
      161 NETDEVICES="$(cat /proc/net/dev | awk -F: '/eth.:|tr.:/{print $1}')"
       161NETDEVICES="$(cat /proc/net/dev | awk -F: '/eth.:|tr.:|br.:/{print $1}')"
      162162for DEVICE in $NETDEVICES; do
      163163  IP_tmp="$(ifconfig $DEVICE | grep -A1 $DEVICE | grep -v $DEVICE | grep "inet addr" | sed -e 's/^.*inet addr:\([0-9\.]\+\).*$/\1/')"
      164164  if [ -n "$IP_tmp" ]; then

  • rock 的作法是在 udhcpc 已經用 eth0 取得 IP 之後,才去把 eth0 暫停,新增 br0,再將 eth0 加入 br0,就能正常掛載 NFS 當 ROOTFS。這真的是需要一些觀念啊~
  • start_xen_bridge.sh 其實有點小複雜,會先去判斷網卡個數,然後透過 /bin/network-bridge 的 netdev 參數來指定網卡。但是這種作法的優點是如果有多張網卡的話,就可以把每一張網卡都對應到 Bridge 裝置
    # cat /usr/lib/mkpxeinitrd-net/initrd-skel/bin/start_xen_bridge.sh
    #!/bin/bash
    # drbl-virt add
    NICs=$(/sbin/ifconfig | grep eth | awk '{print $1}')
    declare -i NICs_nu=$(echo $NICs |wc -w)
    NIC=""
    
    if [ $NICs_nu -lt 1  ]; then
        for (( i=0 ; i<$NICs_nu ; i++ ))
        do
            NIC_IP=$(ifconfig eth${i} | grep "inet addr" | sed 's/inet addr://g' | sed 's/Bcast.*$//g' | sed 's/^[ ]*//')
            if [ -n $NIC_IP ]; then
                NIC="eth${i}"
                break
            fi
        done
    else
        NIC=$NICS
    fi
    
    bash /bin/network-bridge start netdev=$NIC
    

Attachments (7)

Download all attachments as: .zip