Changes between Version 2 and Version 3 of krg_DRBL


Ignore:
Timestamp:
Feb 21, 2008, 12:06:02 PM (16 years ago)
Author:
rock
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • krg_DRBL

    v2 v3  
    1 {{{
    2 #!html
    3 <html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="content-type" content="text/html; charset=utf-8" /><title /><meta name="generator" content="StarOffice/OpenOffice.org XSLT (http://xml.openoffice.org/sx2ml)" /><meta name="created" content="2007-11-09T17:16:18" /><meta name="changedby" content="rock kuo" /><meta name="changed" content="2007-12-06T12:04:00" /><base href="." /><style type="text/css">
    4         @page { size: 20.999cm 29.699cm; margin-top: 2cm; margin-bottom: 2cm; margin-left: 2cm; margin-right: 2cm }
    5         table { border-collapse:collapse; border-spacing:0; empty-cells:show }
    6         td, th { vertical-align:top; }
    7         h1, h2, h3, h4, h5, h6 { clear:both }
    8         ol, ul { padding:0; }
    9         * { margin:0; }
    10         *.fr1 { color:#000000; font-size:12pt; vertical-align:top; }
    11         *.fr2 { color:#000000; font-size:12pt; vertical-align:top; text-align:center; }
    12         *.fr3 { color:#000000; font-size:12pt; vertical-align:top; text-align:center; }
    13         *.Graphics { color:#000000; font-size:12pt; vertical-align:top; text-align:center; }
    14         *.Contents1 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:0cm; margin-right:0cm; text-indent:0cm; }
    15         *.Contents10 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:4.493cm; margin-right:0cm; text-indent:0cm; }
    16         *.Contents2 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:0.499cm; margin-right:0cm; text-indent:0cm; }
    17         *.Contents3 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:0.998cm; margin-right:0cm; text-indent:0cm; }
    18         *.Contents4 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:1.498cm; margin-right:0cm; text-indent:0cm; }
    19         *.Contents5 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:1.997cm; margin-right:0cm; text-indent:0cm; }
    20         *.Contents6 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:2.496cm; margin-right:0cm; text-indent:0cm; }
    21         *.Contents7 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:2.995cm; margin-right:0cm; text-indent:0cm; }
    22         *.Contents8 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:3.494cm; margin-right:0cm; text-indent:0cm; }
    23         *.Contents9 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:3.993cm; margin-right:0cm; text-indent:0cm; }
    24         *.ContentsHeading { color:#000000; font-family:Arial; font-size:16pt; margin-top:0.423cm; margin-bottom:0.499cm; margin-left:0cm; margin-right:0cm; text-indent:0cm; font-weight:bold; }
    25         *.Heading { color:#000000; font-family:Arial; font-size:14pt; margin-top:0.423cm; margin-bottom:0.499cm; }
    26         *.Heading1 { color:#000000; font-family:'Times New Roman'; font-size:24pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; }
    27         *.Heading2 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; }
    28         *.Index { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    29         *.IndexHeading { color:#000000; font-family:Arial; font-size:16pt; margin-top:0.423cm; margin-bottom:0.499cm; margin-left:0cm; margin-right:0cm; text-indent:0cm; font-weight:bold; }
    30         *.P1 { color:#000000; font-family:'Times New Roman'; font-size:18pt; text-align:center ! important; }
    31         *.P10 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-weight:normal; }
    32         *.P100 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    33         *.P101 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    34         *.P102 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    35         *.P103 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    36         *.P104 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    37         *.P105 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    38         *.P106 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    39         *.P107 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    40         *.P108 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    41         *.P109 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    42         *.P11 { color:#000000; font-family:'Times New Roman'; font-size:18pt; text-align:center ! important; font-weight:bold; }
    43         *.P110 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    44         *.P111 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    45         *.P112 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    46         *.P113 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    47         *.P114 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    48         *.P115 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    49         *.P116 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    50         *.P117 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    51         *.P118 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    52         *.P119 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    53         *.P12 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; font-weight:normal; }
    54         *.P120 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    55         *.P121 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    56         *.P122 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    57         *.P123 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    58         *.P124 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    59         *.P125 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:bold; }
    60         *.P126 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:bold; }
    61         *.P127 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:bold; }
    62         *.P128 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-weight:bold; }
    63         *.P129 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:bold; }
    64         *.P13 { color:#000000; font-family:'Times New Roman'; font-size:24pt; text-align:center ! important; font-weight:bold; }
    65         *.P130 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; font-weight:normal; }
    66         *.P131 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:normal; }
    67         *.P132 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:normal; }
    68         *.P133 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    69         *.P134 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    70         *.P135 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    71         *.P136 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    72         *.P137 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    73         *.P138 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    74         *.P139 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    75         *.P14 { color:#000000; font-family:'Times New Roman'; font-size:18pt; font-weight:bold; }
    76         *.P140 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    77         *.P141 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    78         *.P142 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; }
    79         *.P143 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    80         *.P144 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    81         *.P145 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    82         *.P146 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:italic; }
    83         *.P147 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    84         *.P148 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; }
    85         *.P149 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    86         *.P15 { color:#000000; font-family:'Times New Roman'; font-size:18pt; text-align:center ! important; font-weight:bold; }
    87         *.P150 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    88         *.P151 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    89         *.P152 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    90         *.P153 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    91         *.P154 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    92         *.P155 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    93         *.P156 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    94         *.P157 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    95         *.P158 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    96         *.P159 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    97         *.P16 { color:#000000; font-family:'Times New Roman'; font-size:18pt; }
    98         *.P160 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    99         *.P161 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    100         *.P162 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    101         *.P163 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:-1.251cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; }
    102         *.P164 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:-1.251cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; }
    103         *.P165 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:-1.251cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:italic; font-weight:normal; }
    104         *.P166 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    105         *.P167 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    106         *.P168 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    107         *.P169 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    108         *.P17 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    109         *.P170 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    110         *.P171 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    111         *.P172 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    112         *.P18 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    113         *.P19 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; }
    114         *.P2 { color:#000000; font-family:'Times New Roman'; font-size:18pt; font-weight:bold; }
    115         *.P20 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; }
    116         *.P21 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; font-weight:bold; }
    117         *.P22 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    118         *.P23 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    119         *.P24 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; }
    120         *.P25 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; }
    121         *.P26 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; font-weight:bold; }
    122         *.P27 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-weight:bold; }
    123         *.P28 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-style:italic; }
    124         *.P29 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-style:italic; }
    125         *.P3 { color:#ff0000; font-family:'Times New Roman'; font-size:12pt; }
    126         *.P30 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-style:italic; font-weight:normal; }
    127         *.P31 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; }
    128         *.P32 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; }
    129         *.P33 { color:#dc2300; font-family:'Times New Roman'; font-size:10pt; text-align:left ! important; }
    130         *.P34 { color:#dc2300; font-family:'Times New Roman'; font-size:10pt; text-align:left ! important; font-style:normal; }
    131         *.P35 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    132         *.P36 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    133         *.P37 { color:#000000; font-family:'Times New Roman'; font-size:14pt; margin-top:0cm; margin-bottom:0.499cm; text-align:center ! important; font-style:normal; }
    134         *.P38 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; }
    135         *.P39 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:normal; }
    136         *.P4 { color:#ff0000; font-family:'Times New Roman'; font-size:12pt; }
    137         *.P40 { color:#000000; font-family:'Times New Roman'; font-size:14pt; margin-top:0cm; margin-bottom:0.499cm; text-align:left ! important; font-style:normal; font-weight:bold; }
    138         *.P41 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    139         *.P42 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    140         *.P43 { color:#000080; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    141         *.P44 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    142         *.P45 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; }
    143         *.P46 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:normal; }
    144         *.P47 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:italic; }
    145         *.P48 { color:#000000; font-family:'Times New Roman'; font-size:10pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    146         *.P49 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; }
    147         *.P5 { color:#ff0000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; }
    148         *.P50 { color:#dc2300; font-family:'Times New Roman'; font-size:10pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    149         *.P51 { color:#dc2300; font-family:'Times New Roman'; font-size:10pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:normal; }
    150         *.P52 { color:#cc0000; font-family:'Times New Roman'; font-size:10pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; background-color:transparent; }
    151         *.P53 { color:#cc0000; font-family:'Times New Roman'; font-size:10pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    152         *.P54 { color:#cc0000; font-family:'Times New Roman'; font-size:10pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:normal; }
    153         *.P55 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:normal; background-color:transparent; }
    154         *.P56 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; }
    155         *.P57 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    156         *.P58 { color:#000000; font-family:'Times New Roman'; font-size:14pt; margin-top:0cm; margin-bottom:0cm; margin-left:0cm; margin-right:0cm; text-align:left ! important; text-indent:0cm; font-style:normal; font-weight:bold; background-color:transparent; }
    157         *.P59 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:0cm; margin-right:0cm; margin-top:0cm; margin-bottom:0cm; text-align:left ! important; text-indent:0cm; font-style:italic; }
    158         *.P6 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    159         *.P60 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-style:italic; font-weight:normal; }
    160         *.P61 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; font-style:italic; }
    161         *.P62 { color:#dc2300; font-family:'Times New Roman'; font-size:10pt; text-align:left ! important; }
    162         *.P63 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; }
    163         *.P64 { color:#000000; font-family:'Times New Roman'; font-size:14pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; }
    164         *.P65 { color:#000000; font-family:'Times New Roman'; font-size:14pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; }
    165         *.P66 { color:#000000; font-family:'Times New Roman'; font-size:14pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; }
    166         *.P67 { color:#000000; font-family:'Times New Roman'; font-size:18pt; margin-top:0.423cm; margin-bottom:0.499cm; font-weight:bold; text-align:center ! important; }
    167         *.P68 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:0cm; margin-right:0cm; text-indent:0cm; }
    168         *.P69 { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-left:0.499cm; margin-right:0cm; text-indent:0cm; }
    169         *.P7 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; }
    170         *.P70 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    171         *.P71 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    172         *.P72 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    173         *.P73 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    174         *.P74 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    175         *.P75 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    176         *.P76 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    177         *.P77 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    178         *.P78 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    179         *.P79 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    180         *.P8 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:left ! important; }
    181         *.P80 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    182         *.P81 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    183         *.P82 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    184         *.P83 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-weight:bold; }
    185         *.P84 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    186         *.P85 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    187         *.P86 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    188         *.P87 { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    189         *.P88 { color:#000000; font-family:'Times New Roman'; font-size:14pt; }
    190         *.P89 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    191         *.P9 { color:#000000; font-family:'Times New Roman'; font-size:12pt; text-align:center ! important; font-weight:normal; }
    192         *.P90 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    193         *.P91 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    194         *.P92 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    195         *.P93 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    196         *.P94 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    197         *.P95 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    198         *.P96 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    199         *.P97 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    200         *.P98 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    201         *.P99 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; font-weight:bold; }
    202         *.Standard { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    203         *.TableContents { color:#000000; font-family:'Times New Roman'; font-size:12pt; }
    204         *.Textbody { color:#000000; font-family:'Times New Roman'; font-size:12pt; margin-top:0cm; margin-bottom:0.499cm; }
    205         *.Sect1 { }
    206         *.conf { width:16.999cm; float:none; }
    207         *.drbl { width:16.999cm; float:none; }
    208         *.init { width:17.014cm; margin-left:-0.016cm; margin-right:0cm; float:none; }
    209         *.pack { width:16.999cm; float:none; }
    210         *.reboot { width:16.999cm; float:none; }
    211         *.wget { width:16.999cm; float:none; }
    212         *.表格1 { width:13.388cm; margin-left:1.579cm; margin-right:2.032cm; float:none; }
    213         *.表格10 { width:16.999cm; float:none; }
    214         *.表格2 { width:16.977cm; }
    215         *.表格3 { width:16.999cm; float:none; }
    216         *.表格4 { width:16.999cm; float:none; }
    217         *.表格5 { width:16.999cm; float:none; }
    218         *.表格6 { width:16.999cm; float:none; }
    219         *.表格7 { width:16.999cm; float:none; }
    220         *.表格8 { width:16.999cm; float:none; }
    221         *.表格9 { width:16.999cm; float:none; }
    222         *.confA1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    223         *.drblA1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    224         *.initA1 { vertical-align:middle; padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    225         *.packA1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    226         *.rebootA1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    227         *.wgetA1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    228         *.表格1A1 { background-color:#e6e6ff; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-width:0.0133cm; border-top-style:solid; border-top-color:#000000; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    229         *.表格1A2 { background-color:#e6e6ff; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    230         *.表格1B1 { background-color:transparent; padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    231         *.表格1B2 { background-color:transparent; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-width:0.0133cm; border-right-style:solid; border-right-color:#000000; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    232         *.表格10A1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    233         *.表格2A1 { background-color:#e6e6e6; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-width:0.0133cm; border-top-style:solid; border-top-color:#000000; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    234         *.表格2A2 { background-color:#e6e6e6; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    235         *.表格2B1 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-width:0.0133cm; border-top-style:solid; border-top-color:#000000; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    236         *.表格2B2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    237         *.表格2K1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    238         *.表格2K2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-width:0.0133cm; border-right-style:solid; border-right-color:#000000; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    239         *.表格3A1 { background-color:#ffff99; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-width:0.0133cm; border-top-style:solid; border-top-color:#000000; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    240         *.表格3A2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    241         *.表格3B1 { background-color:#ffff99; padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    242         *.表格3B2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-width:0.0133cm; border-right-style:solid; border-right-color:#000000; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    243         *.表格4A1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    244         *.表格5A1 { background-color:#ffff99; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-width:0.0133cm; border-top-style:solid; border-top-color:#000000; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    245         *.表格5A2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    246         *.表格5B1 { background-color:#ffff99; padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    247         *.表格5B2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-width:0.0133cm; border-right-style:solid; border-right-color:#000000; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    248         *.表格6A1 { background-color:#ffff99; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-width:0.0133cm; border-top-style:solid; border-top-color:#000000; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    249         *.表格6A2 { background-color:#ffff99; padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-style:none; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    250         *.表格6B1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    251         *.表格6B2 { padding:0.097cm; border-left-width:0.0133cm; border-left-style:solid; border-left-color:#000000; border-right-width:0.0133cm; border-right-style:solid; border-right-color:#000000; border-top-style:none; border-bottom-width:0.0133cm; border-bottom-style:solid; border-bottom-color:#000000; }
    252         *.表格7A1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    253         *.表格8A1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    254         *.表格9A1 { padding:0.097cm; border-width:0.0133cm; border-style:solid; border-color:#000000; }
    255         *.confA { width:16.999cm; }
    256         *.drblA { width:16.999cm; }
    257         *.initA { width:17.014cm; }
    258         *.packA { width:16.999cm; }
    259         *.rebootA { width:16.999cm; }
    260         *.wgetA { width:16.999cm; }
    261         *.表格1A { width:3.362cm; }
    262         *.表格1B { width:10.026cm; }
    263         *.表格10A { width:16.999cm; }
    264         *.表格2A { width:2.302cm; }
    265         *.表格2B { width:1.468cm; }
    266         *.表格3A { width:8.498cm; }
    267         *.表格3B { width:8.5cm; }
    268         *.表格4A { width:16.999cm; }
    269         *.表格5A { width:8.994cm; }
    270         *.表格5B { width:8.005cm; }
    271         *.表格6A { width:3.521cm; }
    272         *.表格6B { width:13.478cm; }
    273         *.表格7A { width:16.999cm; }
    274         *.表格8A { width:16.999cm; }
    275         *.表格9A { width:16.999cm; }
    276         *.BulletSymbols { font-family:StarSymbol; font-size:9pt; }
    277         *.Internetlink { color:#000080; text-decoration:underline; }
    278         *.NumberingSymbols { }
    279         *.T1 { }
    280         *.T10 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; }
    281         *.T11 { font-size:12pt; font-style:normal; font-weight:normal; }
    282         *.T12 { font-size:12pt; font-style:normal; font-weight:normal; }
    283         *.T13 { font-family:'Times New Roman'; }
    284         *.T14 { font-family:'Times New Roman'; }
    285         *.T15 { font-family:'Times New Roman'; font-size:12pt; }
    286         *.T16 { font-family:'Times New Roman'; font-size:12pt; font-weight:normal; }
    287         *.T17 { font-family:'Times New Roman'; font-size:12pt; font-weight:normal; }
    288         *.T18 { font-family:'Times New Roman'; font-size:12pt; font-weight:normal; }
    289         *.T19 { font-family:'Times New Roman'; font-size:12pt; font-style:italic; }
    290         *.T2 { }
    291         *.T20 { font-family:'Times New Roman'; font-size:12pt; font-style:italic; background-color:transparent; }
    292         *.T21 { font-family:'Times New Roman'; font-size:12pt; font-style:normal; }
    293         *.T22 { font-family:'Times New Roman'; font-style:italic; }
    294         *.T23 { font-style:italic; }
    295         *.T24 { font-style:italic; }
    296         *.T25 { font-style:italic; }
    297         *.T26 { font-style:italic; font-weight:normal; }
    298         *.T27 { font-style:italic; }
    299         *.T28 { color:#ff0000; font-family:'Times New Roman'; }
    300         *.T29 { }
    301         *.T3 { }
    302         *.T30 { }
    303         *.T31 { color:#cc0000; }
    304         *.T32 { background-color:transparent; }
    305         *.T4 { font-weight:bold; }
    306         *.T5 { color:#000000; }
    307         *.T6 { color:#000000; }
    308         *.T7 { color:#000000; font-family:'Times New Roman'; }
    309         *.T8 { color:#000000; font-family:'Times New Roman'; font-style:italic; }
    310         *.T9 { color:#000000; font-family:'Times New Roman'; font-size:12pt; font-style:italic; }
    311         *.VisitedInternetLink { color:#800000; text-decoration:underline; }
    312         </style></head><body dir="ltr"><p class="fr1"><img width="774" height="607" alt="" src="file:///home/rock/%E6%A1%8C%E9%9D%A2/" class="fr1" /></p><p class="P13"> </p><p class="P13"> </p><p class="P13"> </p><p class="P13"> </p><p class="P13"> </p><p class="P13">於DRBL環境建構擬SMP之 </p><p class="P13">Cluster計算平台 </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"><span class="T1">格網技術組        </span>郭文傑</p><p class="P1"><a href="mailto:rock@nchc.org.tw"><span class="T13">rock@nchc.org.tw</span></a></p><p class="P16"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18" /><p class="P18"> </p><p class="P18"> </p><p class="P14"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15">大綱 </p><table border="0" cellspacing="0" cellpadding="0" class="P68"><colgroup><col style="width: 16.999cm" /></colgroup><tr><td>一、前言</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P68"><colgroup><col style="width: 16.999cm" /></colgroup><tr><td>二、文獻探討</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>2.1 DRBL</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>2.2 Kerrighed</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P68"><colgroup><col style="width: 16.999cm" /></colgroup><tr><td>三、測試結果</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>3.1 DRBL merge Kerrighed</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>3.2 Kerrighed 測試結果</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P68"><colgroup><col style="width: 16.999cm" /></colgroup><tr><td>四、結論</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P68"><colgroup><col style="width: 16.999cm" /></colgroup><tr><td>五、參考文獻</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>英文文獻</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>網站資料</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P68"><colgroup><col style="width: 16.999cm" /></colgroup><tr><td>六、附錄</td></tr></table><table border="0" cellspacing="0" cellpadding="0" class="P69"><colgroup><col style="width: 16.499cm" /></colgroup><tr><td>DRBL Merge Kerrighed</td></tr></table><p class="P18"> </p><p class="P18"> </p><p class="P15"> </p><p class="P20"> </p><p class="P10"> </p><p class="P10"> </p><p class="P10"> </p><p class="P10"> </p><p class="P10"> </p><p class="P10"> </p><p class="P15"> </p><p class="P8"> </p><p class="P8"> </p><p class="P18"> </p><p class="P18"> </p><p class="P15"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18" /><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><ol style="margin-left:0.5cm;list-style-type:; "><li class="P166" style="margin-left:0cm;"><h1 class="P166"><a name="_E5_89_8D_E8_A8_80" />前言</h1></li></ol><p class="P18">    高效能計算一直是時代所趨，時代的演進代表<span class="T1">龐大資料的不斷</span>累積，人們面臨<span class="T1">越趨越複雜、規模遇龐大的</span>的問題，<span class="T1">舊時代</span>的計算資源<span class="T1">已</span>不敷使用，需要的是<span class="T1">更</span>強大且穩定的計算資源，這也是<span class="T1">高效能計算</span>領域不斷面對的<span class="T1">挑戰</span>。</p><p class="P18"> </p><p class="P18"> <span class="T1">   各國的研發單位也不斷的找尋適當的方法來整合計算資源，進而提供服務，有的走大型主機方式、有的為了降低成本走叢集方式，在此環境下也激盪出不同的Cluster Middleware和Cluster Software，簡單來說，Middleware可以提供一些基本且必要的功能來讓Cluster順利運作，Middleware通常包含了以下四種功能：</span></p><ul style="margin-left:1.25cm;"><li class="P70" style="margin-left:0cm;"><p class="P70" style="margin-left:0.25cm;">工作管理系統 </p></li><li class="P70" style="margin-left:0cm;"><p class="P70" style="margin-left:0.25cm;">Cluster<span class="T1">監視系統</span></p></li><li class="P70" style="margin-left:0cm;"><p class="P70" style="margin-left:0.25cm;">平行執行函式庫 </p></li><li class="P70" style="margin-left:0cm;"><p class="P70" style="margin-left:0.25cm;">C<span class="T1">luster管理工具</span></p></li><li class="P70" style="margin-left:0cm;"><p class="P70" style="margin-left:0.25cm;">全域行程空間 </p></li></ul><p class="P18"> <span class="T1">   </span></p><p class="P17"><span class="T1">    本專案考量整合性和效能方面，並考慮能與DRBL的管理工具無縫結合，考量的各家Open Source的SSI(Single System Image) Middleware，選擇Open Source的Middleware方案是因可從程式碼中改善或是客製化，最後本專案採用Kerrighed來建置計算的平台，Kerrighed是一套SSI的Cluster Middleware，它的運作哲學是將一群Cluster視為一台SMP，在運作過程中不需使用太多繁瑣的使用者工具或是函式庫來協助平行，工作於底層便由Kerrighed patched過的Kernel來自動化評行，可以達到即下指令即可Real-Time的平行運作，從官方的效能評估報告來看(可參考“</span><span class="T11">OpenMosix, OpenSSI and Kerrighed: </span><span class="T12">A Comparative </span><span class="T11">Study”測試報告</span><span class="T1">)，其效能優於Open MOSIX、Open SSI，Kerrighed展現的高效能表現令人驚艷。但Kerrighed的安裝過程稍嫌煩瑣，所以本研究採用DRBL來協助Kerrighed的安裝流程，此外DRBL提供相當多的管理工具來協助管理Client。</span></p><p class="P4"> </p><p class="P4"> <span class="T1">   </span><span class="T6">專案的最終目的是希望提供一個穩定的Cluster計算平台，卻不會因穩定而喪失優越的效能表現，在使用的簡易性亦可讓使用者簡單方便使用，以Condor來說使用者必須撰寫</span><span class="T6">描述檔，才能讓Condor的排程器來平行程式，以MPI函式庫來說，編譯程式時必須配合函式庫，或是程式設計師在撰寫程式時就指定平行的部分程式碼，而對使用者來說Kerrighed提供的計算平台，可以省卻這些繁瑣步驟，且Kerrighed在平行過程中是由較底層的Kernel來分配，相較於這些軟體是從較高階使用者層級或是程式層級來平行的，Kerrighed效能表現會相當亮眼。</span></p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><p class="P15"> </p><ol style="margin-left:0.5cm;list-style-type:; "><li class="P168" style="margin-left:0cm;"><h1 class="P168"><a name="_E6_96_87_E7_8D_BB_E6_8E_A2_E8_A8_8E" />文獻探討</h1></li></ol><h2 class="P65"><a name="2.1_DRBL" />2.1 DRBL</h2><p class="P18">    DRBL<span class="T1">全名為Diskless Remote Boot in Linux，能透過DRBL Server來設置一套無碟的系統環境，利用遠端Server提供Client開機所需的檔案(kernel、initrd image)，這些檔案文件是透過NFS分享給Client使用，Client本身使用到的只有其自身硬體資源，像是CPU、記憶體、網卡、顯示器等，Client在運作時使用和產生的檔案文件皆存於Server的硬碟，所以Client可以不需有硬碟，應用於Cluster的環境下，可以讓使用者快速的佈署好所需的計算環境，只需將軟體安裝於Server，整個環境內的Client都可使用到此軟體的功能，另外DRBL也提供管理Cluster的指令。</span></p><p class="P18"> </p><p class="P18"> <span class="T1">   Server可根據安裝的Linux Distribution於DRBL下載安裝合適的rpm或deb檔，安裝過程可以根據其訊息提示來設定環境，DRBL提供三種模式：</span></p><ul style="margin-left:1.25cm;"><li class="P75" style="margin-left:0cm;"><p class="P75" style="margin-left:0.25cm;"><span class="T4">Full DRBL：</span>Client都有自己的/etc和/var，存於透過NFS存於Server端。</p></li><li class="P75" style="margin-left:0cm;"><p class="P75" style="margin-left:0.25cm;"><span class="T4">DRBL SSI：</span>Client都有各自的/etc和/var，但是是存於本機的tmpfs，當一旦關機或電源關閉，資料夾內的文件也會消失。</p></li><li class="P83" style="margin-left:0cm;"><p class="P83" style="margin-left:0.25cm;">不提供<span class="T1">D</span>iskless模式</p></li></ul><p class="P18">   </p><p class="P18">    <span class="T1">安裝完成後，Client只需於BIOS設定成PXE網路開機即可使用，DRBL提供相當豐富的Client管理工具，像是從Server遠端重新啟動多台Client、設定Client開機後為圖形或文字介面、Client Wake On Lan和Clonezilla等功能。</span></p><p class="P18"> </p><h2 class="P65"><a name="2.2_Kerrighed" />2.2 Kerrighed</h2><p class="P18">    <span class="T1">Kerrighed為一SSI(Single Ststem Image)的Cluster作業系統，Kerrighed將一群由多台PC構成的Cluster視為一台SMP。Kerrighed的最終目標希望能讓使用者能更簡單的使用Cluster、促進高效能的Cluster應用、Cluster的高可用性、資源的有效管理和Cluster作業系統的高度客製化。</span></p><p class="P18"> </p><p class="P18"> <span class="T1">   Kerrighed是以Linux為基礎去擴展的實現，包含了修改和擴充部分的模組(Module)和Patch過的核心(kernel)。Kerrighed主要的特色有：</span></p><ul style="margin-left:1.25cm;"><li class="P77" style="margin-left:0cm;"><p class="P77" style="margin-left:0.25cm;">Cluster <span class="T1">廣域的行程管理</span></p></li><li class="P77" style="margin-left:0cm;"><p class="P77" style="margin-left:0.25cm;"><span class="T1">支援</span>Cluster廣域的共享記憶體</p></li><li class="P77" style="margin-left:0cm;"><p class="P77" style="margin-left:0.25cm;">Cluster<span class="T1">檔案系統</span></p></li><li class="P77" style="margin-left:0cm;"><p class="P77" style="margin-left:0.25cm;">透明化的行程Checkpoint </p></li><li class="P77" style="margin-left:0cm;"><p class="P77" style="margin-left:0.25cm;"><span class="T1">提供</span>高可用性<span class="T1">的環境給使用者的應用行程</span></p></li><li class="P77" style="margin-left:0cm;"><p class="P77" style="margin-left:0.25cm;">客製的<span class="T1">單一系統印象之特色</span></p></li></ul><p class="P18">        </p><p class="P18">    Kerrighed的架構主要可以分為14個模組，可參考圖一所示，最底層的RPC和Hotplug模組主要是做為Kerrighed的溝通子系統，中間KDDM模組是一個實現Kerrighed分散服務的重要機制。上層的其它模組則實現了其它Kerrighed服務，下面一段落有各模組的詳細介紹：</p><p class="P18"> </p><div class="P18"><img width="694" height="280" alt="" src="file:///home/rock/%E6%A1%8C%E9%9D%A2/" class="fr2" /></div><p class="P19">圖2.1 <span class="T1">Kerrighed架構</span></p><p class="P4"> </p><ul style="margin-left:1.25cm;"><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">R<span class="T1">PC</span></p><p class="P103" style="margin-left:0.25cm;">R<span class="T1">PC模組實現高層級的分散服務管理，它提供功能讓在RPC機制底下如何去找到另一個機制，許多明確的溝通是透過此介面來實現。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Hotplug </p><p class="P103" style="margin-left:0.25cm;">此Hotplug模組負責如何追蹤node的增加(<span class="T1">這邊的node是指Kerrighed client)、移除和偵測錯誤情況，此外，它也管理Cluster的重新設定。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">KDDM </p><p class="P103" style="margin-left:0.25cm;">此K<span class="T1">errighed Distributed Data Manager(KDDM) 機制是一個分散資料的管理，它提供</span><span class="T1">高層級的機制去讓存取遠端資料更簡單更有效率。許多的Kerrighed服務以於此容器(Container)上層去實現。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Capability </p><p class="P103" style="margin-left:0.25cm;">Kerrighed Capability<span class="T1">讓每個行程在Kerrighed系統下能更細微的調整，並且讓Kerrighed系統能允許每個行程的Capability。它可以賦予和取消一些Kerrighed Capability在每個行程，而這些行程準則可在一個正在執行的系統中動態修改。Kerrighed將行程分為四個Capability sets：Effective、Permitted、Inheritable Permited、Inheritable Effective。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">ProcFS </p><p class="P103" style="margin-left:0.25cm;">此ProcFS模組實現了全域的/proc<span class="T1">目錄，他提供和linux系統正規的檔案，不同的是這些資料描繪整個Cluster狀況，包含了全域的記憶體使用率、CPU個數和全部正在執行的行程。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Sync </p><p class="P103" style="margin-left:0.25cm;">此Sync模組實現了分散式同步機制<span class="T1">，像是locks、Semaphores或是condition變數。它被使用來食做使用者層級的同步化(IPC semaphores、POSIX thread 同步化等)</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">IPC </p><p class="P103" style="margin-left:0.25cm;">此IPC模組實做分散式版本的IPC機制。它依靠mm<span class="T1">模組來分享記憶體segments和sync模組來使用semaphores。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">FS </p><p class="P103" style="margin-left:0.25cm;">此FS模組支援開放檔案的migrate<span class="T1">和分享檔案的pointer給在不同node執行的行程。之後此模組會實現針對分散式架構的Cluster檔案管理。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">MM </p><p class="P103" style="margin-left:0.25cm;">此<span class="T1">mm</span>模組負責<span class="T1">行程記憶體空間的migrate和Cluster廣域記憶體分享。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Dynamic Streams </p><p class="P103" style="margin-left:0.25cm;">此<span class="T1">Dynamic Streams模組實做有效migrate各種開放的串流，像是pipe、sockets、char裝置等。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Ghost </p><p class="P103" style="margin-left:0.25cm;">此Ghost模組<span class="T1">使用來輸出和輸入行程的核心meta-data，它通常被使用來migrate、checkpoint和duplicate行程。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Proc </p><p class="P103" style="margin-left:0.25cm;">此proc模組<span class="T1">實做了分散式的行程管理，他負責全域的行程命名和distant signaling等。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">EPM </p><p class="P103" style="margin-left:0.25cm;">此En<span class="T1">hanced Process Management模組實現了行程的migrate、行程的checkpoint、distant fork和分散式thread管理。</span></p></li><li class="P89" style="margin-left:0cm;"><p class="P89" style="margin-left:0.25cm;">Global Scheduler </p><p class="P103" style="margin-left:0.25cm;">此模組實做了不同<span class="T1">的</span>Kerrighed <span class="T1">全域行程排程策略。</span></p></li></ul><p class="P4"> </p><p class="P6"> <span class="T1">   Kerrighed目前釋出的最新版本為Kerrighed 2.2.0，配合使用的核心版本為2.6.20，在此一新版本他增加的Kerrighed支援SMP、多核心的功能和記憶體64bit定址的功能，雖然後者還在實驗階段，但官方說明目前可以正常使用，相較於先前的版本此版本修正許多bug，相對的穩定性也提高許多。</span></p><p class="P6"> </p><p class="P6"> <span class="T1">   Kerrighed的安裝和使用可以參考本文件的附錄，它的安裝哲學是透過一台安裝Kerrighed的Server將其印像檔透過NFS分享給Client使用，Client使用的核心是Kerrighed path過的核心，全部的client開機完成後，可以啟動Kerrighed服務(存放於/etc/init.d/kerrighed)，hotplug模組和rpc模組可以偵測到此網域內有多少台Kerrighed-enabled的機器被開啟，可以透過指令krgadm來將全部的機器組成Cluster。</span></p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P2"> </p><ol style="margin-left:0.5cm;list-style-type:; "><li class="P168" style="margin-left:0cm;"><h1 class="P168"><a name="_E6_B8_AC_E8_A9_A6_E7_B5_90_E6_9E_9C" />測試結果</h1></li></ol><p class="P18"> </p><p class="P4">    <span class="T6">本章節描述系統的安裝步驟和測試結果。有趣的是DRBL可以提供Full DRBL模式，讓個別client可以有各自的組態檔，安裝順序為先安裝Kerrighed在安裝DRBL，再讓DRBL將已安裝Kerrighed的印象分配給Client，使用DRBL可以減省安裝時間，它簡化了Kerrighed安裝過程中的NFSROOT階段，也就是將安裝DHCP server、TFTP server、NFS server步驟及設定檔部分透過一個DRBL的指令完成。</span></p><p class="P6"> </p><p class="P6"><span class="T1">    本章內文</span>可以分為兩個階段，一個是透過<span class="T1">DRBL協助Kerrighed的安裝部分，第二部分則為Kerrighed的效能測試結果。</span></p><h2 class="P65"><a name="3.1_DRBL_merge_Kerrighed" />3.1 DRBL merge Kerrighed</h2><p class="P3"><span class="T7">    以Cluster安裝和使用的角度來看，DRBL是一套可以協助Cluster達成管理和快速安裝Cluster軟體的工具，只需將Cluster安裝於DRBL server，其它的Client只需設定成網路開機的模式，即可開機完成使用，節省了Client安裝OS和軟體的時間成本，像著名的Cluster排成軟體Condor亦能與DRBL搭配來使用，方法也雷同安裝Kerrighed的方法，不同點在於Condor Client必須修改各自的組態檔來指定job 輸入和管理分配的機器為何，此處不對Condor做過多的解釋，有興趣可以參考</span><a href="http://www.linux.com/articles/49654"><span class="T13">http://www.linux.com/articles/49654</span></a><span class="T7">。</span></p><p class="P6"> </p><p class="P6">    以Kerrighed的安裝來看，<span class="T1">在</span>Kerrighed的安裝過程中，可以<span class="T1">透過</span>DRBL來完成SSI的環境安裝和設定，詳細的安裝<span class="T1">指令和</span>文件可以參考附錄。</p><h2 class="P66"><a name="3.2_Kerrighed__E6_B8_AC_E8_A9_A6_E7_B5_90_E6_9E_9C" />3.2 Kerrighed 測試結果</h2><p class="P6">    測試的每台機器規格皆為一致，每台Kerrighed Client都配有4核心的中央處理器，和1G的記憶體，擁有giga byte等級的網卡，彼此間透過高速的交換器連接起來，下表為機器的硬體和軟體規格表。</p><p class="P6"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格1"><colgroup><col width="147" /><col width="438" /></colgroup><tr><td style="text-align:left;width:3.362cm; " class="表格1A1"><p class="P21">C<span class="T1">PU</span></p></td><td style="text-align:left;width:10.026cm; " class="表格1B1"><p class="P22">Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz</p></td></tr><tr><td style="text-align:left;width:3.362cm; " class="表格1A2"><p class="P21">Memory </p></td><td style="text-align:left;width:10.026cm; " class="表格1B2"><p class="P23">2GB DDR2 667  </p></td></tr><tr><td style="text-align:left;width:3.362cm; " class="表格1A2"><p class="P21">NIC </p></td><td style="text-align:left;width:10.026cm; " class="表格1B2"><p class="P23">Intel 82566DM 10/100/1000 Mbps  </p></td></tr><tr><td style="text-align:left;width:3.362cm; " class="表格1A2"><p class="P21">Switch </p></td><td style="text-align:left;width:10.026cm; " class="表格1B2"><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P172" style="margin-left:0cm;"><p class="P172" style="margin-left:0.25cm;">24埠10/100/1000Base-T 高速乙太網路交換器  </p></li></ol></td></tr></table></div><p class="P7">表3.1 測試機器規格 </p><p class="P7"> </p><p class="P6"> <span class="T1">   本實驗測試部分，分為兩個測試方法，第一個主要是測試Kerrighed的穩定性和效能，使用三台Kerrighed Client組成的Cluster來build Kernel，透過make -j指令來於編譯過程中產生幾個行程，從一開始的1個行程逐漸增加到13個行程。</span></p><ol style="margin-left:0.5cm;list-style-type:decimal; "><li class="P125" style="margin-left:0cm;"><p class="P125" style="margin-left:0.25cm;"> </p></li></ol><p class="P4"> <span class="T1">   </span><span class="T6">本實驗的目的，是希望測試Kerrighed的穩定性，和穩定下的效能表現，因先前的Kerrighed的版本像有許多bug，雖然可提供較高的效能，但卻常常會導致系統的不穩定和核心的錯誤，實驗一使用的Kerrighed Cluster的總體資源為12顆中央處理器和3G的記憶體，實驗的過程主要是透過編譯核心來測試效能，詳細的步驟為(1)Client的Kerrigthed模組載入、(2)確認Cluster的資源訊息是否符合目前狀態、(3)設定Kerrighed行程的能力、(4)開始編譯</span></p><p class="P6">核心。本實驗使用的核心選項接採用預設的make defconfig，在每次完成後，會編譯一些不相關的程式，來避免記憶體或Cache已存有相關編譯核心的資料。 </p><ul style="margin-left:1.25cm;"><li class="P117" style="margin-left:0cm;"><p class="P117" style="margin-left:0.25cm;"> </p></li></ul><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P126" style="margin-left:0cm;"><p class="P126" style="margin-left:0.25cm;">Client載入Kerrighed 模組 </p></li></ol><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P118" style="margin-left:0cm;"><p class="P118" style="margin-left:0.25cm;">此步驟透過<span class="T1">啟動Kerrighed的服務，只需將存於/etc/init.d/kerrighed的服務搞啟動及可，啟動後便將第二章介紹的各個服務模組載入，載入的訊息可以透過dmesg指令來查看。</span></p></li></ol><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P126" style="margin-left:0cm;"><p class="P126" style="margin-left:0.25cm;">確認Kerrighed Cluster的資源 </p></li></ol><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P118" style="margin-left:0cm;"><p class="P118" style="margin-left:0.25cm;"><span class="T1">載入後，可以透過krgadm指定來將載入Kerrighed模組的Client結合成Cluster，</span><span class="T7">結合後，亦可透過krgadmk的相關指令來查看有哪幾台client在此Cluster，細節的資源可以透過top指令來查看。下圖為結合6台的Cluster的top指令顯示訊息，共有24顆中央處理器(每個Client的CPU為4核心架構)，和12GB的記憶體。</span></p><p class="P130" style="margin-left:0.25cm;">圖3.1 Cluster的top指令顯示資源訊息 </p></li></ol><p class="P12"> </p><p class="P12"> </p><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P128" style="margin-left:0cm;"><p class="P128" style="margin-left:0.25cm;">設定Kerrighed行程的能力 </p><p class="P131" style="margin-left:0.25cm;">將目前使用的<span class="T1">行程設置為具有CAN_MIGRATE的能力，可以透過指令krgcapset指令來設定。</span></p><p class="P131" style="margin-left:0.25cm;"> </p></li></ol><div class="P9"><img width="639" height="272" alt="" src="file:///home/rock/%E6%A1%8C%E9%9D%A2/" class="fr3" />圖3.2 krgcapset賦予<span class="T1">行程能力</span></div><p class="P9" /><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P126" style="margin-left:0cm;"><p class="P126" style="margin-left:0.25cm;">開始測試(編譯核心) </p><p class="P133" style="margin-left:0.25cm;"><span class="T7">實驗用來測試的核心版本為2.6.23，此</span><span class="T7">核心</span><span class="T7">版本是從The Linux Kernel Archives (</span><a href="http://www.kernel.org/">http://www.kernel.org/</a><span class="T7">)下載的，透過make的-j來指定可以產生幾個行程，依序編譯的行程個數從1個行程逐漸增加到32個，每次都增加四個行程，在編譯過程中可以透過dmesg來查</span><span class="T7">看Kerrihghed排程的情況。圖3.3為編譯核心時透過dmesg指令所顯示的行程分派訊息，圖3.4為實驗結果。</span></p><p class="P120" style="margin-left:0.25cm;"> </p></li></ol><div class="P7"><img width="635" height="349" alt="" src="file:///home/rock/%E6%A1%8C%E9%9D%A2/" class="fr3" />圖3.3 <span class="T1">dmesg指令顯示的訊息</span></div><div class="P7"><img width="594" height="420" alt="" src="file:///home/rock/%E6%A1%8C%E9%9D%A2/" class="fr3" />圖3.4 <span class="T1">實驗結果</span></div><p class="P5"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格2"><colgroup><col width="101" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /><col width="64" /></colgroup><tr><td style="text-align:left;width:2.302cm; " class="表格2A1"><p class="P24">行程<span class="T1">個</span>數</p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">1 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">4 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">8 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">12 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">16 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">20 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">24 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">25 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B1"><p class="P24">28 </p></td><td style="text-align:left;width:1.468cm; " class="表格2K1"><p class="P24">32 </p></td></tr><tr><td style="text-align:left;width:2.302cm; " class="表格2A2"><p class="P24">完成時間 </p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P24">8m50s </p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P25">3m34<span class="T1">s</span></p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P25">2m3<span class="T1">2s</span></p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P25">2m5<span class="T1">s</span></p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P24">2m2s </p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P24">1m59s </p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P25">1m5<span class="T1">4s</span></p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P25">1m51<span class="T1">s</span></p></td><td style="text-align:left;width:1.468cm; " class="表格2B2"><p class="P25">1m57<span class="T1">s</span></p></td><td style="text-align:left;width:1.468cm; " class="表格2K2"><p class="P25">1m5<span class="T1">8s</span></p></td></tr></table></div><p class="P7">表3.2 核心編譯<span class="T1">完成時間</span></p><p class="P5"> </p><p class="P6"><span class="T28">    </span><span class="T7">當行程為1個時，核心編譯完成時間約為8分50秒，當行程增加為4個時，完成時間縮短為3分34秒，當行程增加到8個時，完成時間降到2分31秒，當行程增加到12個時，完成時間則為2分5秒，一直到最後的行程增加為32個時，完成時間約為1分58秒。</span>可以明顯的看出效能提升的趨勢，隨著行程個數的增加，編譯核心所花的時間越來越短，到最<span class="T1">快</span>的只需1<span class="T1">分51秒</span>便能完成<span class="T1">，從8分50秒到1分51秒，替使用者節省了約7分鐘的時</span><span class="T1">間成本</span>。</p><p class="P6"> </p><p class="P6">    但是並非行程的個數一直增加便能無窮的減少完成時間，可以從圖3.4最後的走勢看到，本案例最後完成時間收斂在約1分51秒到1分58秒左右。收斂的部分是因為程式有些部份是無法平行的，簡單來說效能瓶頸便是卡在這些必須循序執行的部分程式。另一個有趣的議題是當行程增加到28時，執行時間不減反增，因為本實驗的Cluster只有24顆中央處理器，過多的行程，無法有效分配到閒置的中央處理器來執行，必須等待上一個進入中央處理器的行程完成後，多餘的行程才能擁有中央處理器的執行權，另一個拖慢效能的原因在於行程間的溝通上，行程間的資料為了確保一致性，過多的行程需要花費的溝通成本相對的也會提升。</p><p class="P6"> </p><p class="P6"><span class="T7">     本案例特別有跑一個行程數目為25的實驗，因為</span>有個經驗法<span class="T1">則為：在</span>考慮到效能和成本下，最佳的行程效能為<span class="T1">所有</span>中央處理器數目<span class="T1">和</span>加1，所以以此案例來說，當行程為25個時效能為1<span class="T1">分51秒，剛好是全部實驗中花費最短的時間將核心編譯完成的，也應證了此經驗法則</span>。</p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><p class="P4"> </p><ol style="margin-left:0.5cm;list-style-type:; "><li class="P166" style="margin-left:0cm;"><h1 class="P166"><a name="_E7_B5_90_E8_AB_96" />結論</h1></li></ol><p class="P18"> </p><p class="P18">    選擇一個<span class="T1">具</span>高效能且穩定的Cluster計算平台是<span class="T1">一項有趣的挑戰</span>，<span class="T1">許多Cluster Middleware為了穩定性卻犧牲了效能，運算過程中加入了許多的確認和除錯的機制，導致大部分的時間都花費於此，確實計算平台的穩定性是非常重要的，但效能亦不可偏廢，兩者看似相對，但其實卻可取得平衡，Kerrighed的定位本就在於以高效能著稱，但先前的版本，系統測試結果是非常不穩定的，雖可獲得高效能，但卻常導致系統的不穩定和核心死當，而現今新版的Kerrighed2.2.0，大大的提高了穩定性，在此穩定的基底下還能有高效能的表現，也是本專案拿來做研究的對象。</span></p><p class="P18"> </p><p class="P18">    採用Kerrighed的優點<span class="T1">尚有</span>，使用者可以不需改寫<span class="T1">程式碼</span>，<span class="T1">也</span>不需配合一些函式庫或是軟體來協助平行，Kerrighed核心於底層便將工作<span class="T1">平行</span>分配完成，預設的工作排程演算法是採用負載平衡，當有CPU的使用度較低或是無使用時，Kerrighed核心便將任務分配給此顆<span class="T1">負載較低的</span>CPU。此點優勢可以提升使用者的使用<span class="T1">上的簡易</span>性，因非所有需要高效能運算的使用者，都受過平行分工的訓練，也就是在撰寫程式時，會採用函式庫或使用者自定來平行程式，此點可以降低使用者在使用和執行Cluster時的使用複雜性。</p><p class="P4"> </p><p class="P4"><span class="T1">    </span><span class="T6">在未來可以預見的是，採用DRBL配合Kerrighed的使用會逐漸增加，一來可以簡化安裝步驟，二來可以使用DRBL指令來協助管理機器，另一方面Kerrighed平台的成熟，會有更多的應用產生，而Kerrighed的官網也指出，在未來Kerrighed將支援到thread的平行、客製化的排程策略，一致的Cluster檔案系統命名空間等。</span></p><p class="P4"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><p class="P18"> </p><ol style="margin-left:0.5cm;list-style-type:; "><li class="P168" style="margin-left:0cm;"><h1 class="P168"><a name="_E5_8F_83_E8_80_83_E6_96_87_E7_8D_BB" />參考文獻</h1></li></ol><h2 class="P64"><a name="_E8_8B_B1_E6_96_87_E6_96_87_E7_8D_BB" />英文文獻</h2><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P84" style="margin-left:0cm;"><p class="P84" style="margin-left:0.25cm;"><span class="T30">Emir I. and Damir D. Z., </span><span class="T24">Cluster Distributions Review</span><span class="T30">, 2004</span></p></li><li class="P85" style="margin-left:0cm;"><p class="P85" style="margin-left:0.25cm;"><span class="T2">Renaud L., Benoit B., Pascal G., Geoffroy V. and Christine </span><span class="T30">M.</span>, <span class="T26">OpenMosix, OpenSSI and Kerrighed: A Comparative </span><span class="T23">Study</span>, 2004 </p></li><li class="P85" style="margin-left:0cm;"><p class="P85" style="margin-left:0.25cm;"><span class="T1">David M., Geoffroy V., Renaud L.</span><span class="T16">, </span><span class="T18">C</span><span class="T16">hristine </span><span class="T18">M.</span><span class="T16">, </span><span class="T18">J</span><span class="T16">ean-yves </span><span class="T18">B.</span><span class="T17">,</span><span class="T15"> </span><span class="T19">Kerrighed: A SSI Cluster OS Running OpenMP</span><span class="T15">, 2003</span></p></li><li class="P85" style="margin-left:0cm;"><p class="P85" style="margin-left:0.25cm;"><span class="T3">Pascal G. and Christine M.</span><span class="T3">, </span><span class="T27">Dynamic Streams For Efficient C</span><span class="T25">ommunications </span><span class="T27">B</span><span class="T25">etween </span><span class="T27">M</span><span class="T25">igrating </span><span class="T27">P</span><span class="T25">rocesses </span><span class="T27">In A C</span><span class="T25">luster</span><span class="T29">, </span><span class="T3">2003</span></p><p class="P88" style="margin-left:0.25cm;"> </p></li></ol><h2 class="P64"><a name="_E7_B6_B2_E7_AB_99_E8_B3_87_E6_96_99" />網站資料</h2><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P121" style="margin-left:0cm;"><p class="P121" style="margin-left:0.25cm;"><span class="T14">Single-system image，取自</span><span class="T14">http://en.wikipedia.org/wiki/Single-system_image，2007</span></p></li><li class="P121" style="margin-left:0cm;"><p class="P121" style="margin-left:0.25cm;"><span class="T1">Kerrighed官網，取自</span>http://<span class="T5">www.kerrighed.org/wiki/index.php/Main_Page</span><span class="T6">，2007</span></p></li><li class="P121" style="margin-left:0cm;"><p class="P121" style="margin-left:0.25cm;">DRBL官網，<span class="T1">取自</span>http://drbl.nchc.org.tw/<span class="T1">，2007</span></p></li></ol><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><p class="P6"> </p><ol style="margin-left:0.5cm;list-style-type:; "><li class="P168" style="margin-left:0cm;"><h1 class="P168"><a name="_E9_99_84_E9_8C_84" />附錄</h1></li></ol><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11" /><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><p class="P11"> </p><h2 class="P67"><a name="DRBL_Merge_Kerrighed" />DRBL Merge Kerrighed</h2><p class="P37">rock, rock@nchc.org.tw </p><p class="P38">People need powerful computing resources to resolve complex problems. A important issue is how to easy build cheap and effective computing resource. We use DRBL and Kerighed to build SSI cluster. It quick to deploy your cluster environment. You just install them in one machine (Server), clients don't install any OS. Client only setup PXE (network boot) in BIOS, when client finish network booting that your cluster is OK! </p><p class="P38">We use below Software: </p><ul style="margin-left:1.25cm;"><li class="P134" style="margin-left:0cm;"><p class="P134" style="margin-left:0.25cm;">Ubuntu or Debian </p><p class="P142" style="margin-left:0.25cm;">Linux Distribution. </p></li><li class="P134" style="margin-left:0cm;"><p class="P134" style="margin-left:0.25cm;">DRBL </p><p class="P143" style="margin-left:0.25cm;">Diskless Remote Boot in Linux (DRBL) provides a diskless or systemless environment for client machines. It works on Debian, Ubuntu, Mandriva, Red Hat, Fedora, CentOS and SuSE. DRBL uses distributed hardware resources and makes it possible for clients to fully access local hardware. It also includes Clonezilla, a partitioning and disk cloning utility similar to Symantec Ghost.  </p><p class="P147" style="margin-left:0.25cm;"><a href="http://drbl.sourceforge.net/"><span class="T13">http://drbl.sourceforge.net/</span></a></p><p class="P143" style="margin-left:0.25cm;" /></li><li class="P134" style="margin-left:0cm;"><p class="P134" style="margin-left:0.25cm;">Kerrighed </p><p class="P143" style="margin-left:0.25cm;">Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs. The new version is Kerrighed 2.2.0, and it used for 2.6.20 kernel. </p><p class="P148" style="margin-left:0.25cm;"><a href="http://www.kerrighed.org/wiki/index.php/Main_Page"><span class="T21">http://www.kerrighed.org/wiki/index.php/Main_Page</span></a></p></li></ul><p class="P44"> </p><p class="P44"> </p><p class="P40">0.Install Kerrighed </p><ul style="margin-left:1.25cm;"><li class="P137" style="margin-left:0cm;"><p class="P137" style="margin-left:0.25cm;">OS Setup  </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格3"><colgroup><col width="371" /><col width="371" /></colgroup><tr><td style="text-align:left;width:8.498cm; " class="表格3A1"><p class="P26">Ubuntu </p></td><td style="text-align:left;width:8.5cm; " class="表格3B1"><p class="P26">Debian </p></td></tr><tr><td style="text-align:left;width:8.498cm; " class="表格3A2"><p class="P28">sudo mv /bin/sh /bin/sh.old </p><p class="P29">sudo ln -s /bin/bash /bin/sh </p><p class="P48"><span class="T31">(Ubuntu default sh is link to dash)</span> </p><p class="P44"> </p><p class="P59">sudo ln -s /bin/gcc-3.3 /bin/gcc  </p><p class="P50">(We use gcc-3.3 to compiler kernel) </p></td><td style="text-align:left;width:8.5cm; " class="表格3B2"><p class="P59">sudo ln -s /bin/gcc-3.3 /bin/gcc  </p></td></tr></table></div><p class="P39"> </p><ul style="margin-left:1.25cm;"><li class="P138" style="margin-left:0cm;"><p class="P138" style="margin-left:0.25cm;">Install Basic Package </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="pack"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="packA1"><p class="P47">sudo aptitude install gcc-3.3 automake autoconf libtool initramfs-tools make </p><p class="P47">sudo aptitude install kernel-package libncurses5-dev build-essential fakeroot wget bzip2  </p><p class="P47">sudo aptitude install xmlto lsb-release  </p><p class="P47">sudo aptitude install nfsbooted  </p></td></tr></table></div><p class="P39"> </p><ul style="margin-left:1.25cm;"><li class="P139" style="margin-left:0cm;"><p class="P139" style="margin-left:0.25cm;">Get Kerrighed </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="wget"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="wgetA1"><p class="P47">cd /usr/src </p><p class="P49"><span class="T20">sudo wget </span><a href="https://gforge.inria.fr/frs/download.php/3522/kerrighed-2.2.0.tar.gz"><span class="T20">https://gforge.inria.fr/frs/download.php/3522/kerrighed-2.2.0.tar.gz</span></a></p><p class="P49"><span class="T20">sudo wget </span><a href="http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2"><span class="T20">http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2</span></a><span class="T20"> </span></p><p class="P47"> </p><p class="P49"><span class="T20">sudo tar -zxvf </span><a href="http://gforge.inria.fr/frs/download.php/2163/kerrighed-2.1.0.tar.gz"><span class="T20">kerrighed-2.2.0.tar.gz</span></a></p><p class="P49"><span class="T20">sudo tar -jxvf </span><a href="http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2"><span class="T20">linux-2.6.20.tar.bz2</span></a><span class="T20"> </span></p></td></tr></table></div><p class="P35"> </p><ul style="margin-left:1.25cm;"><li class="P139" style="margin-left:0cm;"><p class="P139" style="margin-left:0.25cm;">Config and make kernel, modules </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="conf"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="confA1"><p class="P47">cd kerrighed-2.2.0  </p><p class="P47">sudo ./configure --with-kernel=/usr/src/linux-2.6.20 --disable-kernel-config --with-kernel-config=/usr/src/linux-2.6.20/.config  </p><p class="P47">sudo make patch  </p><ul style="margin-left:1.25cm;"><li class="P146" style="margin-left:0cm;"><p class="P146" style="margin-left:0.25cm;"> </p><p class="P163" style="margin-left:0.25cm;">(1)Choose kernel options </p></li></ul><p class="P47">cd ../linux-2.6.20  </p><p class="P47">sudo make defconfig </p><p class="P47">(sudo make menuconfig )</p><p class="P52">(0.If your CPU arch is intel, choose i386  </p><p class="P53"><span class="T32"> 1.Cancel Preemption, KEYS  and SMP</span> options </p><p class="P53"> 2.Enable netwok block device, plug and play, NIC Driver and boot from nfs must to build-in</p><p class="P54">) </p><p class="P44"> </p><ul style="margin-left:1.25cm;"><li class="P163" style="margin-left:-1.25cm;"><p class="P163" style="margin-left:0.25cm;">(2)Make </p></li><li class="P165" style="margin-left:-1.25cm;"><p class="P165" style="margin-left:0.25cm;">cd /usr/src/kerrighed-2.2.0 </p></li></ul><p class="P47">sudo make kernel  </p><p class="P47">sudo make  </p><p class="P47">sudo make kernel-install  </p><p class="P47">sudo make install  </p></td></tr></table></div><p class="P36"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Make initrd image </p><p class="P149" style="margin-left:0.25cm;"> </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="init"><colgroup><col width="744" /></colgroup><tr><td style="text-align:left;width:17.014cm; " class="initA1"><p class="P47">sudo mkinitramfs -o /boot/initrd.img-2.6.20-krg 2.6.20-krg </p><p class="P47">sudo vim /boot/grub/menu.lst  </p><p class="P47"> </p><p class="P50">(ex: </p><p class="P50">title           Kerrighed 2.6.20-krg</p><p class="P50">root            (hd0,0)</p><p class="P50">kernel          /boot/vmlinuz-2.6.20-krg root=UUID=8bc93e80-35f9-41db-a349-02e839f2ed7c ro quiet splash</p><p class="P50">initrd          /boot/initrd.img-2.6.20-krg</p><p class="P50">quiet </p><p class="P50">savedefault </p><p class="P50">) </p></td></tr></table></div><p class="P55"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Reboot and choose 2.6.20-krg kernel </p><p class="P149" style="margin-left:0.25cm;"> </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="reboot"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="rebootA1"><p class="P47">sudo reboot </p></td></tr></table></div><p class="P39"> </p><ul style="margin-left:1.25cm;"><li class="P139" style="margin-left:0cm;"><p class="P139" style="margin-left:0.25cm;">Edit nodes' information </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格4"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="表格4A1"><p class="P47">sudo vim /etc/kerrighed_nodes  </p><p class="P50">(ex: </p><p class="P51">session=99 </p><p class="P51">nbmin=8 </p><p class="P51">krg001:0:eth0 </p><p class="P51">krg002:1:eth0 </p><p class="P51">krg003:2:eth0 </p><p class="P51">krg004:3:eth0 </p><p class="P51">krg005:4:eth0 </p><p class="P51">krg006:5:eth0 </p><p class="P51">krg007:6:eth0 </p><p class="P51">krg008:7:eth0 </p><p class="P51">) </p></td></tr></table></div><p class="P46"> </p><p class="P46"> </p><p class="P40">1.Install DRBL </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Add deb source </p><p class="P149" style="margin-left:0.25cm;" /></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格5"><colgroup><col width="393" /><col width="350" /></colgroup><tr><td style="text-align:left;width:8.994cm; " class="表格5A1"><p class="P26">Ubuntu </p></td><td style="text-align:left;width:8.005cm; " class="表格5B1"><p class="P26">Debian </p></td></tr><tr><td style="text-align:left;width:8.994cm; " class="表格5A2"><p class="P47">sudo vim /etc/apt/sources.list  </p><p class="P31"> </p><p class="P33">(add : </p><p class="P33">deb http://free.nchc.org.tw/ubuntu feisty main restricted universe multiverse<br />deb http://free.nchc.org.tw/drbl-core drbl stable)</p><p class="P31"> </p><p class="P60">wget http://drbl.nchc.org.tw/GPG-KEY-DRBL sudo apt-key add GPG-KEY-DRBL  </p><p class="P60"> </p><p class="P60">sudo apt-get update </p></td><td style="text-align:left;width:8.005cm; " class="表格5B2"><p class="P28">sudo vim /etc/apt/sources.list  </p><p class="P31"> </p><p class="P33">(add: </p><p class="P33">deb http://free.nchc.org.tw/debian etch main<br />deb http://free.nchc.org.tw/drbl-core drbl stable)</p><p class="P33"> </p><p class="P30">wget http://drbl.nchc.org.tw/GPG-KEY-DRBL  </p><p class="P30">sudo apt-key add GPG-KEY-DRBL  </p><p class="P30"> </p><p class="P60">sudo apt-get update </p></td></tr></table></div><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;"> </p></li><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Install DRBL </p></li></ul><p class="P42"> </p><p class="P55">Before we install DRBL, we must clear plan our DRBL environment. The below layout is our environment, eth0 used to connect WAN, eth1 used for DRBL internal clients. </p><p class="P56"> </p><p class="P56">             NIC     NIC IP                              Clients </p><p class="P56">+-----------------------------+  </p><p class="P56">|     DRBL SERVER         | </p><p class="P56">|                                         | </p><p class="P56">|     +-- [eth0] 140.110.X.X  +- to WAN </p><p class="P56">|                                         | </p><p class="P56">|     +-- [eth1] 192.168.0.1 +- to clients group 1 [ 7 clients, their IP from 192.168.0.2 - 192.168.0.8] </p><p class="P56">|                                         |</p><p class="P57">+-----------------------------+  </p><p class="P42"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="drbl"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="drblA1"><p class="P47">sudo aptitude install drbl </p><p class="P50">(DRBL will be installed in directory /opt/drbl ) </p><p class="P47"> </p><p class="P47">sudo /opt/drbl/sbin/drblsrv -i  </p><p class="P50">(The command used interactive mothod help user to install. It install related packages (nfs, dhcp, tftp......) and create /tftpboot directory. The /tftpboot include: </p><p class="P50">nbi_img: kenrel , initrd image and grub menu </p><p class="P50">node_root: server directories copy </p><p class="P50">nodes: each nodes' individual directories </p><p class="P50">) </p><p class="P47"> </p><p class="P47">sudo /opt/drbl/sbin/drblpush -i  </p><p class="P50">(the command will deploy client environment, like client name, DRBL mode, swap ...) </p></td></tr></table></div><p class="P42"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Setup each node grub menu </p></li></ul><p class="P42"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格7"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="表格7A1"><p class="P47">cd /tftpboot/nbi_im/pxelinux.cfg </p><p class="P50">(named rule is IP's hexadecimal  </p><p class="P50">ex. 192.168.0.2  -&gt;  C0A80002)</p><p class="P44"> </p><p class="P47">cp default  C0A80002</p><p class="P47">vim  C0A80002</p><p class="P50">( add node_id in append line: </p><p class="P50">ex. </p><p class="P50">label drbl</p><p class="P50">  MENU DEFAULT</p><p class="P50">  # MENU HIDE</p><p class="P50">  MENU LABEL Ubuntu 7.04 Linux (DRBL mode, mostly local resources)</p><p class="P50">  # MENU PASSWD</p><p class="P50">  kernel vmlinuz-pxe</p><p class="P50">  append initrd=initrd-pxe.img devfs=nomount drblthincli=off selinux=0 <span class="T4">node_id=0</span></p><p class="P50">) </p><p class="P44"> </p></td></tr></table></div><p class="P42"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">When client booting, you maybe meet some problems </p></li></ul><p class="P42"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格6"><colgroup><col width="154" /><col width="589" /></colgroup><tr><td style="text-align:left;width:3.521cm; " class="表格6A1"><p class="P45">don't find nobody  </p></td><td style="text-align:left;width:13.478cm; " class="表格6B1"><p class="P47">sudo cp -f /etc/group /tftboot/node_root/group </p><p class="P47">sudo cp -f /etc/group /tftboot/nodes/{IP}/group </p><p class="P47"><br />sudo cp -f /etc/passwd /tftpboot/node_root/passwd </p><p class="P47">sudo cp -f /etc/group /tftboot/nodes/{IP}/passwd </p></td></tr><tr><td style="text-align:left;width:3.521cm; " class="表格6A2"><p class="P27">/var wait too long </p></td><td style="text-align:left;width:13.478cm; " class="表格6B2"><p class="P47">sudo vim /tftpboot/etc/node_root/mountnfs </p><p class="P47">sudo vim /tftpboot/etc/nodes/{IP}/mountnfs </p><p class="P50">(Timeout=900   -&gt;   Timeout=200)</p></td></tr><tr><td style="text-align:left;width:3.521cm; " class="表格6A2"><p class="P27">don't auto load kerrighed module </p></td><td style="text-align:left;width:13.478cm; " class="表格6B2"><p class="P28">sudo cp /etc/init.d/kerrighed krg </p><p class="P28">sudo vim krg </p><p class="P33">(insmod /usr/src/linux-2.6.20/kerrighed.ko </p><p class="P33">mount -a -t kerfs) </p><p class="P31"> </p><p class="P28">sudo cp krg /tftpboot/nodes/{IP}/etc/init.d </p><p class="P28">ssh {IP} </p><p class="P61">sudo update-rc.d krg start 99 2 3 4 .  </p></td></tr></table></div><p class="P58">2.Test It </p><p class="P42"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Test run </p><p class="P149" style="margin-left:0.25cm;"> </p></li></ul><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格8"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="表格8A1"><p class="P27">If kerrighed module don't auto load: </p><p class="P61">sudo /etc/init.d/kerrighed start </p><p class="P33">(all node must load, and we can use command dmesg see node message </p><p class="P33">ex. </p><p class="P33">TIPC: Established link &lt;1.1.1:eth0-1.1.3:eth0&gt; on network plane A </p><p class="P33">krg_node_arrival: 2 </p><p class="P33">) </p><p class="P33"> </p><p class="P28">sudo krgadm cluster start </p><p class="P62">(Kerrighed is running on 7 nodes) </p><p class="P62"> </p><p class="P28">top </p><p class="P33">(we cane see all clients' CPU and Memory are combined </p><p class="P33">ex. </p><p class="P33">top - 18:53:16 up 10 min,  2 users,  load average: 0.10, 0.07, 0.04</p><p class="P33">Tasks: 221 total,   1 running, 220 sleeping,   0 stopped,   0 zombie</p><p class="P33">Cpu0  :  0.7%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Cpu1  :  0.7%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Cpu2  :  0.7%us,  0.3%sy,  0.0%ni, 98.7%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Cpu3  :  0.8%us,  0.4%sy,  0.0%ni, 98.4%id,  0.3%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Cpu4  :  0.8%us,  0.4%sy,  0.0%ni, 98.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Cpu5  :  0.7%us,  0.4%sy,  0.0%ni, 98.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Cpu6  :  0.8%us,  0.3%sy,  0.0%ni, 98.6%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st</p><p class="P33">Mem:  14530264k total,  1508584k used, 13021680k free,      560k buffers</p><p class="P33">Swap:  2650684k total,        0k used,  2650684k free,  1282652k cached</p><p class="P33"> </p><p class="P33">  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND</p><p class="P33">    1 root      15   0  1580  548  480 S    0  0.0   0:00.30 init.orig</p><p class="P33">    2 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0</p><p class="P33">    3 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0</p><p class="P33">    4 root      10  -5     0    0    0 S    0  0.0   0:00.00 events/0</p><p class="P33">) </p></td></tr></table></div><p class="P42"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Test Kerrighed Command </p></li></ul><p class="P42"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格9"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="表格9A1"><p class="P31">sudo krgadm nodes status </p><p class="P62">(ex. </p><p class="P62">[rock@krg002 ~]$ krgadm nodes </p><p class="P62">  0:1   1:1   2:1   3:1   4:1   5:1   6:1</p><p class="P62">) </p><p class="P31"> </p><p class="P31">sudo krgcapset -s </p><p class="P62">(ex. </p><p class="P62">Permitted Capabilities: 037777777777 </p><p class="P62">        CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY</p><p class="P62">        CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS</p><p class="P62">        USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS</p><p class="P62">        SEE_LOCAL_PROC_STAT</p><p class="P62">Effective Capabilities: 01 </p><p class="P62">        CHANGE_KERRIGHED_CAP</p><p class="P62">Inheritable Permitted Capabilities: 037777777777 </p><p class="P62">        CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY</p><p class="P62">        CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS</p><p class="P62">        USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS</p><p class="P62">        SEE_LOCAL_PROC_STAT</p><p class="P62">Inheritable Effective Capabilities: 01 </p><p class="P62">        CHANGE_KERRIGHED_CAP</p><p class="P62">) </p></td></tr></table></div><p class="P42"> </p><ul style="margin-left:1.25cm;"><li class="P149" style="margin-left:0cm;"><p class="P149" style="margin-left:0.25cm;">Test process migration </p></li></ul><p class="P42"> </p><div style="text-align:left"><table border="0" cellspacing="0" cellpadding="0" class="表格10"><colgroup><col width="743" /></colgroup><tr><td style="text-align:left;width:16.999cm; " class="表格10A1"><p class="P27">DRBL Server: </p><p class="P28">mkdir /home/ker ; chmod 777 /home/ker </p><p class="P28">cd /home/ker </p><p class="P32"><span class="T22">wget</span><a href="../【NCHC】/【NCHC】/【NCHC】/【G%20World】/【Kerrighed】/My/%20http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz"><span class="T22"> http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz</span></a></p><p class="P63"><span class="T8">t</span><span class="T9">ar zxvf</span><span class="T10"> </span><a href="http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz">linux-2.6.22.tar.gz</a></p><p class="P31"> </p><p class="P27">Client: </p><p class="P28">krgcapset -d +CAN_MIGRATE </p><p class="P28">cd /home/linux-2.6.22 </p><p class="P28">sudo make -j 24 bzImage </p><p class="P28">sudo dmesg </p><p class="P34">(you can command dmesg to see message of process migration </p><p class="P34">ex.</p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 36885 (bzip2) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 37449 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39064 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39234 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39269 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39325 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39402 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39465 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39543 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39599 (cc1) </p><p class="P34">send_kerrighed_signal: 36647 (Migration Mgr) -&gt; 39660 (cc1) </p><p class="P34">) </p></td></tr></table></div><p class="P42"> </p><p class="P42"> </p><p class="P58">3.Reference </p><p class="P42"> </p><p class="P57">DRBL </p><p class="P43"><a href="http://drbl.sourceforge.net/"><span class="T13">http://drbl.sourceforge.net/</span></a></p><p class="P42"> </p><p class="P57">Kerrighed </p><p class="P41"><a href="http://www.kerrighed.org/wiki/index.php/Main_Page"><span class="T21">http:///www.kerrighed.org/wiki/index.php/Main_Page</span></a></p><ol style="margin-left:1.25cm;list-style-type:decimal; "><li class="P121" style="margin-left:0cm;"><p class="P121" style="margin-left:0.25cm;"> </p></li></ol></body></html>
    313 }}}
     1 = DRBL Merge Kerrighed =
     2[[BR]]
     3
     4 == Introduction ==
     5People need powerful computing resources to resolve complex problems. A important issue is how to easy build cheap and effective computing resource. We use DRBL and Kerighed to build SSI cluster. It quick to deploy your cluster environment. You just install them in one machine (Server), clients don't install any OS. Client only setup PXE (network boot) in BIOS, when client finish network booting that your cluster is OK!
     6[[BR]]
     7
     8 == Software ==
     9We use below Software:
     10 * Ubuntu or Debian
     11||Linux Distribution.||
     12[[BR]]
     13
     14*DRBL
     15||Diskless Remote Boot in Linux (DRBL) provides a diskless or systemless environment for client machines. It works on Debian, Ubuntu, Mandriva, Red Hat, Fedora, CentOS and SuSE. DRBL uses distributed hardware resources and makes it possible for clients to fully access local hardware. It also includes Clonezilla, a partitioning and disk cloning utility similar to Symantec Ghost.
     16http://drbl.sourceforge.net/||
     17[[BR]]
     18
     19 * Kerrighed
     20||Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs. The new version is Kerrighed 2.2.0, and it used for 2.6.20 kernel.
     21http://www.kerrighed.org/wiki/index.php/Main_Page||
     22[[BR]]
     23