wiki:waue/Hadoop_DRBL

Context Navigation

Version 29 (modified by waue, 16 years ago) (diff)
--

DRBL叢集上運行HADOOP

Hadoop Cluster Based on DRBL

此篇的目的在於利用DRBL統整一個Cluster，並在上面運行Hadoop。
由於DRBL為無碟系統，並非一般的Cluster，因此有些地方需要注意。

零、環境說明

環境中共有七台機器，一台為drbl server，也是hadoop的namenode，其他節點則client 與datanode，如下：

名稱	ip	drbl用途	hadoop 用途
hadoop	192.168.1.254	drbl server	namenode
hadoop	192.168.1.1~12	drbl client	datanode

介紹drbl server環境如下：

debian

etch (4.0)

server - 64 bit

DRBL為無碟系統，因此只要將drbl server系統與所需服務安裝好，則其他的client網路開機後，就會載入以server為依據的檔案系統，也就是說，只有某些特定資料夾內的內容(如 /etc /root /home /tmp /var ...)會各自不同之外，其他都一樣。舉例若改了server內/etc/hosts檔的，則其他的client都會自動即時一起更改（因為是用NFS mount 上來的）。

因此，只要先在drbl server上完成了一、安裝,二、設定之後，在將其他的client開機然後依照三、操作 就可以了。

一、安裝

1.1 安裝drbl

詳見 DRBL的安裝

1.2 安裝 java 6

在套件庫裡 /etc/apt/sources.list 加入 non-free 庫以及 backports 網址才能安裝 sun-java6

deb http://free.nchc.org.tw/debian/ etch main contrib non-free
deb-src http://free.nchc.org.tw/debian/ etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free
deb-src http://security.debian.org/ etch/updates main contrib non-free
deb http://www.backports.org/debian etch-backports main non-free
deb http://free.nchc.org.tw/drbl-core drbl stable

安裝key及java6

$ wget http://www.backports.org/debian/archive.key
$ sudo apt-key add archive.key
$ apt-get update
$ apt-get install sun-java6-bin  sun-java6-jdk sun-java6-jre

1.3 安裝 Hadoop 0.18.3

$ cd /opt
$ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/core/hadoop-0.18.3/hadoop-0.18.3.tar.gz
$ tar zxvf hadoop-0.18.3.tar.gz

1.4 設定使用者

$ su - 
$ addgroup hdfsgrp
$ adduser --ingroup hdfsgrp hdfsadm
$ chown -R hdfsadm:hdfsgrp /opt/hadoop-0.18.3
$ chmod -R 775 /opt/hadoop-0.18.3
$ su - hdfsadm
$ cd /opt/hadoop
$ ln -sf hadoop-0.18.3 hadoop

二、設定 Hadoop

在 /etc/bash.bashrc 的最末加入以下資訊

PATH=$PATH:/opt/drbl/bin:/opt/drbl/sbin
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_HOME=/opt/hadoop/

載入設定值
```
$ source /etc/bash.bashrc
```

編輯 /etc/hosts 把下面內容貼在最後

192.168.1.254 gm2.nchc.org.tw
192.168.1.1 hadoop101
192.168.1.10 hadoop110
192.168.1.11 hadoop111
192.168.1.2 hadoop102
192.168.1.3 hadoop103
192.168.1.4 hadoop104
192.168.1.5 hadoop105
192.168.1.6 hadoop106
192.168.1.7 hadoop107
192.168.1.8 hadoop108
192.168.1.9 hadoop109

編輯 /opt/hadoop-0.18.3/conf/hadoop-env.sh

hadoop-0.18.3/conf/hadoop-env.sh

-                      old
+                      new
 # remote nodes.
 # The java implementation to use.  Required.
+# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
+export JAVA_HOME=/usr/lib/jvm/java-6-sun
+export HADOOP_HOME=/opt/hadoop
+export HADOOP_CONF_DIR=$HADOOP_HOME/conf
+export HADOOP_LOG_DIR=/home/hdfsadm/hdfs/logs
+export HADOOP_PID_DIR=/home/hdfsadm/hdfs/pids
 # Extra Java CLASSPATH elements.  Optional.
 # export HADOOP_CLASSPATH=

編輯 /opt/hadoop-0.18.3/conf/hadoop-site.xml

hadoop-0.18.3/conf/hadoop-site.xml

-                      old
+                      new
 <!-- Put site-specific property overrides in this file. -->
 <configuration>
+  <property>
+    <name>fs.default.name</name>
+    <value>hdfs://gm2.nchc.org.tw:9000/</value>
+    <description>
+      The name of the default file system. Either the literal string
+      "local" or a host:port for NDFS.
+    </description>
+  </property>
+  <property>
+    <name>mapred.job.tracker</name>
+    <value>hdfs://gm2.nchc.org.tw:9001</value>
+    <description>
+      The host and port that the MapReduce job tracker runs at. If
+      "local", then jobs are run in-process as a single map and
+      reduce task.
+    </description>
+  </property>
+  <property>
+    <name>hadoop.tmp.dir</name>
+    <value>/tmp/hadoop/hadoop-${user.name}</value>
+    <description>A base for other temporary directories.</description>
+  </property>
 </configuration>

編輯 /opt/hadoop/conf/slaves

192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
192.168.1.7
192.168.1.8
192.168.1.9
192.168.1.10
192.168.1.11
192.168.1.12

三、操作

3.1 開啟DRBL Client

將所有的 client 開啟，並且如下

******************************************************
          NIC    NIC IP                    Clients
+------------------------------+
|         DRBL SERVER          |
|                              |
|    +-- [eth2] 140.110.X.X    +- to WAN
|                              |
|    +-- [eth1] 192.168.1.254  +- to clients group 1 [ 6 clients, their IP
|                              |             from 192.168.1.1 - 192.168.1.12]
+------------------------------+
******************************************************
Total clients: 12
******************************************************

3.2 設定ssh

編寫 /etc/ssh/ssh_config
```
StrictHostKeyChecking no
```

執行

$ su -
$ /opt/drbl/sbin/drbl-useradd -s hdfsadm hdfsgrp
$ su - hdfsadm
$ ssh-keygen -t rsa -b 1024 -N "" -f ~/.ssh/id_rsa
$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
$ /etc/init.d/ssh restart

正確無誤則可免密碼登入

3.2.1 dsh

$ sudo apt-get install dsh
$ mkdir -p .dsh
$ for ((i=1;i<=12;i++)); do echo "192.168.1.$i" >> .dsh/machines.list; done

測試並執行

$ dsh -a hostname
$ dsh -a source /etc/bash.bashrc

3.3 啟動 Hadoop

啟動

$ cd /opt/hadoop
$ bin/hadoop namenode -format
$ bin/start-all.sh

3.4 Hadoop 測試範例

運作WordCount以測試

$ mkdir input
$ cp *.txt input/
$ bin/hadoop dfs -put input input 
$ bin/hadoop jar hadoop-*-examples.jar wordcount input ouput

執行畫面：

hadoop:/opt/hadoop# bin/hadoop jar hadoop-*-examples.jar wordcount input ouput
09/02/26 06:16:34 INFO mapred.FileInputFormat: Total input paths to process : 4
09/02/26 06:16:34 INFO mapred.FileInputFormat: Total input paths to process : 4
09/02/26 06:16:35 INFO mapred.JobClient: Running job: job_200902260615_0001
09/02/26 06:16:36 INFO mapred.JobClient:  map 0% reduce 0%
09/02/26 06:16:39 INFO mapred.JobClient:  map 80% reduce 0%
09/02/26 06:16:40 INFO mapred.JobClient:  map 100% reduce 0%
09/02/26 06:16:50 INFO mapred.JobClient: Job complete: job_200902260615_0001
09/02/26 06:16:50 INFO mapred.JobClient: Counters: 16
09/02/26 06:16:50 INFO mapred.JobClient:   File Systems
09/02/26 06:16:50 INFO mapred.JobClient:     HDFS bytes read=267854
09/02/26 06:16:50 INFO mapred.JobClient:     HDFS bytes written=100895
09/02/26 06:16:50 INFO mapred.JobClient:     Local bytes read=133897
09/02/26 06:16:50 INFO mapred.JobClient:     Local bytes written=292260
09/02/26 06:16:50 INFO mapred.JobClient:   Job Counters 
09/02/26 06:16:50 INFO mapred.JobClient:     Launched reduce tasks=1
09/02/26 06:16:50 INFO mapred.JobClient:     Rack-local map tasks=5
09/02/26 06:16:50 INFO mapred.JobClient:     Launched map tasks=5
09/02/26 06:16:50 INFO mapred.JobClient:   Map-Reduce Framework
09/02/26 06:16:50 INFO mapred.JobClient:     Reduce input groups=8123
09/02/26 06:16:50 INFO mapred.JobClient:     Combine output records=17996
09/02/26 06:16:50 INFO mapred.JobClient:     Map input records=6515
09/02/26 06:16:50 INFO mapred.JobClient:     Reduce output records=8123
09/02/26 06:16:50 INFO mapred.JobClient:     Map output bytes=385233
09/02/26 06:16:50 INFO mapred.JobClient:     Map input bytes=265370
09/02/26 06:16:50 INFO mapred.JobClient:     Combine input records=44786
09/02/26 06:16:50 INFO mapred.JobClient:     Map output records=34913
09/02/26 06:16:50 INFO mapred.JobClient:     Reduce input records=8123
hadoop:/opt/hadoop#

http://gm2.nchc.org.tw:50030/
- 網頁中可以看到剛剛有在工作的node數
  gm2 Hadoop Map/Reduce Administration
  State: RUNNING
  Started: Tue Mar 17 16:22:46 EDT 2009
  Version: 0.18.3, r736250
  Compiled: Thu Jan 22 23:12:08 UTC 2009 by ndaley
  Identifier: 200903171622
  
  Cluster Summary
  
  Maps Reduces Total Submissions Nodes Map Task Capacity Reduce Task Capacity Avg. Tasks/Node
  
  0 0 1 9 18 18 4.00
  
  Running Jobs
  
  Running Jobs
  
  none
  
  Completed Jobs
  
  Completed Jobs
  
  Jobid User Name Map % Complete Map Total Maps Completed Reduce % Complete Reduce Total Reduces Completed
  
  job_200903171622_0001 hdfsadm wordcount 100.00%
  5 5 100.00%
  1 1

http://gm2.nchc.org.tw:50075/browseDirectory.jsp?dir=%2Fuser%2Froot&namenodeInfoPort=50070
- 可以看到輸出結果

3.5 停止hadoop

$ bin/stop-all.sh

3.6 重新建立 hadoop

$ bin/stop-all.sh
$ dsh -a rm -rf ~/hdfs/logs/* ~/hdfs/pids/* /tmp/hadoop/*
$ bin/hadoop namenode -format
$ bin/start-all.sh

四、操作

4.1 帳號

增加一個hadoop帳號huser，使之可以在hdfs上自己的目錄內進行存取瀏覽的操作

在drbl系統新增帳號 huser

$ su -
$ /opt/drbl/sbin/drbl-useradd -s huser huser

用hdfs的superuser（此篇文章為hdfsadm）在hdfs上建立資料夾
```
$ su - hdfsadm
$ /opt/hadoop/bin/hadoop dfs -mkdir /user/huser
```

用superuser 設定hdfs上該資料夾的權限與擁有者

$ /opt/hadoop/bin/hadoop dfs -chown -R huser /user/huser
$ /opt/hadoop/bin/hadoop dfs -chmod -R 775 /user/huser

測試：用huser瀏覽或寫入檔案

<root>$ su - huser
<huser>$ cd /opt/hadoop/
<huser>$ /opt/hadoop/bin/hadoop dfs -put input /user/huser/input
<huser>$ /opt/hadoop/bin/hadoop dfs -ls /user/huser/input

4.2 多帳號執行

測試兩個user： rock , waue 同時執行，沒有問題

bin/hadoop jar hadoop-*-examples.jar wordcount input/ ouput/

網頁結果：

Completed Jobs

Completed Jobs

Jobid

User

Name

Map % Complete

Map Total

Maps Completed

Reduce % Complete

Reduce Total

Reduces Completed

job_200903061742_0001

waue

wordcount

100.00%

job_200903061742_0002

rock

wordcount

100.00%

job_200903061742_0003

waue

wordcount

100.00%

Failed Jobs

五、參考資料

六、問題排解

drbl似乎安裝不順

drblsrv -i 出現以下錯誤訊息

Kernel 2.6 was found, so default to use initramfs.
The requested kernel "" 2.6.18-6-amd64 kernel files are NOT found in  /tftpboot/node_root/lib/modules/s and /tftpboot/node_root/boot in the server! The necessary modules in the network initrd can NOT be created!
Client will NOT remote boot correctly!
Program terminated!
Done!

原因： apt 的鏡像站台沒有複製到資料因此無法安裝新kernel，導致出現問題

Hadoop 空間問題

hdfsadm@hadoop102:~$ df /tmp
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                  1018360       712   1017648   1% /tmp
hdfsadm@hadoop102:~$ df /
Filesystem           1K-blocks      Used Available Use% Mounted on
192.168.1.254:/tftpboot/node_root
                      28834720  20961600   6408384  77% /

datanode hadoop102的空間大小指稱有0.98G的空間，因此研判datanode的空間大小為hadoop.tmp.dir所設定的目錄剩餘空間

如何增加drbl_hadoop 模式下 datanode的可用空間
1. 將硬碟mount起來到/hdfs_data資料夾
2. 將hadoop.tmp.dir設在各個datanode的 /hdfs_data中

Download in other formats:

Plain Text

Running Jobs
none

Context Navigation

DRBL叢集上運行HADOOP

零、環境說明

一、安裝

1.1 安裝drbl

1.2 安裝 java 6

1.3 安裝 Hadoop 0.18.3

1.4 設定使用者

二、設定 Hadoop

hadoop-0.18.3/conf/hadoop-env.sh

hadoop-0.18.3/conf/hadoop-site.xml

三、操作

3.1 開啟DRBL Client

3.2 設定ssh

3.2.1 dsh

3.3 啟動 Hadoop

3.4 Hadoop 測試範例

gm2 Hadoop Map/Reduce Administration

Cluster Summary

Running Jobs

Completed Jobs

3.5 停止hadoop

3.6 重新建立 hadoop

四、操作

4.1 帳號

4.2 多帳號執行

Completed Jobs

Failed Jobs

五、參考資料

六、問題排解

Download in other formats: