- java is required for Hadoop, so you need to install java runtime or jdk first.
~$ echo "deb http://free.nchc.org.tw/debian/ etch non-free" > /tmp/etch-non-free.list
~$ sudo mv /tmp/etch-non-free.list /etc/apt/sources.list.d/.
~$ sudo apt-get update
~$ sudo apt-get install sun-java5-jdk
- download Hadoop 0.18.2
~$ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/core/hadoop-0.18.2/hadoop-0.18.2.tar.gz
~$ tar zxvf hadoop-0.18.2.tar.gz
- setup JAVA_HOME environment variable
~$ echo "export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun" >> ~/.bash_profile
~$ source ~/.bash_profile
- edit hadoop-0.18.2/conf/hadoop-env.sh
-
old
|
new
|
|
6 | 6 | # remote nodes. |
7 | 7 | |
8 | 8 | # The java implementation to use. Required. |
9 | | # export JAVA_HOME=/usr/lib/j2sdk1.5-sun |
| 9 | export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun |
| 10 | export HADOOP_HOME=/home/jazz/hadoop-0.18.2 |
| 11 | export HADOOP_CONF_DIR=$HADOOP_HOME/conf |
10 | 12 | |
11 | 13 | # Extra Java CLASSPATH elements. Optional. |
12 | 14 | # export HADOOP_CLASSPATH= |
- here is current DRBL setup
你的DRBL環境配置:
******************************************************
NIC NIC IP Clients
+-----------------------------+
| DRBL SERVER |
| |
| +-- [eth0] 140.110.25.101 +- to WAN
| |
| +-- [eth1] 192.168.61.254 +- to clients group 1 [ 16 clients, their IP
| | from 192.168.61.1 - 192.168.61.16]
+-----------------------------+
******************************************************
Total clients: 16
******************************************************
- Hadoop will use ssh connections for internal connection, thus we have to do SSH key exchange.
~$ ssh-keygen
~$ cp .ssh/id_rsa.pub .ssh/authorized_keys
~$ sudo apt-get install dsh
~$ mkdir -p .dsh
~$ nmap -v -sP 192.168.61-63.1-11 | grep '(.*) .* up' | awk '{ print $3 }' | sort -n | sed 's#(##' | sed 's#)##' > .dsh/machines.list
- edit