2008-11-04

• Devaraj Das 來訪
  • 09:30 - 10:30 拜會中心主任
  • 11:00 - 12:30 Public Talk Session: "Introduction to Hadoop and Cloud Computing" @ 北群多媒體
  • 14:00 - 17:00 Hands-on Labs (1): "Basics of DFS commands + How to develop MapReduce program using Hadoop?" @ 北群多媒體

• Hadoop Hands-on Labs (1)
  • download hadoop-0.18.2

    $ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/core/hadoop-0.18.2/hadoop-0.18.2.tar.gz
    $ tar zxvf hadoop-0.18.2.tar.gz
    

  • [注意] 需要 JAVA_HOME 環境變數才能執行 hadoop namenode

    $ echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre" >> ~/.bash_profile
    $ cd hadoop-0.18.2
    ~/hadoop-0.18.2$ bin/hadoop namenode -format
    

  • [注意] 還需要編輯 conf/hadoop-evn.sh (HADOOP_HOME要設定到你的hadoop安裝目錄)

    export JAVA_HOME=/usr/lib/jvm/java-6-sun
    export HADOOP_HOME=/home/jazz/hadoop-0.18.2/
    export HADOOP_LOG_DIR=$HADOOP_HOME/logs
    export HADOOP_SLAVES=$HADOOP_HOME/conf/slaves
    export HADOOP_CONF_DIR=$HADOOP_HOME/conf
    

  • 編輯 conf/hadoop-site.xml 在 configuration 那一段加入以下設定

    <property>
      <name>fs.default.name</name>
      <value>hdfs://localhost:9000/</value>
      <description>
        The name of the default file system. Either the literal string
        "local" or a host:port for NDFS.
      </description>
    </property>
    
    <property>
      <name>mapred.job.tracker</name>
      <value>localhost:9001</value>
      <description>
        The host and port that the MapReduce job tracker runs at. If
        "local", then jobs are run in-process as a single map and
        reduce task.
      </description>
    </property>
    
    

• 執行 bin/start-all.sh

  ~/hadoop-0.18.2$ bin/start-all.sh
  starting namenode, logging to /home/jazz/hadoop-0.18.2/bin/../logs/hadoop-jazz-namenode-hadoop.out
  The authenticity of host 'localhost (127.0.0.1)' can't be established.
  RSA key fingerprint is 70:3f:8b:f2:b9:a8:de:ea:90:f4:bf:ce:cb:85:7a:eb.
  Are you sure you want to continue connecting (yes/no)? yes
  localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
  

• 從這邊的訊息可以知道 Hadoop 會用 SSH 進行內部連線，因此需要做 SSH Key exchange

  ~$ ssh-keygen
  ~$ cp .ssh/id_rsa.pub .ssh/authorized_keys
  

• 完成後可以看到以下三個網頁
  • ​http://localhost:50030/
  • ​http://localhost:50060/
  • ​http://localhost:50070/

• 可以放的東西上去看看

  ~$ bin/hadoop dfs -put conf conf
  ~$ bin/hadoop dfs -ls
  Found 1 items
  drwxr-xr-x   - jazz supergroup          0 2008-11-04 15:56 /user/jazz/conf