wiki:waue/2009/0617

Version 19 (modified by waue, 15 years ago) (diff)

--

hadoop 0.20 程式開發
eclipse plugin + Makefile

零. 前言

  • 開發hadoop 需要用到許多的物件導向語法,包括繼承關係、介面類別,而且需要匯入正確的classpath,否則寫hadoop程式只是打字練習...
  • 用類 vim 來處理這種複雜的程式,有可能會變成一場惡夢,因此用eclipse開發,搭配mapreduce-plugin會事半功倍。
  • 早在hadoop 0.19~0.16之間的版本,筆者就試過各個plugin,每個版本的plugin都確實有大大小小的問題,如:hadoop plugin 無法正確使用、無法run as mapreduce。hadoop0.16搭配IBM的hadoop_plugin 可以提供完整的功能,但是,老兵不死,只是凋零...
  • 子曰:"逝者如斯夫,不捨晝夜",以前寫的文件也落伍了,要跟上潮流,因此此篇的重點在:用eclipse 3.4.2 開發hadoop 0.20程式,並且測試撰寫的程式運作在hadoop平台上
  • 以下是我的作法,如果你有更好的作法,或有需要更正的地方,請與我聯絡
單位 作者 Mail
國家高速網路中心-格網技術組 Wei-Yu Chen waue @ nchc.org.tw

0.1 環境說明

  • ubuntu 8.10
  • sun-java-6
  • eclipse 3.4.2
  • hadoop 0.20.0

0.2 目錄說明

  • 使用者:waue
  • 使用者家目錄: /home/waue
  • 專案目錄 : /home/waue/workspace
  • hadoop目錄: /opt/hadoop

一、安裝

  • 安裝的部份沒必要都一模一樣,僅提供參考,反正只要安裝好java , hadoop , eclipse,並清楚自己的路徑就可以了

1.1. 安裝java

  • java 基本套件
    $ sudo apt-get install java-common sun-java6-bin sun-java6-jdk sun-java6-jre
    

1.1.1. 安裝sun-java6-doc

  • 1 將javadoc下載下來 下載點
  • 2 放在 /tmp/ 下
  • 3 執行
    $ sudo apt-get install sun-java6-doc
    

1.3. 安裝hadoop

  • 安裝hadoop0.20到/opt/並取目錄名為hadoop
$ cd ~
$ wget http://apache.ntu.edu.tw/hadoop/core/hadoop-0.20.0/hadoop-0.20.0.tar.gz
$ tar zxvf hadoop-0.20.0.tar.gz
$ sudo mv hadoop-0.20.0 /opt/
$ sudo chown -R waue:waue /opt/hadoop-0.20.0
$ sudo ln -sf /opt/hadoop-0.20.0 /opt/hadoop
  • 編輯 /opt/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:/opt/hadoop/bin
  • 編輯 /opt/hadoop/conf/core-site.xml
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/tmp/hadoop/hadoop-${user.name}</value>
  </property>
</configuration>

  • 編輯 /opt/hadoop/conf/hdfs-site.xml
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>
  • 編輯 /opt/hadoop/conf/mapred-site.xml
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>
  • 啟動
    $ cd /opt/hadoop
    $ source /opt/hadoop/conf/hadoop-env.sh
    $ hadoop namenode -format
    $ start-all.sh
    $ hadoop fs -put conf input
    $ hadoop -ls 
    
  • 沒有錯誤訊息則代表無誤

1.2. 安裝eclipse

  • 下載 eclipse SDK 3.4.2 Classic
  • 放這檔案到家目錄
    $ cd ~
    $ tar -zxvf eclipse-SDK-3.4.2-linux-gtk.tar.gz
    $ sudo mv eclipse /opt
    $ sudo ln -sf /opt/eclipse/bin/eclipse /usr/local/bin/
    
    
  • 建立連結 (圖形介面要重新啟動才有吧!)
$ cat >> ~/.gnome2/panel2.d/default/launchers/eclipse.desktop << EOF
[Desktop Entry]
Encoding=UTF-8
Version=1.0
Type=Application
Terminal=false
Name[zh_TW]=eclipse
Exec=/usr/bin/eclipse
Name=eclipse
Icon[zh_TW]=/opt/eclipse/plugins/org.eclipse.platform_3.3.101.v200902111700/eclipse48.png
Icon=/opt/eclipse/plugins/org.eclipse.platform_3.3.101.v200902111700/eclipse48.png
EOF

二、 建立專案

  • 匯入hadoop 0.20.0 eclipse plugin
    $ cd /opt/hadoop
    $ sudo cp /opt/hadoop/contrib/eclipse-plugin/hadoop-0.20.0-eclipse-plugin.jar /opt/eclipse/plugins
    
    $ sudo vim /opt/eclipse/eclipse.ini
    
  • 可斟酌參考eclipse.ini內容(非必要)
    -startup
    plugins/org.eclipse.equinox.launcher_1.0.101.R34x_v20081125.jar
    --launcher.library
    plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.0.101.R34x_v20080805
    -showsplash
    org.eclipse.platform
    --launcher.XXMaxPermSize
    512m
    -vmargs
    -Xms40m
    -Xmx512m
    
  • window -> open pers.. -> other.. -> map/reduce
  • file -> new -> project -> Map/Reduce? -> Map/Reduce? Project -> next
    project name-> 輸入 : icas (隨意)
    use default hadoop -> Configur Hadoop install... -> 輸入: "/opt/hadoop" -> ok
    Finish
    
  • Project -> properties ->
    • java Build Path -> Libraries -> hadoop-0.20.0-ant.jar
    • java Build Path -> Libraries -> hadoop-0.20.0-core.jar
    • java Build Path -> Libraries -> hadoop-0.20.0-tools.jar
      • 以 hadoop-0.20.0-core.jar 的設定內容如下,其他依此類推
        source ...-> 輸入:/opt/opt/hadoop-0.20.0/src/core
        javadoc ...-> 輸入:file:/opt/hadoop/docs/api/
        
  • Project -> properties
    • javadoc location -> 輸入:file:/usr/lib/jvm/java-6-sun/docs/api/
  • 視窗右下角黃色大象圖示"Map/Reduce? Locations tag" -> 點選齒輪右邊的藍色大象圖示:
    Location Name -> 輸入:hadoop  (隨意)
    Map/Reduce Master -> Host-> 輸入:localhost 
    Map/Reduce Master -> Port-> 輸入:9001 
    DFS Master -> Host-> 輸入:9000
    Finish
    

三、 撰寫範例程式

  • 之前在eclipse上已經開了個專案icas,因此這個目錄在:
    • /home/waue/workspace/icas
  • 在這個目錄內有兩個資料夾:
    • src : 用來裝程式原始碼
    • bin : 用來裝編譯後的class檔
  • 如此一來原始碼和編譯檔就不會混在一起,對之後產生jar檔會很有幫助

3.1 我的第一隻程式

  • File -> new -> mapper
    source folder-> 輸入: icas/src
    Package : Sample
    Name -> : mapper
    
  • 編輯mapper.java
    package Sample;
    
    import java.io.IOException;
    import java.util.StringTokenizer;
    
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Mapper;
    
    public class mapper extends Mapper<Object, Text, Text, IntWritable> {
    
      private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();
    
      public void map(Object key, Text value, Context context)
          throws IOException, InterruptedException {
        StringTokenizer itr = new StringTokenizer(value.toString());
        while (itr.hasMoreTokens()) {
          word.set(itr.nextToken());
          context.write(word, one);
        }
      }
    }
    
  • File -> new -> reducer
source folder-> 輸入: icas/src
Package : Sample
Name -> : reducer
package Sample;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class reducer extends Reducer<Text, IntWritable, Text, IntWritable> {
  private IntWritable result = new IntWritable();

  public void reduce(Text key, Iterable<IntWritable> values, Context context)
      throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
      sum += val.get();
    }
    result.set(sum);
    context.write(key, result);
  }
}
source folder-> 輸入: icas/src
Package : Sample
Name -> : WordCount.java
package Sample;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf, args)
        .getRemainingArgs();
    if (otherArgs.length != 2) {
      System.err.println("Usage: wordcount <in> <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(mapper.class);

    job.setCombinerClass(reducer.class);
    job.setReducerClass(reducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}
  • 三個檔都存檔後,可以看到icas專案下的src,bin都有檔案產生
$ cd workspace/icas
$ ls src/Sample/
mapper.java  reducer.java  WordCount.java
$ ls bin/Sample/
mapper.class  reducer.class  WordCount.class

四、編譯

  • 由於hadoop 0.20 此版本的eclipse-plugin依舊不完整,如:1. 齒輪圖示沒有作用 2. 右鍵點選WordCount.java -> run as -> run on Hadoop :沒有效果
  • 因此編譯hadoop程式就要靠指令來編輯,然而用一行一行指令來產生太沒效率,在此介紹用Makefile來編譯

4.1 產生Makefile

$ cd /home/waue/workspace/icas/
$ gedit Makefile
JarFile="sample-0.1.jar"
MainFunc="Sample.WordCount"
LocalOutDir="/tmp/output"

all:help
jar:
  jar -cvf ${JarFile} -C bin/ .

run:
  hadoop jar ${JarFile} ${MainFunc} input output

clean:
  hadoop fs -rmr output

output:
  rm -rf ${LocalOutDir}
  hadoop fs -get output ${LocalOutDir}
  gedit ${LocalOutDir}/part-r-00000 & 

help:
  @echo "Usage:"
  @echo " make jar     - Build Jar File."
  @echo " make clean   - Clean up Output directory on HDFS."
  @echo " make run     - Run your MapReduce code on Hadoop."
  @echo " make output  - Download and show output file"
  @echo " make help    - Show Makefile options."
  @echo " "
  @echo "Example:"
  @echo " make jar; make run; make output; make clean"

4.2. 執行

$ cd /home/waue/workspace/icas/
$ make jar; make run; make output; make clean

4.3 screenshot

  • 圖一: 完成後的eclipse視窗快照

No image "1.png" attached to waue/2009/0617


  • 圖二: 因為有設定完整的javadoc, 因此可以得到詳細的解說與輔助

No image "2.png" attached to waue/2009/0617

Attachments (26)