Hadoop Streaming

Hadoop streaming是Hadoop的一個工具， 它幫助用戶創建和運行一類特殊的map/reduce作業， 這些特殊的map/reduce作業是由一些可執行文件或腳本文件充當mapper或者reducer

• 用法：

$ bin/hadoop jar contrib/streaming/hadoop-0.18.3-streaming.jar \
-input $INPUT -output $OUTPUT -mapper $MAPPER -reducer $REDUCER

• 格式分析：

bin/hadoop	呼叫使用hadoop程式
jar contrib/streaming/hadoop-0.18.3-streaming.jar	使用streaming這個功能
-input $INPUT	設定hdfs上的輸入資料夾
-output $OUTPUT	設定hdfs上的輸出資料夾
-mapper $MAPPER	設定mapper程式
-reducer $REDUCER	設定reducer程式

用 shell實做mapReduce

此範例以 cat 當mapper , wc 作 reducer

• 運算方法如下

$ bin/hadoop jar contrib/streaming/hadoop-0.18.3-streaming.jar \
-input lab3_input -output stream-out1 -mapper /bin/cat -reducer /usr/bin/wc

• 輸出的結果為：

$ bin/hadoop fs -cat stream-out1/part-00000

行	字數	字元數
1528	4612	48644

用php實做mapReduce

• ​用 "單機" 跟 "PHP" 開發 Hadoop 程式 from Hadoop Taiwan User Group

  $ cd /opt/hadoop/
  $ sudo apt-get install php5-cli
  

$ gedit mapper.php

#!/usr/bin/php
<?php

$word2count = array();

// 標準輸入為 STDIN (standard input)
while (($line = fgets(STDIN)) !== false) {
   // 移除小寫與空白
   $line = strtolower(trim($line));
   // 將行拆解成各個字於words 陣列中
   $words = preg_split('/\W/', $line, 0, PREG_SPLIT_NO_EMPTY);
   // 將字+1
   foreach ($words as $word) {
       $word2count[$word] += 1;
   }
}

// 將結果寫到 STDOUT (standard output)
foreach ($word2count as $word => $count) {
   // 印出 [字 , "tab符號" ,  "數字" , "結束字元"]
   echo $word, chr(9), $count, PHP_EOL;
}
?>

$ gedit reducer.php

#!/usr/bin/php
<?php

$word2count = array();

// 輸入為 STDIN
while (($line = fgets(STDIN)) !== false) {
    // 移除多餘的空白
    $line = trim($line);
    // 每一行的格式為 (單字 "tab" 數字) ，紀錄到($word, $count)
    list($word, $count) = explode(chr(9), $line);
    // 轉換格式string -> int
    $count = intval($count);
    // 加總
    if ($count > 0) $word2count[$word] += $count;
}

// 此行不必要，但可讓output排列更完整
ksort($word2count);

// 將結果寫到 STDOUT (standard output)
foreach ($word2count as $word => $count) {
    echo $word, chr(9), $count, PHP_EOL;
}

?> 

$ chmod 755 *.php
$ echo "i love hadoop, hadoop love u" | ./mapper.php | ./reducer.php
$ bin/hadoop jar hadoop/contrib/hadoop-*-streaming.jar -mapper mapper.php -reducer reducer.php -input lab4_input -output stream_out2

• 檢查結果

$ haddop dfs -cat stream_out2/part-00000 | more

Python 實做

• ​Hadoop Example Program from brandeis University