Changes between Initial Version and Version 1 of III141129/Lab6


Ignore:
Timestamp:
Nov 28, 2014, 9:45:37 PM (10 years ago)
Author:
jazz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • III141129/Lab6

    v1 v1  
     1[[PageOutline]]
     2
     3◢ <[wiki:III141129/Lab5 實作五]> | <[wiki:III141129 回課程大綱]> ▲ | <[wiki:III141129/Lab7 實作七]> ◣
     4
     5= 實作六 Lab 6 =
     6
     7{{{
     8#!html
     9<div style="text-align: center;"><big style="font-weight: bold;"><big>在完全分散模式執行 MapReduce 基本運算<br/>Running MapReduce in Full Distributed Mode by Examples</big></big></div>
     10}}}
     11{{{
     12#!text
     13以下練習，請連線至 hdp01.3du.me 操作。底下的 userXX 等於您的用戶名稱。
     14以下練習，請連線至 hdp02.3du.me 操作。底下的 userXX 等於您的用戶名稱。
     15以下練習，請連線至 hdp03.3du.me 操作。底下的 userXX 等於您的用戶名稱。
     16以下練習，請連線至 hdp04.3du.me 操作。底下的 userXX 等於您的用戶名稱。
     17}}}
     18
     19== Sample 1 : WordCount ==
     20 
     21 * 如名稱，WordCount會對所有的字作字數統計，並且從a-z作排列[[BR]]WordCount example will count each word shown in documents and sorting from a to z.
     22{{{
     23~$ hadoop fs -put /opt/hadoop/conf lab5_input
     24~$ hadoop fs -rmr lab5_out2        ## 這行是預防上次有執行過這個範例的防呆步驟
     25~$ hadoop jar hadoop-examples.jar wordcount lab5_input lab5_out2
     26}}}
     27
     28 * 連線到 http://hdp01.3du.me:50030 查詢您剛剛提交的任務(Job)狀態
     29
     30 * 檢查輸出結果的方法同之前方法[[BR]]Let's check the computed result of '''wordcount''' from HDFS :
     31{{{
     32$ hadoop fs -ls lab5_out2
     33$ hadoop fs -cat lab5_out2/part-r-00000
     34}}}
     35 * 結果如下[[BR]]You should see results like this:
     36{{{
     37"".     4
     38"*"     9
     39"127.0.0.1"     3
     40"AS     2
     41"License");     2
     42"_logs/history/"        1
     43"alice,bob      9
     44
     45( ... skip ... )
     46}}}
     47
     48== Sample 2: grep ==
     49 
     50 * grep 這個命令是擷取文件裡面特定的字元，在Hadoop example中此指令可以擷取文件中有此指定文字的字串，並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings.
     51{{{
     52$ hadoop fs -ls lab5_input
     53$ hadoop jar hadoop-examples.jar grep lab5_input lab5_out3 'dfs[a-z.]+'
     54}}}
     55 * 運作的畫面如下：[[BR]]You should see procedure like this: 
     56{{{
     5711/04/19 10:00:20 INFO mapred.FileInputFormat: Total input paths to process : 25
     5811/04/19 10:00:20 INFO mapred.JobClient: Running job: job_201104120101_0645
     5911/04/19 10:00:21 INFO mapred.JobClient:  map 0% reduce 0%
     60( ... skip ... )
     61}}}
     62 * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS :
     63{{{
     64$ hadoop fs -ls lab5_out3
     65Found 2 items
     66drwx------   - hXXXX supergroup          0 2011-04-19 10:00 /user/hXXXX/lab5_out1/_logs
     67-rw-r--r--   2 hXXXX supergroup       1146 2011-04-19 10:00 /user/hXXXX/lab5_out1/part-00000
     68$ hadoop fs -cat lab5_out3/part-00000
     69}}}
     70 * 結果如下[[BR]]You should see results like this:
     71{{{
     724       dfs.permissions
     734       dfs.replication
     744       dfs.name.dir
     753       dfs.namenode.decommission.interval.
     763       dfs.namenode.decommission.nodes.per.interval
     773       dfs.
     78( ... skip ... )
     79}}}
     80
     81== More Examples ==
     82 
     83 可執行的指令一覽表：[[BR]]Here is a list of hadoop examples :
     84
     85 || aggregatewordcount ||  An Aggregate based map/reduce program that counts the words in the input files. ||
     86 || aggregatewordhist || An Aggregate based map/reduce program that computes the histogram of the words in the input files. ||
     87 || grep ||  A map/reduce program that counts the matches of a regex in the input. ||
     88 || join || A job that effects a join over sorted, equally partitioned datasets ||
     89 || multifilewc ||  A job that counts words from several files. ||
     90 || pentomino  || A map/reduce tile laying program to find solutions to pentomino problems. ||
     91 || pi ||  A map/reduce program that estimates Pi using monte-carlo method. ||
     92 || randomtextwriter ||  A map/reduce program that writes 10GB of random textual data per node. ||
     93 || randomwriter || A map/reduce program that writes 10GB of random data per node. ||
     94 || sleep ||  A job that sleeps at each map and reduce task. ||
     95 || sort || A map/reduce program that sorts the data written by the random writer. ||
     96 || sudoku ||  A sudoku solver. ||
     97 || wordcount || A map/reduce program that counts the words in the input files. ||
     98
     99You could find more detail at [http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/package-summary.html org.apache.hadoop.examples]