| 1 | ◢ <[wiki:NTUOSS160412/Lab4 實作四]> | <[wiki:NTUOSS160412 回課程大綱]> ▲ | <[wiki:NTUOSS160412/Lab6 實作六]> ◣ |
| 2 | |
| 3 | = 實作五 Lab5 = |
| 4 | |
| 5 | {{{ |
| 6 | #!html |
| 7 | <p style="text-align: center;"><big style="font-weight: bold;"><big>MapReduce 基本指令操作<br/>Basic Commands of Hadoop MapReduce</big></big></p> |
| 8 | }}} |
| 9 | |
| 10 | [[PageOutline]] |
| 11 | |
| 12 | == Sample 1 : WordCount == |
| 13 | |
| 14 | * 如名稱,WordCount會對所有的字作字數統計,並且從a-z作排列[[BR]]WordCount example will count each word shown in documents and sorting from a to z. |
| 15 | {{{ |
| 16 | ~$ hadoop fs -put ${HOME}/hadoop/conf lab11_input |
| 17 | ~$ hadoop fs -rmr lab11_out2 |
| 18 | ~$ hadoop jar ${HOME}/hadoop/hadoop-examples-*.jar wordcount lab11_input lab11_out2 |
| 19 | }}} |
| 20 | * 檢查輸出結果的方法同之前方法[[BR]]Let's check the computed result of '''wordcount''' from HDFS : |
| 21 | {{{ |
| 22 | ~$ hadoop fs -ls lab11_out2 |
| 23 | ~$ hadoop fs -cat lab11_out2/part-r-00000 |
| 24 | }}} |
| 25 | * 結果如下[[BR]]You should see results like this: |
| 26 | {{{ |
| 27 | "". 4 |
| 28 | "*" 9 |
| 29 | "127.0.0.1" 3 |
| 30 | "AS 2 |
| 31 | "License"); 2 |
| 32 | "_logs/history/" 1 |
| 33 | "alice,bob 9 |
| 34 | |
| 35 | ( ... skip ... ) |
| 36 | }}} |
| 37 | |
| 38 | == Sample 2: grep == |
| 39 | |
| 40 | * grep 這個命令是擷取文件裡面特定的字元,在Hadoop example中此指令可以擷取文件中有此指定文字的字串,並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings. |
| 41 | {{{ |
| 42 | ~$ hadoop fs -ls lab11_input |
| 43 | ~$ hadoop jar ${HOME}/hadoop/hadoop-examples-*.jar grep lab11_input lab11_out3 'dfs[a-z.]+' |
| 44 | }}} |
| 45 | * 運作的畫面如下:[[BR]]You should see procedure like this: |
| 46 | {{{ |
| 47 | 11/04/19 10:00:20 INFO mapred.FileInputFormat: Total input paths to process : 25 |
| 48 | 11/04/19 10:00:20 INFO mapred.JobClient: Running job: job_201104120101_0645 |
| 49 | 11/04/19 10:00:21 INFO mapred.JobClient: map 0% reduce 0% |
| 50 | ( ... skip ... ) |
| 51 | }}} |
| 52 | * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS : |
| 53 | {{{ |
| 54 | ~$ hadoop fs -ls lab11_out3 |
| 55 | ~$ hadoop fs -cat lab11_out3/part-00000 |
| 56 | }}} |
| 57 | * 結果如下[[BR]]You should see results like this: |
| 58 | {{{ |
| 59 | 4 dfs.permissions |
| 60 | 4 dfs.replication |
| 61 | 4 dfs.name.dir |
| 62 | 3 dfs.namenode.decommission.interval. |
| 63 | 3 dfs.namenode.decommission.nodes.per.interval |
| 64 | 3 dfs. |
| 65 | ( ... skip ... ) |
| 66 | }}} |
| 67 | |
| 68 | == More Examples == |
| 69 | |
| 70 | 可執行的指令一覽表:[[BR]]Here is a list of hadoop examples : |
| 71 | |
| 72 | || aggregatewordcount || An Aggregate based map/reduce program that counts the words in the input files. || |
| 73 | || aggregatewordhist || An Aggregate based map/reduce program that computes the histogram of the words in the input files. || |
| 74 | || grep || A map/reduce program that counts the matches of a regex in the input. || |
| 75 | || join || A job that effects a join over sorted, equally partitioned datasets || |
| 76 | || multifilewc || A job that counts words from several files. || |
| 77 | || pentomino || A map/reduce tile laying program to find solutions to pentomino problems. || |
| 78 | || pi || A map/reduce program that estimates Pi using monte-carlo method. || |
| 79 | || randomtextwriter || A map/reduce program that writes 10GB of random textual data per node. || |
| 80 | || randomwriter || A map/reduce program that writes 10GB of random data per node. || |
| 81 | || sleep || A job that sleeps at each map and reduce task. || |
| 82 | || sort || A map/reduce program that sorts the data written by the random writer. || |
| 83 | || sudoku || A sudoku solver. || |
| 84 | || wordcount || A map/reduce program that counts the words in the input files. || |
| 85 | |
| 86 | You could find more detail at [http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/package-summary.html org.apache.hadoop.examples] |