{{{ #!html
實作三、執行 MapReduce 基本運算
Lab3: Running MapReduce by Examples
}}} [[PageOutline]] = Step 0 : rebuild your Hadoop = * download the script of install_hadoop-0.20.2 {{{ ~$ wget http://trac.nchc.org.tw/cloud/raw-attachment/wiki/NCHCCloudCourse100802/install_hadoop_0.20.2 ~$ chmod a+x install_hadoop_0.20.2 ~$ ./install_hadoop_0.20.2 }}} = Sample 1: grep = * grep 這個命令是擷取文件裡面特定的字元，在Hadoop example中此指令可以擷取文件中有此指定文字的字串，並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings. {{{ $ cd /opt/hadoop $ bin/hadoop fs -put conf lab3_input $ bin/hadoop fs -ls lab3_input $ bin/hadoop jar hadoop-*-examples.jar grep lab3_input lab3_out1 'dfs[a-z.]+' }}} 運作的畫面如下：[[BR]]You should see procedure like this: {{{ 09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9 09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9 09/03/24 12:33:45 INFO mapred.JobClient: Running job: job_200903232025_0003 09/03/24 12:33:46 INFO mapred.JobClient: map 0% reduce 0% 09/03/24 12:33:47 INFO mapred.JobClient: map 10% reduce 0% 09/03/24 12:33:49 INFO mapred.JobClient: map 20% reduce 0% 09/03/24 12:33:51 INFO mapred.JobClient: map 30% reduce 0% 09/03/24 12:33:52 INFO mapred.JobClient: map 40% reduce 0% 09/03/24 12:33:54 INFO mapred.JobClient: map 50% reduce 0% 09/03/24 12:33:55 INFO mapred.JobClient: map 60% reduce 0% 09/03/24 12:33:57 INFO mapred.JobClient: map 70% reduce 0% 09/03/24 12:33:59 INFO mapred.JobClient: map 80% reduce 0% 09/03/24 12:34:00 INFO mapred.JobClient: map 90% reduce 0% 09/03/24 12:34:02 INFO mapred.JobClient: map 100% reduce 0% 09/03/24 12:34:10 INFO mapred.JobClient: map 100% reduce 10% 09/03/24 12:34:12 INFO mapred.JobClient: map 100% reduce 13% 09/03/24 12:34:15 INFO mapred.JobClient: map 100% reduce 20% 09/03/24 12:34:20 INFO mapred.JobClient: map 100% reduce 23% 09/03/24 12:34:22 INFO mapred.JobClient: Job complete: job_200903232025_0003 09/03/24 12:34:22 INFO mapred.JobClient: Counters: 16 09/03/24 12:34:22 INFO mapred.JobClient: File Systems 09/03/24 12:34:22 INFO mapred.JobClient: HDFS bytes read=48245 09/03/24 12:34:22 INFO mapred.JobClient: HDFS bytes written=1907 09/03/24 12:34:22 INFO mapred.JobClient: Local bytes read=1549 09/03/24 12:34:22 INFO mapred.JobClient: Local bytes written=3584 09/03/24 12:34:22 INFO mapred.JobClient: Job Counters ...... }}} * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS : {{{ $ bin/hadoop fs -ls lab3_out1 $ bin/hadoop fs -cat lab3_out1/part-00000 }}} 結果如下[[BR]]You should see results like this: {{{ 3 dfs.class 3 dfs. 2 dfs.period 1 dfs.http.address 1 dfs.balance.bandwidth 1 dfs.block.size 1 dfs.blockreport.initial 1 dfs.blockreport.interval 1 dfs.client.block.write.retries 1 dfs.client.buffer.dir 1 dfs.data.dir 1 dfs.datanode.address 1 dfs.datanode.dns.interface 1 dfs.datanode.dns.nameserver 1 dfs.datanode.du.pct 1 dfs.datanode.du.reserved 1 dfs.datanode.handler.count 1 dfs.datanode.http.address 1 dfs.datanode.https.address 1 dfs.datanode.ipc.address 1 dfs.default.chunk.view.size 1 dfs.df.interval 1 dfs.file 1 dfs.heartbeat.interval 1 dfs.hosts 1 dfs.hosts.exclude 1 dfs.https.address 1 dfs.impl 1 dfs.max.objects 1 dfs.name.dir 1 dfs.namenode.decommission.interval 1 dfs.namenode.decommission.interval. 1 dfs.namenode.decommission.nodes.per.interval 1 dfs.namenode.handler.count 1 dfs.namenode.logging.level 1 dfs.permissions 1 dfs.permissions.supergroup 1 dfs.replication 1 dfs.replication.consider 1 dfs.replication.interval 1 dfs.replication.max 1 dfs.replication.min 1 dfs.replication.min. 1 dfs.safemode.extension 1 dfs.safemode.threshold.pct 1 dfs.secondary.http.address 1 dfs.servers 1 dfs.web.ugi 1 dfsmetrics.log }}} = Sample 2 : WordCount = * 如名稱，WordCount會對所有的字作字數統計，並且從a-z作排列[[BR]]WordCount example will count each word shown in documents and sorting from a to z. {{{ /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar wordcount lab3_input lab3_out2 }}} 檢查輸出結果的方法同之前方法[[BR]]Let's check the computed result of '''wordcount''' from HDFS : {{{ $ bin/hadoop fs -ls lab3_out2 $ bin/hadoop fs -cat lab3_out2/part-r-00000 }}} = Browsing MapReduce and HDFS via Web GUI = * [http://localhost:50030 JobTracker Web Interface] * [http://localhost:50070 NameNode Web Interface] = More Examples = 可執行的指令一覽表：[[BR]]Here is a list of hadoop examples : || aggregatewordcount || An Aggregate based map/reduce program that counts the words in the input files. || || aggregatewordhist || An Aggregate based map/reduce program that computes the histogram of the words in the input files. || || grep || A map/reduce program that counts the matches of a regex in the input. || || join || A job that effects a join over sorted, equally partitioned datasets || || multifilewc || A job that counts words from several files. || || pentomino || A map/reduce tile laying program to find solutions to pentomino problems. || || pi || A map/reduce program that estimates Pi using monte-carlo method. || || randomtextwriter || A map/reduce program that writes 10GB of random textual data per node. || || randomwriter || A map/reduce program that writes 10GB of random data per node. || || sleep || A job that sleeps at each map and reduce task. || || sort || A map/reduce program that sorts the data written by the random writer. || || sudoku || A sudoku solver. || || wordcount || A map/reduce program that counts the words in the input files. || You could find more detail at [http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/package-summary.html org.apache.hadoop.examples]