Changes between Version 1 and Version 2 of 0428Hadoop_Lab2

Timestamp:

Apr 25, 2009, 11:26:36 PM (15 years ago)

Author:

waue

Comment:

--

0428Hadoop_Lab2

@@ -107 +107 @@
  
   
- == Content 2. Hadoop 運算命令 == 
-  
- === 2.1 Hadoop運算命令 grep ===  
  
- * grep 這個命令是擷取文件裡面特定的字元，在Hadoop example中此指令可以擷取文件中有此指定文字的字串，並作計數統計
- 
-{{{
- /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar grep input grep_output 'dfs[a-z.]+' 
- 
-}}}
- 
- 運作的畫面如下：
- 
-{{{
-
-09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9
-09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9
-09/03/24 12:33:45 INFO mapred.JobClient: Running job: job_200903232025_0003
-09/03/24 12:33:46 INFO mapred.JobClient:  map 0% reduce 0%
-09/03/24 12:33:47 INFO mapred.JobClient:  map 10% reduce 0%
-09/03/24 12:33:49 INFO mapred.JobClient:  map 20% reduce 0%
-09/03/24 12:33:51 INFO mapred.JobClient:  map 30% reduce 0%
-09/03/24 12:33:52 INFO mapred.JobClient:  map 40% reduce 0%
-09/03/24 12:33:54 INFO mapred.JobClient:  map 50% reduce 0%
-09/03/24 12:33:55 INFO mapred.JobClient:  map 60% reduce 0%
-09/03/24 12:33:57 INFO mapred.JobClient:  map 70% reduce 0%
-09/03/24 12:33:59 INFO mapred.JobClient:  map 80% reduce 0%
-09/03/24 12:34:00 INFO mapred.JobClient:  map 90% reduce 0%
-09/03/24 12:34:02 INFO mapred.JobClient:  map 100% reduce 0%
-09/03/24 12:34:10 INFO mapred.JobClient:  map 100% reduce 10%
-09/03/24 12:34:12 INFO mapred.JobClient:  map 100% reduce 13%
-09/03/24 12:34:15 INFO mapred.JobClient:  map 100% reduce 20%
-09/03/24 12:34:20 INFO mapred.JobClient:  map 100% reduce 23%
-09/03/24 12:34:22 INFO mapred.JobClient: Job complete: job_200903232025_0003
-09/03/24 12:34:22 INFO mapred.JobClient: Counters: 16
-09/03/24 12:34:22 INFO mapred.JobClient:   File Systems
-09/03/24 12:34:22 INFO mapred.JobClient:     HDFS bytes read=48245
-09/03/24 12:34:22 INFO mapred.JobClient:     HDFS bytes written=1907
-09/03/24 12:34:22 INFO mapred.JobClient:     Local bytes read=1549
-09/03/24 12:34:22 INFO mapred.JobClient:     Local bytes written=3584
-09/03/24 12:34:22 INFO mapred.JobClient:   Job Counters 
-......
-}}}
-
- 
- * 接著查看結果
-
-{{{
-  /opt/hadoop$ bin/hadoop fs -ls grep_output
-  /opt/hadoop$ bin/hadoop fs -cat grep_output/part-00000 
-}}}
-
- 結果如下
-
-{{{
-3       dfs.class
-3       dfs.
-2       dfs.period
-1       dfs.http.address
-1       dfs.balance.bandwidth
-1       dfs.block.size
-1       dfs.blockreport.initial
-1       dfs.blockreport.interval
-1       dfs.client.block.write.retries
-1       dfs.client.buffer.dir
-1       dfs.data.dir
-1       dfs.datanode.address
-1       dfs.datanode.dns.interface
-1       dfs.datanode.dns.nameserver
-1       dfs.datanode.du.pct
-1       dfs.datanode.du.reserved
-1       dfs.datanode.handler.count
-1       dfs.datanode.http.address
-1       dfs.datanode.https.address
-1       dfs.datanode.ipc.address
-1       dfs.default.chunk.view.size
-1       dfs.df.interval
-1       dfs.file
-1       dfs.heartbeat.interval
-1       dfs.hosts
-1       dfs.hosts.exclude
-1       dfs.https.address
-1       dfs.impl
-1       dfs.max.objects
-1       dfs.name.dir
-1       dfs.namenode.decommission.interval
-1       dfs.namenode.decommission.interval.
-1       dfs.namenode.decommission.nodes.per.interval
-1       dfs.namenode.handler.count
-1       dfs.namenode.logging.level
-1       dfs.permissions
-1       dfs.permissions.supergroup
-1       dfs.replication
-1       dfs.replication.consider
-1       dfs.replication.interval
-1       dfs.replication.max
-1       dfs.replication.min
-1       dfs.replication.min.
-1       dfs.safemode.extension
-1       dfs.safemode.threshold.pct
-1       dfs.secondary.http.address
-1       dfs.servers
-1       dfs.web.ugi
-1       dfsmetrics.log
-
- }}}
-
- === 2.2 Hadoop運算命令 WordCount ===
- 
- * 如名稱，WordCount會對所有的字作字數統計，並且從a-z作排列
- 
- {{{
- /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar wordcount input wc_output
- }}}
- 
- 檢查輸出結果的方法同2.1的方法
-
- === 2.3 更多運算命令 ===
- 
- 可執行的指令一覽表：
-
- || aggregatewordcount ||  An Aggregate based map/reduce program that counts the words in the input files. || 
- || aggregatewordhist || An Aggregate based map/reduce program that computes the histogram of the words in the input files. || 
- || grep ||  A map/reduce program that counts the matches of a regex in the input. || 
- || join || A job that effects a join over sorted, equally partitioned datasets || 
- || multifilewc ||  A job that counts words from several files. || 
- || pentomino  || A map/reduce tile laying program to find solutions to pentomino problems. || 
- || pi ||  A map/reduce program that estimates Pi using monte-carlo method. || 
- || randomtextwriter ||  A map/reduce program that writes 10GB of random textual data per node. || 
- || randomwriter || A map/reduce program that writes 10GB of random data per node. || 
- || sleep ||  A job that sleeps at each map and reduce task. || 
- || sort || A map/reduce program that sorts the data written by the random writer. || 
- || sudoku ||  A sudoku solver. || 
- || wordcount || A map/reduce program that counts the words in the input files. || 
-
- 請參考 [http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/package-summary.html org.apache.hadoop.examples]
-
-{{{
-#!html
-<html lang="zh-tw"><head>
-
-<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>a.html</title>
-
-</head><body>
-<br>
-
-<p>
-</p><table summary="" border="1" cellpadding="3" cellspacing="0" width="100%">
-<tbody><tr class="TableHeadingColor" bgcolor="#ccccff">
-<th colspan="2" align="left"><font size="+2">
-<b>Class Summary</b></font></th>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/AggregateWordCount.html" title="class in org.apache.hadoop.examples">AggregateWordCount</a></b></td>
-<td>This is an example Aggregated Hadoop Map/Reduce application. It
-reads the text input files, breaks each line into words and counts
-them. The output is a locally sorted list of words and the count of how
-often they occurred. To run: bin/hadoop jar hadoop-*-examples.jar
-aggregatewordcount in-dir out-dir numOfReducers textinputformat </td>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/AggregateWordHistogram.html" title="class in org.apache.hadoop.examples">AggregateWordHistogram</a></b></td>
-<td>This is an example Aggregated Hadoop Map/Reduce application.
-Computes the histogram of the words in the input texts. To run:
-bin/hadoop jar hadoop-*-examples.jar aggregatewordhist in-dir out-dir
-numOfReducers textinputformat </td>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/ExampleDriver.html" title="class in org.apache.hadoop.examples">ExampleDriver</a></b></td>
-<td>A description of an example program based on its class and a human-readable description.</td>
-</tr>
-
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/Grep.html" title="class in org.apache.hadoop.examples">Grep</a></b></td>
-<td>&nbsp;</td>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/Join.html" title="class in org.apache.hadoop.examples">Join</a></b></td>
-<td>This is the trivial map/reduce program that does absolutely nothing
-other than use the framework to fragment and sort the input values. To
-run: bin/hadoop jar build/hadoop-examples.jar join [-m maps] [-r
-reduces] [-inFormat input format class] [-outFormat output format
-class] [-outKey output key class] [-outValue output value class]
-[-joinOp] [in-dir]* in-dir out-dir</inner></td>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/RandomTextWriter.html" title="class in org.apache.hadoop.examples">RandomTextWriter</a></b></td>
-<td>This program uses map/reduce to just run a distributed job where
-there is
-no interaction between the tasks and each task writes a large unsorted
-random sequence of words.To run: bin/hadoop jar
-hadoop-${version}-examples.jar randomtextwriter [-outFormat output
-format class] output</td>
-
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/RandomWriter.html" title="class in org.apache.hadoop.examples">RandomWriter</a></b></td>
-<td>This program uses map/reduce to just run a distributed job where
-there is
-no interaction between the tasks and each task write a large unsorted
-random binary sequence file of BytesWritable.To run: bin/hadoop jar
-hadoop-${version}-examples.jar randomwriter [-outFormat output format
-class] output</td>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/Sort.html" title="class in org.apache.hadoop.examples">Sort&lt;K,V&gt;</a></b></td>
-<td>This is the trivial map/reduce program that does absolutely nothing
-other than use the framework to fragment and sort the input values.To
-run: bin/hadoop jar build/hadoop-examples.jar sort [-m maps] [-r
-reduces] [-inFormat input format class] [-outFormat output format
-class] [-outKey output key class] [-outValue output value class]
-[-totalOrder pcnt num samples max splits] in-dir out-dir</td>
-</tr>
-<tr class="TableRowColor" bgcolor="white">
-<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/WordCount.html" title="class in org.apache.hadoop.examples">WordCount</a></b></td>
-
-<td>This is an example Hadoop Map/Reduce application.</td>
-</tr>
-</tbody></table>
-</body></html>
-}}} 
- 
-
- == Content 3. 使用網頁Gui瀏覽資訊 ==
+ == Content 2. 使用網頁Gui瀏覽資訊 ==
  
  * [http://localhost:50030 Map/Reduce Administration]
  * [http://localhost:50070 NameNode ]
  
- == Content 4. 練習 ==
- 
- * 刪除在 hdfs 內的一整個的資料夾 input
- * 用網頁秀出你在 wordcount練習的輸出結果[[BR]]
- [[Image(2009-03-24-135001_872x741_scrot.png)]]