Changes between Initial Version and Version 1 of Hinet120702/Lab5


Ignore:
Timestamp:
Jul 2, 2012, 1:41:14 AM (12 years ago)
Author:
jazz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Hinet120702/Lab5

    v1 v1  
     1◢ <[wiki:Hinet120702/Lab4 實作四]> | <[wiki:Hinet120702 回課程大綱]> ▲ | <[wiki:Hinet120702/Lab6 實作六]> ◣
     2
     3= 實作五 Lab 5 =
     4[[PageOutline]]
     5{{{
     6#!html
     7<div style="text-align: center;"><big style="font-weight: bold;"><big>在單機模式執行 MapReduce 基本運算<br/>Running MapReduce in local mode by Examples</big></big></div>
     8}}}
     9
     10{{{
     11#!text
     12以下練習，請在本機的 Hadoop4Win 環境操作。
     13}}}
     14
     15== 範例一『字數統計(WordCount)』 ==
     16
     17 * STEP 1 : 練習 MapReduce 丟 Job 指令: 『__'''hadoop jar <local jar file> <class name> <parameters>'''__』
     18{{{
     19Jazz@human ~
     20$ cd /opt/hadoop/
     21
     22Jazz@human /opt/hadoop
     23$ hadoop jar hadoop-*-examples.jar wordcount input output
     2411/10/21 14:08:58 INFO input.FileInputFormat: Total input paths to process : 12
     2511/10/21 14:09:00 INFO mapred.JobClient: Running job: job_201110211130_0001
     2611/10/21 14:09:01 INFO mapred.JobClient:  map 0% reduce 0%
     2711/10/21 14:09:31 INFO mapred.JobClient:  map 16% reduce 0%
     2811/10/21 14:10:29 INFO mapred.JobClient:  map 100% reduce 27%
     2911/10/21 14:10:33 INFO mapred.JobClient:  map 100% reduce 100%
     3011/10/21 14:10:35 INFO mapred.JobClient: Job complete: job_201110211130_0001
     3111/10/21 14:10:35 INFO mapred.JobClient: Counters: 17
     3211/10/21 14:10:35 INFO mapred.JobClient:   Job Counters
     3311/10/21 14:10:35 INFO mapred.JobClient:     Launched reduce tasks=1
     3411/10/21 14:10:35 INFO mapred.JobClient:     Launched map tasks=12
     3511/10/21 14:10:35 INFO mapred.JobClient:     Data-local map tasks=12
     3611/10/21 14:10:35 INFO mapred.JobClient:   FileSystemCounters
     3711/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_READ=16578
     3811/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     3911/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=32636
     4011/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=10922
     4111/10/21 14:10:35 INFO mapred.JobClient:   Map-Reduce Framework
     4211/10/21 14:10:35 INFO mapred.JobClient:     Reduce input groups=592
     4311/10/21 14:10:35 INFO mapred.JobClient:     Combine output records=750
     4411/10/21 14:10:35 INFO mapred.JobClient:     Map input records=553
     4511/10/21 14:10:35 INFO mapred.JobClient:     Reduce shuffle bytes=15674
     4611/10/21 14:10:35 INFO mapred.JobClient:     Reduce output records=592
     4711/10/21 14:10:35 INFO mapred.JobClient:     Spilled Records=1500
     4811/10/21 14:10:35 INFO mapred.JobClient:     Map output bytes=24438
     4911/10/21 14:10:35 INFO mapred.JobClient:     Combine input records=1755
     5011/10/21 14:10:35 INFO mapred.JobClient:     Map output records=1755
     5111/10/21 14:10:35 INFO mapred.JobClient:     Reduce input records=750
     52}}}
     53   * [[BR]][[Image(Hadoop4Win:hadoop4win_14.jpg,width=600)]]
     54
     55 * STEP 2 : 練習從 http://localhost:50030 查看目前 MapReduce Job 的運作情形
     56   * [[BR]][[Image(Hadoop4Win:hadoop4win_15.jpg,width=600)]]
     57
     58 * STEP 3 : 使用 HDFS 指令: 『__'''hadoop fs -get <HDFS file/dir> <local file/dir>'''__』，並了解輸出檔案檔名均為 part-r-*****，且執行參數會紀錄於 <HOSTNAME>_<TIME>_job_<JOBID>_0001_conf.xml，不妨可以觀察 xml 內容與 hadoop config 檔的參數關聯。
     59{{{
     60Jazz@human /opt/hadoop
     61$ hadoop fs -get output my_output
     62
     63Jazz@human /opt/hadoop
     64$ ls -alR my_output
     65my_output:
     66total 12
     67drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 .
     68drwxr-xr-x+ 15 Jazz None     0 Oct 21 14:12 ..
     69drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 _logs
     70-rwxr-xr-x   1 Jazz None 10922 Oct 21 14:12 part-r-00000
     71
     72my_output/_logs:
     73total 0
     74drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 .
     75drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 ..
     76drwxr-xr-x+ 2 Jazz None 0 Oct 21 14:12 history
     77
     78my_output/_logs/history:
     79total 48
     80drwxr-xr-x+ 2 Jazz None     0 Oct 21 14:12 .
     81drwxr-xr-x+ 3 Jazz None     0 Oct 21 14:12 ..
     82-rwxr-xr-x  1 Jazz None 26004 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_Jazz_word+count
     83-rwxr-xr-x  1 Jazz None 16984 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_conf.xml
     84}}}
     85   * [[BR]][[Image(Hadoop4Win:hadoop4win_22.jpg,width=600)]]
     86
     87== 範例二『用標準表示法過濾內容 grep』 ==
     88
     89 * grep 這個命令是擷取文件裡面特定的字元，在 Hadoop example 中此指令可以擷取文件中有此指定文字的字串，並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings.
     90{{{
     91Jazz@human /opt/hadoop
     92$ hadoop jar hadoop-*-examples.jar  grep input lab5_out1 'dfs[a-z.]+'
     93}}}
     94 * 運作的畫面如下：[[BR]]You should see procedure like this:
     95{{{
     96Jazz@human /opt/hadoop
     97$ hadoop jar hadoop-*-examples.jar  grep input lab5_out1 'dfs[a-z.]+'
     9811/10/21 14:17:39 INFO mapred.FileInputFormat: Total input paths to process : 12
     99
     10011/10/21 14:17:39 INFO mapred.JobClient: Running job: job_201110211130_0002
     10111/10/21 14:17:40 INFO mapred.JobClient:  map 0% reduce 0%
     10211/10/21 14:17:54 INFO mapred.JobClient:  map 8% reduce 0%
     10311/10/21 14:17:57 INFO mapred.JobClient:  map 16% reduce 0%
     10411/10/21 14:18:03 INFO mapred.JobClient:  map 33% reduce 0%
     10511/10/21 14:18:13 INFO mapred.JobClient:  map 41% reduce 0%
     10611/10/21 14:18:16 INFO mapred.JobClient:  map 50% reduce 11%
     10711/10/21 14:18:19 INFO mapred.JobClient:  map 58% reduce 11%
     10811/10/21 14:18:23 INFO mapred.JobClient:  map 66% reduce 11%
     10911/10/21 14:18:30 INFO mapred.JobClient:  map 83% reduce 16%
     11011/10/21 14:18:33 INFO mapred.JobClient:  map 83% reduce 22%
     11111/10/21 14:18:36 INFO mapred.JobClient:  map 91% reduce 22%
     11211/10/21 14:18:39 INFO mapred.JobClient:  map 100% reduce 22%
     11311/10/21 14:18:42 INFO mapred.JobClient:  map 100% reduce 27%
     11411/10/21 14:18:48 INFO mapred.JobClient:  map 100% reduce 30%
     11511/10/21 14:18:54 INFO mapred.JobClient:  map 100% reduce 100%
     11611/10/21 14:18:56 INFO mapred.JobClient: Job complete: job_201110211130_0002
     11711/10/21 14:18:56 INFO mapred.JobClient: Counters: 18
     11811/10/21 14:18:56 INFO mapred.JobClient:   Job Counters
     11911/10/21 14:18:56 INFO mapred.JobClient:     Launched reduce tasks=1
     12011/10/21 14:18:56 INFO mapred.JobClient:     Launched map tasks=12
     12111/10/21 14:18:56 INFO mapred.JobClient:     Data-local map tasks=12
     12211/10/21 14:18:56 INFO mapred.JobClient:   FileSystemCounters
     12311/10/21 14:18:56 INFO mapred.JobClient:     FILE_BYTES_READ=888
     12411/10/21 14:18:56 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     12511/10/21 14:18:56 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=1496
     12611/10/21 14:18:56 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=280
     12711/10/21 14:18:56 INFO mapred.JobClient:   Map-Reduce Framework
     12811/10/21 14:18:56 INFO mapred.JobClient:     Reduce input groups=7
     12911/10/21 14:18:56 INFO mapred.JobClient:     Combine output records=7
     13011/10/21 14:18:56 INFO mapred.JobClient:     Map input records=553
     13111/10/21 14:18:56 INFO mapred.JobClient:     Reduce shuffle bytes=224
     13211/10/21 14:18:56 INFO mapred.JobClient:     Reduce output records=7
     13311/10/21 14:18:56 INFO mapred.JobClient:     Spilled Records=14
     13411/10/21 14:18:56 INFO mapred.JobClient:     Map output bytes=193
     13511/10/21 14:18:56 INFO mapred.JobClient:     Map input bytes=18312
     13611/10/21 14:18:56 INFO mapred.JobClient:     Combine input records=10
     13711/10/21 14:18:56 INFO mapred.JobClient:     Map output records=10
     13811/10/21 14:18:56 INFO mapred.JobClient:     Reduce input records=7
     13911/10/21 14:18:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
     140e arguments. Applications should implement Tool for the same.
     14111/10/21 14:18:57 INFO mapred.FileInputFormat: Total input paths to process : 1
     14211/10/21 14:18:57 INFO mapred.JobClient: Running job: job_201110211130_0003
     143( ... skip ... )
     144}}}
     145 * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS :
     146 * 這個例子是要從 input 目錄中的所有檔案中找出符合 dfs 後面接著 a-z 字母一個以上的字串
     147{{{
     148Jazz@human /opt/hadoop
     149$ hadoop fs -ls lab5_out1
     150Found 2 items
     151drwxr-xr-x   - Jazz supergroup          0 2011-10-21 14:18 /user/Jazz/lab5_out1/_logs
     152-rw-r--r--   1 Jazz supergroup         96 2011-10-21 14:19 /user/Jazz/lab5_out1/part-00000
     153
     154Jazz@human /opt/hadoop
     155$ hadoop fs -cat lab5_out1/part-00000
     1563       dfs.class
     1572       dfs.period
     1581       dfs.file
     1591       dfs.replication
     1601       dfs.servers
     1611       dfsadmin
     1621       dfsmetrics.log
     163}}}