Changes between Initial Version and Version 1 of ITRI0521/Lab5


Ignore:
Timestamp:
May 21, 2013, 12:19:12 AM (12 years ago)
Author:
jazz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ITRI0521/Lab5

    v1 v1  
     1[[PageOutline]]
     2
     3◢ <[wiki:ITRI0521/Lab4 實作四]> | <[wiki:ITRI0521 回課程大綱]> ▲ | <[wiki:ITRI0521/Lab6 實作六]> ◣
     4
     5= 實作五 Lab 5 =
     6
     7{{{
     8#!html
     9<div style="text-align: center;"><big style="font-weight: bold;"><big>在單機模式執行 MapReduce 基本運算<br/>Running MapReduce in local mode by Examples</big></big></div>
     10}}}
     11
     12{{{
     13#!text
     14以下練習，請在本機的 Hadoop4Win 環境操作。
     15}}}
     16
     17== 範例一『字數統計(WordCount)』 ==
     18
     19 * STEP 1 : 練習 MapReduce 丟 Job 指令: 『__'''hadoop jar <local jar file> <class name> <parameters>'''__』
     20{{{
     21Jazz@human ~
     22$ cd /opt/hadoop/
     23
     24Jazz@human /opt/hadoop
     25$ hadoop jar hadoop-*-examples.jar wordcount input output
     2611/10/21 14:08:58 INFO input.FileInputFormat: Total input paths to process : 12
     2711/10/21 14:09:00 INFO mapred.JobClient: Running job: job_201110211130_0001
     2811/10/21 14:09:01 INFO mapred.JobClient:  map 0% reduce 0%
     2911/10/21 14:09:31 INFO mapred.JobClient:  map 16% reduce 0%
     3011/10/21 14:10:29 INFO mapred.JobClient:  map 100% reduce 27%
     3111/10/21 14:10:33 INFO mapred.JobClient:  map 100% reduce 100%
     3211/10/21 14:10:35 INFO mapred.JobClient: Job complete: job_201110211130_0001
     3311/10/21 14:10:35 INFO mapred.JobClient: Counters: 17
     3411/10/21 14:10:35 INFO mapred.JobClient:   Job Counters
     3511/10/21 14:10:35 INFO mapred.JobClient:     Launched reduce tasks=1
     3611/10/21 14:10:35 INFO mapred.JobClient:     Launched map tasks=12
     3711/10/21 14:10:35 INFO mapred.JobClient:     Data-local map tasks=12
     3811/10/21 14:10:35 INFO mapred.JobClient:   FileSystemCounters
     3911/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_READ=16578
     4011/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     4111/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=32636
     4211/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=10922
     4311/10/21 14:10:35 INFO mapred.JobClient:   Map-Reduce Framework
     4411/10/21 14:10:35 INFO mapred.JobClient:     Reduce input groups=592
     4511/10/21 14:10:35 INFO mapred.JobClient:     Combine output records=750
     4611/10/21 14:10:35 INFO mapred.JobClient:     Map input records=553
     4711/10/21 14:10:35 INFO mapred.JobClient:     Reduce shuffle bytes=15674
     4811/10/21 14:10:35 INFO mapred.JobClient:     Reduce output records=592
     4911/10/21 14:10:35 INFO mapred.JobClient:     Spilled Records=1500
     5011/10/21 14:10:35 INFO mapred.JobClient:     Map output bytes=24438
     5111/10/21 14:10:35 INFO mapred.JobClient:     Combine input records=1755
     5211/10/21 14:10:35 INFO mapred.JobClient:     Map output records=1755
     5311/10/21 14:10:35 INFO mapred.JobClient:     Reduce input records=750
     54}}}
     55   * [[BR]][[Image(Hadoop4Win:hadoop4win_14.jpg,width=600)]]
     56
     57 * STEP 2 : 練習從 http://localhost:50030 查看目前 MapReduce Job 的運作情形
     58   * [[BR]][[Image(Hadoop4Win:hadoop4win_15.jpg,width=600)]]
     59
     60 * STEP 3 : 使用 HDFS 指令: 『__'''hadoop fs -get <HDFS file/dir> <local file/dir>'''__』，並了解輸出檔案檔名均為 part-r-*****，且執行參數會紀錄於 <HOSTNAME>_<TIME>_job_<JOBID>_0001_conf.xml，不妨可以觀察 xml 內容與 hadoop config 檔的參數關聯。
     61{{{
     62Jazz@human /opt/hadoop
     63$ hadoop fs -get output my_output
     64
     65Jazz@human /opt/hadoop
     66$ ls -alR my_output
     67my_output:
     68total 12
     69drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 .
     70drwxr-xr-x+ 15 Jazz None     0 Oct 21 14:12 ..
     71drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 _logs
     72-rwxr-xr-x   1 Jazz None 10922 Oct 21 14:12 part-r-00000
     73
     74my_output/_logs:
     75total 0
     76drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 .
     77drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 ..
     78drwxr-xr-x+ 2 Jazz None 0 Oct 21 14:12 history
     79
     80my_output/_logs/history:
     81total 48
     82drwxr-xr-x+ 2 Jazz None     0 Oct 21 14:12 .
     83drwxr-xr-x+ 3 Jazz None     0 Oct 21 14:12 ..
     84-rwxr-xr-x  1 Jazz None 26004 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_Jazz_word+count
     85-rwxr-xr-x  1 Jazz None 16984 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_conf.xml
     86}}}
     87   * [[BR]][[Image(Hadoop4Win:hadoop4win_22.jpg,width=600)]]
     88
     89== 範例二『用標準表示法過濾內容 grep』 ==
     90
     91 * grep 這個命令是擷取文件裡面特定的字元，在 Hadoop example 中此指令可以擷取文件中有此指定文字的字串，並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings.
     92{{{
     93Jazz@human /opt/hadoop
     94$ hadoop jar hadoop-*-examples.jar  grep input lab5_out1 'dfs[a-z.]+'
     95}}}
     96 * 運作的畫面如下：[[BR]]You should see procedure like this:
     97{{{
     98Jazz@human /opt/hadoop
     99$ hadoop jar hadoop-*-examples.jar  grep input lab5_out1 'dfs[a-z.]+'
     10011/10/21 14:17:39 INFO mapred.FileInputFormat: Total input paths to process : 12
     101
     10211/10/21 14:17:39 INFO mapred.JobClient: Running job: job_201110211130_0002
     10311/10/21 14:17:40 INFO mapred.JobClient:  map 0% reduce 0%
     10411/10/21 14:17:54 INFO mapred.JobClient:  map 8% reduce 0%
     10511/10/21 14:17:57 INFO mapred.JobClient:  map 16% reduce 0%
     10611/10/21 14:18:03 INFO mapred.JobClient:  map 33% reduce 0%
     10711/10/21 14:18:13 INFO mapred.JobClient:  map 41% reduce 0%
     10811/10/21 14:18:16 INFO mapred.JobClient:  map 50% reduce 11%
     10911/10/21 14:18:19 INFO mapred.JobClient:  map 58% reduce 11%
     11011/10/21 14:18:23 INFO mapred.JobClient:  map 66% reduce 11%
     11111/10/21 14:18:30 INFO mapred.JobClient:  map 83% reduce 16%
     11211/10/21 14:18:33 INFO mapred.JobClient:  map 83% reduce 22%
     11311/10/21 14:18:36 INFO mapred.JobClient:  map 91% reduce 22%
     11411/10/21 14:18:39 INFO mapred.JobClient:  map 100% reduce 22%
     11511/10/21 14:18:42 INFO mapred.JobClient:  map 100% reduce 27%
     11611/10/21 14:18:48 INFO mapred.JobClient:  map 100% reduce 30%
     11711/10/21 14:18:54 INFO mapred.JobClient:  map 100% reduce 100%
     11811/10/21 14:18:56 INFO mapred.JobClient: Job complete: job_201110211130_0002
     11911/10/21 14:18:56 INFO mapred.JobClient: Counters: 18
     12011/10/21 14:18:56 INFO mapred.JobClient:   Job Counters
     12111/10/21 14:18:56 INFO mapred.JobClient:     Launched reduce tasks=1
     12211/10/21 14:18:56 INFO mapred.JobClient:     Launched map tasks=12
     12311/10/21 14:18:56 INFO mapred.JobClient:     Data-local map tasks=12
     12411/10/21 14:18:56 INFO mapred.JobClient:   FileSystemCounters
     12511/10/21 14:18:56 INFO mapred.JobClient:     FILE_BYTES_READ=888
     12611/10/21 14:18:56 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     12711/10/21 14:18:56 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=1496
     12811/10/21 14:18:56 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=280
     12911/10/21 14:18:56 INFO mapred.JobClient:   Map-Reduce Framework
     13011/10/21 14:18:56 INFO mapred.JobClient:     Reduce input groups=7
     13111/10/21 14:18:56 INFO mapred.JobClient:     Combine output records=7
     13211/10/21 14:18:56 INFO mapred.JobClient:     Map input records=553
     13311/10/21 14:18:56 INFO mapred.JobClient:     Reduce shuffle bytes=224
     13411/10/21 14:18:56 INFO mapred.JobClient:     Reduce output records=7
     13511/10/21 14:18:56 INFO mapred.JobClient:     Spilled Records=14
     13611/10/21 14:18:56 INFO mapred.JobClient:     Map output bytes=193
     13711/10/21 14:18:56 INFO mapred.JobClient:     Map input bytes=18312
     13811/10/21 14:18:56 INFO mapred.JobClient:     Combine input records=10
     13911/10/21 14:18:56 INFO mapred.JobClient:     Map output records=10
     14011/10/21 14:18:56 INFO mapred.JobClient:     Reduce input records=7
     14111/10/21 14:18:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
     142e arguments. Applications should implement Tool for the same.
     14311/10/21 14:18:57 INFO mapred.FileInputFormat: Total input paths to process : 1
     14411/10/21 14:18:57 INFO mapred.JobClient: Running job: job_201110211130_0003
     145( ... skip ... )
     146}}}
     147 * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS :
     148 * 這個例子是要從 input 目錄中的所有檔案中找出符合 dfs 後面接著 a-z 字母一個以上的字串
     149{{{
     150Jazz@human /opt/hadoop
     151$ hadoop fs -ls lab5_out1
     152Found 2 items
     153drwxr-xr-x   - Jazz supergroup          0 2011-10-21 14:18 /user/Jazz/lab5_out1/_logs
     154-rw-r--r--   1 Jazz supergroup         96 2011-10-21 14:19 /user/Jazz/lab5_out1/part-00000
     155
     156Jazz@human /opt/hadoop
     157$ hadoop fs -cat lab5_out1/part-00000
     1583       dfs.class
     1592       dfs.period
     1601       dfs.file
     1611       dfs.replication
     1621       dfs.servers
     1631       dfsadmin
     1641       dfsmetrics.log
     165}}}