Changes between Initial Version and Version 1 of III110813/Lab5


Ignore:
Timestamp:
Oct 21, 2011, 2:13:29 PM (13 years ago)
Author:
jazz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • III110813/Lab5

    v1 v1  
     1[[PageOutline]]
     2
     3◢ <[wiki:III110813/Lab3 實作三]> | <[wiki:III110813 回課程大綱]> ▲ | <[wiki:III110813/Lab5 實作五]> ◣
     4
     5= 實作五 Lab 5 =
     6
     7{{{
     8#!html
     9<div style="text-align: center;"><big style="font-weight: bold;"><big>在單機模式執行 MapReduce 基本運算<br/>Running MapReduce in local mode by Examples</big></big></div>
     10}}}
     11
     12== MapReduce 範例一『字數統計(WordCount)』 ==
     13
     14 * STEP 1 : 練習 MapReduce 丟 Job 指令: 『__'''hadoop jar <local jar file> <class name> <parameters>'''__』
     15{{{
     16Jazz@human ~
     17$ cd /opt/hadoop/
     18
     19Jazz@human /opt/hadoop
     20$ hadoop jar hadoop-*-examples.jar wordcount input output
     2111/10/21 14:08:58 INFO input.FileInputFormat: Total input paths to process : 12
     2211/10/21 14:09:00 INFO mapred.JobClient: Running job: job_201110211130_0001
     2311/10/21 14:09:01 INFO mapred.JobClient:  map 0% reduce 0%
     2411/10/21 14:09:31 INFO mapred.JobClient:  map 16% reduce 0%
     2511/10/21 14:10:29 INFO mapred.JobClient:  map 100% reduce 27%
     2611/10/21 14:10:33 INFO mapred.JobClient:  map 100% reduce 100%
     2711/10/21 14:10:35 INFO mapred.JobClient: Job complete: job_201110211130_0001
     2811/10/21 14:10:35 INFO mapred.JobClient: Counters: 17
     2911/10/21 14:10:35 INFO mapred.JobClient:   Job Counters
     3011/10/21 14:10:35 INFO mapred.JobClient:     Launched reduce tasks=1
     3111/10/21 14:10:35 INFO mapred.JobClient:     Launched map tasks=12
     3211/10/21 14:10:35 INFO mapred.JobClient:     Data-local map tasks=12
     3311/10/21 14:10:35 INFO mapred.JobClient:   FileSystemCounters
     3411/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_READ=16578
     3511/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     3611/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=32636
     3711/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=10922
     3811/10/21 14:10:35 INFO mapred.JobClient:   Map-Reduce Framework
     3911/10/21 14:10:35 INFO mapred.JobClient:     Reduce input groups=592
     4011/10/21 14:10:35 INFO mapred.JobClient:     Combine output records=750
     4111/10/21 14:10:35 INFO mapred.JobClient:     Map input records=553
     4211/10/21 14:10:35 INFO mapred.JobClient:     Reduce shuffle bytes=15674
     4311/10/21 14:10:35 INFO mapred.JobClient:     Reduce output records=592
     4411/10/21 14:10:35 INFO mapred.JobClient:     Spilled Records=1500
     4511/10/21 14:10:35 INFO mapred.JobClient:     Map output bytes=24438
     4611/10/21 14:10:35 INFO mapred.JobClient:     Combine input records=1755
     4711/10/21 14:10:35 INFO mapred.JobClient:     Map output records=1755
     4811/10/21 14:10:35 INFO mapred.JobClient:     Reduce input records=750
     49}}}
     50   * [[BR]][[Image(Hadoop4Win:hadoop4win_14.jpg,width=600)]]
     51
     52 * STEP 2 : 練習從 http://localhost:50030 查看目前 MapReduce Job 的運作情形
     53   * [[BR]][[Image(Hadoop4Win:hadoop4win_15.jpg,width=600)]]
     54
     55 * STEP 3 : 使用 HDFS 指令: 『__'''hadoop fs -get <HDFS file/dir> <local file/dir>'''__』，並了解輸出檔案檔名均為 part-r-*****，且執行參數會紀錄於 <HOSTNAME>_<TIME>_job_<JOBID>_0001_conf.xml，不妨可以觀察 xml 內容與 hadoop config 檔的參數關聯。
     56{{{
     57Jazz@human /opt/hadoop
     58$ hadoop fs -get output my_output
     59
     60Jazz@human /opt/hadoop
     61$ ls -alR my_output
     62my_output:
     63total 12
     64drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 .
     65drwxr-xr-x+ 15 Jazz None     0 Oct 21 14:12 ..
     66drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 _logs
     67-rwxr-xr-x   1 Jazz None 10922 Oct 21 14:12 part-r-00000
     68
     69my_output/_logs:
     70total 0
     71drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 .
     72drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 ..
     73drwxr-xr-x+ 2 Jazz None 0 Oct 21 14:12 history
     74
     75my_output/_logs/history:
     76total 48
     77drwxr-xr-x+ 2 Jazz None     0 Oct 21 14:12 .
     78drwxr-xr-x+ 3 Jazz None     0 Oct 21 14:12 ..
     79-rwxr-xr-x  1 Jazz None 26004 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_Jazz_word+count
     80-rwxr-xr-x  1 Jazz None 16984 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_conf.xml
     81}}}
     82   * [[BR]][[Image(Hadoop4Win:hadoop4win_22.jpg,width=600)]]