Changes between Version 3 and Version 4 of NCHCCloudCourse100928/Lab1


Ignore:
Timestamp:
Sep 28, 2010, 9:21:13 AM (14 years ago)
Author:
jazz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • NCHCCloudCourse100928/Lab1

    v3 v4  
    3131== Hadoop Streaming with commands ==
    3232
     33 * 範例一：使用 cat 當 mapper，使用 wc 當 reducer
    3334{{{
    3435hadoop@lucid:~$ cd /opt/hadoop
    35 hadoop@lucid:/opt/hadoop$ hadoop fs -rmr input output
    36 hadoop@lucid:/opt/hadoop$ hadoop fs -put /etc/hadoop/conf input
    37 hadoop@lucid:/opt/hadoop$ hadoop jar hadoop-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
     36hadoop@lucid:/opt/hadoop$ bin/start-all.sh
     37hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
     38hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
    3839}}}
    39 
     40 * 範例二：使用 Bash Shell Script 當 Mapper 與 Reducer
    4041{{{
    4142hadoop@lucid:/opt/hadoop$ echo "sed -e \"s/ /\n/g\" | grep ." > streamingMapper.sh
    … …  
    4344hadoop@lucid:/opt/hadoop$ chmod a+x streamingMapper.sh
    4445hadoop@lucid:/opt/hadoop$ chmod a+x streamingReducer.sh
    45 hadoop@lucid:/opt/hadoop$ hadoop fs -rmr input output
    46 hadoop@lucid:/opt/hadoop$ hadoop fs -put /etc/hadoop/conf input
    47 hadoop@lucid:/opt/hadoop$ hadoop jar hadoop-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
     46hadoop@lucid:/opt/hadoop$ bin/hadoop fs -rmr input output
     47hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
     48hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
    4849}}}