Changes between Version 1 and Version 2 of waue/2011/0426_1

Timestamp:

Apr 25, 2011, 10:11:09 PM (14 years ago)

Author:

waue

Comment:

--

waue/2011/0426_1

@@ -16 +16 @@
  * 範例一：使用 cat 當 mapper，使用 wc 當 reducer
 {{{
-hadoop@lucid:~$ cd /opt/hadoop
-hadoop@lucid:/opt/hadoop$ bin/start-all.sh
-hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
-hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
+$ cd /opt/hadoop
+$ bin/start-all.sh
+$ bin/hadoop fs -put conf input
+$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
 }}}
  * 範例二：使用 Bash Shell Script 當 Mapper 與 Reducer
 {{{
-hadoop@lucid:/opt/hadoop$ echo "sed -e \"s/ /\n/g\" | grep ." > streamingMapper.sh
-hadoop@lucid:/opt/hadoop$ echo "uniq -c | awk '{print \$2 \"\t\" \$1}'" > streamingReducer.sh
-hadoop@lucid:/opt/hadoop$ chmod a+x streamingMapper.sh
-hadoop@lucid:/opt/hadoop$ chmod a+x streamingReducer.sh
-hadoop@lucid:/opt/hadoop$ bin/hadoop fs -rmr input output
-hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
-hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
+$ echo "sed -e \"s/ /\n/g\" | grep ." > streamingMapper.sh
+$ echo "uniq -c | awk '{print \$2 \"\t\" \$1}'" > streamingReducer.sh
+$ chmod 755 streaming*.sh
+$ bin/hadoop fs -rmr input output
+$ bin/hadoop fs -put conf input
+$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
 }}}