Changes between Version 3 and Version 4 of NCHCCloudCourse100928/Lab1

Timestamp:

Sep 28, 2010, 9:21:13 AM (15 years ago)

Author:

jazz

Comment:

--

NCHCCloudCourse100928/Lab1

@@ -31 +31 @@
 == Hadoop Streaming with commands ==
 
+ * 範例一：使用 cat 當 mapper，使用 wc 當 reducer
 {{{
 hadoop@lucid:~$ cd /opt/hadoop
-hadoop@lucid:/opt/hadoop$ hadoop fs -rmr input output
-hadoop@lucid:/opt/hadoop$ hadoop fs -put /etc/hadoop/conf input
-hadoop@lucid:/opt/hadoop$ hadoop jar hadoop-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
+hadoop@lucid:/opt/hadoop$ bin/start-all.sh
+hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
+hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
 }}}
-
+ * 範例二：使用 Bash Shell Script 當 Mapper 與 Reducer
 {{{
 hadoop@lucid:/opt/hadoop$ echo "sed -e \"s/ /\n/g\" | grep ." > streamingMapper.sh
@@ -43 +44 @@
 hadoop@lucid:/opt/hadoop$ chmod a+x streamingMapper.sh
 hadoop@lucid:/opt/hadoop$ chmod a+x streamingReducer.sh
-hadoop@lucid:/opt/hadoop$ hadoop fs -rmr input output
-hadoop@lucid:/opt/hadoop$ hadoop fs -put /etc/hadoop/conf input
-hadoop@lucid:/opt/hadoop$ hadoop jar hadoop-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
+hadoop@lucid:/opt/hadoop$ bin/hadoop fs -rmr input output
+hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
+hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
 }}}