Changes between Initial Version and Version 1 of waue/2011/0426_1

Timestamp:

Apr 25, 2011, 3:08:48 PM (13 years ago)

Author:

waue

Comment:

--

waue/2011/0426_1

@@ +1 @@
+
+{{{
+#!html
+<div style="text-align: center; color:#151B8D"><big style="font-weight: bold;"><big><big>
+ITRI HBase 進階課程 
+</big></big></big></div> <div style="text-align: center; color:#7E2217"><big style="font-weight: bold;"><big>
+1. Streaming 
+</big></big></div>
+}}}
+
+[wiki:waue/2011/0426 回課程大綱 << ] Stream 範例
+
+
+== Hadoop Streaming with commands ==
+
+ * 範例一：使用 cat 當 mapper，使用 wc 當 reducer
+{{{
+hadoop@lucid:~$ cd /opt/hadoop
+hadoop@lucid:/opt/hadoop$ bin/start-all.sh
+hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
+hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
+}}}
+ * 範例二：使用 Bash Shell Script 當 Mapper 與 Reducer
+{{{
+hadoop@lucid:/opt/hadoop$ echo "sed -e \"s/ /\n/g\" | grep ." > streamingMapper.sh
+hadoop@lucid:/opt/hadoop$ echo "uniq -c | awk '{print \$2 \"\t\" \$1}'" > streamingReducer.sh
+hadoop@lucid:/opt/hadoop$ chmod a+x streamingMapper.sh
+hadoop@lucid:/opt/hadoop$ chmod a+x streamingReducer.sh
+hadoop@lucid:/opt/hadoop$ bin/hadoop fs -rmr input output
+hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
+hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh
+}}}