ITRI HBase 進階課程

1. Streaming

回課程大綱 << Stream 範例

Hadoop Streaming with commands

• 範例一：使用 cat 當 mapper，使用 wc 當 reducer

  hadoop@lucid:~$ cd /opt/hadoop
  hadoop@lucid:/opt/hadoop$ bin/start-all.sh
  hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
  hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper /bin/cat -reducer /usr/bin/wc
  

• 範例二：使用 Bash Shell Script 當 Mapper 與 Reducer

  hadoop@lucid:/opt/hadoop$ echo "sed -e \"s/ /\n/g\" | grep ." > streamingMapper.sh
  hadoop@lucid:/opt/hadoop$ echo "uniq -c | awk '{print \$2 \"\t\" \$1}'" > streamingReducer.sh
  hadoop@lucid:/opt/hadoop$ chmod a+x streamingMapper.sh
  hadoop@lucid:/opt/hadoop$ chmod a+x streamingReducer.sh
  hadoop@lucid:/opt/hadoop$ bin/hadoop fs -rmr input output
  hadoop@lucid:/opt/hadoop$ bin/hadoop fs -put conf input
  hadoop@lucid:/opt/hadoop$ bin/hadoop jar ./contrib/streaming/hadoop-0.20.2-streaming.jar -input input -output output -mapper streamingMapper.sh -reducer streamingReducer.sh -file streamingMapper.sh -file streamingReducer.sh