wiki:YMU110509/Lab9

Version 3 (modified by jazz, 13 years ago) (diff)

--

◢ <實作八> | <回課程大綱> ▲ | <實作十> ◣

實作九 Lab9

HDFS、MapReduce 與 Hadoop Streaming 觀念驗證
Running Hadoop Streaming with HDFS and MapReduce

準備輸入資料集 Input Dataset

  • 首先準備輸入,包含兩部份:(1) 供 Velvet 計算的 *.fa 檔案,這裡為了方便示範起見,採用 test_long.fa 當範本,並複製 99 份不同檔名,當輸入 (2) 供 Mapper 運算的輸入檔案(內含 HDFS 的檔名路徑)
    ~$ cp /usr/share/doc/velvet-example/examples/data/test_long.fa.gz .
    ~$ gunzip test_long.fa.gz
    ~$ for ((i=1;i<100;i++)); do hadoop fs -put test_long.fa sample-$i.fa; done
    ~$ for ((i=1;i<20;i++)); do echo /user/h998/sample-$i.fa; done > sample-01.txt
    ~$ for ((i=20;i<40;i++)); do echo /user/h998/sample-$i.fa; done > sample-02.txt
    ~$ for ((i=40;i<60;i++)); do echo /user/h998/sample-$i.fa; done > sample-03.txt
    ~$ for ((i=60;i<80;i++)); do echo /user/h998/sample-$i.fa; done > sample-04.txt
    ~$ for ((i=80;i<100;i++)); do echo /user/h998/sample-$i.fa; done > sample-05.txt
    ~$ hadoop fs -mkdir lab9_input
    ~$ hadoop fs -put sample-0* lab9_input