wiki:jazz/10-05-10

2010-05-10

Hadoop : Matrix Multiply 用 MapReduce 作矩陣運算

  • 測試 A MapReduce Algorithm for Matrix Multiplication 所提供之矩陣相乘範例
    • 發現會有 IOException 錯誤訊息,肇因於 /tmp/MatrixMultiply/out/_logs 是目錄而不是檔案
      10/05/10 15:23:23 INFO input.FileInputFormat: Total input paths to process : 1
      10/05/10 15:23:23 INFO mapred.JobClient: Running job: job_201005101012_0016
      10/05/10 15:23:24 INFO mapred.JobClient:  map 0% reduce 0%
      10/05/10 15:23:33 INFO mapred.JobClient:  map 100% reduce 0%
      10/05/10 15:23:45 INFO mapred.JobClient:  map 100% reduce 100%
      10/05/10 15:23:47 INFO mapred.JobClient: Job complete: job_201005101012_0016
      ..........
      Exception in thread "main" java.io.IOException: Cannot open filename /tmp/MatrixMultiply/out/_logs
              at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1497)
      
    • [解法] 修改 TestMatrixMultiply.java,並以單機 LocalRunner 執行。( hadoop 0.20.2 預設用 LocalRunner )
      jazz@drbl:~$ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz
      jazz@drbl:~$ tar zxvf hadoop-0.20.2.tar.gz
      jazz@drbl:~$ cd hadoop-0.20.2
      jazz@drbl:~/hadoop-0.20.2$ echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun/" >> conf/hadoop-env.sh
      jazz@drbl:~/hadoop-0.20.2$ wget http://trac.nchc.org.tw/grid/raw-attachment/wiki/jazz/10-05-10/matrix.tar.gz
      jazz@drbl:~/hadoop-0.20.2$ tar zxvf matrix.tar.gz
      jazz@drbl:~/hadoop-0.20.2$ cd matrix/
      jazz@drbl:~/hadoop-0.20.2/matrix$ ant
      jazz@drbl:~/hadoop-0.20.2/matrix$ mv matrix.jar ../.
      jazz@drbl:~/hadoop-0.20.2/matrix$ cd ..
      jazz@drbl:~/hadoop-0.20.2$ bin/hadoop jar matrix.jar TestMatrixMultiply
      
      • TestMatrixMultiply.java

        old new  
        7272               for (int i = 0; i < rowDim; i++)
        7373                       for (int j = 0; j < colDim; j++)
        7474                               result[i][j] = 0;
        75                if (fs.isFile(path)) {
        76                        fillMatrix(result, path);
        77                } else {
        78                        FileStatus[] fileStatusArray = fs.listStatus(path);
        79                        for (FileStatus fileStatus : fileStatusArray) {
        80                                fillMatrix(result, fileStatus.getPath());
        81                        }
         75               FileStatus[] fileStatusArray = fs.listStatus(path);
         76               for (FileStatus fileStatus : fileStatusArray) {
         77                 if (fs.isFile(fileStatus.getPath())) {
         78                   fillMatrix(result, fileStatus.getPath());
         79                 }
        8280               }
        8381               return result;
        8482       }
         
        104102               int[][] Y = readMatrix(I, J, OUTPUT_DIR_PATH);
        105103               for (int i = 0; i < I; i++) {
        106104                       for (int j = 0; j < J; j++) {
         105                               System.out.println("......X["+i+"]["+j+"]="+X[i][j]+", Y["+i+"]["+j+"]="+Y[i][j]);
        107106                               if (X[i][j] != Y[i][j]) {
        108107                                       throw new Exception("Bad answer!");
        109108                               }
         
        283282               }
        284283       }
Last modified 15 years ago Last modified on May 11, 2010, 1:36:06 PM

Attachments (1)

Download all attachments as: .zip