2010-05-10
Hadoop : Matrix Multiply 用 MapReduce 作矩陣運算
- 測試 A MapReduce Algorithm for Matrix Multiplication 所提供之矩陣相乘範例
- 發現會有 IOException 錯誤訊息,肇因於 /tmp/MatrixMultiply/out/_logs 是目錄而不是檔案
10/05/10 15:23:23 INFO input.FileInputFormat: Total input paths to process : 1 10/05/10 15:23:23 INFO mapred.JobClient: Running job: job_201005101012_0016 10/05/10 15:23:24 INFO mapred.JobClient: map 0% reduce 0% 10/05/10 15:23:33 INFO mapred.JobClient: map 100% reduce 0% 10/05/10 15:23:45 INFO mapred.JobClient: map 100% reduce 100% 10/05/10 15:23:47 INFO mapred.JobClient: Job complete: job_201005101012_0016 .......... Exception in thread "main" java.io.IOException: Cannot open filename /tmp/MatrixMultiply/out/_logs at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1497)
- [解法] 修改 TestMatrixMultiply.java,並以單機 LocalRunner 執行。( hadoop 0.20.2 預設用 LocalRunner )
jazz@drbl:~$ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz jazz@drbl:~$ tar zxvf hadoop-0.20.2.tar.gz jazz@drbl:~$ cd hadoop-0.20.2 jazz@drbl:~/hadoop-0.20.2$ echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun/" >> conf/hadoop-env.sh jazz@drbl:~/hadoop-0.20.2$ wget http://trac.nchc.org.tw/grid/raw-attachment/wiki/jazz/10-05-10/matrix.tar.gz jazz@drbl:~/hadoop-0.20.2$ tar zxvf matrix.tar.gz jazz@drbl:~/hadoop-0.20.2$ cd matrix/ jazz@drbl:~/hadoop-0.20.2/matrix$ ant jazz@drbl:~/hadoop-0.20.2/matrix$ mv matrix.jar ../. jazz@drbl:~/hadoop-0.20.2/matrix$ cd .. jazz@drbl:~/hadoop-0.20.2$ bin/hadoop jar matrix.jar TestMatrixMultiply
-
TestMatrixMultiply.java
old new 72 72 for (int i = 0; i < rowDim; i++) 73 73 for (int j = 0; j < colDim; j++) 74 74 result[i][j] = 0; 75 if (fs.isFile(path)) { 76 fillMatrix(result, path); 77 } else { 78 FileStatus[] fileStatusArray = fs.listStatus(path); 79 for (FileStatus fileStatus : fileStatusArray) { 80 fillMatrix(result, fileStatus.getPath()); 81 } 75 FileStatus[] fileStatusArray = fs.listStatus(path); 76 for (FileStatus fileStatus : fileStatusArray) { 77 if (fs.isFile(fileStatus.getPath())) { 78 fillMatrix(result, fileStatus.getPath()); 79 } 82 80 } 83 81 return result; 84 82 } … … 104 102 int[][] Y = readMatrix(I, J, OUTPUT_DIR_PATH); 105 103 for (int i = 0; i < I; i++) { 106 104 for (int j = 0; j < J; j++) { 105 System.out.println("......X["+i+"]["+j+"]="+X[i][j]+", Y["+i+"]["+j+"]="+Y[i][j]); 107 106 if (X[i][j] != Y[i][j]) { 108 107 throw new Exception("Bad answer!"); 109 108 } … … 283 282 } 284 283 }
-
- 發現會有 IOException 錯誤訊息,肇因於 /tmp/MatrixMultiply/out/_logs 是目錄而不是檔案
Last modified 15 years ago
Last modified on May 11, 2010, 1:36:06 PM
Attachments (1)
- matrix.tar.gz (7.1 KB) - added by jazz 15 years ago.
Download all attachments as: .zip