| 83 | |
| 84 | == 範例二『用標準表示法過濾內容 grep』 == |
| 85 | |
| 86 | * grep 這個命令是擷取文件裡面特定的字元,在 Hadoop example 中此指令可以擷取文件中有此指定文字的字串,並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings. |
| 87 | {{{ |
| 88 | Jazz@human /opt/hadoop |
| 89 | $ hadoop jar hadoop-*-examples.jar grep input lab5_out1 'dfs[a-z.]+' |
| 90 | }}} |
| 91 | * 運作的畫面如下:[[BR]]You should see procedure like this: |
| 92 | {{{ |
| 93 | Jazz@human /opt/hadoop |
| 94 | $ hadoop jar hadoop-*-examples.jar grep input lab5_out1 'dfs[a-z.]+' |
| 95 | 11/10/21 14:17:39 INFO mapred.FileInputFormat: Total input paths to process : 12 |
| 96 | |
| 97 | 11/10/21 14:17:39 INFO mapred.JobClient: Running job: job_201110211130_0002 |
| 98 | 11/10/21 14:17:40 INFO mapred.JobClient: map 0% reduce 0% |
| 99 | 11/10/21 14:17:54 INFO mapred.JobClient: map 8% reduce 0% |
| 100 | 11/10/21 14:17:57 INFO mapred.JobClient: map 16% reduce 0% |
| 101 | 11/10/21 14:18:03 INFO mapred.JobClient: map 33% reduce 0% |
| 102 | 11/10/21 14:18:13 INFO mapred.JobClient: map 41% reduce 0% |
| 103 | 11/10/21 14:18:16 INFO mapred.JobClient: map 50% reduce 11% |
| 104 | 11/10/21 14:18:19 INFO mapred.JobClient: map 58% reduce 11% |
| 105 | 11/10/21 14:18:23 INFO mapred.JobClient: map 66% reduce 11% |
| 106 | 11/10/21 14:18:30 INFO mapred.JobClient: map 83% reduce 16% |
| 107 | 11/10/21 14:18:33 INFO mapred.JobClient: map 83% reduce 22% |
| 108 | 11/10/21 14:18:36 INFO mapred.JobClient: map 91% reduce 22% |
| 109 | 11/10/21 14:18:39 INFO mapred.JobClient: map 100% reduce 22% |
| 110 | 11/10/21 14:18:42 INFO mapred.JobClient: map 100% reduce 27% |
| 111 | 11/10/21 14:18:48 INFO mapred.JobClient: map 100% reduce 30% |
| 112 | 11/10/21 14:18:54 INFO mapred.JobClient: map 100% reduce 100% |
| 113 | 11/10/21 14:18:56 INFO mapred.JobClient: Job complete: job_201110211130_0002 |
| 114 | 11/10/21 14:18:56 INFO mapred.JobClient: Counters: 18 |
| 115 | 11/10/21 14:18:56 INFO mapred.JobClient: Job Counters |
| 116 | 11/10/21 14:18:56 INFO mapred.JobClient: Launched reduce tasks=1 |
| 117 | 11/10/21 14:18:56 INFO mapred.JobClient: Launched map tasks=12 |
| 118 | 11/10/21 14:18:56 INFO mapred.JobClient: Data-local map tasks=12 |
| 119 | 11/10/21 14:18:56 INFO mapred.JobClient: FileSystemCounters |
| 120 | 11/10/21 14:18:56 INFO mapred.JobClient: FILE_BYTES_READ=888 |
| 121 | 11/10/21 14:18:56 INFO mapred.JobClient: HDFS_BYTES_READ=18312 |
| 122 | 11/10/21 14:18:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1496 |
| 123 | 11/10/21 14:18:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=280 |
| 124 | 11/10/21 14:18:56 INFO mapred.JobClient: Map-Reduce Framework |
| 125 | 11/10/21 14:18:56 INFO mapred.JobClient: Reduce input groups=7 |
| 126 | 11/10/21 14:18:56 INFO mapred.JobClient: Combine output records=7 |
| 127 | 11/10/21 14:18:56 INFO mapred.JobClient: Map input records=553 |
| 128 | 11/10/21 14:18:56 INFO mapred.JobClient: Reduce shuffle bytes=224 |
| 129 | 11/10/21 14:18:56 INFO mapred.JobClient: Reduce output records=7 |
| 130 | 11/10/21 14:18:56 INFO mapred.JobClient: Spilled Records=14 |
| 131 | 11/10/21 14:18:56 INFO mapred.JobClient: Map output bytes=193 |
| 132 | 11/10/21 14:18:56 INFO mapred.JobClient: Map input bytes=18312 |
| 133 | 11/10/21 14:18:56 INFO mapred.JobClient: Combine input records=10 |
| 134 | 11/10/21 14:18:56 INFO mapred.JobClient: Map output records=10 |
| 135 | 11/10/21 14:18:56 INFO mapred.JobClient: Reduce input records=7 |
| 136 | 11/10/21 14:18:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing th |
| 137 | e arguments. Applications should implement Tool for the same. |
| 138 | 11/10/21 14:18:57 INFO mapred.FileInputFormat: Total input paths to process : 1 |
| 139 | 11/10/21 14:18:57 INFO mapred.JobClient: Running job: job_201110211130_0003 |
| 140 | ( ... skip ... ) |
| 141 | }}} |
| 142 | * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS : |
| 143 | * 這個例子是要從 input 目錄中的所有檔案中找出符合 dfs 後面接著 a-z 字母一個以上的字串 |
| 144 | {{{ |
| 145 | Jazz@human /opt/hadoop |
| 146 | $ hadoop fs -ls lab5_out1 |
| 147 | Found 2 items |
| 148 | drwxr-xr-x - Jazz supergroup 0 2011-10-21 14:18 /user/Jazz/lab5_out1/_logs |
| 149 | -rw-r--r-- 1 Jazz supergroup 96 2011-10-21 14:19 /user/Jazz/lab5_out1/part-00000 |
| 150 | |
| 151 | Jazz@human /opt/hadoop |
| 152 | $ hadoop fs -cat lab5_out1/part-00000 |
| 153 | 3 dfs.class |
| 154 | 2 dfs.period |
| 155 | 1 dfs.file |
| 156 | 1 dfs.replication |
| 157 | 1 dfs.servers |
| 158 | 1 dfsadmin |
| 159 | 1 dfsmetrics.log |
| 160 | }}} |