close
Warning:
Can't synchronize with repository "(default)" (Unsupported version control system "svn": /usr/lib/python2.7/dist-packages/libsvn/_repos.so: failed to map segment from shared object: Cannot allocate memory). Look in the Trac log for more information.
- Timestamp:
-
Aug 4, 2010, 12:22:16 PM (14 years ago)
- Author:
-
jazz
- Comment:
-
--
Legend:
- Unmodified
- Added
- Removed
- Modified
-
v2
|
v3
|
|
2 | 2 | #!html |
3 | 3 | <div style="text-align: center;"><big |
4 | | style="font-weight: bold;"><big><big>實作二: HDFS Shell操作練習</big></big></big><big>Lab2: HDFS Shell in practice</big></big></div> |
| 4 | style="font-weight: bold;"><big><big>實作二: HDFS Shell操作練習</big></big></big><br><big><big>Lab2: HDFS Shell in practice</big></big></div> |
5 | 5 | }}} |
6 | 6 | |
… |
… |
|
11 | 11 | * 此部份接續[wiki:NCHCCloudCourse100802/Lab1 實作一][[BR]]Please refer to [wiki:NCHCCloudCourse100802/Lab1 Lab1] |
12 | 12 | |
13 | | = Content 1: HDFS Shell 基本操作 = |
14 | | = Content 1: Basic HDFS Shell Commands = |
15 | | |
16 | | == 1.1 瀏覽你HDFS目錄 == |
17 | | == 1.1 Browsing Your HDFS Folder == |
| 13 | == Content 1: HDFS Shell 基本操作 == |
| 14 | == Content 1: Basic HDFS Shell Commands == |
| 15 | |
| 16 | === 1.1 瀏覽你HDFS目錄 === |
| 17 | === 1.1 Browsing Your HDFS Folder === |
18 | 18 | |
19 | 19 | {{{ |
… |
… |
|
22 | 22 | }}} |
23 | 23 | |
24 | | == 1.2 上傳資料到 HDFS 目錄 == |
25 | | == 1.2 Upload Files or Folder to HDFS == |
| 24 | === 1.2 上傳資料到 HDFS 目錄 === |
| 25 | === 1.2 Upload Files or Folder to HDFS === |
26 | 26 | |
27 | 27 | * 上傳 Upload |
… |
… |
|
38 | 38 | }}} |
39 | 39 | |
40 | | == 1.3 下載 HDFS 的資料到本地目錄 == |
41 | | == 1.3 Download HDFS Files or Folder to Local == |
| 40 | === 1.3 下載 HDFS 的資料到本地目錄 === |
| 41 | === 1.3 Download HDFS Files or Folder to Local === |
42 | 42 | |
43 | 43 | * 下載 Download |
… |
… |
|
54 | 54 | }}} |
55 | 55 | |
56 | | == 1.4 刪除檔案 == |
57 | | == 1.4 Remove Files or Folder == |
| 56 | === 1.4 刪除檔案 === |
| 57 | === 1.4 Remove Files or Folder === |
58 | 58 | |
59 | 59 | {{{ |
… |
… |
|
62 | 62 | }}} |
63 | 63 | |
64 | | == 1.5 直接看檔案 == |
65 | | == 1.5 Browse Files Directly == |
| 64 | === 1.5 直接看檔案 === |
| 65 | === 1.5 Browse Files Directly === |
66 | 66 | |
67 | 67 | {{{ |
… |
… |
|
70 | 70 | }}} |
71 | 71 | |
72 | | == 1.6 更多指令操作 == |
73 | | == 1.6 More Commands -- Help message == |
| 72 | === 1.6 更多指令操作 === |
| 73 | === 1.6 More Commands -- Help message === |
74 | 74 | |
75 | 75 | {{{ |
… |
… |
|
119 | 119 | }}} |
120 | 120 | |
121 | | |
122 | | |
123 | | = Content 2: 使用網頁 GUI 瀏覽資訊 = |
124 | | = Content 2: User Web GUI to browse HDFS = |
125 | | |
126 | | * [http://localhost:50030 Map/Reduce Administration] |
127 | | * [http://localhost:50070 NameNode ] |
128 | | |
129 | | = Content 3: 更多 HDFS Shell 的用法 = |
130 | | = Content 3: More about HDFS Shell = |
| 121 | == Content 2: 使用網頁 GUI 瀏覽資訊 == |
| 122 | == Content 2: User Web GUI to browse HDFS == |
| 123 | |
| 124 | * [http://localhost:50030 JobTracker Web Interface] |
| 125 | * [http://localhost:50070 NameNode Web Interface] |
| 126 | |
| 127 | == Content 3: 更多 HDFS Shell 的用法 == |
| 128 | == Content 3: More about HDFS Shell == |
131 | 129 | |
132 | 130 | * bin/hadoop fs <args> ,下面則列出 <args> 的用法[[BR]]Following are the examples of hadoop fs related commands. |
… |
… |
|
150 | 148 | }}} |
151 | 149 | |
152 | | == -cat == |
153 | | * 將路徑指定文件的內容輸出到stdout |
| 150 | === -cat === |
| 151 | |
| 152 | * 將路徑指定文件的內容輸出到 STDOUT [[BR]] Print given file content to STDOUT |
154 | 153 | {{{ |
155 | 154 | $ bin/hadoop fs -cat quota/hadoop-env.sh |
156 | 155 | }}} |
157 | | == -chgrp == |
158 | | * 改變文件所屬的組 |
| 156 | |
| 157 | === -chgrp === |
| 158 | |
| 159 | * 改變文件所屬的組 [[BR]] Change '''owner group''' of given file or folder |
159 | 160 | {{{ |
160 | 161 | $ bin/hadoop fs -chgrp -R hadooper own |
161 | 162 | }}} |
162 | | == -chmod == |
163 | | * 改變文件的權限 |
| 163 | |
| 164 | === -chmod === |
| 165 | |
| 166 | * 改變文件的權限 [[BR]] Change '''read and write permission''' of given file or folder |
164 | 167 | {{{ |
165 | 168 | $ bin/hadoop fs -chmod -R 755 own |
166 | 169 | }}} |
167 | | == -chown == |
168 | | * 改變文件的擁有者 |
| 170 | |
| 171 | === -chown === |
| 172 | |
| 173 | * 改變文件的擁有者 [[BR]] Change '''owner''' of given file or folder |
169 | 174 | {{{ |
170 | 175 | $ bin/hadoop fs -chown -R hadooper own |
171 | 176 | }}} |
172 | | == -copyFromLocal, -put == |
173 | | * 從local放檔案到hdfs |
| 177 | |
| 178 | === -copyFromLocal, -put === |
| 179 | |
| 180 | * 從 local 放檔案到 hdfs [[BR]] Both commands will copy given file or folder from local to HDFS |
174 | 181 | {{{ |
175 | 182 | $ bin/hadoop fs -put input dfs_input |
176 | 183 | }}} |
177 | | == -copyToLocal, -get == |
178 | | * 把hdfs上得檔案下載到 local |
| 184 | |
| 185 | === -copyToLocal, -get === |
| 186 | |
| 187 | * 把hdfs上得檔案下載到 local [[BR]] Both commands will copy given file or folder from HDFS to local |
179 | 188 | {{{ |
180 | 189 | $ bin/hadoop fs -get dfs_input input1 |
181 | 190 | }}} |
182 | | == -cp == |
183 | | * 將文件從hdfs原本路徑複製到hdfs目標路徑 |
| 191 | |
| 192 | === -cp === |
| 193 | |
| 194 | * 將文件從 hdfs 原本路徑複製到 hdfs 目標路徑 [[BR]] Copy given file or folder from HDFS source path to HDFS target path |
184 | 195 | {{{ |
185 | 196 | $ bin/hadoop fs -cp own hadooper |
186 | 197 | }}} |
187 | | == -du == |
188 | | * 顯示目錄中所有文件的大小 |
| 198 | |
| 199 | === -du === |
| 200 | |
| 201 | * 顯示目錄中所有文件的大小 [[BR]] Display the size of files in given folder |
189 | 202 | {{{ |
190 | 203 | $ bin/hadoop fs -du input |
… |
… |
|
196 | 209 | 25844527 hdfs://gm1.nchc.org.tw:9000/user/hadooper/input/4.txt |
197 | 210 | }}} |
198 | | == -dus == |
199 | | * 顯示該目錄/文件的總大小 |
| 211 | === -dus === |
| 212 | |
| 213 | * 顯示該目錄/文件的總大小 [[BR]] Display total size of given folder |
200 | 214 | {{{ |
201 | 215 | $ bin/hadoop fs -dus input |
… |
… |
|
203 | 217 | hdfs://gm1.nchc.org.tw:9000/user/hadooper/input 143451003 |
204 | 218 | }}} |
205 | | == -expunge == |
206 | | * 清空垃圾桶 |
| 219 | |
| 220 | === -expunge === |
| 221 | |
| 222 | * 清空垃圾桶 [[BR]] Clean up Recycled |
207 | 223 | {{{ |
208 | 224 | $ bin/hadoop fs -expunge |
209 | 225 | }}} |
210 | | == -getmerge == |
211 | | * 將來源目錄<src>下所有的文件都集合到本地端一個<localdst>檔案內 |
212 | | * bin/hadoop fs -getmerge <src> <localdst> |
| 226 | |
| 227 | === -getmerge === |
| 228 | |
| 229 | * 將來源目錄<src>下所有的文件都集合到本地端一個<localdst>檔案內 [[BR]] Merge all files in HDFS source folder <src> into one local file |
| 230 | {{{ |
| 231 | $ bin/hadoop fs -getmerge <src> <localdst> |
| 232 | }}} |
213 | 233 | {{{ |
214 | 234 | $ echo "this is one; " >> in1/input |
… |
… |
|
219 | 239 | }}} |
220 | 240 | |
221 | | == -ls == |
222 | | * 列出文件或目錄的資訊 |
223 | | * 文件名 <副本數> 文件大小 修改日期 修改時間 權限 用戶ID 組ID |
224 | | * 目錄名 <dir> 修改日期 修改時間 權限 用戶ID 組ID |
| 241 | == -ls === |
| 242 | |
| 243 | * 列出文件或目錄的資訊 [[BR]] List files and folders |
| 244 | * 文件名 <副本數> 文件大小 修改日期 修改時間 權限 用戶ID 組ID [[BR]] <file name> <replication> <size> <modified date> <modified time> <permission> <user id> <group id> |
| 245 | * 目錄名 <dir> 修改日期 修改時間 權限 用戶ID 組ID [[BR]] <folder name> <modified date> <modified time> <permission> <user id> <group id> |
225 | 246 | {{{ |
226 | 247 | $ bin/hadoop fs -ls |
227 | 248 | }}} |
228 | | == -lsr == |
229 | | * ls命令的遞迴版本 |
| 249 | |
| 250 | === -lsr === |
| 251 | |
| 252 | * ls 命令的遞迴版本 [[BR]] list files and folders with recursive |
230 | 253 | {{{ |
231 | 254 | $ bin/hadoop fs -lsr / |
232 | 255 | }}} |
233 | | == -mkdir == |
234 | | * 建立資料夾 |
| 256 | |
| 257 | === -mkdir === |
| 258 | |
| 259 | * 建立資料夾 [[BR]] create directories |
235 | 260 | {{{ |
236 | 261 | $ bin/hadoop fs -mkdir a b c |
237 | 262 | }}} |
238 | | == -moveFromLocal == |
239 | | * 將local端的資料夾剪下移動到hdfs上 |
| 263 | |
| 264 | === -moveFromLocal === |
| 265 | |
| 266 | * 將 local 端的資料夾剪下移動到 hdfs 上 [[BR]] move local files or folder to HDFS ( it will delete local files or folder. ) |
240 | 267 | {{{ |
241 | 268 | $ bin/hadoop fs -moveFromLocal in1 in2 |
242 | 269 | }}} |
243 | | == -mv == |
244 | | * 更改資料的名稱 |
| 270 | |
| 271 | === -mv === |
| 272 | |
| 273 | * 更改資料的名稱 [[BR]] Change file name or folder name. |
245 | 274 | {{{ |
246 | 275 | $ bin/hadoop fs -mv in2 in3 |
247 | 276 | }}} |
248 | | == -rm == |
249 | | * 刪除指定的檔案(不可資料夾) |
| 277 | |
| 278 | === -rm === |
| 279 | |
| 280 | * 刪除指定的檔案(不可資料夾)[[BR]] Remove given files (not folders) |
250 | 281 | {{{ |
251 | 282 | $ bin/hadoop fs -rm in1/input |
252 | 283 | }}} |
253 | | == -rmr == |
254 | | * 遞迴刪除資料夾(包含在內的所有檔案) |
| 284 | === -rmr === |
| 285 | |
| 286 | * 遞迴刪除資料夾(包含在內的所有檔案) [[BR]] Remove given files and folders with recursive |
255 | 287 | {{{ |
256 | 288 | $ bin/hadoop fs -rmr in1 |
257 | 289 | }}} |
258 | | == -setrep == |
259 | | * 設定副本係數 |
260 | | * bin/hadoop fs -setrep [-R] [-w] <rep> <path/file> |
| 290 | |
| 291 | === -setrep === |
| 292 | |
| 293 | * 設定副本係數 [[BR]] setup replication numbers of given files or folder |
| 294 | {{{ |
| 295 | $ bin/hadoop fs -setrep [-R] [-w] <rep> <path/file> |
| 296 | }}} |
261 | 297 | {{{ |
262 | 298 | $ bin/hadoop fs -setrep -w 2 -R input |
… |
… |
|
270 | 306 | Waiting for hdfs://gm1.nchc.org.tw:9000/user/hadooper/input/4.txt ... done |
271 | 307 | }}} |
272 | | == -stat == |
273 | | * 印出時間資訊 |
| 308 | |
| 309 | === -stat === |
| 310 | |
| 311 | * 印出時間資訊 [[BR]] Print Status of time stamp of folder |
274 | 312 | {{{ |
275 | 313 | $ bin/hadoop fs -stat input |
276 | 314 | 2009-04-02 03:51:29 |
277 | 315 | }}} |
278 | | == -tail == |
279 | | * 將文件的最後1k內容輸出 |
280 | | * 用法 : bin/hadoop fs -tail [-f] 檔案 (-f 參數用來顯示如果檔案增大,則秀出被append上得內容) |
| 316 | === -tail === |
| 317 | |
| 318 | * 將文件的最後 1K 內容輸出 [[BR]] Display the last 1K contents of given file |
| 319 | * 用法 Usage |
| 320 | {{{ |
| 321 | bin/hadoop fs -tail [-f] 檔案 (-f 參數用來顯示如果檔案增大,則秀出被append上得內容) |
| 322 | bin/hadoop fs -tail [-f] <path/file> (-f is used when file had appended) |
| 323 | }}} |
281 | 324 | {{{ |
282 | 325 | $ bin/hadoop fs -tail input/1.txt |
283 | 326 | }}} |
284 | | == -test == |
285 | | * 測試檔案, -e 檢查文件是否存在(1=存在, 0=否), -z 檢查文件是否為空(1=空, 0=不為空), -d 檢查是否為目錄(1=存在, 0=否) |
286 | | * 要用echo $? 來看回傳值為 0 or 1 |
287 | | * 用法: bin/hadoop fs -test -[ezd] URI |
| 327 | |
| 328 | === -test === |
| 329 | |
| 330 | * 測試檔案, -e 檢查文件是否存在(1=存在, 0=否), -z 檢查文件是否為空(1=空, 0=不為空), -d 檢查是否為目錄(1=存在, 0=否) [[BR]] test files or folders [[BR]] -e : check if file or folder existed ( 1 = exist , 0 = false )[[BR]] -z : check if file is empty ( 1 = empty , 0 = false ) [[BR]] -d : check if given path is folder ( 1 = it's folder , 0 = false ) |
| 331 | * 要用 echo $? 來看回傳值為 0 or 1 [[BR]] You have to use '''echo $?''' to get the return value |
| 332 | * 用法 Usage |
| 333 | {{{ |
| 334 | $ bin/hadoop fs -test -[ezd] URI |
| 335 | }}} |
288 | 336 | |
289 | 337 | {{{ |
… |
… |
|
295 | 343 | test: File does not exist: /user/hadooper/input/5.txt |
296 | 344 | }}} |
297 | | == -text == |
298 | | * 將檔案(如壓縮檔, textrecordinputstream)輸出為純文字格式 |
299 | | * hadoop fs -text <src> |
| 345 | |
| 346 | === -text === |
| 347 | |
| 348 | * 將檔案(如壓縮檔, textrecordinputstream)輸出為純文字格式 [[BR]] Display archive file contents into STDOUT |
| 349 | {{{ |
| 350 | $ hadoop fs -text <src> |
| 351 | }}} |
300 | 352 | {{{ |
301 | 353 | $ hadoop fs -text macadr-eth1.txt.gz |
… |
… |
|
303 | 355 | 00:1b:fc:58:9c:23 |
304 | 356 | }}} |
305 | | * ps : 目前沒支援zip的函式庫 |
| 357 | * ps : 目前沒支援zip的函式庫 [[BR]] PS. It does not support zip files yet. |
306 | 358 | {{{ |
307 | 359 | $ bin/hadoop fs -text b/a.txt.zip |
… |
… |
|
314 | 366 | ��a.txtUTb��IUxPK@C |
315 | 367 | }}} |
316 | | == -touchz == |
317 | | * 建立一個空文件 |
| 368 | |
| 369 | === -touchz === |
| 370 | |
| 371 | * 建立一個空文件 [[BR]] creat an empty file |
318 | 372 | {{{ |
319 | 373 | $ bin/hadoop fs -touchz b/kk |