Hadoop分佈式文件系統使用指南二
升級
- 停止hdfs
- 注意不可使用bin/stop-all.sh來停止
$ bin/stop-dfs.sh
退回
- 停止集群
$ bin/stop-dfs.sh
- 部署老版本的Hadoop
Hadoop指令手冊
$ hadoop [--config confdir] [COMMAND] [GENERIC_OPTIONS] [COMMAND_OPTIONS]
[GENERIC_OPTIONS] :
| -conf <configuration file> | 指定應用程序的配置文件。
|
| -D <property=value> | 為指定property指定值value。
|
| -fs <local|namenode:port> | 指定namenode。
|
| -jt <local|jobtracker:port> | 指定job tracker。只適用於job。
|
archieve
- ps: 官方文件介紹的 hadoop dfs -lsr har:///user/hadoop/output/foo.har 會出現錯誤！
lsr: could not get get listing for 'har:/user/waue/output/foo.har/user/waue' : File: har://hdfs-gm1.nchc.org.tw:9000/user/waue/output/foo.har/user/waue/input does not exist in har:///user/waue/output/foo.har
distCp
？？ 然而8020 port 在機器上沒有開，且不是應該檔案會均勻散佈在每個節點上嗎？怎麼還會知道nn1的節點上有這個檔要複製到nn2呢？
fsck
$ bin/hadoop fsck /
.
/user/waue/input/1.txt: Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
/user/waue/input/1.txt: Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
.
/user/waue/input/2.txt: Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
.
/user/waue/input/3.txt: Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
.
/user/waue/input/4.txt: Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 143451003 B
Total dirs: 8
Total files: 4
Total blocks (validated): 5 (avg. block size 28690200 B)
Minimally replicated blocks: 5 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 5 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 5 (50.0 %)
Number of data-nodes: 2
Number of racks: 1
The filesystem under path '/' is HEALTHY
- 加不同的參數有不同的用處，如
$ bin/hadoop fsck / -files
/tmp <dir>
/tmp/hadoop <dir>
/tmp/hadoop/hadoop-waue <dir>
/tmp/hadoop/hadoop-waue/mapred <dir>
/tmp/hadoop/hadoop-waue/mapred/system <dir>
/user <dir>
/user/waue <dir>
/user/waue/input <dir>
/user/waue/input/1.txt 115045564 bytes, 2 block(s): Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
/user/waue/input/2.txt 987864 bytes, 1 block(s): Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
/user/waue/input/3.txt 1573048 bytes, 1 block(s): Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
/user/waue/input/4.txt 25844527 bytes, 1 block(s): Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
....（同上）
| -move | -delete | -openforwrite
|
| 移動受損文件到/lost+found | 刪除受損文件 | 印出寫打開的文件
|
| -files | -blocks | -locations | -racks
|
| 印出正被檢查的文件 | 印出區塊的資訊 | 印出每個區塊的位置 | 印出data-node的網絡拓撲結構
|
如：
$ bin/hadoop fsck /user/waue/input/1.txt -files -blocks -locations
/user/waue/input/1.txt 115045564 bytes, 2 block(s): Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
0. blk_-90085106852013388_1001 len=67108864 repl=2 [140.110.138.191:50010, 140.110.141.129:50010]
1. blk_-4027196261436469955_1001 len=47936700 repl=2 [140.110.138.191:50010, 140.110.141.129:50010]
Status: HEALTHY
Total size: 115045564 B
Total dirs: 0
Total files: 1
....（略）
job
- 用以跟Map Reduce 的作業程序溝通
- 在測試此指令之前，請確認已經先執行過mapReduce的程序過
- 可到JobTracker:50030網頁來看程序的Jobid
-status
-kill
-list
-history
streaming
