wiki:waue/2009/0402

Version 8 (modified by waue, 15 years ago) (diff)

--

Hadoop分佈式文件系統使用指南二

升級

  • 由於換版本的話,資料夾內的conf設定檔也勢必被更改,因此目前作法為: 把conf 移至/opt/conf ,hadoop 0.16 與 hadoop 0.18用 ln 做捷徑代換。由於conf已不在hadoop_home內,因此記得匯入conf/hadoop-env.sh
    $ source /opt/conf/hadoop-env.sh
    
  • 先看狀態
    $ bin/hadoop dfsadmin -upgradeProgress status
    
    There are no upgrades in progress.
    
  • 停止hdfs
    • 注意不可使用bin/stop-all.sh來停止
      $ bin/stop-dfs.sh
      
  • 部署新版本的Hadoop
    • 注意每個node的版本都要統一,否則會出現問題
  • 啟動
    $ bin/start-dfs.sh -upgrade
    
  • namenode管理網頁會出現升級狀態

退回

  • 停止集群
    $ bin/stop-dfs.sh
    
  • 部署老版本的Hadoop
  • 退回之前版本
    $ bin/start-dfs.sh -rollback
    

檢查hdfs磁碟狀況

  • fsck 可以
    $ bin/hadoop fsck /
    
    .
    /user/waue/input/1.txt:  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
    /user/waue/input/1.txt:  Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
    .
    /user/waue/input/2.txt:  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
    .
    /user/waue/input/3.txt:  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
    .
    /user/waue/input/4.txt:  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
    Status: HEALTHY
     Total size:	143451003 B
     Total dirs:	8
     Total files:	4
     Total blocks (validated):	5 (avg. block size 28690200 B)
     Minimally replicated blocks:	5 (100.0 %)
     Over-replicated blocks:	0 (0.0 %)
     Under-replicated blocks:	5 (100.0 %)
     Mis-replicated blocks:		0 (0.0 %)
     Default replication factor:	3
     Average block replication:	2.0
     Corrupt blocks:		0
     Missing replicas:		5 (50.0 %)
     Number of data-nodes:		2
     Number of racks:		1
    The filesystem under path '/' is HEALTHY
    
  • 加不同的參數有不同的用處,如
    $ bin/hadoop fsck / -files
    
    
    /tmp <dir>
    
    /tmp/hadoop <dir>
    
    /tmp/hadoop/hadoop-waue <dir>
    
    /tmp/hadoop/hadoop-waue/mapred <dir>
    
    /tmp/hadoop/hadoop-waue/mapred/system <dir>
    
    /user <dir>
    
    /user/waue <dir>
    
    /user/waue/input <dir>
    
    /user/waue/input/1.txt 115045564 bytes, 2 block(s):  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
    
     Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
    
    /user/waue/input/2.txt 987864 bytes, 1 block(s):  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
    
    /user/waue/input/3.txt 1573048 bytes, 1 block(s):  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
    
    /user/waue/input/4.txt 25844527 bytes, 1 block(s):  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
    
    Status: HEALTHY
    
    ....(同上)