Changes between Initial Version and Version 1 of 0428Hadoop_Lab4

Timestamp:

Apr 26, 2009, 1:50:30 AM (15 years ago)

Author:

waue

Comment:

--

0428Hadoop_Lab4

@@ +1 @@
+
+
+
+== 升級 ==
+ * 由於換版本的話，資料夾內的conf設定檔也勢必被更改，因此目前作法為： 把conf 移至/opt/conf ，hadoop 0.16 與 hadoop 0.18用 ln 做捷徑代換。由於conf已不在hadoop_home內，因此記得匯入conf/hadoop-env.sh
+{{{
+$ source /opt/conf/hadoop-env.sh
+}}}
+
+ * 先看狀態
+{{{
+$ bin/hadoop dfsadmin -upgradeProgress status
+
+There are no upgrades in progress.
+}}}
+
+ * 停止hdfs
+   * 注意不可使用bin/stop-all.sh來停止
+{{{
+$ bin/stop-dfs.sh
+}}}
+
+ * 部署新版本的Hadoop
+   * 注意每個node的版本都要統一，否則會出現問題
+
+
+ * 啟動
+{{{
+$ bin/start-dfs.sh -upgrade
+}}}
+
+ ps：之後有介紹到 bin/hadoop namenode -upgrade ，應該要查查看與 $ bin/start-dfs.sh -upgrade 有何不同
+
+ * namenode管理網頁會出現升級狀態
+
+
+-----
+
+== 退回 ==
+
+ * 停止集群
+{{{
+$ bin/stop-dfs.sh
+}}}
+ * 部署老版本的Hadoop
+
+ * 退回之前版本
+{{{
+$ bin/start-dfs.sh -rollback
+}}}
+ ps：之後有介紹到 bin/hadoop namenode -rollback ，應該要查查看與 $ bin/start-dfs.sh -rollback 有何不同
+
+
+-----
+ == fsck ==
+ * HDFS文件系統檢查工具
+
+{{{
+$ bin/hadoop fsck /
+
+.
+/user/waue/input/1.txt:  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
+/user/waue/input/1.txt:  Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
+.
+/user/waue/input/2.txt:  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
+.
+/user/waue/input/3.txt:  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
+.
+/user/waue/input/4.txt:  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
+Status: HEALTHY
+ Total size:    143451003 B
+ Total dirs:    8
+ Total files:   4
+ Total blocks (validated):      5 (avg. block size 28690200 B)
+ Minimally replicated blocks:   5 (100.0 %)
+ Over-replicated blocks:        0 (0.0 %)
+ Under-replicated blocks:       5 (100.0 %)
+ Mis-replicated blocks:         0 (0.0 %)
+ Default replication factor:    3
+ Average block replication:     2.0
+ Corrupt blocks:                0
+ Missing replicas:              5 (50.0 %)
+ Number of data-nodes:          2
+ Number of racks:               1
+The filesystem under path '/' is HEALTHY
+}}}
+
+ * 加不同的參數有不同的用處，如
+{{{
+$ bin/hadoop fsck / -files
+
+/tmp <dir>
+/tmp/hadoop <dir>
+/tmp/hadoop/hadoop-waue <dir>
+/tmp/hadoop/hadoop-waue/mapred <dir>
+/tmp/hadoop/hadoop-waue/mapred/system <dir>
+/user <dir>
+/user/waue <dir>
+/user/waue/input <dir>
+/user/waue/input/1.txt 115045564 bytes, 2 block(s):  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
+ Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
+/user/waue/input/2.txt 987864 bytes, 1 block(s):  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
+/user/waue/input/3.txt 1573048 bytes, 1 block(s):  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
+/user/waue/input/4.txt 25844527 bytes, 1 block(s):  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
+Status: HEALTHY
+....（同上）
+}}}
+
+-----
+
+== job  == 
+ * 用以跟Map Reduce 的作業程序溝通
+ * 在測試此指令之前，請確認已經先執行過mapReduce的程序過
+ * 可到JobTracker:50030網頁來看程序的Jobid
+=== -status ===
+ * 查看工作狀態
+{{{
+$ bin/hadoop job -status job_200904021140_0001
+
+}}}
+=== -kill ===
+ * 終止正在執行的程序，其id為 job_200904021140_0001
+{{{
+$ bin/hadoop job -kill job_200904021140_0001
+}}}
+=== -list ===
+ * 印出所有程序的狀態
+{{{
+$ bin/hadoop job -list all
+
+5 jobs submitted
+States are:
+        Running : 1     Succeded : 2    Failed : 3      Prep : 4
+JobId   State   StartTime       UserName
+job_200904021140_0001   2       1238652150499   waue
+job_200904021140_0002   3       1238657754096   waue
+job_200904021140_0004   3       1238657989495   waue
+job_200904021140_0005   2       1238658076347   waue
+job_200904021140_0006   2       1238658644666   waue
+}}}
+
+=== -history ===
+ * 印出程序的歷史狀態
+{{{
+$ bin/hadoop job -history /user/waue/stream-output1
+
+Hadoop job: job_200904021140_0005
+=====================================
+Job tracker host name: gm1.nchc.org.tw
+job tracker start time: Thu Apr 02 11:40:06 CST 2009
+User: waue
+JobName: streamjob9019.jar
+JobConf: hdfs://gm1.nchc.org.tw:9000/tmp/hadoop/hadoop-waue/mapred/system/job_200904021140_0005/job.xml
+Submitted At: 2-四月-2009 15:41:16
+Launched At: 2-四月-2009 15:41:16 (0sec)
+Finished At: 2-四月-2009 15:42:04 (48sec)
+Status: SUCCESS
+=====================================
+...略
+}}}
+-----
+
+== version ==
+ * 印出目前的hadoop 版本
+{{{
+bin/hadoop version
+
+Hadoop 0.18.3
+Subversion https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250
+Compiled by ndaley on Thu Jan 22 23:12:08 UTC 2009
+}}}
+
+-----
+= HDFS權限管理用戶 =
+
+ * hdfs的權限有owner, group, other三種
+ * 而用戶的身份取決於client上的使用者 （用 whoami），群組為（bash -c groups）
+ * 相關的操作：
+{{{
+$ bin/hadoop dfs -mkdir own
+$ bin/hadoop dfs -chmod -R 755 own
+$ bin/hadoop dfs -chgrp -R waue own
+$ bin/hadoop dfs -chown -R waue own
+$ bin/hadoop dfs -lsr own
+}}}
+ * conf/hadoop-site.xml 可用參數：
+{{{
+#!php
+dfs.permissions = true 
+dfs.web.ugi = webuser,webgroup
+dfs.permissions.supergroup = supergroup
+dfs.upgrade.permission = 777
+dfs.umask = 022
+}}}
+
+=== -setQuota ===
+ * 目錄配額是對目錄樹上該目錄下的名字數量做硬性限制
+ * 設定配額，數字代表個數 （如：我上傳了一個2個block的檔案可以上傳，但我上傳兩個檔案很小的檔上去卻不行）
+ * 配額為1可以強制目錄保持為空
+ * 重命名不會改變該目錄的配額
+{{{
+$ bin/hadoop dfs -mkdir quota
+$ bin/hadoop dfsadmin -setQuota 2 quota
+$ bin/hadoop dfs -put ../conf/hadoop-env.sh quota/
+$ bin/hadoop dfs -put ../conf/hadoop-site.xml quota/
+
+put: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/waue/quota is exceeded: quota=2 count=3
+}}}
+
+ * 檢查目錄的配額方法： "bin/hadoop fs -count -q <目錄> "
+
+{{{
+$ bin/hadoop fs -count -q own
+   none  inf  1    0     0 hdfs://gm1.nchc.org.tw:9000/user/waue/own
+$ bin/hadoop dfsadmin -setQuota 4 own
+$ bin/hadoop fs -count -q own
+    4     3   1    0     0 hdfs://gm1.nchc.org.tw:9000/user/waue/own
+}}}
+
+ === -clrQuota ===
+ * 清除之前設定的配額
+{{{
+$ bin/hadoop dfsadmin -clrQuota quota/
+}}}
+
+-----
+= Hadoop Streaming 函式庫用法 =
+ * Hadoop streaming是Hadoop的一個工具， 它幫助用戶創建和運行一類特殊的map/reduce作業， 這些特殊的map/reduce作業是由一些可執行文件或腳本文件充當mapper或者reducer
+ * 最簡單的透過shell執行stream的map reduce:
+{{{
+$ bin/hadoop jar hadoop-0.18.3-streaming.jar -input input -output stream-output1 -mapper /bin/cat -reducer /usr/bin/wc
+}}}
+   * 輸出的結果為： (代表 行、字數、字元數)
+{{{
+#!sh
+2910628 24507806 143451003
+}}}