Changes between Version 5 and Version 6 of jazz/09-08-27

Timestamp:

Aug 28, 2009, 2:46:53 AM (15 years ago)

Author:

jazz

Comment:

--

jazz/09-08-27

@@ -71 +71 @@
  * [計畫] Hadoop 叢集維護
   * [狀況] 發現把副本設為 3 會造成空間不足，此外因為檔案數多，但都是小檔案，或許不該把 block size 設為 64Mbytes (67108864)
-  * [解法] 把副本設為 2 並該把 block size 設為 4kbytes (4096)
+  * [解法] 
+　　　1. 把副本設為 2 並該把 block size 設為 4kbytes (4096)
+   2. '由於 Block Size 只有__'''檔案產生'''__的時候才會生效,因此目前看起來除了跑 fsck 外，還要跑 balancer，有搬動過的才有機會變小。其次就是用 hadoop fs -cp /user /user2 比較有機會全面性的造成影響。
+{{{
+09/08/27 14:37:31 INFO dfs.Balancer: Decided to move block -2997460447318763427 with a length of 4 KB bytes from 192.168.1.18:50010 to 192.168.1.2:50010 using proxy source 192.168.1.18:50010
+09/08/27 14:37:31 INFO dfs.Balancer: Starting moving -2997460447318763427 from 192.168.1.18:50010 to 192.168.1.2:50010
+09/08/27 14:37:31 INFO dfs.Balancer: Decided to move block -2997680912701056691 with a length of 4 KB bytes from 192.168.1.18:50010 to 192.168.1.2:50010 using proxy source 192.168.1.18:50010
+09/08/27 14:37:31 INFO dfs.Balancer: Starting moving -2997680912701056691 from 192.168.1.18:50010 to 192.168.1.2:50010
+09/08/27 14:37:31 INFO dfs.Balancer: Decided to move block 248102854142724550 with a length of 13.36 KB bytes from 192.168.1.6:50010 to 192.168.1.5:50010 using proxy source 192.168.1.6:50010
+}}}
   * [統計] 目前平均 file size 大概都落在 1Kbytes 以下。
   * [[Image(wiki:jazz/09-08-27:hadoop_file_size_block_size.png,width=400)]]