Changes between Version 8 and Version 9 of crawlzilla/stress_testing

Timestamp:

Sep 24, 2010, 10:27:15 AM (14 years ago)

Author:

shunfa

Comment:

--

crawlzilla/stress_testing

@@ -76 +76 @@
  * 爬取3層後，執行"echo 1>/proc/sys/vm/drop_caches"清除記憶體
  * 爬取4層後，執行"echo 1>/proc/sys/vm/drop_caches"清除記憶體
- * 爬取第5層時，系統停在下列進度超過12HR，無Fail Job, log檔案也無其他錯誤訊息，系統進度如下表：
+ * 爬取第5層時，系統進度及相關錯誤訊息如下：
 ||Jobid||Priority||User||Name||Map % Complete||Reduce % Complete||
 ||job_201009231019_0059||NORMAL||crawler||fetch NCHC_5/segments/20100923162900||100.00%||0.00%||
- * go.sh 仍執行中
-  * crawler  30226  0.0  0.1   4332  1124 ?        S    Sep23   0:00 /bin/bash /home/crawler/crawlzilla/system/go.sh 5 NCHC_5
+ 
+ * 錯誤訊息：
+{{{
+org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/crawler/NCHC_5/segments/20100923162900/crawl_fetch/part-00000/index could only be replicated to 0 nodes, instead of 1
+        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1280)
+        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
+        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+        at java.lang.reflect.Method.invoke(Method.java:597)
+        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
+        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
+
+        at org.apache.hadoop.ipc.Client.call(Client.java:697)
+        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
+        at $Proxy1.addBlock(Unknown Source)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+        at java.lang.reflect.Method.invoke(Method.java:597)
+        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
+        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
+        at $Proxy1.addBlock(Unknown Source)
+        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2814)
+        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2696)
+        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996)
+        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
+}}}
+
  * 記憶體狀態(單位/M)：
 ||||total||used||free||