Changes between Version 2 and Version 3 of waue/2010/1029

Timestamp:

Oct 29, 2010, 3:25:54 PM (14 years ago)

Author:

waue

Comment:

--

waue/2010/1029

@@ -28 +28 @@
 
 == 手動修復步驟 ==
+
+
+{{{
+cd /opt/crawlzilla/nutch
+
+}}}
+
+ === index ===
+ * linkdb tw_yahoo_com_6/linkdb
+
+{{{
+#!java
+Usage: LinkDb <linkdb> (-dir <segmentsDir> | <seg1> <seg2> ...) 
+}}}
+
+{{{
+$ /opt/crawlzilla/nutch/bin/nutch invertlinks /user/crawler/cw_yahoo_5/linkdb -dir /user/crawler/cw_yahoo_5/segments/
+}}}
+
+ === index-lucene ===
+ 
+ * index-lucene tw_yahoo_com_6/indexes
+ 
+{{{
+#!java
+Usage: Indexer <index> <crawldb> <linkdb> <segment> ...
+}}}
+
+{{{
+$ /opt/crawlzilla/nutch/bin/nutch index /user/crawler/cw_yahoo_5/index /user/crawler/cw_yahoo_5/crawldb /user/crawler/cw_yahoo_5/linkdb /user/crawler/cw_yahoo_5/segments/20101027234843 /user/crawler/cw_yahoo_5/segments/20101027234956 /user/crawler/cw_yahoo_5/segments/20101027235315 /user/crawler/cw_yahoo_5/segments/20101028000804 /user/crawler/cw_yahoo_5/segments/20101028002826
+}}}
+
+
+ === dedup ===
+ 
+ * dedup 1: urls by time        100.00%
+ * dedup 2: content by hash     100.00%
+ * dedup 3: delete from index(es)
+{{{
+/opt/crawlzilla/nutch/bin/nutch dedup /user/crawler/cw_yahoo_5/index
+}}}