Changes between Version 4 and Version 5 of waue/2009/0609

Timestamp:

Jun 9, 2009, 4:33:59 PM (16 years ago)

Author:

waue

Comment:

--

waue/2009/0609

@@ -8 +8 @@
 
  === readdb ===
-
+ - read / dump crawl db
+ - Usage: CrawlDbReader <crawldb> (-stats | -dump <out_dir> | -topN <nnnn> <out_dir> [<min>] | -url <url>)
+ - -stats [-sort]       print overall statistics to System.out
 {{{
 $ nutch readdb /tmp/search/crawldb -stats
 
 09/06/09 12:18:13 INFO mapred.MapTask: data buffer = 79691776/99614720
-
 09/06/09 12:18:13 INFO mapred.MapTask: record buffer = 262144/327680
-
 09/06/09 12:18:14 INFO crawl.CrawlDbReader: TOTAL urls: 1072
 09/06/09 12:18:14 INFO crawl.CrawlDbReader: status 1 (db_unfetched):    1002
+09/06/09 12:18:14 INFO crawl.CrawlDbReader: status 2 (db_fetched):      68
+}}} 
+ - -dump <out_dir> [-format normal|csv ]        dump the whole db to a text file in <out_dir>
+ - -url <url>   print information on <url> to System.out
+ - -topN <nnnn> <out_dir> [<min>]       dump top <nnnn> urls sorted by score to <out_dir>
 
-09/06/09 12:18:14 INFO crawl.CrawlDbReader: status 2 (db_fetched):      68
+ === inject ===
+ - inject new urls into the database
+ - Usage: Injector <crawldb> <url_dir>
 
-}}}
- === convdb ===
- - 沒啥用
- === inject ===
- 
+ === readlinkdb ===
+ - read / dump link db
+ - Usage: LinkDbReader <linkdb> {-dump <out_dir> | -url <url>)
 {{{
-
-}}}
- === readlinkdb ===
-{{{
-
-}}}
-
- ===  ===
-{{{
-
+$ nutch readlinkdb /tmp/search/linkdb/ -dump ./dump
+$ vim ./dump/part-00000
 }}}
  === readseg ===
  - read / dump segment data
- - Usage: SegmentReader (-dump ... | -list ... | -get ...) [general options]
- - SegmentReader -dump <segment_dir> <output> [general options]
- - SegmentReader -list (<segment_dir1> ... | -dir <segments>) [general options]
- - SegmentReader -get <segment_dir> <keyValue> [general options]
+ - Usage: !SegmentReader (-dump ... | -list ... | -get ...) [general options]
+ - !SegmentReader -dump <segment_dir> <output> [general options]
+ - !SegmentReader -list (<segment_dir1> ... | -dir <segments>) [general options]
+ - !SegmentReader -get <segment_dir> <keyValue> [general options]
 {{{
 
@@ -49 +46 @@
  === updatedb ===
  - update crawl db from segments after fetching
- - Usage: CrawlDb <crawldb> (-dir <segments> | <seg1> <seg2> ...) [-force] [-normalize] [-filter] [-noAdditions]
+ - Usage: !CrawlDb <crawldb> (-dir <segments> | <seg1> <seg2> ...) [-force] [-normalize] [-filter] [-noAdditions]
 {{{
 $ nutch updatedb /tmp/search/crawldb/ -dir /tmp/search/segments/
@@ -55 +52 @@
  === dedup ===
  - remove duplicates from a set of segment indexes
- - Usage: DeleteDuplicates <indexes> ...
+ - Usage: !DeleteDuplicates <indexes> ...
 {{{
 $ nutch dedup /tmp/search/indexes/
-
 }}}
  == 筆記 ==