Changes between Initial Version and Version 1 of waue/2011/0727

Timestamp:

Jul 27, 2011, 3:39:10 PM (13 years ago)

Author:

waue

Comment:

--

waue/2011/0727

@@ +1 @@
+{{{
+#!html
+<div style="text-align: center; color:#151B8D"><big style="font-weight: bold;"><big><big>
+Nutch 1.3
+</big></big></big></div> <div style="text-align: center; color:#7E2217"><big style="font-weight: bold;"><big>
+
+</big></big></div>
+}}}
+[[PageOutline]]
+
+7 June 2011 - Apache Nutch 1.3 Released
+
+[http://ftp.twaren.net/Unix/Web/apache/nutch/ get nutch]
+
+ * conf/nutch-site.xml 
+
+{{{
+#!text
+<property>
+  <name>http.agent.name</name>
+  <value>waue_test</value>
+</property>
+}}}
+
+ * conf/crawl-urlfilter.txt
+
+{{{
+#!text
+-^(file|ftp|mailto):
+
+# skip image and other suffixes we can't yet parse
+-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
+
+# skip URLs containing certain characters as probable queries, etc.
+-[*!]
+
+# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
+#-.*(/[^/]+)/[^/]+\1/[^/]+\1/
+
+# accept anything else
++.
+}}}
+
+
+{{{
+echo "http://lucene.apache.org/nutch/" >urls/url.txt
+bin/nutch crawl urls -dir crawl2 -depth 2 -topN 50
+}}}