= Crawlzilla 2.0 = * 近期發現的 Bug / 缺點 * install 程式不支援無線網卡 * 升級/反安裝 -> 舊的資料如何保存或移植延續?!(Stateless) * Recrawl 進行時必須保留原本的 CrawlDB,等完成後才覆蓋掉。 * Fix Job 流程忘了刪除 HDFS 的 crawldb ? {{{ crawler@CrawlzillaServ:~$ /opt/crawlzilla/nutch/bin/hadoop fs -lsr jazz drwxr-xr-x - crawler supergroup 0 2012-09-14 21:59 /user/crawler/jazz drwxr-xr-x - crawler supergroup 0 2012-09-14 21:59 /user/crawler/jazz/wang drwxr-xr-x - crawler supergroup 0 2012-09-14 21:59 /user/crawler/jazz/wang/crawldb }}} * 想法: * 套件化(分離 Nutch, Lucene, Hadoop 的部份) - 預設用單機版 * 與 Solr 結合 -> 與 ElasticSearch 結合?! * 與 Carrot2 結合? 搜尋結果分群分類(Search Clustering) * 支援網頁截圖?!(2012-12-04) * http://blog.jangmt.com/2009/10/cutycapt.html * http://blog.saymoon.com/2009/11/take-snapshot-in-linux-command-line/ * https://github.com/istvan-antal/CutyCapt * http://code.google.com/p/minemine/wiki/WebPageGrabber * 套件相依: * bc {{{ 480: large16=$(echo "$JAVA_version >= 1.6" | bc) }}} * dialog * expect * lsb_release {{{ ./install: line 593: expect: command not found ./install: line 968: lsb_release: command not found }}} * 過去有人提報過的 Nutch Debian 套件 WNPP * http://lists.debian.org/debian-wnpp/2006/02/msg00225.html * 有人問我說抓抓龍與商業版的龍捲風有何優勢?! * http://www.tornado.com.tw/gov/ts