Develop Customizable Web Crawler Using WebSphinx

WebSPHINX ( Website-Specific Processors for HTML INformation eXtraction) is a Java class library

and interactive development environment for web crawlers. As its home page's title, Websphinx is

aimed to be a personal , customizable Web crawler.

 

1 Package Overview

The architecture of Websphinx is quite straight forward.  As shown blow,

Figure1

the package websphinx contains the core function of the crawler.

websphinx.workbench implements the userinterface.

websphinx.searchengine has classes to call several search engines such as Google, Excite, etc.

Most of the search engine classifiers were written in 1998.  Search engines have changed the

format of their results many times since then, so the classifiers are out of date.

rcm.awt, rcm.util, org.apache.regexp are utility package. websphinx.workbench gets the graphical

support from rcm.awt. websphinx using PrioritizeQueue provided by rcm.util. org.apache.regexp

helps websphinx with pattern matching work.

 

2 Implement Your Own Crawler

Figure2

I suggest you focus on package websphinx and leave websphinx.workbench and

websphinx.searchengine behind, since it completely implements the funcion of the cawler.

Figure2 is inheritance diagram of package websphinx.

Class Crawler is the main class of the crawler. Pages are download in class Page. Class Access

provide the method that generate a connection to the server. Class es such asLinkPredicate,

PagePredicate, are used to decide whether the content should be visited or not. Pay attention to

DownloadParameters, which holds the configuration of downloading a page, like max threads, max

pagesize, download timeout, etc. You can add your configuration to it.