WEB数据挖掘（十六）——Aperture数据抽取（9）：数据源

One of the central concepts of Aperture is the notion of a DataSource. A DataSource contains all information necessary to locate the individual information resources in a physical source. For example, a FileSystemDataSource holds a root directory, a set of patterns that describe what files to include or exclude, a maximum depth, etc., thereby effectively describing a set of files.
One of the main purposes of a DataSource is to hold all data needed by a Crawlers to crawl the physical source and retrieve all the individual resources in it. There are quite a few DataSource subclasses in Aperture. The following diagram contains a selection of them.

The specific DataSource implementations available at the moment contain specific 'get' and 'set' methods for the configuration properties accepted by the data source. Thus providing a convenient interface and abstracting from the underlying RDF properties. All configuration data is stored in a RDFContainer. Each data source type comes with it's own specific properties. There is also a set of generic properties used by many data source types (username, password etc.). You can have a look at the source code of the DataSource implementation class of your choosing to see which properties are used. Note that the data source classes are not stored in the SVN. They are generated automatically from an RDF file with the description of the class. (like this one). The classes are generated by a maven plugin, by adding appropriate entries in the datasource module pom.xml file similar to these. If you'd like to develop your own data source implementation, try to mimic the existing implementations or ask at the aperture-devel for help.

It is worth mentioning, that DataSource classes only DESCRIBE a data source. They don't contain any resources that would enable direct access to the source (such as InputStreams, or Readers, whatever...). (At least it was not the intention of the designers). Any such resource is obtained by the crawler at the start of crawl and may be encapsulated in a DataObject returned by an Accessor or crawler. The following code demonstrates how to create and configure a FileSystemDataSource:

// determine the root folder of the source
File rootFolder = new File("D:\\path\\to\\the\\root\\folder");
// create the model that will store the data source configure
Model model = RDF2Go.getModelFactory().createModel();
// don't forget to open it before use
model.open();
// determine a URI to identify the DataSource
URI id = model.createURI("urn:test:testsource");
// wrap the model in an RDFContainer
RDFContainer configuration = new RDFContainerImpl(model,id);
// create the DataSource instance
FileSystemDataSource source = new FileSystemDataSource();
// set the configuration (it is empty at the moment)
source.setConfiguration(configuration)
// and set the rootFolder (you can do it now)
source.setRootFolder(rootFolder.getAbsolutePath());

转载于:https://www.cnblogs.com/chenying99/p/3194778.html