org.semanticdesktop.aperture.subcrawler
Interface SubCrawler

All Known Implementing Classes:

AbstractArchiverSubCrawler, AbstractCompressorSubCrawler, AbstractSubCrawler, BZip2SubCrawler, CompressSubCrawler, GZipSubCrawler, IcalCrawler, MboxSubCrawler, MimeSubCrawler, TarSubCrawler, VcardSubCrawler, ZipSubCrawler

public interface SubCrawler

A SubCrawler accesses an InputStream and produces a stream of other DataObjects representing the resources found "inside".

An AccessData instance can optionally be specified to a SubCrawler, allowing it to perform incremental crawling, i.e. to scan and report the differences in the stream since the last crawl.

The uris of the data objects found inside other data objects have a fixed form, consisting of three basic parts:

 <prefix>:<parent-object-uri>!/<path>
 

• <prefix> - the uri prefix, characteristic for a particular SubCrawler, returned by the SubCrawlerFactory.getUriPrefix() method
• <parent-object-uri> - the uri of the parent data object, it is obtained from the parentMetadata parameter to the subCrawl(org.ontoware.rdf2go.model.node.URI, java.io.InputStream, org.semanticdesktop.aperture.subcrawler.SubCrawlerHandler, org.semanticdesktop.aperture.datasource.DataSource, org.semanticdesktop.aperture.accessor.AccessData, java.nio.charset.Charset, java.lang.String, org.semanticdesktop.aperture.rdf.RDFContainer) method, by calling RDFContainer.getDescribedUri()
• <path> - an internal path of the 'child' data object inside the 'parent' data object

This scheme has been inspired by the apache commons VFS project, homepaged under http://commons.apache.org/vfs

Method Summary

DataObject	getDataObject(URI parentUri, String path, InputStream stream, DataSource dataSource, Charset charset, String mimeType, RDFContainerFactory factory) Get a DataObject from the specified stream with the given path.
void	stopSubCrawler() Stops a running crawl as fast as possible.
void	subCrawl(URI id, InputStream stream, SubCrawlerHandler handler, DataSource dataSource, AccessData accessData, Charset charset, String mimeType, RDFContainer parentMetadata) Starts crawling the given stream and to report the encountered DataObjects to the given SubCrawlerHandler.

Method Detail

subCrawl

void subCrawl(URI id,
              InputStream stream,
              SubCrawlerHandler handler,
              DataSource dataSource,
              AccessData accessData,
              Charset charset,
              String mimeType,
              RDFContainer parentMetadata)
              throws SubCrawlerException

Starts crawling the given stream and to report the encountered DataObjects to the given SubCrawlerHandler. If an AccessData instance is passed, it is used to check if the data objects are to be reported as new, modified, or unmodified. Note that the SubCrawler will not report deleted objects.

Parameters:

id - the URI identifying the object (e.g. a file or web page) from which the stream was obtained. This URI is treated as the URI of the parent object, all objects encountered in the stream are considered to be contained within the parent object. (optional, the implementation may use this uri or the one returned from the RDFContainer.getDescribedUri() method of the parentMetadata)

stream - the stream to be crawled. (obligatory)

accessData - the AccessData used to determine if the encountered objects are to be returned as new, modified, unmodified or deleted. Information about new or modified objects is stored within for use in future crawls. This parameter may be null if this functionality is not desired, in which case all DataObjects will be reported as new. (optional)

handler - The crawler handler that is to receive the notifications from the SubCrawler (obligatory)

dataSource - the data source that will be returned by the DataObject.getDataSource() method of the returned data objects. Some implementations may require that this reference is not null and that it contains some particular information

charset - the charset in which the input stream is encoded (optional).

mimeType - the MIME type of the passed stream (optional).

parentMetadata - The 'parent' RDFContainer, that will contain the metadata about the top-level entity in the stream. A SubCrawler may (in some cases) limit itself to augmenting the metadata in this RDFContainer without delivering any additional DataObjects. (obligatory)

Throws:

SubCrawlerException - if any of the obligatory parameters is null or if any error during the crawling process occured

getDataObject

DataObject getDataObject(URI parentUri,
                         String path,
                         InputStream stream,
                         DataSource dataSource,
                         Charset charset,
                         String mimeType,
                         RDFContainerFactory factory)
                         throws SubCrawlerException,
                                PathNotFoundException

Get a DataObject from the specified stream with the given path.

Parameters:

parentUri - the URI of the parent object where the path will be looked for

path - the path of the requested resource

stream - the stream that contains the resource

dataSource - data source that will be returned by the DataObject.getDataSource() method of the returned data object. Some implementations may require that this reference is not null and that it contains some particular information

charset - the charset in which the input stream is encoded (optional).

mimeType - the MIME type of the passed stream (optional).

factory - An RDFContainerFactory that delivers the RDFContainer to which the metadata of the DataObject should be added. The provided RDFContainer can later be retrieved as the DataObject's metadata container.

Returns:

The DataObject extracted from the given stream with the given path

Throws:

SubCrawlerException - if any I/O error occurs

PathNotFoundException - if the requested path is not found

stopSubCrawler

void stopSubCrawler()

Stops a running crawl as fast as possible. This method may return before the crawling has actually stopped.