Focused Web Crawler: “Closed” Directories should be run first

by bose on May 25, 2011

Our focused crawler sometimes only finds results within the starting URL directory. This is most common with Applegate, Zibb and EngineeringTalk.

I call these “closed” because they only produce results from within the domain.

The results are accurate and precise, so this directory run could be used to find relevant targets for a similarity set quickly and so aid the wider search later on.

Previous post:

Next post: