Frontera

How to use Frontera in a web scraping pipeline

Frontera manages a crawl frontier: which URLs should be visited, when, and in what order. It complements a fetcher such as Scrapy instead of replacing the code that downloads pages and extracts records.

A practical workflow

1. Supply seed URLs and choose the frontier backend and scheduling strategy for your crawl goal. Keep URL identity rules consistent with the collector.
2. Let workers obtain scheduled requests and return response information plus newly discovered links to the frontier. Use those observations to guide subsequent visits.
3. Send extracted business records to their own destination while the frontier retains the request state needed for continued discovery and revisits.

When it fits

It fits crawls where URL prioritization, frontier size or coordination across workers has become a separate engineering problem. A focused or recurring crawl can then change scheduling strategy without rewriting every spider.

Boundaries

A frontier decision is not a successful fetch or a validated record. Your workers still implement HTTP or browser access, parsing and output handling. Backend and message-bus choices affect the operational setup; assess the documented stack and its current compatibility before adopting it.