Frontera

Frontera is a scalable, modular crawl frontier framework that manages crawl scheduling, prioritization, and goal tracking for web crawlers of any size.

Key Features:

• Crawl Frontier - Manages URL prioritization, scheduling, and deduplication at scale
• Scrapy Integration - Drop-in components for distributing Scrapy crawlers
• Distributed Workers - Scale crawling across multiple workers with message bus backends
• Pluggable Backend - Support for SQLAlchemy, HBase, and custom storage backends
• Goal Tracking - Monitor crawl progress and detect when objectives are accomplished

Whether you're orchestrating large-scale crawls, managing URL frontiers for distributed scrapers, or building custom crawling infrastructure, Frontera provides the scheduling framework to crawl smarter at any scale.