Example-Based HTML Extraction For Python
Library or tool · Library / Self-hosted
Stars: 8.1K · Code license: MIT Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.
Scalable Java Crawlers For Teams
Library or tool · Self-hosted
Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.
Large-Scale & Distributed Crawling
Distributed on-demand scraping with Scrapy
Library or tool · Self-hosted
Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.
Large-Scale & Distributed Crawling
Scalable crawl frontier framework
Library or tool · Self-hosted
Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.
Large-Scale & Distributed Crawling
Highly extensible and scalable web crawler
Library or tool · Self-hosted
Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.
Large-Scale & Distributed Crawling
Web-scale archival web crawler
Library or tool · Self-hosted
Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.
Large-Scale & Distributed Crawling
Automate website interaction with Python
Library or tool · Library / Self-hosted
Stars: 4.9K · Code license: MIT MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.
Scraping Frameworks
Redis-based distributed components for Scrapy
Library or tool · Self-hosted
Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.
Large-Scale & Distributed Crawling
Structured Web Extraction with LLMs
Library or tool · Library / Self-hosted
Stars: 6.9K · Code license: MIT LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.
Adaptive web scraping for the modern web
Library or tool · Framework / Self-hosted
Stars: 87.9K · Code license: BSD-3-Clause Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.
Scraping Frameworks
AI Browser Automation with Computer Vision
Library or tool · Framework / Self-hosted
Stars: 23.2K · Code license: AGPL-3.0 Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that can adapt workflows to page layouts.
Build reliable web scrapers. Fast.
Library or tool · Framework / Self-hosted
Stars: 26.2K · Code license: Apache-2.0 Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.
Browser Automation +2 more
AI-Powered Web Scraping API for Agents
Library or tool · Library / Self-hosted
Stars: 31.8K · Code license: MIT ScrapeGraphAI extracts structured data from web pages using AI, handling proxies, rendering, and adapting to site changes automatically.
Elegant scraping and crawler framework for Golang
Library or tool · Framework / Self-hosted
Stars: 25.6K · Code license: Apache-2.0 Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.
Scraping Frameworks
Open-Source LLM-Friendly Web Crawler
Library or tool · Framework / Self-hosted
Stars: 85.6K · Code license: Apache-2.0 Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.
Fast, high-level web crawling framework for Python
Library or tool · Framework / Self-hosted
Stars: 64.8K · Code license: BSD-3-Clause Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.
Scraping Frameworks
Open-Source AI Browser Automation Platform
Library or tool · Self-hosted
Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.
