Self-Healing AI Web Data Pipelines
Platform or service · SaaS
AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.
Add Vector Search To PostgreSQL
Infrastructure · Self-hosted
Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.
Vector Databases
Scalable Java Crawlers For Teams
Library or tool · Self-hosted
Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.
Large-Scale & Distributed Crawling
Browser-Like Requests For Reliable Scraping
Library or tool · Library / Self-hosted
Stars: 776 · Code license: Apache-2.0 A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.
Scraping Frameworks
Extract Web Data Cleanly With Python
Library or tool · Library / Self-hosted
Stars: 1.4K · Code license: BSD-3-Clause A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.
HTML Parsers
Scalable Search For Scraped Content
Infrastructure · Self-hosted
Open-source Java search software for indexing scraped content with full-text search and faceted navigation.
Search Engines
Parse Real-World HTML In Java
Library or tool · Library / Self-hosted
Stars: 11.4K · Code license: MIT A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.
HTML Parsers
Fast HTML Parsing for Python Workflows
Library or tool · Library / Self-hosted
Stars: 1.7K · Code license: MIT Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.
HTML Parsers
Reliable Browser Automation For Python
Library or tool · Self-hosted
An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.
Browser Automation
Stealth Plugins For Puppeteer Scraping
Library or tool · Self-hosted
A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.
Fingerprint & Stealth Tooling
Browser-Like Requests Without Full Browsers
Library or tool · Self-hosted
Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.
Fingerprint & Stealth Tooling
Clean Documents For AI Pipelines
Library or tool · Self-hosted
An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.
Document Extraction
Turn Web Pages Into Usable Data
Platform or service · API
An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.
AI Web Scraping
Fast XML And HTML Parsing For Python
Library or tool · Library / Self-hosted
Stars: 3.1K · Code license: BSD-3-Clause A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.
HTML Parsers
Extract Text From Any Document
Library or tool · Self-hosted
Detects file types and extracts text and document details from 1,000+ formats through one Java interface.
Document Extraction
Turn Documents Into AI-Ready Data
Library or tool · Self-hosted
Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.
Document Extraction
AI-Ready Web Data Extraction
Platform or service · API
A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.
Scraping APIs
AI-Ready Scraping With Anti-Bot Bypass
Platform or service · API
A commercial scraping API for bypassing bot defenses, rendering JavaScript, using residential proxies, and returning AI-ready results.
Scraping APIs
Automate Web Workflows With AI
Library or tool · Library / Self-hosted
Stars: 25.7K · Code license: MIT Stagehand is an open-source SDK for AI-assisted browser automation with natural-language actions, structured extraction, and Playwright-style APIs.
Reliable Cloud Browsers for AI Agents
Platform or service · API
Run hosted browser sessions for AI automation and scraping with search, fetch, stealth, observability, and replay built in.
Headless Browsers
Turn web pages into structured data, no code
Platform or service · Desktop app / SaaS / API
Build scrapers visually and turn web pages into structured data in minutes, with scheduling and an optional managed data service.
Web Scraping
Generative AI Search API with Real-Time Research
Platform or service · Legacy API · API
Perplexity Sonar provides models for web-grounded answers. Support for Sonar Chat Completions ended on September 27, 2026. Perplexity recommends Agent API for new integrations; existing clients should follow its migration guide.
Web Search APIs
Multi-Engine Real-Time SERP API
Platform or service · API
SearchAPI.io delivers real-time SERP data from Google, Bing, Baidu, YouTube, and Amazon with structured JSON output.
SERP Extraction APIs
All-in-One Web Scraping API with AI Extraction
Platform or service · API
Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.
Scraping APIs
Semantic Web Search API Powered by AI
Platform or service · API
Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.
Web Search APIs
Real-Time Search API for AI Agents and RAG
Platform or service · API
Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.
Web Search APIs
Independent Web Search API with 35B+ Pages
Platform or service · API
Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.
Web Search APIs
S3-compatible storage with zero egress fees
Infrastructure · API / SaaS
Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.
Object Storage
Serverless SQL Queries on Amazon S3
Platform or service · API
Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.
Data Processing
Serverless Data Integration Service by AWS
Platform or service · API
AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.
ETL Tools
Fast and affordable proxy servers
Platform or service · API
Webshare provides self-serve proxy servers across many locations, offering datacenter and residential proxy options for scraping workflows.
Proxy Services
Premium proxies at affordable prices
Platform or service · API
IPRoyal offers residential, datacenter, ISP, and mobile proxy products with self-service access for scraping workflows.
Proxy Services
AI-ready proxy and scraping solutions
Platform or service · API
Smartproxy (now Decodo) offers residential and datacenter proxy products with rotating and sticky sessions for scalable scraping.
Proxy Services
Cloud platform for web services and cron jobs
Infrastructure · API / SaaS
Render is a cloud platform that deploys web services, background workers, and cron jobs with simple scaling for scraping workloads.
Cloud Compute
Deploy apps from GitHub through configured exporters or custom item pipelines
Infrastructure · API / SaaS
Railway deploys web apps and services from GitHub with auto-scaling, built-in databases, and instant previews for scraping workloads.
Cloud Compute
Serverless compute for event-driven pipelines
Infrastructure · API / SaaS
AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.
Cloud Compute
