Run serverless code at the edge worldwide
Infrastructure · API / SaaS
Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.
Cloud Compute
Scale data collection with a simple API
Platform or service · API
ScraperAPI provides managed web scraping with proxy rotation, JavaScript rendering and CAPTCHA handling for supported public websites.
Scraping APIs
Web scraping API with proxy and browser handling
Platform or service · API
ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.
Scraping APIs
AI-powered automatic CAPTCHA solving service
Platform or service · API
CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.
CAPTCHA Solvers
Proxy services and web scraping APIs
Platform or service · API
Oxylabs provides proxy services, web scraping APIs and managed browser tools for collecting public web data.
Real-Time Google Search Results API
Platform or service · API
SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.
SERP Extraction APIs
Python library for parsing HTML and XML
Library or tool · Library / Self-hosted
Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.
HTML Parsers
Serverless Python platform for AI and data teams
Infrastructure · API / SaaS
Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with configurable containers and autoscaling.
Cloud Compute
CAPTCHA solving service with API integration
Platform or service · API
2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.
CAPTCHA Solvers
Async-first undetected browser automation
Library or tool · Self-hosted
Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.
Browser Automation +1 more
Scalable crawl frontier framework
Library or tool · Self-hosted
Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.
Large-Scale & Distributed Crawling
Deploy containers globally on edge servers
Infrastructure · API / SaaS
Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.
Cloud Compute
CLI-First ELT Engine Built on Singer
Library or tool · Self-hosted
Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.
ETL Tools
End-of-life proxy orchestration project
Library or tool · Retired · Self-hosted
Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.
Proxy Services
Highly extensible and scalable web crawler
Library or tool · Self-hosted
Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.
Large-Scale & Distributed Crawling
Managed search and AI retrieval platform
Infrastructure · API / SaaS
Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.
Search Engines
Durable Functions for Workflows and Queues
Library or tool · Self-hosted
Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.
Workflow Orchestration
The fast and forgiving HTML/XML parser
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: MIT htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.
HTML Parsers
Python Library for Data Loading
Library or tool · Self-hosted
dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.
Data Transformation & Validation +1 more
Automate website interaction with Python
Library or tool · Library / Self-hosted
Stars: 4.9K · Code license: MIT MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.
Scraping Frameworks
Lightweight Python Task Queue
Library or tool · Self-hosted
Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.
Task Queues
Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
AWS managed serverless NoSQL database
Infrastructure · API / SaaS
Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.
NoSQL Databases
Message Queue and Background Jobs for Node.js
Library or tool · Self-hosted
BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.
Task Queues
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Open-Source Lakehouse Storage Framework
Infrastructure · Self-hosted
Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.
Lakehouse & Table Formats
Serverless vector database, no server management
Infrastructure · Self-hosted
LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.
Vector Databases
AI-Powered Data Pipeline Builder
Library or tool · Self-hosted
Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.
Workflow Orchestration
Adaptive web scraping for the modern web
Library or tool · Framework / Self-hosted
Stars: 87.9K · Code license: BSD-3-Clause Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.
Scraping Frameworks
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Next-Gen Browser and Mobile Automation for Node.js
Library or tool · Self-hosted
WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.
Browser Automation +1 more
WebDriver-Free End-to-End Browser Testing
Library or tool · Self-hosted
TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.
Browser Automation
Simple Job Queues for Python
Library or tool · Self-hosted
RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.
Task Queues
Transform Data in Your Warehouse with SQL
Library or tool · Self-hosted
dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.
Data Transformation & Validation
Data Quality Testing and Validation Platform
Library or tool · Self-hosted
Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.
Data Transformation & Validation
Sub-second analytics for enterprise-scale data
Infrastructure · Self-hosted
StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.
Analytics Databases
bisibility
Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.
Try bisibility free Our product 
