JavaScript Web Scraping Tools

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last recorded push
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
137
Forks
83
Last recorded push
4 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
774
Forks
60
Last recorded push
4 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last recorded push
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloudflare Workers

Run serverless code at the edge worldwide

Cloud Compute

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last recorded push
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
132
Forks
4
Last recorded push
8 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
51
Forks
12
Last recorded push
8 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
74
Forks
10
Last recorded push
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

SerpAPI

Real-Time Google Search Results API

Stars
176
Forks
11
Last recorded push
10 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
180
Forks
21
Last recorded push
8 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.

2Captcha

CAPTCHA solving service with API integration

Stars
803
Forks
126
Last recorded push
8 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.4K
Forks
400
Last recorded push
8 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last recorded push
8 months ago

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

Algolia

Managed search and AI retrieval platform

Stars
4.1K
Forks
557
Last recorded push
8 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Inngest

Durable Functions for Workflows and Queues

Stars
5.9K
Forks
240
Last recorded push
8 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last recorded push
8 months ago

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

LLM-Scraper

Structured Web Extraction with LLMs

Stars
6.9K
Forks
369
Last recorded push
10 months ago

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
9.5K
Forks
547
Last recorded push
8 months ago

BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.

LanceDB

Serverless vector database, no server management

Stars
11.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Cassandra

Distributed NoSQL with high availability at scale

Stars
10.1K
Forks
3.8K
Last recorded push
8 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.8K
Forks
2.7K
Last recorded push
8 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last recorded push
8 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Manticore Search

Open-source search database with SQL and HTTP APIs

Stars
12.1K
Forks
622
Last recorded push
8 months ago

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

OpenSearch

Open-source search and analytics suite

Stars
13.8K
Forks
2.4K
Last recorded push
8 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Browserless

Headless browsers as a service

Stars
13.8K
Forks
949
Last recorded push
8 months ago

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with self-hosted or cloud deployment options.

KeyDB

Multithreaded Redis fork with higher throughput

Stars
12.5K
Forks
647
Last recorded push
2 years ago

KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.

Memcached

Simple, fast distributed memory caching system

Stars
14.3K
Forks
3.3K
Last recorded push
9 months ago

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last recorded push
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
137
Forks
83
Last recorded push
4 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
774
Forks
60
Last recorded push
4 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last recorded push
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloudflare Workers

Run serverless code at the edge worldwide

Cloud Compute

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last recorded push
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
132
Forks
4
Last recorded push
8 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
51
Forks
12
Last recorded push
8 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
74
Forks
10
Last recorded push
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

SerpAPI

Real-Time Google Search Results API

Stars
176
Forks
11
Last recorded push
10 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
180
Forks
21
Last recorded push
8 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.

2Captcha

CAPTCHA solving service with API integration

Stars
803
Forks
126
Last recorded push
8 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.4K
Forks
400
Last recorded push
8 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last recorded push
8 months ago

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

Algolia

Managed search and AI retrieval platform

Stars
4.1K
Forks
557
Last recorded push
8 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Inngest

Durable Functions for Workflows and Queues

Stars
5.9K
Forks
240
Last recorded push
8 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last recorded push
8 months ago

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

LLM-Scraper

Structured Web Extraction with LLMs

Stars
6.9K
Forks
369
Last recorded push
10 months ago

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
9.5K
Forks
547
Last recorded push
8 months ago

LanceDB

Serverless vector database, no server management

Stars
11.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Cassandra

Distributed NoSQL with high availability at scale

Stars
10.1K
Forks
3.8K
Last recorded push
8 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.8K
Forks
2.7K
Last recorded push
8 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last recorded push
8 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Manticore Search

Open-source search database with SQL and HTTP APIs

Stars
12.1K
Forks
622
Last recorded push
8 months ago

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

OpenSearch

Open-source search and analytics suite

Stars
13.8K
Forks
2.4K
Last recorded push
8 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Browserless

Headless browsers as a service

Stars
13.8K
Forks
949
Last recorded push
8 months ago

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with self-hosted or cloud deployment options.

KeyDB

Multithreaded Redis fork with higher throughput

Stars
12.5K
Forks
647
Last recorded push
2 years ago

KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.

Memcached

Simple, fast distributed memory caching system

Stars
14.3K
Forks
3.3K
Last recorded push
9 months ago

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

JavaScript Web Scraping Tools

Discover web scraping tools and libraries for JavaScript. Compare features, community support, and documentation.