JavaScript Web Scraping Tools

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Library or tool · Library / Self-hosted

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemap Scrapers

Sitemapper

Parse Sitemaps And Indexes In Node.js

Library or tool · Library / Self-hosted

Stars: 137
Code license: MIT

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Sitemap Scrapers

got-scraping

Browser-Like Requests For Reliable Scraping

Library or tool · Library / Self-hosted

Stars: 776
Code license: Apache-2.0

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Scraping Frameworks

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Library or tool · Self-hosted

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Fingerprint & Stealth Tooling

Cloudflare R2

S3-compatible storage with zero egress fees

Infrastructure · API / SaaS

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

Object Storage

AWS Lambda

Serverless compute for event-driven pipelines

Infrastructure · API / SaaS

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloud Compute

Cloudflare Workers

Run serverless code at the edge worldwide

Infrastructure · API / SaaS

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

Cloud Compute

Patchright

Undetected Playwright fork for bot evasion

Library or tool · Self-hosted

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Browser Automation

+1 more

Scrapoxy

End-of-life proxy orchestration project

Library or tool · Retired · Self-hosted

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

Proxy Services

BeeAI Framework

AI agents and multi-agent workflows

Library or tool · Framework / Self-hosted

Stars: 3.4K
Code license: Apache-2.0

Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.

AI Agent Tooling

parse5

HTML5 spec-compliant parser for Node.js

Library or tool · Library / Self-hosted

Stars: 3.9K
Code license: MIT

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

HTML Parsers

Inngest

Durable Functions for Workflows and Queues

Library or tool · Self-hosted

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

Workflow Orchestration

htmlparser2

The fast and forgiving HTML/XML parser

Library or tool · Library / Self-hosted

Stars: 4.8K
Code license: MIT

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

HTML Parsers

LLM-Scraper

Structured Web Extraction with LLMs

Library or tool · Library / Self-hosted

Stars: 6.9K
Code license: MIT

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

AI Web Scraping

+1 more

BullMQ

Message Queue and Background Jobs for Node.js

Library or tool · Self-hosted

BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.

Task Queues

LanceDB

Serverless vector database, no server management

Infrastructure · Self-hosted

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Vector Databases

Cassandra

Distributed NoSQL with high availability at scale

Infrastructure · Self-hosted

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

NoSQL Databases

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Library or tool · Self-hosted

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

Browser Automation

+1 more

TestCafe

WebDriver-Free End-to-End Browser Testing

Library or tool · Self-hosted

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Browser Automation

Manticore Search

Open-source search database with SQL and HTTP APIs

Infrastructure · Self-hosted

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

Search Engines

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

KeyDB

Multithreaded Redis fork with higher throughput

Infrastructure · Self-hosted

KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.

Caching Databases

Memcached

Simple, fast distributed memory caching system

Infrastructure · Self-hosted

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

Caching Databases

Weaviate

AI-native vector database with built-in ML models

Infrastructure · Self-hosted

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Vector Databases

Windmill

Build and Deploy Internal Tools at Scale

Library or tool · Self-hosted

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Workflow Orchestration

QuestDB

Time-series database with ultra-low latency

Infrastructure · Self-hosted

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Analytics Databases

Temporal

Durable Execution for Invincible Applications

Library or tool · Self-hosted

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Workflow Orchestration

Mastra

TypeScript AI Agent Framework with Workflows

Library or tool · Framework / Self-hosted

Stars: 28.8K
Code license: Apache-2.0 (outside ee/)

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

AI Agent Tooling

Crawlee

Build reliable web scrapers. Fast.

Library or tool · Framework / Self-hosted

Stars: 26.2K
Code license: Apache-2.0

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Browser Automation

+2 more

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

Chroma

AI-native embedding database with simple API

Infrastructure · Self-hosted

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Vector Databases

Kestra

Declarative Data Orchestration Platform

Library or tool · Self-hosted

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Workflow Orchestration

MongoDB

Flexible document database for any data shape

Infrastructure · API / SaaS / Self-hosted

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

NoSQL Databases

Qdrant

High-performance vector search engine in Rust

Infrastructure · Self-hosted

Stars: 35.1K
Code license: Apache-2.0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

Vector Databases

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Library or tool · Library / Self-hosted

Stars: 30.5K
Code license: MIT

Cheerio is a JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

HTML Parsers

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Library or tool · Library / Self-hosted

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemap Scrapers

Sitemapper

Parse Sitemaps And Indexes In Node.js

Library or tool · Library / Self-hosted

Stars: 137
Code license: MIT

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Sitemap Scrapers

got-scraping

Browser-Like Requests For Reliable Scraping

Library or tool · Library / Self-hosted

Stars: 776
Code license: Apache-2.0

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Scraping Frameworks

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Library or tool · Self-hosted

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Fingerprint & Stealth Tooling

Cloudflare R2

S3-compatible storage with zero egress fees

Infrastructure · API / SaaS

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

Object Storage

AWS Lambda

Serverless compute for event-driven pipelines

Infrastructure · API / SaaS

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloud Compute

Cloudflare Workers

Run serverless code at the edge worldwide

Infrastructure · API / SaaS

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

Cloud Compute

Patchright

Undetected Playwright fork for bot evasion

Library or tool · Self-hosted

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Browser Automation

+1 more

Scrapoxy

End-of-life proxy orchestration project

Library or tool · Retired · Self-hosted

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

Proxy Services

BeeAI Framework

AI agents and multi-agent workflows

Library or tool · Framework / Self-hosted

Stars: 3.4K
Code license: Apache-2.0

Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.

AI Agent Tooling

parse5

HTML5 spec-compliant parser for Node.js

Library or tool · Library / Self-hosted

Stars: 3.9K
Code license: MIT

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

HTML Parsers

Inngest

Durable Functions for Workflows and Queues

Library or tool · Self-hosted

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

Workflow Orchestration

htmlparser2

The fast and forgiving HTML/XML parser

Library or tool · Library / Self-hosted

Stars: 4.8K
Code license: MIT

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

HTML Parsers

LLM-Scraper

Structured Web Extraction with LLMs

Library or tool · Library / Self-hosted

Stars: 6.9K
Code license: MIT

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

AI Web Scraping

+1 more

BullMQ

Message Queue and Background Jobs for Node.js

Library or tool · Self-hosted

Task Queues

LanceDB

Serverless vector database, no server management

Infrastructure · Self-hosted

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Vector Databases

Cassandra

Distributed NoSQL with high availability at scale

Infrastructure · Self-hosted

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

NoSQL Databases

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Library or tool · Self-hosted

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

Browser Automation

+1 more

TestCafe

WebDriver-Free End-to-End Browser Testing

Library or tool · Self-hosted

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Browser Automation

Manticore Search

Open-source search database with SQL and HTTP APIs

Infrastructure · Self-hosted

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

Search Engines

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

KeyDB

Multithreaded Redis fork with higher throughput

Infrastructure · Self-hosted

KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.

Caching Databases

Memcached

Simple, fast distributed memory caching system

Infrastructure · Self-hosted

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

Caching Databases

Weaviate

AI-native vector database with built-in ML models

Infrastructure · Self-hosted

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Vector Databases

Windmill

Build and Deploy Internal Tools at Scale

Library or tool · Self-hosted

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Workflow Orchestration

QuestDB

Time-series database with ultra-low latency

Infrastructure · Self-hosted

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Analytics Databases

Temporal

Durable Execution for Invincible Applications

Library or tool · Self-hosted

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Workflow Orchestration

Mastra

TypeScript AI Agent Framework with Workflows

Library or tool · Framework / Self-hosted

Stars: 28.8K
Code license: Apache-2.0 (outside ee/)

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

AI Agent Tooling

Crawlee

Build reliable web scrapers. Fast.

Library or tool · Framework / Self-hosted

Stars: 26.2K
Code license: Apache-2.0

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Browser Automation

+2 more

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

Chroma

AI-native embedding database with simple API

Infrastructure · Self-hosted

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Vector Databases

Kestra

Declarative Data Orchestration Platform

Library or tool · Self-hosted

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Workflow Orchestration

MongoDB

Flexible document database for any data shape

Infrastructure · API / SaaS / Self-hosted

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

NoSQL Databases

Qdrant

High-performance vector search engine in Rust

Infrastructure · Self-hosted

Stars: 35.1K
Code license: Apache-2.0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

Vector Databases

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Library or tool · Library / Self-hosted

Stars: 30.5K
Code license: MIT

Cheerio is a JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

HTML Parsers

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation

JavaScript Web Scraping Tools

Discover web scraping tools and libraries for JavaScript. Compare features, community support, and documentation.