Best Java Script Web Scraping Tools

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
4 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
last month

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
760
Forks
60
Last commit
last month

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last commit
last year

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloudflare Workers

Run serverless code at the edge worldwide

Cloud Compute

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last commit
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
29
Forks
4
Last commit
5 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
39
Forks
12
Last commit
5 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
63
Forks
10
Last commit
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

SerpAPI

Real-Time Google Search Results API

Stars
116
Forks
11
Last commit
7 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
161
Forks
21
Last commit
5 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with 19,000+ ready-made tools and code templates.

2Captcha

CAPTCHA solving service with API integration

Stars
714
Forks
126
Last commit
5 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Patchright

Undetected Playwright fork for bot evasion

Stars
2.2K
Forks
88
Last commit
5 months ago

Patchright is a patched fork of Playwright that evades bot detection while maintaining full API compatibility for seamless migration.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.1K
Forks
400
Last commit
5 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last commit
5 months ago

parse5 is the fastest spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

Algolia

Managed search and AI retrieval platform

Stars
4K
Forks
557
Last commit
5 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Inngest

Durable Functions for Workflows and Queues

Stars
4.7K
Forks
240
Last commit
5 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last commit
5 months ago

htmlparser2 is the fastest Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

LLM-Scraper

Turn Any Webpage into Structured Data with LLMs

Stars
6.2K
Forks
369
Last commit
7 months ago

LLM Scraper is a TypeScript library that extracts structured data from any webpage using LLMs with Zod schemas, Playwright, and streaming support.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
8.3K
Forks
547
Last commit
5 months ago

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

LanceDB

Serverless vector database, no server management

Stars
8.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Cassandra

Distributed NoSQL with high availability at scale

Stars
9.6K
Forks
3.8K
Last commit
5 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.7K
Forks
2.7K
Last commit
5 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last commit
5 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Manticore Search

Fast open-source search database, 2.8x faster

Stars
11.6K
Forks
622
Last commit
5 months ago

Manticore Search is a fast, open-source search database that delivers 2.8x faster performance than Elasticsearch for full-text and analytical queries.

OpenSearch

Open-source search and analytics suite

Stars
12.3K
Forks
2.4K
Last commit
5 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Browserless

Headless browsers as a service

Stars
12.3K
Forks
949
Last commit
5 months ago

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with built-in bot detection bypass and CAPTCHA solving.

KeyDB

Multithreaded Redis fork with higher throughput

Stars
12.4K
Forks
647
Last commit
2 years ago

KeyDB is a multithreaded Redis-compatible database delivering over 1M ops/sec per node for high-throughput caching and queuing.

Memcached

Simple, fast distributed memory caching system

Stars
14.1K
Forks
3.3K
Last commit
7 months ago

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
4 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
last month

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
760
Forks
60
Last commit
last month

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last commit
last year

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloudflare Workers

Run serverless code at the edge worldwide

Cloud Compute

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last commit
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
29
Forks
4
Last commit
5 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
39
Forks
12
Last commit
5 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
63
Forks
10
Last commit
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

SerpAPI

Real-Time Google Search Results API

Stars
116
Forks
11
Last commit
7 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
161
Forks
21
Last commit
5 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with 19,000+ ready-made tools and code templates.

2Captcha

CAPTCHA solving service with API integration

Stars
714
Forks
126
Last commit
5 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Patchright

Undetected Playwright fork for bot evasion

Stars
2.2K
Forks
88
Last commit
5 months ago

Patchright is a patched fork of Playwright that evades bot detection while maintaining full API compatibility for seamless migration.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.1K
Forks
400
Last commit
5 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last commit
5 months ago

parse5 is the fastest spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

Algolia

Managed search and AI retrieval platform

Stars
4K
Forks
557
Last commit
5 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Inngest

Durable Functions for Workflows and Queues

Stars
4.7K
Forks
240
Last commit
5 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last commit
5 months ago

htmlparser2 is the fastest Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

LLM-Scraper

Turn Any Webpage into Structured Data with LLMs

Stars
6.2K
Forks
369
Last commit
7 months ago

LLM Scraper is a TypeScript library that extracts structured data from any webpage using LLMs with Zod schemas, Playwright, and streaming support.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
8.3K
Forks
547
Last commit
5 months ago

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

LanceDB

Serverless vector database, no server management

Stars
8.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Cassandra

Distributed NoSQL with high availability at scale

Stars
9.6K
Forks
3.8K
Last commit
5 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.7K
Forks
2.7K
Last commit
5 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last commit
5 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Manticore Search

Fast open-source search database, 2.8x faster

Stars
11.6K
Forks
622
Last commit
5 months ago

Manticore Search is a fast, open-source search database that delivers 2.8x faster performance than Elasticsearch for full-text and analytical queries.

OpenSearch

Open-source search and analytics suite

Stars
12.3K
Forks
2.4K
Last commit
5 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Browserless

Headless browsers as a service

Stars
12.3K
Forks
949
Last commit
5 months ago

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with built-in bot detection bypass and CAPTCHA solving.

KeyDB

Multithreaded Redis fork with higher throughput

Stars
12.4K
Forks
647
Last commit
2 years ago

KeyDB is a multithreaded Redis-compatible database delivering over 1M ops/sec per node for high-throughput caching and queuing.

Memcached

Simple, fast distributed memory caching system

Stars
14.1K
Forks
3.3K
Last commit
7 months ago

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

Best Java Script Web Scraping Tools

Discover the best web scraping tools and libraries for Java Script. Compare features, community support, and documentation.