Best Java Web Scraping Tools

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
981
Forks
282
Last commit
21 days ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

Apache Solr

Scalable Search For Scraped Content

Stars
1.6K
Forks
839
Last commit
21 days ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last commit
22 days ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

Apache Tika

Extract Text From Any Document

Stars
3.8K
Forks
941
Last commit
21 days ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Algolia

Managed search and AI retrieval platform

Stars
4K
Forks
557
Last commit
5 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Apache Hudi

Lakehouse Platform for Incremental Processing

Stars
6.1K
Forks
0

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Stars
8.5K
Forks
0

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Cassandra

Distributed NoSQL with high availability at scale

Stars
9.6K
Forks
3.8K
Last commit
5 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

OpenSearch

Open-source search and analytics suite

Stars
12.3K
Forks
2.4K
Last commit
5 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Weaviate

AI-native vector database with built-in ML models

Stars
15.5K
Forks
0

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Temporal

Durable Execution for Invincible Applications

Stars
17.9K
Forks
1.3K
Last commit
5 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Typesense

Typo-tolerant search with sub-50ms responses

Stars
25.1K
Forks
849
Last commit
5 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

MongoDB

Flexible document database for any data shape

Stars
28K
Forks
5.7K
Last commit
5 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Selenium

Browser Automation Framework and Ecosystem

Stars
33.9K
Forks
8.6K
Last commit
5 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

DuckDB

In-process SQL analytics, no server needed

Stars
35.6K
Forks
2.9K
Last commit
5 months ago

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Milvus

Open-source vector database for billion-scale AI

Stars
42.4K
Forks
0

Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.

ClickHouse

Real-time analytics database for petabyte-scale data

Stars
45.4K
Forks
8K
Last commit
5 months ago

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Meilisearch

Lightning-fast search engine with zero config

Stars
55.5K
Forks
2.3K
Last commit
5 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

MinIO

S3-compatible object storage you can self-host

Stars
59.9K
Forks
6.9K
Last commit
6 months ago

MinIO is a high-performance, S3-compatible object store for storing scraped files, datasets, and media at exascale.

Redis

In-memory data store for caching and queues

Stars
72.6K
Forks
24.5K
Last commit
5 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
75.9K
Forks
25.8K
Last commit
5 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Playwright

Cross-Browser Automation for Modern Web Apps

Stars
81.6K
Forks
5K
Last commit
5 months ago

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.