Java Web Scraping Tools

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
1K
Forks
282
Last recorded push
3 months ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

Apache Solr

Scalable Search For Scraped Content

Stars
1.7K
Forks
839
Last recorded push
3 months ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last recorded push
3 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

Apache Tika

Extract Text From Any Document

Stars
4.1K
Forks
941
Last recorded push
3 months ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Algolia

Managed search and AI retrieval platform

Stars
4.1K
Forks
557
Last recorded push
8 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Apache Hudi

Lakehouse Platform for Incremental Processing

Stars
6.3K
Forks
0

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Stars
9.3K
Forks
0

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Cassandra

Distributed NoSQL with high availability at scale

Stars
10.1K
Forks
3.8K
Last recorded push
8 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

OpenSearch

Open-source search and analytics suite

Stars
13.8K
Forks
2.4K
Last recorded push
8 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Weaviate

AI-native vector database with built-in ML models

Stars
16.9K
Forks
0

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Temporal

Durable Execution for Invincible Applications

Stars
23.6K
Forks
1.3K
Last recorded push
8 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Typesense

Typo-tolerant search engine with faceted filtering

Stars
26.9K
Forks
849
Last recorded push
8 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

MongoDB

Flexible document database for any data shape

Stars
28.6K
Forks
5.7K
Last recorded push
8 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Selenium

Browser Automation Framework and Ecosystem

Stars
34.5K
Forks
8.6K
Last recorded push
8 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

DuckDB

In-process SQL analytics, no server needed

Stars
42.1K
Forks
2.9K
Last recorded push
8 months ago

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Milvus

Open-source vector database for billion-scale AI

Stars
46.4K
Forks
0

Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.

ClickHouse

Real-time analytics database for petabyte-scale data

Stars
50.3K
Forks
8K
Last recorded push
8 months ago

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Stars
59.5K
Forks
2.3K
Last recorded push
8 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

MinIO

Archived community S3-compatible object store

Stars
61.3K
Forks
6.9K
Last recorded push
9 months ago

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Redis

In-memory data store for caching and queues

Stars
76.7K
Forks
24.5K
Last recorded push
8 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
78.2K
Forks
25.8K
Last recorded push
8 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Playwright

Cross-Browser Automation for Modern Web Apps

Stars
97.5K
Forks
5K
Last recorded push
8 months ago

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.