Scalable Java Crawlers For Teams
Library or tool · Self-hosted
Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.
Large-Scale & Distributed Crawling
Scalable Search For Scraped Content
Infrastructure · Self-hosted
Open-source Java search software for indexing scraped content with full-text search and faceted navigation.
Search Engines
Parse Real-World HTML In Java
Library or tool · Library / Self-hosted
Stars: 11.4K · Code license: MIT A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.
HTML Parsers
Extract Text From Any Document
Library or tool · Self-hosted
Detects file types and extracts text and document details from 1,000+ formats through one Java interface.
Document Extraction
Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Open-source search and analytics suite
Infrastructure · Self-hosted
OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.
Search Engines
AI-native vector database with built-in ML models
Infrastructure · Self-hosted
Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.
Vector Databases
Durable Execution for Invincible Applications
Library or tool · Self-hosted
Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.
Workflow Orchestration
Typo-tolerant search engine with faceted filtering
Infrastructure · Self-hosted
Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.
Search Engines
Flexible document database for any data shape
Infrastructure · API / SaaS / Self-hosted
MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.
NoSQL Databases
Browser Automation Framework and Ecosystem
Library or tool · Framework / Self-hosted
Stars: 34.6K · Code license: Apache-2.0 Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.
Browser Automation
In-process SQL analytics, no server needed
Infrastructure · Self-hosted
DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.
Analytics Databases
Open-source vector database for billion-scale AI
Infrastructure · Self-hosted
Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.
Vector Databases
Real-time analytics database for petabyte-scale data
Infrastructure · Self-hosted
ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.
Analytics Databases
Lightning-fast search engine through configured exporters or custom item pipelines
Infrastructure · Self-hosted
Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.
Search Engines
Archived community S3-compatible object store
Infrastructure · Self-hosted
MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.
Object Storage
In-memory data store for caching and queues
Infrastructure · Self-hosted
Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.
Caching Databases
Distributed RESTful search and analytics engine
Infrastructure · Self-hosted
Stars: 78.3K · Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers) Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.
Search Engines
Cross-Browser Automation for Modern Web Apps
Library or tool · Library / Self-hosted
Stars: 97.8K · Code license: Apache-2.0 Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.
Browser Automation +1 more
