Databases

Quick answer

Database choice follows data shape and query pattern. Use caching databases for fast key-value access, document stores or SQL plus JSON for flexible records, analytics databases for columnar aggregates, vector databases for embeddings, and lakehouse tables when files and query engines need to share one storage layer.

pgvector

Add Vector Search To PostgreSQL

Stars
21.9K
Forks
1.2K
Last commit
23 days ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Apache Hudi

Lakehouse Platform for Incremental Processing

Stars
6.1K
Forks
0

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

DynamoDB

AWS managed serverless NoSQL database

Stars
7.6K
Forks
1.5K
Last commit
7 months ago

Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Stars
8.5K
Forks
0

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Delta Lake

Open-Source Lakehouse Storage Framework

Stars
8.6K
Forks
0

Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.

LanceDB

Serverless vector database, no server management

Stars
8.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Cassandra

Distributed NoSQL with high availability at scale

Stars
9.6K
Forks
3.8K
Last commit
5 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

StarRocks

Sub-second analytics for enterprise-scale data

Stars
11.3K
Forks
2.3K
Last commit
5 months ago

StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.

KeyDB

Multithreaded Redis fork with higher throughput

Stars
12.4K
Forks
647
Last commit
2 years ago

KeyDB is a multithreaded Redis-compatible database delivering over 1M ops/sec per node for high-throughput caching and queuing.

Apache Druid

Real-time OLAP database for event-driven data

Stars
13.9K
Forks
0

Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.

Memcached

Simple, fast distributed memory caching system

Stars
14.1K
Forks
3.3K
Last commit
7 months ago

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

Apache Doris

Real-time analytical database with MySQL syntax

Stars
14.9K
Forks
3.7K
Last commit
5 months ago

Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.

Weaviate

AI-native vector database with built-in ML models

Stars
15.5K
Forks
0

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

QuestDB

Time-series database with ultra-low latency

Stars
16.5K
Forks
0

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

PostgreSQL + JSONB

Relational SQL database with native JSON document support

Stars
19.7K
Forks
5.4K
Last commit
5 months ago

PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.

Chroma

AI-native embedding database with simple API

Stars
25.7K
Forks
0

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

MongoDB

Flexible document database for any data shape

Stars
28K
Forks
5.7K
Last commit
5 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Qdrant

High-performance vector search engine in Rust

Stars
28.4K
Forks
0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

DuckDB

In-process SQL analytics, no server needed

Stars
35.6K
Forks
2.9K
Last commit
5 months ago

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Milvus

Open-source vector database for billion-scale AI

Stars
42.4K
Forks
0

Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.

ClickHouse

Real-time analytics database for petabyte-scale data

Stars
45.4K
Forks
8K
Last commit
5 months ago

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Redis

In-memory data store for caching and queues

Stars
72.6K
Forks
24.5K
Last commit
5 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.