Add Vector Search To PostgreSQL
Infrastructure · Self-hosted
Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.
Vector Databases
Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
AWS managed serverless NoSQL database
Infrastructure · API / SaaS
Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.
NoSQL Databases
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Open-Source Lakehouse Storage Framework
Infrastructure · Self-hosted
Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.
Lakehouse & Table Formats
Serverless vector database, no server management
Infrastructure · Self-hosted
LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.
Vector Databases
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Sub-second analytics for enterprise-scale data
Infrastructure · Self-hosted
StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.
Analytics Databases
Multithreaded Redis fork with higher throughput
Infrastructure · Self-hosted
KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.
Caching Databases
Real-time OLAP database for event-driven data
Infrastructure · Self-hosted
Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.
Analytics Databases
Simple, fast distributed memory caching system
Infrastructure · Self-hosted
Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.
Caching Databases
Real-time analytical database with MySQL syntax
Infrastructure · Self-hosted
Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.
Analytics Databases
AI-native vector database with built-in ML models
Infrastructure · Self-hosted
Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.
Vector Databases
Time-series database with ultra-low latency
Infrastructure · Self-hosted
QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.
Analytics Databases
Relational SQL database with native JSON document support
Infrastructure · Self-hosted
PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.
Databases
AI-native embedding database with simple API
Infrastructure · Self-hosted
Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.
Vector Databases
Flexible document database for any data shape
Infrastructure · API / SaaS / Self-hosted
MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.
NoSQL Databases
High-performance vector search engine in Rust
Infrastructure · Self-hosted
Stars: 35.1K · Code license: Apache-2.0 Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.
Vector Databases
In-process SQL analytics, no server needed
Infrastructure · Self-hosted
DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.
Analytics Databases
Open-source vector database for billion-scale AI
Infrastructure · Self-hosted
Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.
Vector Databases
Real-time analytics database for petabyte-scale data
Infrastructure · Self-hosted
ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.
Analytics Databases
In-memory data store for caching and queues
Infrastructure · Self-hosted
Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.
Caching Databases
