Add Vector Search To PostgreSQL
Infrastructure · Self-hosted
Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.
Vector Databases
Scalable Search For Scraped Content
Infrastructure · Self-hosted
Open-source Java search software for indexing scraped content with full-text search and faceted navigation.
Search Engines
S3-compatible storage with zero egress fees
Infrastructure · API / SaaS
Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.
Object Storage
Cloud platform for web services and cron jobs
Infrastructure · API / SaaS
Render is a cloud platform that deploys web services, background workers, and cron jobs with simple scaling for scraping workloads.
Cloud Compute
Deploy apps from GitHub through configured exporters or custom item pipelines
Infrastructure · API / SaaS
Railway deploys web apps and services from GitHub with auto-scaling, built-in databases, and instant previews for scraping workloads.
Cloud Compute
Serverless compute for event-driven pipelines
Infrastructure · API / SaaS
AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.
Cloud Compute
Run serverless code at the edge worldwide
Infrastructure · API / SaaS
Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.
Cloud Compute
Serverless Python platform for AI and data teams
Infrastructure · API / SaaS
Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with configurable containers and autoscaling.
Cloud Compute
Deploy containers globally on edge servers
Infrastructure · API / SaaS
Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.
Cloud Compute
AI agents and multi-agent workflows
Library or tool · Framework / Self-hosted
Stars: 3.4K · Code license: Apache-2.0 Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.
AI Agent Tooling
Managed search and AI retrieval platform
Infrastructure · API / SaaS
Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.
Search Engines
Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
AWS managed serverless NoSQL database
Infrastructure · API / SaaS
Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.
NoSQL Databases
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Open-Source Lakehouse Storage Framework
Infrastructure · Self-hosted
Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.
Lakehouse & Table Formats
Serverless vector database, no server management
Infrastructure · Self-hosted
LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.
Vector Databases
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Sub-second analytics for enterprise-scale data
Infrastructure · Self-hosted
StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.
Analytics Databases
Open-source search database with SQL and HTTP APIs
Infrastructure · Self-hosted
Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.
Search Engines
Open-source search and analytics suite
Infrastructure · Self-hosted
OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.
Search Engines
Multithreaded Redis fork with higher throughput
Infrastructure · Self-hosted
KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.
Caching Databases
Real-time OLAP database for event-driven data
Infrastructure · Self-hosted
Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.
Analytics Databases
Simple, fast distributed memory caching system
Infrastructure · Self-hosted
Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.
Caching Databases
Real-time analytical database with MySQL syntax
Infrastructure · Self-hosted
Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.
Analytics Databases
AI-native vector database with built-in ML models
Infrastructure · Self-hosted
Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.
Vector Databases
Time-series database with ultra-low latency
Infrastructure · Self-hosted
QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.
Analytics Databases
Relational SQL database with native JSON document support
Infrastructure · Self-hosted
PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.
Databases
High-performance S3-compatible storage in Rust
Infrastructure · Self-hosted
RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.
Object Storage
TypeScript AI Agent Framework with Workflows
Library or tool · Framework / Self-hosted
Stars: 28.8K · Code license: Apache-2.0 (outside ee/) Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.
AI Agent Tooling
Typo-tolerant search engine with faceted filtering
Infrastructure · Self-hosted
Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.
Search Engines
AI-native embedding database with simple API
Infrastructure · Self-hosted
Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.
Vector Databases
Flexible document database for any data shape
Infrastructure · API / SaaS / Self-hosted
MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.
NoSQL Databases
High-performance vector search engine in Rust
Infrastructure · Self-hosted
Stars: 35.1K · Code license: Apache-2.0 Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.
Vector Databases
In-process SQL analytics, no server needed
Infrastructure · Self-hosted
DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.
Analytics Databases
Open-source vector database for billion-scale AI
Infrastructure · Self-hosted
Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.
Vector Databases
Real-time analytics database for petabyte-scale data
Infrastructure · Self-hosted
ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.
Analytics Databases
