Infrastructure

Apache Solr

Scalable Search For Scraped Content

Infrastructure · Self-hosted

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

Search Engines

Cloudflare R2

S3-compatible storage with zero egress fees

Infrastructure · API / SaaS

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

Object Storage

Render

Cloud platform for web services and cron jobs

Infrastructure · API / SaaS

Render is a cloud platform that deploys web services, background workers, and cron jobs with simple scaling for scraping workloads.

Cloud Compute

Railway

Deploy apps from GitHub through configured exporters or custom item pipelines

Infrastructure · API / SaaS

Railway deploys web apps and services from GitHub with auto-scaling, built-in databases, and instant previews for scraping workloads.

Cloud Compute

AWS Lambda

Serverless compute for event-driven pipelines

Infrastructure · API / SaaS

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloud Compute

Cloudflare Workers

Run serverless code at the edge worldwide

Infrastructure · API / SaaS

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

Cloud Compute

Modal

Serverless Python platform for AI and data teams

Infrastructure · API / SaaS

Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with configurable containers and autoscaling.

Cloud Compute

Fly.io

Deploy containers globally on edge servers

Infrastructure · API / SaaS

Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.

Cloud Compute

Algolia

Managed search and AI retrieval platform

Infrastructure · API / SaaS

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Search Engines

Manticore Search

Open-source search database with SQL and HTTP APIs

Infrastructure · Self-hosted

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

Search Engines

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

RustFS

High-performance S3-compatible storage in Rust

Infrastructure · Self-hosted

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

Object Storage

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

MinIO

Archived community S3-compatible object store

Infrastructure · Self-hosted

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Object Storage

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

Browse 3 subcategories of Infrastructure tools and find the perfect solution for your needs.

Quick answer

Related categories

Apache Solr

Cloudflare R2

Render

Railway

AWS Lambda

Cloudflare Workers

Modal

Fly.io

Algolia

Manticore Search

OpenSearch

RustFS

Typesense

Meilisearch

MinIO

Elasticsearch

Apache Solr

Cloudflare R2

Render

Railway

AWS Lambda

Cloudflare Workers

Modal

Fly.io

Algolia

Manticore Search

OpenSearch

RustFS

Typesense

Meilisearch

MinIO

Elasticsearch