Infrastructure

Apache Solr

Scalable Search For Scraped Content

Stars
1.6K
Forks
839
Last commit
21 days ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

Render

Cloud platform for web services and cron jobs

Cloud Compute

Render is a cloud platform that deploys web services, background workers, and cron jobs with simple scaling for scraping workloads.

Railway

Deploy apps from GitHub with zero config

Cloud Compute

Railway deploys web apps and services from GitHub with auto-scaling, built-in databases, and instant previews for scraping workloads.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Cloudflare Workers

Run serverless code at the edge worldwide

Cloud Compute

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

Modal

Serverless Python platform for AI and data teams

Stars
433
Forks
0

Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with sub-second cold starts and instant autoscaling.

Fly.io

Deploy containers globally on edge servers

Stars
1.6K
Forks
0

Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.

Algolia

Managed search and AI retrieval platform

Stars
4K
Forks
557
Last commit
5 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Manticore Search

Fast open-source search database, 2.8x faster

Stars
11.6K
Forks
622
Last commit
5 months ago

Manticore Search is a fast, open-source search database that delivers 2.8x faster performance than Elasticsearch for full-text and analytical queries.

OpenSearch

Open-source search and analytics suite

Stars
12.3K
Forks
2.4K
Last commit
5 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

RustFS

High-performance S3-compatible storage in Rust

Stars
20.2K
Forks
868
Last commit
5 months ago

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

Typesense

Typo-tolerant search with sub-50ms responses

Stars
25.1K
Forks
849
Last commit
5 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Meilisearch

Lightning-fast search engine with zero config

Stars
55.5K
Forks
2.3K
Last commit
5 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

MinIO

S3-compatible object storage you can self-host

Stars
59.9K
Forks
6.9K
Last commit
6 months ago

MinIO is a high-performance, S3-compatible object store for storing scraped files, datasets, and media at exascale.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
75.9K
Forks
25.8K
Last commit
5 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Browse 3 subcategories of Infrastructure tools and find the perfect solution for your needs.

Quick answer

Related categories

Apache Solr

Cloudflare R2

Render

Railway

AWS Lambda

Cloudflare Workers

Modal

Fly.io

Algolia

Manticore Search

OpenSearch

RustFS

Typesense

Meilisearch

MinIO

Elasticsearch

Apache Solr

Cloudflare R2

Render

Railway

AWS Lambda

Cloudflare Workers

Modal

Fly.io

Algolia

Manticore Search

OpenSearch

RustFS

Typesense

Meilisearch

MinIO

Elasticsearch