Object Storage

Quick answer

Object storage should be chosen as a storage backend, not as a transfer tool. MinIO fits self-hosted S3-compatible storage, while Cloudflare R2 fits managed S3-compatible storage with zero egress fees. Use tools such as rclone for moving data between storage systems, not as the backend itself.

Top picks in Object Storage

MinIO
Self-hosted S3
Fits teams that want to operate their own S3-compatible object storage for raw HTML, screenshots, exports, and datasets.
Cloudflare R2
Zero egress
Fits teams that want managed S3-compatible object storage on Cloudflare with zero egress fees for frequently accessed scraped data.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

RustFS

High-performance S3-compatible storage in Rust

Stars
20.2K
Forks
868
Last commit
5 months ago

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

MinIO

S3-compatible object storage you can self-host

Stars
59.9K
Forks
6.9K
Last commit
6 months ago

MinIO is a high-performance, S3-compatible object store for storing scraped files, datasets, and media at exascale.

A curated collection of the best object storage services for archiving raw HTML, screenshots, and large scraped datasets.

Quick answer

Top picks in Object Storage