C# Web Scraping Tools

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Library or tool · Library / Self-hosted

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

Sitemap Scrapers

Cassandra

Distributed NoSQL with high availability at scale

Infrastructure · Self-hosted

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

NoSQL Databases

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

Temporal

Durable Execution for Invincible Applications

Library or tool · Self-hosted

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Workflow Orchestration

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

MongoDB

Flexible document database for any data shape

Infrastructure · API / SaaS / Self-hosted

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

NoSQL Databases

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation

DuckDB

In-process SQL analytics, no server needed

Infrastructure · Self-hosted

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Analytics Databases

ClickHouse

Real-time analytics database for petabyte-scale data

Infrastructure · Self-hosted

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Analytics Databases

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

MinIO

Archived community S3-compatible object store

Infrastructure · Self-hosted

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Object Storage

Redis

In-memory data store for caching and queues

Infrastructure · Self-hosted

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Caching Databases

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

Playwright

Cross-Browser Automation for Modern Web Apps

Library or tool · Library / Self-hosted

Stars: 97.8K
Code license: Apache-2.0

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.

Browser Automation

+1 more

Puppeteer

JavaScript API for Chrome and Firefox Automation

Library or tool · Library / Self-hosted

Stars: 95.7K
Code license: Apache-2.0

Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.

Browser Automation

+1 more

Discover web scraping tools and libraries for C#. Compare features, community support, and documentation.