C# Web Scraping Tools

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last recorded push
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

Algolia

Managed search and AI retrieval platform

Stars
4.1K
Forks
557
Last recorded push
8 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Cassandra

Distributed NoSQL with high availability at scale

Stars
10.1K
Forks
3.8K
Last recorded push
8 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

OpenSearch

Open-source search and analytics suite

Stars
13.8K
Forks
2.4K
Last recorded push
8 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Temporal

Durable Execution for Invincible Applications

Stars
23.6K
Forks
1.3K
Last recorded push
8 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Typesense

Typo-tolerant search engine with faceted filtering

Stars
26.9K
Forks
849
Last recorded push
8 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

MongoDB

Flexible document database for any data shape

Stars
28.6K
Forks
5.7K
Last recorded push
8 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Selenium

Browser Automation Framework and Ecosystem

Stars
34.5K
Forks
8.6K
Last recorded push
8 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

DuckDB

In-process SQL analytics, no server needed

Stars
42.1K
Forks
2.9K
Last recorded push
8 months ago

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

ClickHouse

Real-time analytics database for petabyte-scale data

Stars
50.3K
Forks
8K
Last recorded push
8 months ago

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Stars
59.5K
Forks
2.3K
Last recorded push
8 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

MinIO

Archived community S3-compatible object store

Stars
61.3K
Forks
6.9K
Last recorded push
9 months ago

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Redis

In-memory data store for caching and queues

Stars
76.7K
Forks
24.5K
Last recorded push
8 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
78.2K
Forks
25.8K
Last recorded push
8 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Playwright

Cross-Browser Automation for Modern Web Apps

Stars
97.5K
Forks
5K
Last recorded push
8 months ago

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.

Puppeteer

JavaScript API for Chrome and Firefox Automation

Stars
95.7K
Forks
9.4K
Last recorded push
8 months ago

Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.

Discover web scraping tools and libraries for C#. Compare features, community support, and documentation.