Ruby Web Scraping Tools

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last recorded push
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

Algolia

Managed search and AI retrieval platform

Stars
4.1K
Forks
557
Last recorded push
8 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

OpenSearch

Open-source search and analytics suite

Stars
13.8K
Forks
2.4K
Last recorded push
8 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Typesense

Typo-tolerant search engine with faceted filtering

Stars
26.9K
Forks
849
Last recorded push
8 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

MongoDB

Flexible document database for any data shape

Stars
28.6K
Forks
5.7K
Last recorded push
8 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Selenium

Browser Automation Framework and Ecosystem

Stars
34.5K
Forks
8.6K
Last recorded push
8 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Stars
59.5K
Forks
2.3K
Last recorded push
8 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Redis

In-memory data store for caching and queues

Stars
76.7K
Forks
24.5K
Last recorded push
8 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
78.2K
Forks
25.8K
Last recorded push
8 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Discover web scraping tools and libraries for Ruby. Compare features, community support, and documentation.