Ruby Web Scraping Tools

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Library or tool · Library / Self-hosted

Stars: 33
Code license: MIT

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

Sitemap Scrapers

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

MongoDB

Flexible document database for any data shape

Infrastructure · API / SaaS / Self-hosted

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

NoSQL Databases

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

Redis

In-memory data store for caching and queues

Infrastructure · Self-hosted

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Caching Databases

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

bisibility

Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.

Try bisibility freeOur product

Discover web scraping tools and libraries for Ruby. Compare features, community support, and documentation.