Utilities

A curated collection of the best general-purpose utility tools and libraries that support web scraping workflows.

Top picks in Utilities

rclone
#1
Listed for an open-source option, 55,100 tracked repository stars, and a direct Utilities category match.
Crawler-Commons
#2
Listed for an open-source option, 259 tracked repository stars, and a direct Utilities category match.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
15 days ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

rclone

Move and sync scraped datasets across cloud storage

Stars
55.1K
Forks
0

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.