dbt

How to use dbt in a web scraping pipeline

dbt transforms data already present in a supported data platform and organizes those transformations as a maintained project. In scraping, it belongs after extraction and loading, where inconsistent source records become analytical models.

A practical workflow

1. A separate scraper and loader populate raw tables with entity keys, source URLs and observation times.
2. Build dbt models that normalize fields, join reference data and distinguish current records from historical observations.
3. Run relevant tests and publish curated models for reports or application consumers, keeping lineage back to the raw inputs.

When it fits

Use it when cleaning rules should be versioned, reviewed and shared across several downstream datasets. A model can make source-specific differences explicit instead of repeating the same cleanup in every dashboard query.

Boundaries

dbt does not fetch pages or provide the initial ingestion path. The chosen adapter and data platform determine supported behavior. Tests can enforce selected expectations, but they cannot establish that every extracted value reflects the source page; preserve capture references and review parser changes separately.