AWS Glue
AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.
AWS Glue is a fully managed, serverless data integration service that streamlines the entire extract, transform, and load process for supported workloads.
Key Features:
- Serverless ETL - Run Spark-based transform jobs without provisioning or managing infrastructure.
- Data Catalog - Discover and organize metadata across 100+ data sources in a centralized catalog.
- Automated Crawlers - Automatically infer schemas and partition structures from your data lakes.
- Visual Pipeline Builder - Design ETL workflows visually or write custom PySpark and Python code.
- AI-Powered Authoring - Accelerate pipeline development with built-in generative AI assistance.
- Auto-Scaling - Scale from gigabytes to petabytes on demand, paying only for resources consumed.
Whether you're loading data into S3, Redshift, or a lakehouse, AWS Glue provides enterprise-grade ETL with managed infrastructure and configurable jobs.
Web scraping workflow
How to use AWS Glue in a web scraping pipeline
AWS Glue provides data integration, transformation and cataloging for analytical data. It can prepare a scraper's stored outputs for downstream querying rather than fetch website content itself.
A practical workflow
- A separate scraper writes structured collection files into S3 with source and run metadata.
- Use the relevant Glue cataloging and ETL components to describe the stored data, normalize fields and separate invalid records.
- Publish curated output for Athena or another analytical consumer, retaining a link to the original collection batch.
When it fits
Consider it when multiple scraping jobs land data in an AWS data lake and need shared schemas or transformation jobs. Keep raw and curated locations distinct so a transformation can be repeated without refetching the website.
Boundaries
A Glue crawler discovers metadata about supported data sources; it is not a browser crawler of arbitrary websites. Acquisition code, business validation and access policy remain separate. Review inferred schemas before promotion, and define how job reruns update destinations instead of assuming every ETL run is safe to repeat.
Frequently asked questions
What is AWS Glue used for?
AWS Glue is a serverless data-integration service for discovering, preparing and moving data. Its Data Catalog stores metadata, while jobs perform processing and loading. It supports analytics pipelines that connect sources, storage and query services.
Sources (checked ): AWS Glue introduction
Is AWS Glue ETL or ELT?
AWS Glue supports ETL, ELT and streaming workloads. Glue Studio runs transformation jobs on Glue’s Apache Spark engine. ELT workflows depend on the supported destination integration and configured processing steps.
Sources (checked ): AWS Glue introduction
How much does it cost to use AWS Glue?
Glue charges vary by feature, including processing resources and run duration, crawlers, Data Catalog usage and optional services. Catalog allowances do not make all job execution free. Estimate the features and resources your pipeline uses from the current pricing table.
Sources (checked ): AWS Glue pricing
What are the key differences between AWS Lambda and AWS Glue?
Lambda runs application functions in response to events. Glue provides data-integration jobs and a shared Data Catalog. Use the required execution model, processing duration and data-integration features to decide which service handles each pipeline step.
Sources (checked ): AWS Glue introduction; Lambda introduction
Similar to AWS Glue
Sources
Verified against the official AWS Glue website and documentation.