Data Engineering Book Scraper
Solo Developer
An automated ETL pipeline that scrapes data engineering book listings from Bol.com, orchestrated with Apache Airflow and containerised with Docker.
Overview
About the project
Tracking book listings, prices, and availability across an online retail site by hand is tedious and does not scale. To stay current, the collection needs to run itself reliably, on a schedule, and without manual babysitting.
I built a three stage ETL workflow and modelled it as an Airflow DAG. One task creates the database table, a second scrapes book data from Bol.com with BeautifulSoup, and a third loads the cleaned records into PostgreSQL, with data passed between tasks through Airflow's XCom. The scraper handles inconsistent HTML and includes retry logic and error handling so a single bad page does not break the run. The entire pipeline is containerised with Docker for consistent, reproducible deployment.
The pipeline runs fully automated on a defined schedule, collecting structured book data reliably. It demonstrates a complete, production style orchestration setup from extraction through to a queryable database.
System Overview
At a glance
An ETL pipeline that scrapes data engineering book listings from Bol.com on a defined schedule, with every step orchestrated as an Apache Airflow DAG and the whole thing containerised with Docker. It demonstrates a small but complete production pattern: scheduled extraction, resilient scraping, and a clean landing zone in PostgreSQL.
How It Works
System Architecture
BeautifulSoup Scraper
Extracts book listings from Bol.com
Apache Airflow
Orchestrates the pipeline as a scheduled DAG
PostgreSQL
Stores the cleaned, structured book records
Docker
Containerises the whole pipeline for reproducible runs
Process Flow
How the project moves from start to finish, step by step.
Create
An Airflow task creates the database table if it does not already exist.
Extract
A scraping task pulls book data from Bol.com with BeautifulSoup.
Load
A final task cleans the records and inserts them into PostgreSQL.
Workflows
Airflow DAG
Three Airflow tasks form a strict dependency chain, with data handed between them via XCom.
Create Table
Purpose: Guarantee the destination exists before any data tries to land.
How it works
- An Airflow task runs the table creation statement.
- The statement is idempotent so re-runs are safe.
- Downstream tasks can assume the table is ready.
Extract
Purpose: Pull current book listings from Bol.com.
How it works
- A scraping task fetches Bol.com pages with BeautifulSoup.
- Inconsistent HTML is handled with defensive parsing.
- Failed pages are retried so a single error does not stop the run.
- The cleaned list is handed off through Airflow XCom.
Load
Purpose: Land structured records ready to query.
How it works
- A loader task picks up the records from XCom.
- Each record is validated and shaped to match the table.
- Rows are inserted into PostgreSQL in a single transaction.
Under The Hood
Technical Implementation
# Task execution order
create_table_task >> fetch_book_data_task >> insert_book_data_task
# Data handed between tasks via XCom
ti.xcom_push(key="books", value=scraped_books)XCom Passing
Scraped data handed cleanly between tasks through Airflow XCom.
Retry Logic
Failed pages retried so a single error does not stop the run.
What It Does
Features & Capabilities
Scheduled Runs
The pipeline runs automatically on a defined cadence.
Resilient Scraping
Retry logic and error handling for unstable web pages.
Containerised
A Docker image gives consistent runs on any machine.
Structured Output
Clean records ready to query directly in PostgreSQL.
Running It