The project called for a data processing platform designed around repeated ingestion from external APIs into PostgreSQL. The system supports multiple pipelines through a shared framework with per-pipeline extract, transform, validate, and load stages. It stores raw API pages for replay, supports checkpointing and resume, and exposes control through a FastAPI service. I also added change detection using content hashes so unchanged records are skipped. The project includes integration tests covering sync, replay, metrics, reset behavior, and multi-pipeline support.
Built with
- Python
- FastAPI
- Data Engineering
- API Development