Dotlabs
Sindri Media
Introduction
This project involved optimizing data processing and pipeline efficiency using Python, SQL, Airflow, DBT, AWS S3, and EC2. By implementing efficient stored procedures and adding indexes to frequently queried columns, we achieved significant performance improvements in database operations.
Additionally, migrating the project to DBT (Data Build Tool) streamlined data transformation workflows, enhancing the overall efficiency and manageability of the data pipelines.