Dotlabs
Hubspot Data Pipeline using Airbyte Connector
Introduction
The Brainvoy Flask App is designed to interact with HubSpot's API to manage user accounts, retrieve data, and synchronize with Airbyte for data extraction. The app is built using the Flask framework and handles OAuth 2.0 authentication to connect users with their HubSpot accounts and synchronize their data to S3 via Airbyte
Project Overview
This project sets up a data pipeline for a HubSpot app, installed by multiple users, to extract and process targeted user data. The pipeline utilizes Airbyte connectors, AWS Glue for transformation, and AWS services for storage and reporting.
The whole process consists of many steps which are explained below:
1. Store user accounts data in DB
2. Configure Airbyte for Data Extraction
3. Data transformation with AWS Glue
4. Store the data in S3