The Problem
Aptitive’s client wanted to move their data, housed in AWS, into a data warehouse on Google Cloud Platform. In doing this, they needed a way to automatically schedule the ETL jobs to run from AWS through the layers of data warehouse without allocating resources to manually trigger the load.
Our Solution
To alleviate this problem, Aptitive leveraged their ETL framework and best practices to run ETL jobs using Google Cloud Composer, a managed version of Apache Airflow. Aptitive created Python scripts that are run daily, at scheduled times, using Composer, to transform tables from AWS and load them into the cloud.
The Outcome
The solution with Google Cloud Composer manages the daily raw loads and business logic loads for the client, ensuring that employees do not need to spend their time manually triggering the jobs. This ensures that the correct data will be available in a cloud based data warehouse , saves the the company time and money, and allows them to expand their Google Cloud platform capabilities.
Client Industry
Sustainable Agricultural Technology
Technology
AWS
Google Cloud Platform
Cloud Composer
Airflow
