Aptitive implemented a solution using Google Cloud Platform to automatically trigger ETL jobs in the cloud and ensure that our client could access their data and manage it automatically.

The Problem

Aptitive’s client wanted to move their data, housed in AWS, into a data warehouse on Google Cloud Platform. In doing this, they needed a way to automatically schedule the ETL jobs to run from AWS through the layers of data warehouse without allocating resources to manually trigger the load.

Our Solution

To alleviate this problem, Aptitive leveraged their ETL framework and best practices to run ETL jobs using Google Cloud Composer, a managed version of Apache Airflow. Aptitive created Python scripts that are run daily, at scheduled times, using Composer, to transform tables from AWS and load them into the cloud.

The Outcome

The solution with Google Cloud Composer manages the daily raw loads and business logic loads for the client, ensuring that employees do not need to spend their time manually triggering the jobs. This ensures that the correct data will be available in a cloud based data warehouse , saves the the company time and money, and allows them to expand their Google Cloud platform capabilities.

Client Industry

Sustainable Agricultural Technology

Technology

AWS
Google Cloud Platform
Cloud Composer
Airflow