Aptitive designed and implemented a scalable ETL framework to collect and transform data from a variety of changing data sources.

The Challenge

Our client had multiple organizations that were combining into a single, upgraded instance of their ERP system. In addition to the transactional data, there was a variety of data sources including semi-structured flat files, API data flows from Salesforce, and manual spreadsheets. The business team required a way to merge all of the data into one seamless application that would act as a single source of the truth for the company’s reporting.

Our Solution

Using a combination of Attunity and Azure Data Factory, our team extracted data from each disparate source and collected the information in a staging-layer of the Data Lake. Next, the data was cleansed and processed into a loading-ready format by:

  • Migrating legacy ERP data to reflect the new structure and values
  • Processing unstructured data into an analyzable business layer
  • Organizing data for more effective export to custom applications and analysis by advanced tools such as Azure Machine Learning Studio

The Outcome

Aptitive’s client achieved a significant cost savings by integrating the different data sources into a central hub. The new central hub allowed the user to access all of the data from each organization and avoided a “spaghetti” data flow caused by multiple applications pulling from the different data sources. The new system helped identify data quality issues with each new data source and fix these issues before they caused inaccurate reports. Moreover, the backend reporting was updated to use the new source data without service interruptions.

Client Industry

Global Manufacturing Company

 

Technology Used

Azure Data Lake
Data Factory
Attunity
Cognos