Data Migration & Validation for a Major Retail Business Group

The client, which is one of North America’s largest retail chains, has 3 entities. They were looking for successful migration of applications and databases to include Entity 1 data by end of Q1. The project involved Data Migration, Validation, and Reporting. The client hired Incepta to:

  • Create new instances of Informatica and Snowflake with Entity 1 only data.
  • Configure Airflow jobs related to Entity 1 and corporate to fetch relevant data from different supporting applications and update the Snowflake database.
  •  Ensure all BI (MicroStrategy & Looker) instances are pointing to the new instance of Informatica and Snowflake and generating reports with Entity 1 only data.
  • Ensure smooth functioning of business after this separation.
  • Run project using Agile – Scrum Framework, Jira

Tools and technologies involved-

  • Data Migration Informatica and Apache AirFlow
  • Data ValidationRPA, AI, and Microstrategy
  • Data Reporting – MicroStrategy Tableau and Lookup

One of North America’s oldest retail business groups with the iconic department stores. It has three major entities. 

  • Entity 1 – A leading online eCommerce platform for luxury fashion
  • Entity 2A premier luxury off-price eCommerce company
  • Entity 3 – A Canadian eCommerce marketplace

Data Migration:

The client has the three entities, with a combined database. The client wanted to separate them into different instances.

Data Validation:

Generate various reports and compare the data between the existing and new setup if they are matching.

Data Migration Diagram

Data Migration: 

New instances for Entity 1 were created migrating data from the common instances on Snowflakes using Informatica and Apache AirFlow following the process of Extract Transformation and Load (ETL). Incepta made sure that the mappings and everything migrated successfully and tested on the new instance of Informatica. 

RPA:

Data lineage was carried out using UIPath bots to ensure the source and target data is validated. 

QA:

Validating the end results by executing queries and capturing results to compare with production using test cases created for Snowflake, Microstrategy, and Airflow. 

Data Engineering: 

Update Data separately for Entity 1 from Entity 2 and Entity 3 using DAGs (Directed Acyclic Graph) which is the core concept of Airflow, collecting tasks together, organized with dependencies and relationships to say how they should run.

Incepta has deployed 183 DAGs that update tables, folders, and SFTPs to make sure that data is up to date. 

Data Reporting: 

The updated data from Data Engineering is then followed by reporting. Incepta created new instances on MicroStrategy, Tableau, and Lookup Looker and validated these dashboards/reports to generate the same results as the existing environment. 

  • 40 ETL jobs migrated to new instances
  • 100s of reports validated
  • 7 Databases migrated and validated

By partnering with Incepta Solutions, our client was able to complete the data migration and validation project smoothly using the Extraction, Transformation, and Load or ETL approach

40 ETL jobs migrated to new instances, finetuned, troubleshoot, and fixed with best practices, 100s of reports validated, and 7 databases migrated and validated.

Contact us by emailing [email protected] or calling +1 877 797 0406 to learn how Incepta Solutions can help with Data Migration, Validation, or any other ETL Project. 

or Connect with us now

Incepta Solutions | Powered by innovation, driven by results

Stay connected:

LinkedIn | Twitter | Facebook | Instagram

More relevant insights

RPA CoE

Business Insights, Digital Transformation, Featured Post Business Insights

5 Steps to Set Up an RPA CoE (Center of Excellence) for Your Company

RPA Center of Excellence is the enabler of RPA adoption across the organization. Learn to set up an RPA CoE for your company in five easy steps.

Read More

MuleSoft GTM Award

#InceptaInnovators, Business Insights, Featured Post Business Insights, Life @ Incepta, News & Insights

InceptaInnovator Earns Go To Market Champion Badge from MuleSoft

Incepta’s Partner Engagement Specialist for Mulesoft earned the Go To Market Champion badge for the retail industry from MuleSoft.

Read More