Data Migration & Validation
The client, which is one of North America’s largest retail chains, has 3 entities. They were looking for successful migration of applications and databases to include Entity 1 data by end of Q1. The project involved Data Migration, Validation, and Reporting. The client hired Incepta to:
- Create new instances of Informatica and Snowflake with Entity 1 only data.
- Configure Airflow jobs related to Entity 1 and corporate to fetch relevant data from different supporting applications and update the Snowflake database.
- Ensure all BI (MicroStrategy & Looker) instances are pointing to the new instance of Informatica and Snowflake and generating reports with Entity 1 only data.
- Ensure smooth functioning of business after this separation.
- Run project using Agile – Scrum Framework, Jira
Tools and technologies involved-
- Data Migration– Informatica and Apache AirFlow
- Data Validation – RPA, AI, and Microstrategy
- Data Reporting – MicroStrategy Tableau and Lookup
One of North America’s oldest retail business groups with iconic department stores. It has three major entities.
- Entity 1 – A leading online eCommerce platform for luxury fashion
- Entity 2 – A premier luxury off-price eCommerce company
- Entity 3 – A Canadian eCommerce marketplace
The client has three entities, with a combined database. The client wanted to separate them into different instances.
Generate various reports and compare the data between the existing and new setup if they are matching.
New instances for Entity 1 were created migrating data from the common instances on Snowflakes using Informatica and Apache AirFlow following the process of Extract Transformation and Load (ETL). Incepta made sure that the mappings and everything migrated successfully and tested on the new instance of Informatica.
Data lineage was carried out using UIPath bots to ensure the source and target data is validated.
Validating the end results by executing queries and capturing results to compare with production using test cases created for Snowflake, Microstrategy, and Airflow.
Update Data separately for Entity 1 from Entity 2 and Entity 3 using DAGs (Directed Acyclic Graph) which is the core concept of Airflow, collecting tasks together, organized with dependencies and relationships to say how they should run.
Incepta has deployed 183 DAGs that update tables, folders, and SFTPs to make sure that data is up to date.
The updated data from Data Engineering is then followed by reporting. Incepta created new instances on MicroStrategy, Tableau, and Lookup Looker and validated these dashboards/reports to generate the same results as the existing environment.
- 40 ETL jobs migrated to new instances
- 100s of reports validated
- 7 Databases migrated and validated
By partnering with Incepta Solutions, our client was able to complete the data migration and validation project smoothly using the Extraction, Transformation, and Load or ETL approach.
40 ETL jobs migrated to new instances, finetuned, troubleshoot, and fixed with best practices, 100s of reports validated, and 7 databases migrated and validated.