The client, which is one of North America’s largest retail chains, has 3 entities. They were looking for successful migration of applications and databases. The project involved Data Migration, Validation, and Reporting. The client hired Incepta to:
- Create new instances of Informatica and Snowflake with Entity 1 only data.
- Configure Airflow jobs related to Entity 1 and corporate to fetch relevant data from different supporting applications and update the Snowflake database.
- Ensure all BI (MicroStrategy & Looker) instances are pointing to the new instance of Informatica and Snowflake and generating reports with Entity-1 only data.
- Ensure smooth functioning of business after this separation.
- Run project using Agile – Scrum Framework, Jira
Tools and technologies involved-
- Data Migration– Informatica and Apache AirFlow
- Data Validation – RPA, AI, and Microstrategy
- Data Reporting – MicroStrategy Tableau and Lookup
One of North America’s oldest retail business groups with iconic department stores. It has three major entities.
- Entity 1 – A leading online eCommerce platform for luxury fashion
- Entity 2 – A premier luxury off-price eCommerce company
- Entity 3 – A Canadian eCommerce marketplace
Data Migration:
The client has three entities, with a combined database. The client wanted to separate them into different instances.
Data Validation:
Compare the data between the existing and new setup to validate that it is matching.
Data Reporting:
Generate various reports and create new user-friendly reporting dashboards with Tableau, Figma, and sometimes automation retaining details from previous reports.
Data Migration:
New instances for Entity 1 were created migrating data from the common instances on Snowflakes using Informatica and Apache AirFlow following the process of Extract Transformation and Load (ETL). The Incepta team ensured that the mappings and data migrated successfully and tested on the new instance of Informatica.
RPA:
Data lineage was carried out using UIPath bots to ensure the source and target data is validated.
QA:
Validating the end results by executing queries and capturing results to compare with production using test cases created for Snowflake, Microstrategy, and Airflow.
Data Engineering:
Update Data separately for Entity 1 from Entity 2 and Entity 3 using DAGs (Directed Acyclic Graph) which is the core concept of Airflow, collecting tasks together, organized with dependencies and relationships to say how they should run.
Incepta has deployed 183 DAGs that update tables, folders, and SFTPs to make sure that data is up to date.
Data Reporting:
The updated data from Data Engineering is then followed by reporting. Incepta created new instances on MicroStrategy, Tableau, and Lookup Looker and validated these dashboards/reports to generate the same results as the previous environment.
- 40 ETL jobs migrated to new instances
- 100s of reports were created using Tableau
- 7 Databases migrated and validated
By partnering with Incepta Solutions, the client could complete the data migration and validation project smoothly using the Extraction, Transformation, and Load or ETL approach.
40 ETL jobs migrated to new instances, finetuned, troubleshoot, and fixed with best practices, 100s of reports were created, and 7 databases migrated and validated.
Dashboards created by the Incepta team are being used by multiple stakeholders every day These dashboards and redesigned reports in user-friendly format can be accessed on multiple devices. The reports enable data-based decision-making faster and easier for executives benefitting the business and its clients.
