What is the Data migration process?, Data Migrated Source to Data Warehouse

What is the Data migration process?

1. Requirements will come as RSD (Requirement Specification Document)

2. RSD document will be shared with Both dev and SIT team

3. Dev Team will start the Developments, SIT (our team) will start with the test plan and test case preparation, mapping document preparations.

 

The reason for the source stage is 

Files can be a data source 

• We can load the file data into the database table, it will happen in the source stage 

Data may from multiple databases 

• 10 tables have to be migrated then

• 5 of them coming from MySQL database 

• 5 of them coming from the oracle database 

• We can merge all into the source stage 

Source stage is not needed, when we're receiving cleansed data and data is from single source 

No cleansed data is 

Like internet provided in the 100mbps of speed and limit of 1000 GB 

Like internet provided in the 100M of speed and limit of 1000G

Like internet provided in the 100M- 1000 GB 

This data has to go to target as 

For speed, column 100

For limit column 1000

1. If Source is a file then, the Source stage will be there

2. In this case file data will load to the source stage environment 

3. From the Source stage data will be migrated to the target stage 

4. In that target stage data has to be validated by the ETL testers 

5. If the ETL tester is given a green signal, then(sign off), data will be migrated to Target DB.

ETL Architecture Explaination

 

·         ETL Architecture

•  ETL is Extract Transformation and Load

•  Extracting the data from the OLTP environment and transforming the data into the ETL tools and    Loading the data into the OLAP environment

• OLTP can have multiple data sources called flat files and Databases and also legacy systems like DB2, mainframe databases

•  ETL will have a Source area and staging area and Landing area

• The source will be OLTP Db or else files

• The stage will be any of the RDBMS databases

• Target will be a database ( RDBMS)

• OLAP target will be designed like fact table and dimension table types and created as snowflake or star schema design.

OLAP Vs OLTP

OLTP

1. OLTP is a data source it can be a flat file or database 

2. It contains normalized data and current year data 

3. It contains regular transaction data 

OLAP 

• OLAP  will contain historical data and it must be a database 

• IT contains denormalized data and a huge volume of data 

• BI – reports will run on OLAP environment 

• OLAP will be a data source of an ETL process 

IT Industry Environment Details | What Will Do Every Department In IT Industry - Basics Of IT Industry

 Environment details

• DEV
• SIT
• UAT
• PROD

For Front end and back end developers, below environment is common

Dev  -  Development Environment 

it is used for development of the code – Developer will have full access for this environment – developers will do the development here – Developer will test the functionality which you developed here (that is called UNIT TESTING)

If unit testing is OK then Code will be promoted to SIT environment

Development will happen in Separate database (separate schema as well)

 

SIT   -  System Integration Testing (Our Environment)

Professional testers will test the functionality which is working or not,

They will understand the functionality and as per their plan they will execute some test cases

If any failures occurs then, code will send back to the dev team.

If everything is OK then, Code will promote to the UAT environment. 

SIT will happen in Separate database (separate schema as well)                  

For SIT database we will not have any DML / DDL privs / We can select the data from the database.

We can't modify anything in SIT

 

UAT – User acceptance Test

Environment is accessed by customers (End users)

Customer will test the functionality with their inputs.

In case of issues then, code will send back to SIT team

If everything is OK then, Code will promote to PROD

It will be in Separate DB – Separate Schema

We can’t access the data, some of them from our dev team can access the data

But any case, we can’t perform any DML / DDL in this environment.

 

PROD  - Production

High secured environment

We can’t perform any DDL / DML here

We can select the data, in case of any production issues validations

It will be in Separate DB – Separate Schema