• 0203 912 0170

  • contactus@transactts.com

chat-icon

Complex AWS Migrations (Technical Blog Series)

July 7, 2022 - Paul Corley

3-3 v.2

Introduction

Welcome to our third and final blog around migrations to AWS. We have looked at ‘Why Migrate to AWS?’ and also ‘How to Migrate?’ this third blog looks at complex AWS migrations which in this case we define as migrations involving databases or migrations that need to be carried out while production systems are live.

This blog will cover:

Migrating a Database to another Database Engine using AWS SCT
Migrating a Live Database using AWS DMS
Migrating Live Data using DataSync and/or Snowball Edge

Migrating a Database to another Database Engine using AWS SCT

Migrating a Database to another Database Engine tends to be difficult because:

  1. Schema and database code objects within a database may not be fully understood or documented
  2. Databases need to refactored to take advantage of cloud native services such as conversion of Microsoft SQL Server to Aurora MySQL

AWS Schema Conversion Tool (AWS SCT) - SCT a can be used to convert your database schema from the source database (e.g. on-premises) to the target database (AWS).

AWS SCT will assess and convert the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database. Any objects that cannot be automatically converted are clearly marked as action items with prescriptive instructions on how to convert, so that they can be manually converted to complete the migration.

AWS SCT can also scan your application source codes for embedded SQL statements and convert them as part of a database-schema-conversion project. During this process, AWS SCT performs cloud-native code optimization by converting legacy Oracle and SQL Server functions to their equivalent AWS service, helping to modernize the applications at the same time of database migration.

Using AWS SCT simplifies the migration process and reduces the time taken and effort involved and supports popular Database Engines including Oracle, Microsoft SQL Server, PostgreSQL, and MySQL.

Migrating a Live Database using AWS DMS

Most methods of database migration require a time delay as you backup the source database and restore it to the target database server. This delay requires the source database to be shutdown to avoid any uncaptured changes to the database. This is not suitable for a production system that needs to be always-on as the application will need to be shutdown during this process. AWS DMS solves this issue.

AWS Database Migration Service (AWS DMS) – helps you migrate your current databases to the AWS Cloud reliably, quickly, and securely. The source database remains fully operational during the migration, minimizing downtime with the following features:

Continuous Data Replication: Perform continuous data replication between an on-premise solution and AWS without the need to stop your application.

Homogeneous Database Migrations: Homogeneous database migration means that the source and target database engines are the same or are compatible like Oracle to Amazon RDS for Oracle, MySQL to Amazon Aurora.

Heterogeneous Database Migrations: In heterogeneous database migrations the source and target databases engines are different such as Oracle to Amazon Aurora or Oracle to PostgreSQL.

There are three main steps in the AWS DMS process:

  1. Create Target Database.
  2. Create a Replication Instance which is an EC2 server that carries out the replication tasks between the Source and Target Databases. The instance will require storage which is dependent on the load expected from the replication task. You should minimize network traffic by keeping the Replication Instance in the same subnet as the target databases where possible in addition to implementing Security Groups and ACLs to allow replication traffic to flow.
  3. Create a Replication Task specifying the Replication Instance to use, the Source and Target endpoints and the Migration type option. The migration type option for a live migration is normally Full load + CDC . Full load migrates existing data while CDC, Change Data Capture replicates ongoing changes. The Replication Task can be tailored further to your requirement using the other options available.

Migrating Live Data using DataSync and/or Snowball Edge

Migrating Live Data between storage systems is complex for the following reasons:

  1. Volume of data may be very large such as TBs or PBs
  2. Number of files my be millions
  3. File integrity must be maintained between Source and Target
  4. Requirement to demonstrate that all files have been moved and quickly resolve any issues where file transfer has failed

AWS DataSync can help can help resolve these issues by enabling you to easily sync your on-premises data with data in the Cloud.

AWS DataSync is an online data transfer service designed to simplify, automate, and accelerate data transfer to AWS. DataSync copies data over the internet or AWS Direct Connect. It automates management of the infrastructure and the transfer processes including encryption and validation. This approach minimizes the time for in-house development and management that is otherwise needed for fast, reliable, and secure transfers. DataSync charges a flat fee per-gigabyte of data transferred.

DataSync is thought of mainly as a tool that allows you to move files from NFS to S3 but has grown over the years to have support for copying data to and from NFS, SMB, HDFS, Object Storage, S3, Amazon EFS, Amazon FSX, Google Cloud Storage and Azure files.

A task needs to be created for each data transfer workflow. The main DataSync Components with descriptions are shown below:

Component Description
Agent A virtual machine hosted on a hypervisor used to read data from or write data to storage.
Location Any source or destination storage used to transfer data. Each location has a defined storage protocol such as S3 or NFS.
Task A task is the complete definition of a data transfer. It includes two locations (source and destination) and the configuration of how to

transfer the data from one location to the other. Configuration settings can include options such as how to treat metadata, deleted files, and copy permission

Task execution An individual run of a task, which includes options such as start time, bandwidth limits and verification methodology.
VPC EndPoint A highly available endpoint service that enables DataSync traffic to privately connect between the source and destination locations.
AWS Management Console Console allowing management of DataSync and related AWS services

DataSync is an online data transfer tool and is dependent on available bandwith which is often an issue for companies who have little bandwidth or little available bandwidth due to production workflows. DataSync can be used in conjunction with the AWS Snowball Family where Snowballs are used to move the majority of the data and then DataSync is used to keep the on-premises and cloud data synchronised until cutover.

The AWS Snowball Edge is Amazon’s primary Snowball Family offering for moving large amounts of data. It is a portable ruggedized device providing up to 80TB of storage and 40vCPUs of compute which you connect to your storage devices allowing you to transfer data.

AWS ship the device to your site, you fill the device with data and then ship it back to Amazon where your data is ingested into AWS storage services such as S3. You can use the AWS Snow Family console to order and then track your data transfer job from Snowball preparation, Delivery, Collection and Ingestion. You are provided with a job report with success and failure logs.

Conclusion

Migrating to AWS is a lot like preparing for a new website to go live. You have to ensure everything is in order and that your site runs as smoothly as possible to avoid any unexpected downtime or latency issues. However, migrating a whole infrastructure to the cloud is an even bigger undertaking, so working with a partner to guide you during the transition is advisable.

We hope you’ve found this series on Migrating to AWS to be a valuable guide and are excited about the many benefits of migrating to AWS. Whatever your requirements are, there is an AWS migration solution to fit, from single applications and operating systems to more complex migrations and hybrid/multi-cloud environments. However, if you require assistance, please reach out to our experienced cloud team; as an accredited AWS partner, we will help you with every stage of your migration journey cloud@transactts.com

AWS DataSync high-level architecture

Feedback-1.2-Datasync.png

Leave a Comment

Your email address will not be published. Required fields are marked *