Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction
2.
Creating Source and Target Endpoints
2.1.
Step 1
2.2.
Step 2
3.
Sources for Data Migration
4.
Targets for Data Migration
4.1.
Using an Oracle Database as a Source for AWS DMS
4.2.
Using Microsoft SQL Server Database as a Source for AWS DMS
4.3.
Using Microsoft Azure SQL Database as a Source for AWS DMS
4.4.
Using Microsoft Azure SQL Managed Instance as a Source for AWS DMS
4.5.
Using Google Cloud for MySQL as a Source for AWS DMS
4.6.
Using PostgreSQL database as a Source for AWS DMS
4.7.
Using MySQL-compatible database as a Source for AWS DMS
4.8.
Using an SAP ASE database as a Source for AWS DMS
4.9.
Using MongoDB as a source for AWS DMS
4.10.
Using Amazon DocumentDB as a Source for AWS DMS
4.11.
Using Amazon S3 as a Source for AWS DMS
4.12.
Using IBM Db2 for Linux, Unix, and Windows Database as a Source for AWS DMS
4.13.
DDL Commands supported
5.
Frequently Asked Questions
5.1.
What is the meaning of Data Migration?
5.2.
List the advantages and disadvantages of Data Migration.
5.3.
Mention the types of Data Migration.
6.
Conclusion
Last Updated: Mar 27, 2024
Easy

AWS DMS Endpoints

Author Rupal Saluja
0 upvote
Master Python: Predicting weather forecasts
Speaker
Ashwin Goyal
Product Manager @

Introduction

In this blog, we will learn about AWS DMS endpoints. An endpoint is a source of connection, holder of type of data store, and storage space for local information. AWS DMS is a service that utilizes this information to form a connection with the data store and facilitates data migration between source and target endpoints. Extra connection attributes control log, file size, and some other parameters. So now, let us start with first understanding how we can create source and target endpoints.

Creating Source and Target Endpoints

To create a source endpoint, we must create a replication instance first. With assumption, AWS DMS console wizard has been chosen; follow the below procedure to create source and target endpoints.

If you use console wizard, you can create endpoints on the same page, while endpoints are created differently on separate pages if we do not use Console Wizard.

Step 1

Mention the connect information for the databases, source, and target on the ‘Connect source and target database endpoints’ page. The following table specifies the compulsory settings.

OPTION

CHOICE

Endpoint Type Either choose source or target as per which is required.
RDB DB Instance Choose this option only if an endpoint is an Amazon RDB DB Instance.
Endpoint Identifier Mention the name you want to use to identify the endpoint. Names should be unique for every replication instance.
Database Engine Choose the type of database engine as per the requirement of endpoint, that is, source or target.
Access to endpoint database

Select the option that specifies endpoint database credentials.

  • Choose AWS Secrets Manager
  • Provide access information manually
AWS Secrets Manager To set the secret credentials.
Secret ID You can type here full ARN, that is, Amazon Resource Name or Partial ARN or any friendly name you might have given for endpoint database access in Secrets Manager.
Secret ID for Oracle ASM You can type here full ARN, that is, Amazon Resource Name or Partial ARN or any friendly name you might have given for Oracle ASM access in Secrets Manager.
IAM Role Fill here the ARN of the secret access role created to provide AWS DMS access in IAM.
IAM Role for Oracle ASM Fill here the ARN of the secret access role created to provide AWS DMS access in IAM for Oracle ASM.
Access Information Set the credentials to provide access to information manually.
Server Name Fill in the server name, which can be the IP address or the public hostname.
Port Here, the details of the port used by the database will be entered.
SSL Mode SSL stands for Secure Socket Layer. For connection encryption to be enabled for the endpoint, we select SSL Mode. You might also have to provide a server certificate or its information here.
User Name Fill in the username with the required permissions.
Password Fill in the password with the required information.
Database Name In certain database engines, you are also required to fill in name of the database you want to use endpoint database.

We cannot use some characters in the endpoint passwords of certain database engines. These characters are mentioned below-

  1. Microsoft Azure- ‘;’
  2. Microsoft SQL Server- ‘, and;’
  3. MySQL, MariaDB, Amazon Aurora MySQL- ‘;’
  4. Oracle- ‘,’
  5. Postgre SQL, Amazon Aurora, Amazon Aurora Serverless- ‘; and + and %’
  6. Amazon Redshift-, and ;

 

Step 2

For Extra connection attributes and AWS KMS key, if required, go for the Advanced Tab.

OPTION

CHOICE

Extra Connection Attributes Here additional parameters would be filled. 
AWS KMS key Choose the suitable credentials with the required information.
Test Endpoint Connection (optional) For this, you have to add VPS and replication instance name, then select Run Test.

To check if the endpoint connection works properly or not, choose the ‘Run Test’ option.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Sources for Data Migration

Since AWS DMS uses many data search engines, it provides various sources for data migration. Some of these are mentioned below in the list.

  1. Oracle database
  2. Microsoft SQL Server database
  3. Microsoft Azure SQL database
  4. Microsoft Azure SQL Managed Instance
  5. Google Cloud for MySQL
  6. PostgreSQL database
  7. MongoDB
  8. Amazon DocumentDB
  9. Amazon S3
  10. SAP ASE database

Targets for Data Migration

AWS DMS also provides numerous targets for data migration. Some of these are mentioned below in the list.

  1. Oracle database
  2. Microsoft SQL Server database
  3. PostgreSQL database
  4. SAP ASE database
  5. Amazon DynamoDB database
  6. Apache Kafka
  7. Amazon DocumentDB
  8. Amazon Neptune
  9. Amazon Kinesis Data Streams
  10. Redis

Using an Oracle Database as a Source for AWS DMS

If we use Oracle Database as a source for AWS DMS, data can be migrated from one or many Oracle databases. Data is migrated to any of the targets supported by AWS DMS. Oracle database editions supported by AWS DMS are mentioned below:

  • Oracle Enterprise Edition
  • Oracle Standard Edition
  • Oracle Express Edition
  • Oracle Personal Edition

Using Microsoft SQL Server Database as a Source for AWS DMS

If we use Microsoft SQL Server Database as a source for AWS DMS, we can migrate data from one or many Microsoft SQL Server databases. Data is migrated to any other SQL Server database or AWS DMS-supported databases.

Various SQL Server versions are available and are used as per the type of Online replication (CDC) being performed.  

Using Microsoft Azure SQL Database as a Source for AWS DMS

If Microsoft Azure SQL Database is used as a source, it is used in the same way as you use SQL Server with AWS DMS. The same list of database versions supported for SQL Server running on-premises or on an Amazon EC2 instance is supported by Microsoft Azure SQL Database.

Using Microsoft Azure SQL Managed Instance as a Source for AWS DMS

Microsoft Azure SQL Managed Instance as a source is used the same way as we use SQL Server with AWS DMS. The same list of database versions supported by SQL Server running on-premises or on an Amazon EC2 instance are supported by Microsoft Azure SQL Managed Instance.

Using Google Cloud for MySQL as a Source for AWS DMS

Google Cloud for MySQL as a source is used in much the same way as you do MySQL, with AWS DMS. As a source, AWS DMS supports Google Cloud Platform (GCP) for MySQL versions of 5.6, 5.7, and 8.0. 

Using PostgreSQL database as a Source for AWS DMS

With the PostgreSQL database as a source, data is migrated from one or many PostgreSQL databases. Data is migrated from either another PostgreSQL database or one of the other supported databases. AWS DMS as a source supports certain types of databases, as mentioned below. 

  • On-premise databases
  • Databases on an Amazon EC2 instance
  • Databases on an Amazon RDS DB instance
  • Database on DB instance based on Amazon Aurora PostgreSQL- compatible edition

Using MySQL-compatible database as a Source for AWS DMS

Data can be migrated from any MySQL-compatible database using AWS DMS. Certain versions of MySQL, MariaDB, or Amazon Aurora MySQL supported on-premises are listed below.

  • MySQL versions- 5.5, 5.6, 5.7 and 8.0
  • MariaDB versions- 10.0.24 to 10.0.28, 10.1, 10.2, 10.3, 10.4 and 10.5
  • All versions of Amazon Aurora MySQL

Using an SAP ASE database as a Source for AWS DMS

Formerly known as Sybase, data can be migrated from an SAP Adaptive Server Enterprise using AWS DMS. The database is migrated to any of the supported AWS DMS target databases. SAP ASE versions of 12.5.3 or higher 15, 15.5, 15.7, 16, and later versions are supported by AWS DMS.

Using MongoDB as a source for AWS DMS

AWS DMS supports MongoDB versions of 3.0 and 4.0. Two Migration modes are supported by AWS DMS when using MongoDB as a source. These modes are Document Mode or Table Mode. You can specify at the time of creating the MongoDB endpoint or by setting the Metadata Mode parameter from the AWS DMS console.

Using Amazon DocumentDB as a Source for AWS DMS

Amazon DocumentDB versions  (3.6 and 4.0) are supported by AWS DMS, with MongoDB compatibility, as a database source. Data can be migrated from one Amazon DocumentDB cluster to another. Also, you can migrate data from an Amazon DocumentDB cluster to one of the other target endpoints supported by AWS DMS.

Using Amazon S3 as a Source for AWS DMS

Data can be migrated from an Amazon S3 bucket using AWS DMS. You just need to provide access to an Amazon S3 bucket containing one or more data files. Also, include a JSON file that describes the mapping between the database and database tables of the data, in that S3 file.

Using IBM Db2 for Linux, Unix, and Windows Database as a Source for AWS DMS

Data can be migrated from IBM Db2 for Linux, Unix, and Windows Database to any supported target database using AWS DMS. 

The following versions of Db2 LUW are supported as migration sources by AWS DMS:

  • Version 9.7, with all Fix Packs, supported
  • Version 10.1, with all Fix Packs, supported
  • Version 10.5, with all Fix Packs, supported except for Fix Pack 5
  • Version 11.1, with all Fix Packs, supported
  • Version 11.5, Mods (0-7) with only Fix Pack 0 supported

DDL Commands supported

Data Definition commands can be executed on the source database to complete the process of Data Migration. These commands are mentioned below in the list.

  1. Create Table
  2. Drop Table
  3. Rename Table
  4. Truncate Table
  5. Add Column
  6. Drop Column
  7. Rename Column
  8. Change Column Datatype

Depending on the search engine, supported DDL Commands vary because every search engine handles commands differently.

Frequently Asked Questions

What is the meaning of Data Migration?

Data Migration is the process of changing data location, data format, or the application on which data is processing. It is required when a new system or location has been introduced.

List the advantages and disadvantages of Data Migration.

The advantages of Data Migration include-

  • Comprehensive Data Integrity
  • Reduced Media and Storage costs
  • Minimized disruptions with minimal manual efforts
  • Upgradation of underlying applications
  • Helps in the scaling of resources

 

The disadvantages of Data Migration include-

  • High cost
  • It takes a lot of time
  • Inheritance of poor data quality
  • Data loss risk
  • Can cause application instability

 

Mention the types of Data Migration.

Some known types of Data Migration are as follows-

  1. Database Migration
  2. Application Migration
  3. Storage Migration
  4. Cloud Migration

Conclusion

In this article, we have extensively discussed AWS DMS endpoints and their creation. We also saw some prominent sources and targets for Data Migration. 

We hope that this blog helped you enhance your knowledge regarding AWS DMS Endpoints. If you want to learn more about such articles, you can refer to our platform Coding Ninjas Studio.

For peeps out there who want to learn more about Data Structures, Algorithms, Power programming, JavaScript, or any other upskilling, please refer to guided paths on Coding Ninjas Studio. Enroll in our courses, go for mock tests, solve problems available, and interview puzzles. Also, you can put your attention towards interview stuff- interview experiences and an interview bundle for placement preparations. 

Do upvote our blog to help other ninjas grow.

Happy Coding!

Previous article
Amazon Data Lifecycle Manager
Next article
AWS Cloud Computing
Live masterclass