Automated VM Migrations using AWS VM Import: A Python Driving Application

Co-Authors : Ashish Singh & Ayanampudi Varsha, Cloud Practice, Persistent Systems

Virtual Machine (VM) Migrations to Amazon Web Services has gained a lot of momentum over the last couple of years. With the introduction of AWS VM Import, VM migrations to AWS have even become easier for developers and companies, who may not want to use third party tools for cloud migration. Using CLI tools offered by Amazon, developers or migration engineers can import VMs onto AWS; this however can be a repetitive and manual task, creating a need of an automated approach of initiating the VM migrations, which requires multiple different parameters. In this post, we present a programmatic way of automating migrations to AWS by a python application that leverages the AWS VM Import utility.

AWS VM Import

The AWS VM Import utility developed by Amazon allows you to import (migrate from on-premise to AWS) and export (migrate back from AWS to on-premise) virtual machines. VMs can be imported by using Amazon EC2 API tools or the Amazon EC2 VM Import connector for VMware vSphere. The utility enables import of ‘raw’ formatted images, along with images compatible to the VMware ESX, Citrix Xen and Hyper-V platform. When importing an image, a flavor of target instance type must be assigned to the source image for booting up in AWS. A Windows image can be imported into an instance of any type that AWS offers, however to import Linux images, only specific instance types are supported. Images of the ‘raw’, ‘vmdk’ and ‘vhd’ formats can be imported into AWS. Additionally, specific operating system versions of Windows and UNIX can be imported. Additional information can be found here. In our exercise, we specifically look at importing a Windows Database server and Unix Web server. Before beginning the import, it is important to analyze which source VM to migrate and prepare the source environment. This also requires installation of the AWS EC2 CLI.

The following schematic is a typical workflow of a VM migration that leverages AWS VM import:

• Identify cloud candidates: You can analyze / identify which source VM needs to be migrated onto the cloud by profiling the VM. 3^rd party discovery and inventory analysis tools can be used here to identify the potential cloud candidates.
• Prepare source VM: Before starting the import, it is imperative to configure the source VM at the Operating system level. This may involve disabling the anti-virus program, enabling remote desktop and SSH, shutting down the VM etc. More details on the preparation step can be found here. In this step, you can also capture your source VM by taking a snapshot of the VM. Amazon refers to this step as ‘exporting the local environment‘. The snapshot can be taken while the VM is running or paused, depending on what the underlying subsystem may offer. VMware vSphere offers live snapshot of a VM, while the VM is automatically paused while taking a snapshot in KVM. AWS recommends shutting down the VM and exporting the image.
• Install EC2 CLI: In order to initiate and manage the migration (import), you need to install the EC2 CLI tool on the machine source VM image resides.
• Execute VM Import: Initiate the migration by invoking the ec2-import-instance command. The command takes the following parameters as input.

• Check VM Import status: The status of the VM Import task can be checked by issuing the ‘ec2-describe-conversion-tasks‘ [3]. If the VM import fails, the import can be restarted (resumed) by issuing the ‘ec2-resume-import‘ command. On the other hand, once the VM import task completes successfully, the VM is launched as an EC2 instance in the ‘stopped’ state. Once the import is successful, the EC2 instance needs to be started using the ‘ec2-start-instances‘ command.
• Integrity Check: In this step the integrity check of the VM can be performed either manually, or by using some automated mechanisms. The primary purpose of doing this is to validate if the VM contents are consistent after the VM instance is launched in EC2.
• Delete image from S3: You can delete the image (image parts) from the S3 bucket to avoid incurring of storage costs. This step is an optional.

Integration of AWS Import a driving application

We now take a closer look at how the AWS Import utility integrates with our driving application. Our driving application is a python based utility which captures a VM data disk file (image) and forklifts (transports) the image to a target destination. We leverage the AWS Import feature to migrate VM’s to AWS. Our python utility combines Apache Libcloud API with AWS VM Import to conduct the VM migration.

Once the image is prepared and ready to be forklifted, our driving application first creates a temporary VM placeholder in AWS, in the form of an S3 bucket. To create a bucket on a target AWS region, an AWS EC2 account is required. The ‘EC2_ACCESS_ID’ and ‘EC2_SECRET_KEY’ must be given as input. The following code snippet represents the creation of an S3 bucket.

For the forklifting piece, we execute the VM Import command to migrate the source VM image into the S3 bucket, and boot the VM instance from the bucket. The AMAZON_KEY and AMAZON_CERT parameters are required for the VM import. The image upload process breaks down the image into smaller chunks, where each chunk (part) is a size of 10 MB. These chunks are uploaded in parallel, and in-case of an upload failure, or link error, the upload process can be resumed from the point it failed. The time taken to import the image is a function of network bandwidth and the disk size.

Once the image import is completed, the EC2 instance is created in a stopped state. Our driving application boots the VM in a ‘running’ state as shown below.

After the VM comes back into the ‘running’ state, the actual VM migration is said to be completed. At this point in time, the migration engineer can perform an integrity check (manual or automated) to validate if all the VM data, file-system structure is consistent as the original source VM.

In this way, our python driving application integrates with the AWS VM Import utility to conduct an automated migration of a Windows Database server on AWS.