To determine the baseline for the cost of migration, we collected the migration statistics of an idle virtual machine. Table 5.1 gives statistics during the migration of an idle VM on a dual core Opteron Xen host at different VM memory (RAM) sizes. The memory page size is 4KB. The results and an inspection of the Xen migration code 1 show that the major cost of migration is copying of the memory from one VMM to another. This means that the cost of migration has a direct relationship with the size of the memory that needs to be transfered and the communication bandwidth of the underlying network infrastructure.
The average CPU utilization during the migration for both the source VMM and the destination VMM was around 95%, suggesting the fact that migration is a CPU intensive operation. The average bandwidth utilization in each case was around 108 MB/Sec, which suggests that the network bandwidth plays a limiting role in migration of the virtual machines. To improve migration speed, one can employ channel bonding or a faster interconnect. The total number of pages transfered is slightly more than the total pages in the memory. This is because
1
5.3 Xen Migration Statistics
Table 5.2: Average statistics of migrating a 512 MB (131072 Page) VM on a 1×4 cluster configuration.
Benchmark Total Pages Transfered Avg. Page Transfer Breakdown Iterations
First Iter Avg. in Between Last Iter
CG.B.4 149951±6% 130017 675 1040 30
EP.B.4 138941±4% 130656 283 359 30
FT.B.4 470278±8% 41334 36256 66385 24
LU.B.4 189551±6% 120910 2108 9617 30
HPL 833263±7% 46390 25859 62094 30
of the fact that the VM is still running and memory pages are being dirtied. The rate of memory pages being dirtied is very low and it takes only two iterations to complete the migration. The downtime of the VM is in the order of milliseconds. The standard deviation for the total number of pages transfered was less than 1%. In the case of an HPC application, we expect the memory dirtying rate to be significant. Table 5.2 gives migration statistics of migrating a single VM on a 1×4 cluster configuration for the NAS Parallel Benchmarks (NPB) and the HPL benchmark. This migration scenario is as shown in Figure 5.2.
VM 1
VMM-1
VM 2
VMM-2
VM 3
VMM-3
VM 4
VMM-4
VMM-5
Figure 5.2: Migration of a single VM from a VMM in a 1×4 configuration .
The CPU utilization of each VM during the benchmark run was 100%. which understandably dropped to 0% in the stop-and-copy operation. The CPU utilization of the Xen VMM (or Dom-0) during the benchmark varied with the communication rate of the benchmark but jumped to 100% during the migration. A comprehensive analysis of CPU utilization is given in Section 5.4.1.
All the benchmarks, with an exception of FT, use the maximum bound set for pre-copy iterative process. The FT benchmark is highly communication and store intensive. Memory is dirtied quickly in FT compared to other benchmarks 73
and thus the migration threshold is achieved earlier. During the migration, the total number of pages transfered from source to destination VM are 3.6 times the actual number of pages in the VM’s memory. Similarly in the case of the LU benchmark, 1.45 times more pages are transfered compared to the total pages in the memory of the VM. The standard deviation for the total number of pages transfered during the migration of a single VM varies with the memory dirtying rate and the communication intensitivity of the benchmark. For EP, we see this to be the lowest as there is no communication between the processes. However, for FT benchmark, we see a higher deviation. As FT is communication intensive and has a higher memory dirtying rate, the VMM not only has to transfer the memory pages of the migrating VM to the destination VMM but it also has to transfer the network packets. This results in the VMM CPU carryout two CPU intensive operations at the given time and hence we see the higher standard deviation.
The pages transfered in the earlier iterations are mostly retransmitted in subsequent iterations as they are dirtied again by the time iteration finishes. This transfer wastes a lot of CPU cycles and communication bandwidth and therefore should be avoided.
We varied the problem size (N) of the HPL benchmark to determine the effects of memory footprint on the migration time. We used problem sizes of 8000, 10000, 12500 and 15000 and kept all the other parameters as described in Section 5.2. It was found that the memory footprint of an application has no affect on the migration time. This is because of the fact that during migration, Xen transfers the memory pages from the source VMM to the destination VMM in a number iterations. In the first iteration, Xen transfers the whole VM memory (used or unused) as shown in Table 5.1. In the subsequent iterations, only those memory pages that were dirtied during the previous iteration are transfered. This means, the memory footprint does not affect the migration time, rather it is the memory dirtying rate that dictates the migration time.