Kubernetes is built for containers, and it’s been doing that since it used to run docker as an engine for its containers. But what if you want to add VMs to the mix? After all, containers are ephemeral and don’t require fixed IPs as they shift the identity toward labels, but VMs on the other hand are tied to IP addresses and in some cases MAC addresses.
This brings us to this blog about VM migration and IP preservation. Unlike a pod that can be part of a deployment and run in a swarm of stateless endpoints, a VM is a stateful machine run by hypervisor like QEMU and extended to Kubernetes via KubeVirt Custom Resource Definitions (CRDs).
There Is Something About KubeVirt
KubeVirt is an abstraction layer between the underlying hypervisor (QEMU) on your machine and Kubernetes. Its job is to manage a VM’s lifecycle and provide the necessary requirements for a VM to be a native resident in Kubernetes. These requirements are CPU, Memory, Networking, etc.
KubeVirt does this by wrapping each VM in an ordinary Kubernetes pod called virt-launcher. Inside that pod, KubeVirt runs libvirt and QEMU, and the “VM” is really just a process scheduled, networked, and accounted for like any other pod. That detail matters a lot once we get to migration: when a VM moves to another node, what Kubernetes actually does is create a brand-new virt-launcher pod on the destination and tear down the old one. Everything hard about live migration comes from making that pod swap invisible to the workload running inside.
CPU
CPU is the part that does the actual work, every instruction the guest operating system and its applications execute runs on a virtual CPU that KubeVirt maps onto real cores of the host node. You can pin the VM to dedicated cores, expose host CPU features, or let it float over shared cores. For migration, the CPU matters for a subtle reason: while a VM is being moved, its CPU keeps running and keeps changing memory. The faster the guest dirties memory, the harder it is to copy that memory to the other node before it changes again. We’ll come back to this race in a moment.
Memory
Other than being expensive these days, RAM or Memory has a crucial role in VM migration, since it is the place where everything that the CPU is working on is stored and referenced. In a physical computer, memory is the expensive stick that you buy and install in your computer. However, in a VM, memory is a region of your computer’s RAM allocated for the tasks that the VM is actively working on.
Memory is the thing migration is really about. When KubeVirt live-migrates a VM, the bulk of what it ships from the source node to the destination node is the VM’s RAM, gigabytes of it, while the guest keeps running and keeps writing to it.
Networking
Another important part of a VM is networking, and KubeVirt supports multiple networking modes. Our focus is going to be on bridge, since that is required for VM migration with Calico, but if you’d like to learn more about other modes feel free to check out the official KubeVirt documentation.
What is a bridge?
A bridge is similar to a playground where all the resources connected to it are able to communicate with each other. In Linux, a software bridge is a virtual switch: you plug interfaces into it and it forwards Ethernet frames between them just like a physical switch would.
In KubeVirt’s bridge mode, the VM is connected to the pod network through a Linux bridge, and the pod’s IP address is handed down to the VM itself (via DHCP). The guest doesn’t get some separate, NAT’d address, it uses the pod IP directly as its own. If the relevant pod interface has a MAC address and the VM doesn’t override it, the VM inherits that MAC too.
That “VM uses the pod IP directly” property is exactly why bridge mode is the only mode that works for live migration with Calico, and it’s the hinge the rest of this post turns on.
How local live migration actually works
Local live migration is the process of moving a running VM from one node to another within a cluster while the guest keeps running and stays reachable. No reboot, no shutdown, ideally the application inside never even notices.
You start the process by posting a VirtualMachineInstanceMigration (VMIM) object, or just running virtctl migrate vm1, and KubeVirt does the rest. The default strategy is pre-copy, and it works roughly like this:
- A new target VM (a fresh
virt-launcherpod) is created on the destination node, while the source machine is still running. - The source starts streaming chunks of VM state, mostly RAM, to the target. This repeats: pages that the guest dirties while the copy is in flight get re-sent.
- Once enough state has transferred that only a tiny delta remains, the guest is briefly paused, the last pages are shipped, and the guest resumes on the target.
- The source VM is removed.
Keep in mind that the VM migration is a race between the source and your network speed. If the source is copying memory to the target and the guest is simultaneously adding new things to that memory and this rate is faster than the network can copy them, the migration may never converge. KubeVirt has knobs for that (auto-converge throttles the guest CPU; post-copy runs the guest on the target immediately and faults memory across on demand), but for most workloads pre-copy just works, and that’s what we’ll see in the real report below.
KubeVirt is deliberately conservative here. Out of the box it runs at most 5 migrations in parallel cluster-wide, no more than 2 outbound per node, and caps each migration at 64 MiB/s of bandwidth, so a busy cluster doesn’t saturate its own network moving VMs around.
The hard part: the VM has to keep its IP
Here’s the thing the KubeVirt docs gloss over and where Calico Unified Platform does the heavy lifting. Remember that a migration is really a pod swap, old virt-launcher pod on node A dies, and a new one on node B is born. Normally, a brand-new pod means a brand-new IP. For a stateless web pod behind a Service, who cares. For a VM, the IP is its identity, every open TCP connection, every client that resolves it, every firewall rule references that address. Change the IP and you’ve effectively killed and rebooted the machine, which is the exact opposite of “live.”
So the job is: the new pod on node B must come up with the same IP the old pod had on node A, traffic must switch over to node B at exactly the right moment, and network policy has to already be in place on node B before that switch, otherwise the VM lands on the new node and gets firewalled off from its own connections. Calico coordinates all of this. Here’s how the pieces fit.
Bridge mode is non-negotiable
Because Calico’s IP persistence depends on the VM IP matching the pod IP, only bridge mode qualifies. Modes like masquerade give the VM a different internal IP than the pod and NAT between them, which breaks both IP persistence and policy enforcement during migration. KubeVirt actually refuses to migrate a bridge-mode VM unless you opt in with an annotation:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm
spec:
template:
metadata:
annotations:
kubevirt.io/allow-pod-bridge-network-live-migration: ""
...VM IP address persistence
Calico keys its IP allocation off the VM’s identity rather than the pod’s. Internally, instead of the usual per-pod IPAM handle, the address is held under a VM-scoped handle like k8s-pod-network.vmi.default.vm1. When the target pod is created during migration, Calico’s CNI plugin recognizes it as a KubeVirt virt-launcher pod, looks up that VM handle, finds the existing IP, and reuses it instead of allocating a fresh one.
This is a cluster-wide setting added in Calico v3.32 (kubeVirtVMAddressPersistence, enabled by default), and it’s mandatory for this type of migration. With persistence off, Calico rejects the migration target outright.
Don’t NAT the VM on the way out
If a VM talks to something outside the cluster and natOutgoing is enabled on its IP pool, the server on the other end sees the VM’s traffic as coming from the node’s IP. Migrate the VM and that source IP changes from node A’s address to node B’s, and any in-flight connection to that external server breaks. So for migratable VMs you disable natOutgoing on their pool, keeping the VM’s own IP on the wire end-to-end.
Switching the traffic with BGP route priority
Both nodes briefly believe they host the VM’s /32 route. Calico breaks the tie with route priority: the target host programs the route with an elevated priority (a lower kernel metric, 512 by default) than the source’s normal priority (1024). Lower metric wins in the Linux kernel, so traffic steers to the target. Within a single AS this propagates automatically (Calico maps the kernel metric onto BGP local_pref); across eBGP rack boundaries you carry the signal with BGP communities via a BGPFilter. After a convergence window (~30s by default), priorities return to normal.
Policy must land before the switch
This is the subtle one. If the VM activated on the target node before its network policy was programmed there, it would arrive into a node that doesn’t yet know its firewall rules, and get cut off. Calico prevents this with an interlock: when the CNI plugin sets up the migration target pod, it returns the IP with empty routes, deliberately not programming the host-side routes that would pull traffic over. Felix only completes that switch once policy is in place on the destination (governed by policy_setup_timeout_seconds). Policy first, traffic second.
Wrapping up
Live migration looks like magic, a running machine teleports between hosts and nobody notices, but it’s really two systems cooperating very carefully. KubeVirt handles the compute side: racing the guest’s memory across the wire with pre-copy until it converges, then cutting the CPU over in under a second. Calico handles the network side: pinning the VM’s IP to the VM’s identity so the new pod reclaims it, withholding routes until policy is programmed on the destination, switching traffic with BGP route priority, and cleaning up dual ownership after a convergence window, all without the VM’s address ever changing.
Get the networking mode wrong (anything but bridge), forget to disable natOutgoing, or skip the policy interlock, and “live” migration becomes a reboot with extra steps. Get them right, as the report above shows, and a stateful VM moves between physical machines while its TCP connections stay open and its identity stays put.
Watch an interactive demo: Live Migration of VMs Running on Kubernetes
