How to resolve time drifts in HyperV and KVM

Table of Contents

Case
Solution
Sources

Case #

You have a hypervisor-based infrastructure and you notice that your virtual machines do not show the correct time, even though you have made arrangements to synchronize time from a trusted NTP server, such as for example time.windows.com or time-a-g.nist.gov or pool.ntp.org. These time synchronization issues create clock time drifts between the time shown on the hypervisors and the time shown on the virtual machines. These issues are also called clock skew issues. This KB article provides guidance on how to resolve time drifts in HyperV and KVM.

If two systems' clocks times differ more than one minute, this can cause a lot of different issues, such as for example the Windows Kerberos protocol to stop functioning properly or the Windows Update mechanism to stop functioning with the following error.

We couldn't install updates because there's a problem with the date and time information on your device. Make sure your date, time and time zone settings are correct and we'll try again later.

How to resolve time drifts in HyperV and Linux KVM guest machines

You check your workstations or servers and realize that there is significant clock time drift from a trusted atomic clock, such as in the following example. In the screenshot below, there is only 0.001 time drift, which is ideal. In an problematic case however, the time drift can be significant and can be the "hidden" root cause of many application errors or custom application software bugs, which can be difficult to pinpoint. Having a health time sync configuration in your infrastructure is therefore of paramount importance.

Solution #

Hypervisor guest tools, integration services and drivers #

Knowing how to resolve time drifts in HyperV and KVM guest machines is crucial for your applicaiton health. In the case of Linux KVM hypervisor, ensure that you have the latest QEMU agent and Virtio drivers installed on your Linux or Windows virtual machine, as per the following instructions: https://docs.openshift.com/container-platform/4.10/virt/virtual_machines/virt-installing-qemu-guest-agent.html.

In the case of Hyper-V virtual machines, ensure that you have installed and enabled Hyper-V integration services, including the "Time Synchronization" and "Guest services" for all your VMs.

In Windows machines there are operating system tools available to configure and check healthy time synchronization, as for example the w32tm.exe tool. You can use the command-line tool W32tm.exe in an elevated command prompt to configure Windows Time service settings and to diagnose computer time problems. Membership in the local Administrators group is required to run W32tm.exe locally, while membership in the Domain Admins group is required to run W32tm.exe remotely.

For basic troubleshooting, you should run the following commands in an elevated command prompt on the AD domain controller and on a domain member server.

w32tm /query /source /verbose
w32tm /query /peers /verbose
w32tm /query /status /verbose
w32tm /query /configuration
w32tm /tz

# Try to resync time on the local machine
w32tm /resync /rediscover

If you run into any issues when checking time sync status, you need to ensure first that the AD domain controller has an active sync status with the external NTP time source, e.g. the time.windows.com server. Also compare the following registry key of the server(s) in question with the same key on a healthy server. An example is shown below.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters

Carry out the steps below for logging w32tm.exe events.

Enable w32tm.exe debugging with the following command: w32tm /debug /enable /file:C:\Scripts\w32time.log /size:10000000 /entries:0-300
Search the log and look for a line similar to "ReadConfig: failed to open 'Software\Policies\Microsoft\W32Time\Config' key,[...]". In this case you should check for another process which is running on UDP 123 (Windows time service port)
For further investigation, TCPView from SysInternals if there is another process working on UDP 123 port.

After making any necessary changes to the w32tm configuration (either with manual w32tm commands or by GPO), run the following commands on each affected machine, to apply changes. Instead of the default time.windows.com NTP server, you may use the appropriate server from https://www.pool.ntp.org/.

net stop W32Time
w32tm /unregister
w32tm /register
net start W32Time
w32tm /resync /rediscover
# You will have to wait for maximum of 15 minutes before the changes of the above commands are applied. An equivalent action instead of the above commands, if you need immediate effect of the configuration changes, is to reboot the machine(s) in question and ensure that the w32tm service is running after reboot.

If issues persist, consult the following troubleshooting articles:

Hypervisor best practices on how to resolve time drifts in HyperV and Linux KVM guest machines #

Review the following configuration best practices for Linux KVM and Linux LXC virtualization clusters to avoid clock time drifts.

Group Policy time sync configuration #

The Windows Time service (W32Time) synchronizes the date and time for all computers managed by Active Directory Domain Services (AD DS). Most domain-joined computers have a time client type of NT5DS, which means that they synchronize time from the domain hierarchy. An exception to this is the domain controller, which functions as the primary domain controller (PDC) emulator operations master for the root forest domain. The PDC emulator operations master in turn is usually configured to synchronize time with an external time source. You can set Active Directory GPO to configure time sync from time.windows.com and internally from the PDC emulator FSMO role but in essence this get overridden by the KVM guest tool which writes the KVM host time to the guest.

The Microsoft best practices for time synchronization in a Windows Server environment are provided in the following articles:

Notes for VDI environments #

If you are running a VDI environment, for instance Microsoft RDS or Citrix Virtual Apps and Desktops, you should consider the following notes as well.