You have a hypervisor-based infrastructure and you notice that your virtual machines do not show the correct time, even though you have made arrangements to synchronize time from a trusted NTP server, such as for example time.windows.com or time-a-g.nist.gov or pool.ntp.org. These time synchronization issues create clock time drifts between the time shown on the hypervisors and the time shown on the virtual machines. These issues are also called clock skew issues. If two systems’ clocks times differ more than one minute, this can cause a lot of different issues, such as for example the Windows Kerberos protocol to stop functioning properly or the Windows Update mechanism to stop functioning with the following error.
We couldn’t install updates because there’s a problem with the date and time information on your device. Make sure your date, time and time zone settings are correct and we’ll try again later.
You check your workstations or servers and realize that there is significant clock time drift from a trusted atomic clock, such as in the following example. In the screenshot below, there is only 0.001 time drift, which is ideal. In an problematic case however, the time drift can be significant and can be the “hidden” root cause of many application errors or custom application software bugs, which can be difficult to pinpoint. Having a health time sync configuration in your infrastructure is therefore of paramount importance.
Hypervisor guest tools, integration services and drivers
In the case of Linux KVM hypervisor, ensure that you have the latest QEMU agent and Virtio drivers installed on your Linux or Windows virtual machine, as per the following instructions: https://docs.openshift.com/container-platform/4.10/virt/virtual_machines/virt-installing-qemu-guest-agent.html.
In the case of Hyper-V virtual machines, ensure that you have installed and enabled Hyper-V integration services, including the “Time Synchronization” and “Guest services” for all your VMs.
In Windows machines there are operating system tools available to configure and check healthy time synchronization, as for example the w32tm.exe tool. You can use the command-line tool W32tm.exe in an elevated command prompt to configure Windows Time service settings and to diagnose computer time problems. Membership in the local Administrators group is required to run W32tm.exe locally, while membership in the Domain Admins group is required to run W32tm.exe remotely.
For basic troubleshooting, you should run the following commands in an elevated command prompt on the AD domain controller and on a domain member server.
w32tm /query /source /verbose w32tm /query /peers /verbose w32tm /query /status /verbose w32tm /query /configuration w32tm /tz # Try to resync time on the local machine w32tm /resync /rediscover
If you run into any issues when checking time sync status, you need to ensure first that the AD domain controller has an active sync status with the external NTP time source, e.g. the time.windows.com server. Also compare the following registry key of the server(s) in question with the same key on a healthy server. An example is shown below.
Carry out the steps below for logging w32tm.exe events.
- Enable w32tm.exe debugging with the following command: w32tm /debug /enable /file:C:\Scripts\w32time.log /size:10000000 /entries:0-300
- Search the log and look for a line similar to “ReadConfig: failed to open ‘Software\Policies\Microsoft\W32Time\Config’ key,[…]”. In this case you should check for another process which is running on UDP 123 (Windows time service port)
- For further investigation, TCPView from SysInternals if there is another process working on UDP 123 port.
After making any necessary changes to the w32tm configuration (either with manual w32tm commands or by GPO), run the following commands on each affected machine, to apply changes. Instead of the default time.windows.com NTP server, you may use the appropriate server from https://www.pool.ntp.org/.
net stop W32Time w32tm /unregister w32tm /register net start W32Time w32tm /resync /rediscover # You will have to wait for maximum of 15 minutes before the changes of the above commands are applied. An equivalent action instead of the above commands, if you need immediate effect of the configuration changes, is to reboot the machine(s) in question and ensure that the w32tm service is running after reboot.
If issues persist, consult the following troubleshooting articles:
- https://docs.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-for-traceability (W32TM event logs)
Hypervisor best practices for time drift avoidance
Review the following configuration best practices for Linux KVM and Linux LXC virtualization clusters to avoid clock time drifts.
Group Policy time sync configuration
The Windows Time service (W32Time) synchronizes the date and time for all computers managed by Active Directory Domain Services (AD DS). Most domain-joined computers have a time client type of NT5DS, which means that they synchronize time from the domain hierarchy. An exception to this is the domain controller, which functions as the primary domain controller (PDC) emulator operations master for the root forest domain. The PDC emulator operations master in turn is usually configured to synchronize time with an external time source. You can set Active Directory GPO to configure time sync from time.windows.com and internally from the PDC emulator FSMO role but in essence this get overridden by the KVM guest tool which writes the KVM host time to the guest.
The Microsoft best practices for time synchronization in a Windows Server environment are provided in the following articles:
Notes for VDI environments
If you are running a VDI environment, for instance Microsoft RDS or Citrix Virtual Apps and Desktops, you should consider the following notes as well.
- https://discussions.citrix.com/topic/399684-xenserver-tools-timesync-disable/ (assuming Citrix XenServer as hypervisor)