Case #
When placing a Hyper-V cluster node in drain mode and then bringing the node back online (re-join), you may come across the following warning message: Cluster Shared Volumes root directory 'C:\ClusterStorage' already exists. The directory 'C:\ClusterStorage' was renamed to 'C:\ClusterStorage.000'. Please verify that applications accessing data in this location have been updated as necessary.
At the same time you observe that a temporary new folder C:\ClusterStorage.000 has been created as shown below.
You observe that the ClusterStorage.000 folder has been created alongside pre-existing "ClusterStorage" folder.
The above may occur on more than one Hyper-V hosts and may create outdated copies of more than one subfolders underneath the Cluster Shared Volume (CSV) volumes (cluster disks). For example for VM named VM001, you may have two copies of the "Virtual Machines" folder, one in C:\ClusterStorage and one in C:\ClusterStorage.000. You observe that the file copies under C:\ClusterStorage.000 are outdated and that the synchronized copy exists under C:\ClusterStorage.
As a side effect of the above, some or all of the cluster shared volumes under the "ClusterStorage" folder may not be properly mounted. This in turn results in Hyper-V cluster live migration failing from any Hyper-V host to the Hyper-V host(s) which have incorrect "ClusterStorage" folder copies, as explained above.
Solution #
Root cause analysis #
The most probable cause of the issue is the Windows Failover Cluster (WFC) service. During the WFC service start, the service will try to create/delete or find a CSV$ share in the C:\ClusterStorage folder. By design, if the WFC service fails to complete this operation and is unable to place a handle on the CSV volume, a new folder will be created, such as C:\ClusterStorage.000, C:\ClusterStorage.001, etc. Failure to read the CSV$ share can occur for a number of reasons, with most common reason being that the WFC did not have permissions to access the C:\ClusterStorage folder at start time. Other third party applicaitions, such as Antimalware and/or backup agents may place handles on the C:\ClusterStorage folder and this could be the reason why the WFC service is not able to access it. Another common reason is that there is temporary network disruption during WFC service startup and therefore the service momentarily cannot access the CSV$ share.
A useful analysis of how Cluster Shared Volumes (CSV) respond to application or infrastructure (e.g. network) outages is provided by Microsoft at https://techcommunity.microsoft.com/t5/failover-clustering/troubleshooting-cluster-shared-volume-auto-pauses-8211-event/ba-p/371994.
Action items #
First of all, place the Hyper-V host in question in drain mode and then reboot the host. Afterwards, place the server back to online mode in the WFC cluster and verify operations of the server and all associated VMs. At this point, refresh also the WFC logs to ensure there is no other error/warning. If all operations are running without issues, then it is ok to proceed with below steps. If there are other cluster errors, investigate and resolve these errors first, before moving on with below steps.
Ensure that the HyperV physical nodes in question are firstly drained from the HyperV cluster. Then boot the affected HyperV physical node(s) in question in safe mode. You can find instructions on how to boot your server in safe mode in the following article: https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234#WindowsVersion=Windows_11.
All the below actions must be carried out while the affected Hyper-V nodes are booted in safe mode (without networking). This is to ensure that no sync is performed while you make the changes, thus resulting in a faulty/corrupt "ClusterStorage" folder state.
Take ownership of the C:\ClusterStorage.000 folder and all its subfolders recursively. To accomplish this, open a command line prompt and enter following commands.
takeown /F "C:\ClusterStorage.000" /R
icacls "C:\ClusterStorage.000" /grant <account>:F /T
After having taken full control of the C:\ClusterStorage.000 folder and all of its subfolders, it is now safe to delete this folder and all subfolders. Also delete the original "ClusterStorage" folder. This will be synced once you reboot the server.
Now reboot the affected Hyper-V server(s) but this time in full service mode (disable safe mode). Upon next normal boot, you should get a health synced copy of the "ClusterStorage" folder under the C:\ drive.
Hyper-V cluster troubleshooting #
For more details on troubleshooting Hyper-V cluster issues, refer to the following article: https://stefanos.cloud/kb/how-to-troubleshoot-hyper-v-cluster-issues/
