When placing a Hyper-V cluster node in drain mode and then bringing the node back online (re-join), you may come across the following warning message: Cluster Shared Volumes root directory ‘C:\ClusterStorage’ already exists. The directory ‘C:\ClusterStorage’ was renamed to ‘C:\ClusterStorage.000’. Please verify that applications accessing data in this location have been updated as necessary.
At the same time you observe that a temporary new folder C:\ClusterStorage.000 has been created as shown below.
You observe that the ClusterStorage.000 folder has been created alongside pre-existing “ClusterStorage” folder.
The above may occur on more than one Hyper-V hosts and may create outdated copies of more than one subfolders underneath the Cluster Shared Volume (CSV) volumes (cluster disks). For example for VM named VM001, you may have two copies of the “Virtual Machines” folder, one in C:\ClusterStorage and one in C:\ClusterStorage.000. You observe that the file copies under C:\ClusterStorage.000 are outdated and that the synchronized copy exists under C:\ClusterStorage.
Root cause analysis #
The most probable cause of the issue is the Windows Failover Cluster (WFC) service. During the WFC service start, the service will try to create/delete or find a CSV$ share in the C:\ClusterStorage folder. By design, if the WFC service fails to complete this operation and is unable to place a handle on the CSV volume, a new folder will be created, such as C:\ClusterStorage.000, C:\ClusterStorage.001, etc. Failure to read the CSV$ share can occur for a number of reasons, with most common reason being that the WFC did not have permissions to access the C:\ClusterStorage folder at start time. Other third party applicaitions, such as Antimalware and/or backup agents may place handles on the C:\ClusterStorage folder and this could be the reason why the WFC service is not able to access it. Another common reason is that there is temporary network disruption during WFC service startup and therefore the service momentarily cannot access the CSV$ share.
Action items #
First of all, place the Hyper-V host in question in drain mode and then reboot the host. Afterwards, place the server back to online mode in the WFC cluster and verify operations of the server and all associated VMs. At this point, refresh also the WFC logs to ensure there is no other error/warning. If all operations are running without issues, then it is ok to proceed with below steps. If there are other cluster errors, investigate and resolve these errors first, before moving on with below steps.
Take ownership of the C:\ClusterStorage.000 folder and all its subfolders recursively. To accomplish this, open a command line prompt and enter following commands.
takeown /F "C:\ClusterStorage.000" /R icacls "C:\ClusterStorage.000" /grant <account>:F /T
After having taken full control of the C:\ClusterStorage.000 folder and all of its subfolders, it is now safe to delete this folder and all subfolders.
Hyper-V cluster troubleshooting #
For more details on troubleshooting Hyper-V cluster issues, refer to the following article: https://stefanos.cloud/kb/how-to-troubleshoot-hyper-v-cluster-issues/