How to resolve error "Group Policy Client service failed the logon. Access denied." in Citrix and FsLogix environments

Case #

You have a Citrix Virtual Apps and Desktops 1912 LTSR CU1 or earlier environment with FsLogix profiles in place on top of Windows Server 2019 session machines. After installing the latest FsLogix version 2.9.7576.47294 you have been getting the following error occasionally from end users when trying to login to their Citrix/RDS session: "Group Policy Client service failed the logon. Access denied." All your Windows Servers are running Windows Server 2019. This KB article provides guidance on how to resolve error "Group Policy Client service failed the logon. Access denied." in Citrix and FsLogix environments.

The issue can occur in any environment running FsLogix as the profile management mechanism (Citrix or Microsoft RDS).

The case is also described in detail in the following Microsoft FsLogix community forum thread: https://docs.microsoft.com/en-us/answers/questions/229309/fslogix-unclean-logoff-causing-locked-files-until.html

Solution #

Follow the steps below:

1. Check the FsLogix profile containers and Office 365 containers SMB storage path permissions as per Microsoft instructions at: https://docs.microsoft.com/en-us/fslogix/fslogix-storage-config-ht.
2. Double check that all antivirus exclusions are set up properly, as per Microsoft recommendations: https://docs.microsoft.com/en-us/azure/architecture/example-scenario/wvd/windows-virtual-desktop-fslogix (antivirus exclusions section).
3. Check the NTUser.dat registry hive permissions.
   1. Open regedit.
   2. Select HKEY_USERS in the left pane. Click the File menu and choose the Load Hive option.
   3. Select All Files in the Files of type box. Navigate to the affected user's profile folder (under C:\Users folder) and select the NTUSER.DAT file. Click Open.
   4. Provide any name for the new hive, e.g. TestHive.
   5. Set proper registry hive permissions under the TestHive section. The owner user as well as the System account at minimum should have full control enabled.
   6. In regedit, select File --> Unload hive.
   7. Logoff the user gracefully and try again.
4. Carry out a graceful FsLogix profile logoff of the Citrix VAD session, ensuring that there is no local profile left in C:\users folder of the VDA server and try logging on again.
5. Experiment with changing a few registry key values affecting the FsLogix profile operations dependign on your environment-wide configuration, as per: https://docs.microsoft.com/en-us/fslogix/profile-container-configuration-reference.
6. Take procmon and etl traces (with logman commands) in a working and non-working scenario, since this is occurring to some of the users. In procmon traces, check the CloseFile events by the FsLogix service (run with NT Authority\SYSTEM credentials) for any access denied events.
7. Stop and disable the "Connected User Experiences and Telemetry" Windows service, as this has been seen in causing issues with profile release in Microsoft RDS/UPD environments.
8. If all else fails, you will need to delete the FsLogix profile container and office365 container folders from the SMB path for the affected user and instruct the user to logon again with a clean profile. However, this requires that the local_ folder under C:\Users has been released properly.

Update (Jan 2021):

It seems that the root cause of this issue in many cases is the fact that the FsLogix profile service or another system service cannot properly release the "local_username" folder under C:\Users, which is the local profile of the user. After running ProcMon and Handle64 traces, it appears that the two folders holding the local_ folder to be released are the following:

AppData\Local\Microsoft\Credentials
Appdata\Roaming\Microsoft\Credentials

In the case of Citrix Profile Management (CPM), the best practice is to exclude the following paths from the CPM mechanism:

AppData\Local\Microsoft\Windows\Caches
AppData\Local\Microsoft\Credentials
Appdata\Roaming\Microsoft\Credentials
Appdata\Roaming\Microsoft\Crypto
Appdata\Roaming\Microsoft\Protect
Appdata\Roaming\Microsoft\SystemCertificates

Citrix CPM is a file sync based profile mechanism, while FsLogix is a profile container based mechanism. So the exclusion process in CPM means that a certain path is retained in the CPM profile copy and not synced to the Session Machine which holds the local copy. The equivalent of this in FsLogix terms is including the respective path inside the Redirections.xml file. So a valid test to perform to resolve the original gpclient error is to include the following paths in the Redirections.xml file of FsLogix:

AppData\Local\Microsoft\Windows\Caches
AppData\Local\Microsoft\Credentials
Appdata\Roaming\Microsoft\Credentials
Appdata\Roaming\Microsoft\Crypto
Appdata\Roaming\Microsoft\Protect
Appdata\Roaming\Microsoft\SystemCertificates

This will force the above two paths marked in bold to be retained inside the FsLogix vhdx container and not be created inside the local_username folder.

Further testing needs to be performed on the above to confirm if it is a working solution or not. This article will be updated as soon as all tests are completed.

Update (8 Feb 2021)

I ran the redirections.xml test as per the previous comment in this post (see above). I added two rows to the inclusions section of the file, as per instructions at https://social.msdn.microsoft.com/Forums/windows/en-US/029e130e-5892-4d1f-88a7-f8046d78f3b0/using-redirectionsxml-to-configure-what-to-copy-to-a-profile-with-fslogix?forum=FSLogix.

The appdata\local\microsoft\credentials folder was created both in the profile container and in the local_username folder. The same error occurred again. So I have reverted the redirections.xml file to the original state.

This issue has been escalated to Microsoft Technical support and now waiting on action plan from Microsoft.

Update (15 Feb 2021)

This case is under investigation by the FsLogix Technical Support backend team. I am waiting on feedback from them on new action plan. It is highly possible that this will be recognized as a defect/bug of FsLogix. I will update this article once I receive an official answer from Microsoft Support.

Update (24 Feb 2021)

After speaking again with Microsoft Technical Support, I was informed that this issue could be a bug, as stated by the FsLogix product group. They are further investigating the case and hopefully there will be a conclusive answer to identify this as bug and wait on the fix in an upcoming FsLogix release. I will update this post notes as soon as I receive the next update from Microsoft technical support.

Update (11 Mar 2021)

A lot of Microsoft customers have been affected by this issue and a lot of tickets have been opened on the same. My ticket has been open since Jan 29th as a critical ticket. I have performed very thorough testing, took verbose traces and carried out all actions requested by MS FsLogix support team, to no avail. For some reason they are very slow in responding. The support team is having discussions with the FsLogix product group team but they have yet to provide a clear and conclusive update on how the issue will be fixed. All evidence so far points to a potential software bug but I have not yet received official confirmation on this from the technical support side, despite my numerous emails to the escalation team. Hopefully there will be action from the Microsoft side soon, since this is affecting a growing number of customers and production environments worldwide.

Update (19 Mar 2021)

I received the following update from FsLogix support:

We have discussed the case with our internal team today and came to know that the issue is still under discussion with Product Group (PG) and it seems like a Bug/known issue however PG has not concluded yet. We have seen a few more cases with the same behavior, we would request you to please allow us some time to investigate the issue further and will keep you posted for the same.

Update (31 Mar 2021)

Microsoft Support requested the following further troubleshooting actions:

1 - Disable Roam Search and test the behavior and if the issue I resolved then we need to setup a task scheduler at the time of logoff - Reference article - Windows Search in Server 2019 and Multi-Session Windows 10 – James Kindon (jkindon.com)   - The Windows Search service is always disabled and stopped and had been set to disabled in my environment since the very beginning of testing, as I was aware of the Search Roaming issue they mentioned since the very beginning.

2 - monitor the computer management window to validate how many VHDX were remains attached even after user logged Off – There is no issue with vhdx files, only with the local_ folder, all FsLogix container vhdx files and Office365 container vhdx files are detached in general after user logoff.

3 - In File Server - Check user Open files hooks and at the same time, in Process explore - try to find out the hooks on that VHDX and if we found any 3rd party hook, disable or uninstall that app – There are no hooks to check because profile/office container vhdx files in general get detached at logoff.

Update (9 April 2021)

Since this case is dragging too long, I have temporarily notified end users to not logoff but only disconnect from their Citrix sessions. I have also implemented a weekly maintenance reboot script for the Citrix VDA servers. Microsoft support have requested that I provide them with a full memory crash dump by utilizing SysInternals NotMyFault tool (https://docs.microsoft.com/en-us/sysinternals/downloads/notmyfault) right after the FsLogix local_ folder issue happens.

Update (11 May 2021)

Upon requesting a status update, the Microsoft Technical Support team informed me that the issue is still under investigation by the Backend Product Group and that they will get back to me when they have an update.

Update (20 May 2021)

Today I received the following reply from the FsLogix technical support team by email:

Hope you are doing well. I had discussed the case with my internal team and as of now there is no ETA from PG regarding Fslogix profile issue: FsLogix - Unclean logoff causing locked files until server reboot - Microsoft Q&A. We are tracking the progress on this and will update you once I heard anything from PG. Meanwhile, I will go ahead and close this case with refund. There is an internal case open with PG however there is no ETA for resolution. We will keep you posted if I heard an update from them.

Update (July 2021)

Please review my recent blog post which re-publishes Microsoft announcement of a new GA release of FsLogix which may be resolving the issue discussed in this thread. I would strongly suggest that you first install the new GA release in a lab environment and perform sufficient testing before applying to production.

Update (January 2022)

The issue still occurs as of Jan 2022. Microsoft recently released the FsLogix 2201 Public Preview release this month. It is strongly suggested to install and test the new Public Preview release in a lab environment. I am still waiting on an official response from Microsoft Support and FsLogix product group on resolution ETA of this bug.

Update (February 2022)

Microsoft has yet to provide a conclusive answer and resolution to this issue, which seems to persist even in FsLogix 2201 Public Preview release. As per this release notes (https://techcommunity.microsoft.com/t5/azure-virtual-desktop/the-fslogix-2201-public-preview-is-now-available/m-p/3070794/page/2), user PdfPeet has suggested the following workaround for releasing FsLogix file handles without a server reboot, until a permanent fix is provided by Microsoft.

---

For now I've scheduled a simple script that runs every 30 minutes during the nightly hours to:

• Check if there are any users still logged on through RDP
• If not, check if there are any local_ of profile folders still there that shouldn't. If so:
  • Run an "frx.exe list-redirects" and cleanup where needed (doesn't seem to be needed anymore since the 2201 release)
  • Run an "frx.exe stop-agent" & "frx.exe start-agent" (this removes the locks from the profile folders; without having to reboot the server)
  • Do additional cleaning
    • Registry: ProfileList and ProfileServices\References
    • Profile folder: take ownership and remove "local_<usersname>" and "<username>" folders

It's not ideal, but it keeps servers clean.

Update (March 2022)

FsLogix 2201 release is now generally available. You can find details of the new FsLogix version and bug fixes at .

Update (October 2022)

As per https://learn.microsoft.com/en-us/answers/questions/229309/fslogix-unclean-logoff-causing-locked-files-until.html?childToView=1051021#answer-1051021, the issue is unfortunately still not fixed by the FsLogix team. In a new Citrix install, 2203 LTSR on W2019 with latest Fslogix version (FSLogix 2201 hotfix 2 (2.9.8228.50276), the "credentials" folders are still not deleted on logoff.

Update (November 2023)

Unfortunately issue is not resolved in latest version of FsLogix and Citrix. I am providing two workarounds which were published by other community members which may give you a resolution.

Workaround 1

Exclude the c:\windows\system32\lsass.exe process from Microsoft Defender (via proper GPO). This is the process keeping the credentials folder locked. This should be part of a broader antivirus exclusions GPO configuration for both Windows OS and Citrix or other third party applications which may be running on your VDI servers, as discussed in the beginning of the "Solution" section of this KB article.

Workaround 2

On each RDS session host server or Citrix VDA server, create a folder C:\Scripts and put the following Powershell script into that folder.

 $Event = Get-WinEvent -MaxEvents 1 -FilterHashtable @{logname='Microsoft-FSLogix-Apps/Operational';ID='31'} $TimeCreated = $Event.TimeCreated $Event = [xml]$Event.ToXml() $User = $Event.Event.EventData.Data | Where-Object {$.name -eq "Username"}
$SID = $Event.Event.EventData.Data | Where-Object {$.name -eq "SID"} $RefCount = (Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService\References\$($SID."#text")").RefCount[0] if( $RefCount -ne 0 )
{
"$TimeCreated, $($User.""#text""), $($SID."#text"), $($env:COMPUTERNAME)" | out-file \resoleasing.com\NETLOGON\RDS\FSLogix\FSlogix.txt -Append
Set-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService\References\$($SID."#text")" -Name RefCount -Value (byte\[\])
}

Create a Windows Scheduled Task to run with highest privileges. Trigger: This event occurs when a profile is unloaded. Program: powershell.exe Arguments: -NoProfile -NoLogo -NonInteractive -ExecutionPolicy Bypass -File C:\Scripts\FixLockoutProfile.ps1

Finally, bear in mind that your file server / cluster on which the FsLogix profiles are hosted needs to have optimal networking and storage IOPS performance configuration. This could also contribute to occassional timeout incidents which could further result in FsLogix operation issues.