How to troubleshoot Citrix Virtual Apps and Desktops random freezing incidents

Case #

You experience random freezing incidents in your Citrix Virtual Apps and Desktops (CVAD) environment. You need to identify is these events are caused by a client-side or server-side root cause. You need to perform Root Cause Analysis (RCA) of the freezing incidents. This KB article provides guidance on how to troubleshoot Citrix Virtual Apps and Desktops random freezing incidents.

Solution #

If you need to know how to troubleshoot Citrix Virtual Apps and Desktops random freezing incidents, You should carry out the following troubleshooting steps in your CVAD environment:

1. Perform full technical assessment of the Citrix environment (Citrix server and client components deployed).
2. Perform a configuration review of the environment and validate that there are no environment-wide misconfigurations. More specifically:
   1. Check the Windows and Citrix infrastructure patching levels. Ensure that all Windows Server updates and Citrix KB patches for CVAD have been deployed server-side. There are various potential VDA software bugs which are resolved by installing corresponding fixes provided in Citrix KB patches. You should search the Citrix KB for these patches depending on your version of Windows Server.
   2. Check the core and distribution networking infrastucture for performance issues
   3. Check the storage infrastructure for IOPS and performance issues
   4. Check the Citrix EDT and DTLS protocol configuration and the MTU/MSS configuration of the various network segments of your infrastructure.
   5. Check the Citrix profiles (UPM) or other third party profile solution (eg. Microsoft FsLogix). There are Citrix profile configuration aspects, such as Active Writeback, which as known to cause performance issues in CVAD environments.
   6. Check TLS certificates and ensure that the certification configuration does not create any certificate warnings.
   7. Check the licensing of all Citrix components to ensure that there is no freezing due to mislicensed components. There are Citrix Provisioning Services (PVS) known cases where misconfigured licensing of the PVS servers caused intermittent freezing issues.
3. Isolate the potential root causes of the freezing incidents. Collect environment conxtextual information and by asking the right questions try to minimize the potential contributing factors. Proper questions to ask in this direction are:
   1. When does the freezing issue occur? Is there any periodic behavior or is it absolutely random?
   2. Which network segments/zones does the issue occur in?
   3. Which users and user groups does the issue affect?
   4. Which Citrix VDA servers does this issue occur in?
   5. Does the CVAD environment have any other third party components which can contribute to this issue?
4. Check the release notes, bug fixes and known issues in the CVAD VDA version and Citrix Workspace App client versions you are running in your environment. Maybe the issue you are encountering in your specific configuration is already a known issue which has been resolved in a certain CVAD VDA or Workspace App client version.
5. Check the configuration of Citrix Session Reliability at the Citrix server side, as per https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/manage-deployment/sessions.html.
6. Activate the Citrix Connection Quality Indicator (CQI) tool. This should help you expedite the Citrix troubleshooting process in relation to network performance metrics between the Citrix client and server. The Citrix CQI tool monitors and logs Citrix performance metrics with focus on network-related metrics such as ICA session Round-Trip Time (RTT) and ICA session latency in ms. More details about deploying this tool and collecting its performance logs can be found at https://support.citrix.com/article/CTX220774.
7. Run the Remote Display Analyzer portable tool inside a Citrix ICA session (virtual desktop) and check the ICA RTT and ICA latency metrics as they evolve over time.
8. Find a way to reproduce the freezing issue and collect verbose logs for further analysis. The following logs should be collected at minimum:
   1. Citrix Scout logs from the Citrix Delivery Controllers.
   2. Citrix PVS logs, if Citrix PVS is involved in the freezing issue.
   3. Citrix VDA logs as part of the Citrix VDA application Windows event logs.
   4. Citrix Workspace App client logs. In the case of Windows clients, you should run the Citrix Receiver Diagnostics tool, or, for newer versions of Citrix Workspace App for Windows, follow the procedure at https://support.citrix.com/article/CTX286820. In the case of Linux clients, you should consult my post on Citrix Workspace App for Linux log collection.

1. Wireshark network traces taken before the freezing event occurs on the Citrix VDA server. You should filter your captured traffic to isolate traffic only related to the VDA server, the Citrix client and possible the Citrix PVS server.
   1. Network traces taken from the Citrix client side, using Wireshark or other similar tool.
   2. Process Monitor trace taken on the VDA server side and ensure that only the Citrix-related processes, (Citrix Windows Services and other .exe files), and any other third party application process are included.
   3. Application logs and traces from any other third party application which is correlated to Citrix server-side or client-side processes. The freezes for example could be due to a software bug of the third application which manifests only under certain infrastructure or application conditions.

For more troubleshooting guidance on a series of known Citrix issues, you can consult my “Citrix Virtual Apps and Desktops Troubleshooting” e-book.