In this article, I explain the steps involved in troubleshooting Citrix Gateway Service connection issues and how to fix them.

Fun fact: so far, the root cause was never Citrix Gateway Service itself 😉

Sometimes a connection issue prevents the successful launch of a resource (e.g., a published desktop or published application) from Citrix Workspace through Citrix Gateway Service. In this article, I explain the troubleshooting steps I go through to determine the root cause.

Table of Contents Setting the scene The troubleshooting steps: Check Citrix Cloud for Cloud Connector errors Communication between the Cloud Connector and the VDA Workspace app and ICA encryption TLS versions and cipher suites SSL inspection on the firewall TCP versus HDX Adaptive Transport (EDT) + MTU

Setting the scene

So what exactly is the issue? A user tries to connect to a resource (such as a virtual desktop or a published application) from Citrix Workspace through Citrix Gateway Service.

The connection is in progress as seen in the following screenshot.

The old GUI

The new GUI

After a while, the connection times out, and the following error is shown:

Please read this first! Citrix Gateway Service can be configured with and without the Rendezvous protocol. This article only covers issues when Rendezvous is not configured (which is the default setting in DaaS). The main difference between the two scenarios is the components in the HDX traffic flow: Without Rendezvous (default): User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA With Rendezvous (V1): User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <->  VDA Please see the following article on how to configure and troubleshoot Rendezvous (V1) for HDX traffic: https://docs.citrix.com/en-us/citrix-daas/hdx-transport/rendezvous-protocol/rendezvous-v1 And one more thing. By default, neither Direct Workload Connection (Network Location Service) nor HDX Direct is configured. This means that all users, whether they are connected to their company’s local network or not, establish a connection to their VDA through Citrix Gateway Service (with or without Rendezvous). Direct Workflow Connection and HDX Direct enable a direct connection to the VDA if possible (if the user’s endpoint is within the local company network or connected via a VPN). In case a direct connection is possible, the HDX traffic flow is as follows: User’s endpoint -> VDA In case Direct Workload Connection or the newer solution HDX Direct is configured and applied to the current user, the main issue will be local to the VDA. For troubleshooting purposes, see the sections Check the communication between the Cloud Connector and the VDA (to check for possible local firewall issues on the VDA) and Workspace app and ICA encryption in this article.

Check Citrix Cloud for Cloud Connector errors

Let us start our troubleshooting journey by looking at the connectivity status of the Cloud Connector in Citrix Cloud. From an HDX traffic flow perspective, we are looking at the components in green.

User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA

When you create a new Resource Location in Citrix Cloud, the connectivity, by default, is set to Gateway Service.

In Citrix Cloud, go to the menu item Resource Locations, select the respective resource location, and click on the item Cloud Connectors.

If the Cloud Connector is 100% up and running, the small color stripe next to the Cloud Connector will be green.

More details are provided if you click on the three dots on the right, run the health check (1) and afterward check the status of the individual services under View Connector Data (2).

 

 

In case of any errors, the color stripe on the left side of the Cloud Connector will be orange. You will also see an error message displayed. Go to View Connector Data to see more details.

The first thing to check is the service connection status of the Cloud Connector. This is either Connected, Unknown, or Disconnected. Unknown and disconnected means there is an issue.

Check the current version and target version of the Cloud Connector and the Cloud Connector components. Make sure these are both running the latest version. If not, the Cloud Connector has an issue with the automatic update initiated by Citrix Cloud. You most likely have a firewall issue and the outbound connection (port 443) cannot reach Citrix Cloud.

Make sure that at least the following addresses are available:

• https://*.citrixworkspacesapi.net (provides access to Citrix Cloud APIs that the services use)
• https://*.cloud.com (provides access to the Citrix Cloud sign-in interface)
• https://*.blob.core.windows.net (provides access to Azure Blob Storage, which stores updates for Citrix Cloud Connector)
• https://*.servicebus.windows.net (provides access to Azure Service Bus, which is used for logging and the Active Directory agent)
• https://*.nssvc.net (Citrix Gateway Service)
• https://*.xendesktop.net (Citrix DaaS service)

References:

• https://docs.citrix.com/en-us/citrix-cloud/overview/requirements/internet-connectivity-requirements.html#cloud-connector-common-service-connectivity-requirements
• Allowed FQDNs for Cloud Connector

Scroll down the list of services. These are the main Citrix services that run on the local Cloud Connector. Check the service Citrix NetScaler Cloud Gateway. The service may be down or may not be able to reach the Gateway Service in Citrix Cloud.

In case of any errors, the first thing to do is to run the Cloud Connector Connectivity Check Utility (https://support.citrix.com/article/CTX260337). This tool will show you if all required URLs can be reached and if the required root and intermediate certificates are installed correctly.

In practice, the errors that I see most are:

• Not all URLs are reachable (HTTP status is not always 200)
  • Solution 1: The URLs are not reachable because the proxy configuration for the local system has not been configured correctly. Make sure that the Default Proxy Config (= local system proxy) is properly configured.
    
  • Solution 2: Allow the Citrix Cloud URLs on the firewall (not the local Windows firewall; the real firewall 😉 ) and/or proxy server as described in the official Citrix documentation System and Connectivity Requirements (for Cloud Connector).
  • Sometimes, errors can (potentially) be ignored. For example, if the URL https://*.sharefile.com is blocked this will not negatively influence the functionality of the Cloud Connector concerning HDX traffic.
• The intermediate certificate is missing.
  This can happen on occasion. You can download the certificate here:
  https://dl.cacerts.digicert.com/DigiCertSHA2AssuredIDCodeSigningCA.crt.
  The above link is also provided in the official Citrix support article: https://support.citrix.com/article/CTX260337. After downloading the certificate, add it to the local computer Intermediate Certification Authorities certificate store on the Citrix Cloud Connector.

• Missing check in the Cloud Connector Connectivity Check Utility
  The tool Cloud Connector Connectivity Check Utility does not show any error, but in Citrix Cloud, you may see the error unable to connect to https://reg.c.nssvc:443/Control/Ping. Unfortunately, this particular URL is not checked by the tool. Allow this URL on the firewall to solve the issue.

 

 

Check the communication between the Cloud Connector and the VDA

In case the previous section did not solve your issue, we move on to the next step, which is to check the communication between the Cloud Connector and the VDA. This concerns the following section of the HDX traffic flow:

User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA

The Cloud Connector should be able to reach the following ports on the VDA:

• 80 TCP
• 1494 TCP/UDP
• 2598 TCP/UDP

The VDA should be able to reach the following port on the Cloud Connector:

• 80 TCP

You can easily test the connectivity using a Telnet or PowerShell command.

Reference: Tech Paper: Communication Ports Used by Citrix Technologies

In case one of the abovementioned ports cannot be reached check (and fix) the following:

1. Check the local Windows firewall on the Cloud Connector or VDA. Make sure the required ports are open.
2. Check the VDA installation routine: the local Windows firewall can be automatically configured during VDA installation (recommended for most environments).
3. Check the Windows firewall configuration in your Group Policies. You may need to add the required addresses and ports.
4. Very tricky (I once wasted two hours on this!): Make sure that the Group Policy setting Apply local firewall rules is not set to No. This would prevent any local Windows firewall rules, including the ones configured by the VDA installation, from being applied!
   
5. Check the Group Policy setting Access this computer from the network. By default, access is granted to Everyone. In case this setting has been modified, you may have to add an Active Directory security group that contains the Cloud Connectors and/or VDAs or you have to add these machines to the Group Policy individually.
   

   Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment

   I added this Group Policy configuration for the sake of completeness. In most cases, it will not be the cause of the HDX traffic issue. The reason for this is simple: in case this policy is blocking the Cloud Connector from communicating with the VDA and visa versa, the VDA would not have been able to properly register with the Cloud Connector in the first place.

6. The last thing to check is if there is a firewall in place between the subnet of the Cloud Connector and the VDA. Make sure that the abovementioned ports are configured accordingly. Just to be clear: I am not talking about the local Windows firewall on either the Cloud Connector or the VDA. I am talking about an actual firewall appliance (physical or virtual).

 

 

Workspace app and ICA encryption

In case of any HDX connection error, make sure to test with a full version of the Citrix Workspace app, such as the Workspace app for Windows or Mac.

User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA

Do not perform your tests with the HTML5 Workspace app for example. One of the reasons is that the HTML5 Workspace app does not support SecureICA (in case this is configured).

This brings us to another setting you may want to check: the ICA (HDX) encryption level configured in your Citrix DaaS environment. The two options are Basic encryption (which is the default) and SecureICA, which can be configured with different RC5 encryption levels. The ICA encryption level can be configured in a Citrix policy. For troubleshooting purposes, make sure to use Basic encryption. Once the HDX connection issue has been solved you can again activate SecureICA.

TLS versions and cipher suites

In the HDX traffic flow, certain TLS versions and cipher suites are used for communication with Citrix Cloud. These TLS versions and cipher suites need to be available/installed on the Cloud Connector.

User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA

Citrix uses industry-standard TLS 1.2 with the strongest cipher suites for data-in-flight. The current TLS version used is 1.2. Make sure that at least TLS 1.2 is active on your Citrix Cloud Connector. Most, if not all, Citrix components now also support TLS 1.3, so it may be wise to also have this version activated on your Cloud Connectors.

You can use PowerShell to check the list of available TLS versions. Run this command on your Citrix Cloud Connector:

The result should be something like this:

TLS 1.2 is enabled by default on Windows Server 2016/W10 operating systems. For Windows Server 2012, additional configurations may be required (https://learn.microsoft.com/en-us/mem/configmgr/core/plan-design/security/enable-tls-1-2-client).

TLS 1.3 is enabled by default on Windows Server 2016/W10 operating systems as well if the latest Windows updates have been installed (https://www.microsoft.com/en-us/security/blog/2020/08/20/taking-transport-layer-security-tls-to-the-next-level-with-tls-1-3/).

Certain cipher suites are required for correct communication with Citrix Cloud. For Windows Server 2016 to 2022, make sure that the following cipher suites are installed in this (priority) order on your Citrix Cloud Connector:

• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
• TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

In case you are still using Windows Server 2012 in your environment I highly recommend you to upgrade. If this is not possible for whatever reason, make sure that the following cipher suites are available in this (priority) order on your Cloud Connectors:

• TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
• TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

You can use PowerShell to check the list of installed cipher suites. Run this command on your Citrix Cloud Connector:

This should be the approximate result (the list may look different depending on the operating system).

Configuring the cipher suites (and order) should be done using a Group Policy (https://learn.microsoft.com/en-us/windows-server/security/tls/manage-tls). You have to restart the server for the new order to take effect.

References:

• Secure Deployment Guide for the Citrix Cloud Platform (section Encryption and key management)
• Citrix Cloud: Changes to Cipher suites for Cloud Connector

This article does not go into HDX connections established with the Rendezvous protocol enabled. However, I do want to point out that with Rendezvous enabled, the required TLS version(s) and cipher suites must be configured on the VDA (since there is no Cloud Connector in play). For more information, see the article Rendezvous V1 on docs.citrix.com.

 

 

SSL inspection on the firewall

SSL inspection (or more precisely TLS inspection or interception) should not be enabled on your firewall for Citrix Gateway Service. We are now analyzing this section of the HDX traffic flow (in green):

User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA

You can verify if SSL inspection is active by opening the Connection Center of your local Workspace app. Click on the Properties button of the active session to view the certificate.

Now, when SSL inspection is not active then you should see the root certificate DigiCert and the intermediate certificate DigiCert TLS RSA SHA256 […]. The URL *.nssvc.net belongs to the Citrix Gateway Service. This is the correct configuration.

When SSL inspection is active, the original certificates are replaced with a custom certificate on the firewall, like in the screenshot below.

As stated in the Citrix Docs, SSL inspection should be disabled (or excluded) for Citrix Gateway Services.

Reference: https://docs.citrix.com/en-us/citrix-cloud/overview/requirements/internet-connectivity-requirements.html#citrix-gateway-service-connectivity

The following URL(s) should be excluded from SSL inspection:

• https://*.*.nssvc.net

In case your organization does not allow enabling all subdomains you can use the following addresses instead:

• https://*.g.nssvc.net
• https://*.c.nssvc.net

 

 

TCP versus HDX Adaptive Transport (EDT) + MTU

For the sake of completeness, I want to mention HDX Adaptive Transport or EDT (the Citrix adaptation of the UDP protocol) versus TCP. Although theoretically, this can be the cause of an HDX connection problem, it cannot be in this particular case.

Why? Because in combination with Citrix Gateway Service, EDT only functions when Rendezvous is enabled. The reason for this is that the Citrix Cloud Connector does not support connections to the VDA with network-level encryption (reference: https://docs.citrix.com/en-us/citrix-daas/secure.html#vda-tlsdtls).

User’s endpoint <-> Citrix Gateway Service (Citrix Cloud) <-> firewall/proxy <-> Cloud Connector <-> firewall (optional) <-> VDA

I do however want to list these very important considerations when Rendezvous is not in use and all HDX communication goes through the Citrix Cloud Connector:

• EDT with Citrix Gateway Service is only available when using Rendezvous. If HDX sessions are being proxied through the Cloud Connector, only TCP is available for data transport.
• When an EDT session establishment fails the session falls back to TCP, causing an increase in the session launch time.
• If you want to continue to proxy HDX sessions through the Cloud Connector, consider disabling Adaptive Transport via the Citrix Studio policy to avoid the potential increase in session launch times introduced by the fallback sequence.

So if you are not planning on using Rendezvous, make sure to disable HDX Adaptive Transport for all your VDAs!

Reference: https://docs.citrix.com/en-us/citrix-gateway-service/hdx-edt-support-for-gateway-service.html#considerations

Another configuration that can potentially cause HDX connection issues is the MTU size. This mainly affects UDP-based connections (= Citrix HDX Adaptive Transport / EDT). As mentioned, UDP is not supported by the Citrix Cloud Connector (only TCP).

The reason why the MTU size mainly affects UDP and not TCP is because of several reasons:

• TCP supports segmentation and reassembling of packages (UDP does not).
• No segmentation of packages means that only one package is sent through the network which in most, if not all, cases results in fragmentation by a network component such as a router.
• Package fragmentation can result in application issues and decreased performance (= negative user experience).

References:

• https://www.zenarmor.com/docs/network-basics/tcp-vs-udp
• https://www.linkedin.com/advice/1/what-benefits-drawbacks-udp-fragmentation-low-bandwidth

You have reached the end of this article. I hope you were able to solve your HDX connection problem. If you have any questions or if you found other root causes that should be documented in this article, please leave a comment below.