Notes from the Field: Citrix Provisioning Server 2402 CU1 LTSR Upgrade

by ⋅ Leave a Comment

While upgrading our Citrix Provisioning Server (2203 LTSR) to the latest LTSR version 2402 CU1, we ran into some difficulties that we didn’t encounter with earlier versions. In addition to issues with the Citrix Provisioning Server Console, we also had some struggles getting both Citrix Studio and Citrix PVS Console to run from a single admin server. In this blog, I’ll share our findings.

Until now, our default upgrade procedure was to upgrade all Citrix Provisioning Server Consoles, test the console, and then upgrade the Citrix Provisioning Server itself. However, we quickly noticed that the new PVS Console (based on LTSR 2402 CU1) was unable to successfully connect to a Provisioning Server running LTSR 2203.

When connecting a 2402 PVS Console to a 2203 PVS Server, the following error messages appear:
FX:{d4362438-2b4a-4f95-91b1-4f813d7b18e7} Type is not resolver for member ‘EnterpriseAccess.EAException

For completeness, an older 2203 PVS Console was able to connect to a 2402 PVS Server without any error messages. The product documentation was recently updated with this information.

Upgrade the Citrix Provisioning console and server on every Citrix Provisioning server. After all servers are upgraded, upgrade the console on any other systems it is installed on. The console is not backward compatible.

Next, we upgraded the first Citrix Provisioning Server to 2024 CU1 LTSR. Although the upgrade went flawlessly and the configuration wizard succeeded without any errors, the upgraded server appeared as “down” in the PVS Console. The first error message that appeared in Event Viewer was Event ID 268: “StreamProcess – Cannot establish a connection to the database because the server cannot be found.” It seemed to be something related to the database. The issue was caused by the ‘Microsoft OLE DB Driver 19.x‘ driver, which, due to stricter security measures, requires certificate validation on the SQL Server. In our case, the Microsoft SQL Server was configured with the ‘force encryption’ option enabled, but did not provide a certificate. Regarding encryption, PVS DOES NOT use the default. It specifies Encrypt=Optional when it connects to the database.  This means the connection is encrypted if the SQL Server has Force Encryption=Yes, and otherwise it is unencrypted.  This is identical to the behavior with earlier versions.

CTX69229 : PVS Server Down In Console After Upgrade to 2402 CU1

Because Citrix Provisioning 2308 and later only supports Microsoft OLE DB Driver 19.3 or newer, downgrading to OLE DB 18.x (which doesn’t require a certificate) was not an option. As a result, we had to involve the database team and ask them to modify our SQL instance. After this change, we were able to successfully start the upgraded Citrix Provisioning Server.

The next issue we encountered was the compatibility with Citrix Studio (2402). As mentioned earlier, we like to have all Citrix consoles installed on our admin servers. Unfortunately, Citrix Studio was completely broken after installing the Citrix Provisioning Server console. We tried several installation sequences, but in the end, the only working sequence was as follows:

  1. Uninstall Citrix Studio
  2. Install Citrix Provisioning Server Console
  3. Uninstall Citrix DaaS Remote PowerShell SDK
  4. Reinstall Citrix Studio

After this sequence, we finally had both Citrix Studio and the Citrix Provisioning Server Console working side by side on the admin servers.

Uninstall Citrix Studio before installing the Citrix Provisioning Server Console.

Depending on your environment, the third step might not apply to your setup. However, in our case, we still use Ivanti Workspace Control to publish Citrix applications (on-prem, non-cloud). Unfortunately, this does not work well when the Citrix DaaS Remote PowerShell SDK is installed. One thing we found very frustrating is that the ‘Citrix DaaS Remote PowerShell SDK’ is installed as a requirement by the ‘Citrix Provisioning Server Console.’ There is no option to skip this component, even if another SDK is already in place. We discussed this issue with Citrix Support, and hopefully, they will make adjustments in the future. For now, manually uninstalling the Citrix DaaS Remote PowerShell SDK was our only option. šŸ˜ž

Ivanti Workspace Control application publishing (on-prem) is not compatible with the “Citrix DaaS Remote PowerShell SDK”

That’s it for now! Hopefully, we’ve saved you some frustrations while upgrading your Citrix Provisioning Servers. šŸ˜‰ Finally, I would like to thank my colleague Hugo Koop for his research efforts and insightful discussions with Citrix Support.

Citrix StoreFront Error “Cannot complete your request”

by ⋅ Leave a Comment

While replacing Citrix StoreFront (2203 CU5 LTSR) servers running Microsoft Server 2019 with servers running Server 2022, we encountered an error message during login attempts to Citrix StoreFront. Users were shown an error message stating: “Cannot complete your request.”

We were running a GSLB setup with a multi-server group (across sites), each containing several Citrix StoreFront servers (2203 CU5 LTSR). As described in “Virtual Apps and Desktops – 1912/2203 – Citrix Infrastructure / OS Upgrade” (CTX278869), we exported the configuration from an old StoreFront server running Server 2019 and imported it onto the newly created StoreFront servers running Server 2022. All servers were deployed in the same OU, received the same GPOs/settings, and apart from the different operating systems, all settings were pretty much identical.

Read More →

Another error: ‘Application can’t be startedā€¦ (Instant Passthru could not be resolved)’ caused by the CWA upgrade.

by ⋅ Leave a Comment

A while ago, I wrote a blog about the error message ‘Application canā€™t be startedā€¦ (Instant Passthru could not be resolved)’ that we encountered when launching a published application from our Ivanti Workspace Control managed session. Recently, we received the same message again, but this time the cause was different.

Application canā€™t be startedā€¦ (Instant Passthru could not be resolved)

Shortly after a relatively simple taskā€”upgrading an outdated version of Citrix Workspace App (CWA) from 1912 LTSR to the slightly newer 2203 LTSRā€”we encountered some inexplicable issues that made it impossible to launch Citrix Published Apps via SelfService.exe.

Read More →

The Citrix Workspace App sometimes detects an incorrect keyboard layout on OSX.

by ⋅ Leave a Comment

Apple OSX users sometimes experience an incorrect keyboard layout loaded within their Citrix session. As a result, special characters are often located in different places. The cause of this issue is that Apple has a different keyboard layout compared to Windows, leading to an Apple US-international keyboard being recognized as a Dutch keyboard in Windows.

How to identify your Apple keyboard layout by country or region

Some time ago, we conducted extensive research together with Citrix Support to investigate the cause of this issue and whether there are possibilities to change this behavior. Unfortunately, it has been found that this behavior cannot be changed through a central solution. This behavior can only be altered by making adjustments on a per OSX system basis. This guide provides detailed instructions on what needs to be adjusted.

Read More →

Managing Citrix Clientname in Double-Hop Scenarios Using Ivanti Workspace Control

by ⋅ Leave a Comment

We manage a Citrix farm where users primarily launch a full desktop environment. From there, they can also connect to other applications running in Citrix Silo’s or access external Citrix farms. As an user environment manager (UEM), we utilize Ivanti Workspace Control (IWC).

When a user logs onto the primary desktop, the endpoint hostname is utilized by Ivanti Workspace Control within that session. Based on the endpoint hostname, we can set specific configurations using features like “location and devices”. In a double-hop scenario, where a user launches a Citrix published application or another Citrix desktop from within the primary session, the hostname of the primary session server is used as the hostname in the secondary session.

Read More →

The Mystery of Why Citrix NetScaler Sometimes Fails to Load the StoreFront Page after Login

by ⋅ Leave a Comment

Due to lifecycle management (LCM), we replaced several Citrix NetScaler appliances with new ones. Although we conducted thorough acceptance tests before putting them into production, unfortunately, we experienced an annoying issue once they were operational.

Some users complained that they saw a spinning progress bar after they successfully logged on to the Citrix NetScaler. It was only reported by a minority of users and was resolved by refreshing their web browser sessions. In the end, users stopped reporting the issue because it occurred infrequently and the solution was simpleā€”just press F5. We initiated an investigation in the hope of completely resolving the issue.

Read More →

Expand you Citrix NetScaler Virtual Servers with a security.txt file

by ⋅ Leave a Comment

A “security.txt” file is a standard proposed by security researcher Ed Foudil in 2017 as a way for websites to define a security policy. It’s akin to the well-known “robots.txt” file which specifies rules for web crawlers. The security.txt file allows website owners to provide information to security researchers about how to report security vulnerabilities or concerns.

Since April 2022, Security.txt has been an Internet Engineering Task Force (IETF) informational standard

Read More →

Check your Citrix Product Eligibility Dates before upgrading your Citrix product!

by ⋅ Leave a Comment

Recently, we worked on upgrading a Citrix NetScaler VPX from version 13.0 to the latest 14.1 build. The Citrix NetScaler VPX, which had been running for quite some time, had not been upgraded because it still used features and functionalities, including Classic Policies, which essentially needed to be replaced by Advanced Policies starting from the 13.1 build.

During the preparation for the upgrade, our main focus was on the legacy configuration in the running ns.conf file that needed to be adjusted.

Citrix ADC scripts for migrating and converting Citrix ADC configuration with deprecated features https://github.com/netscaler/ADC-scripts/tree/master

By using the NSPEPI tool, you can not only check for legacy configuration but also convert it to new configurations in many cases. Always ensure that you download and use the latest version during the analysis. If you are upgrading from a version older than build 13.1, always use NSPEPI beforehand to ensure that everything continues to work as expected after the upgrade.

check_invalid_config /nsconfig/ns.conf

After replacing all legacy configurations in the ns.conf and ensuring there were no blocking issues according to the NSPEPI tool to upgrade to the latest 14.1 build, we conducted a trial upgrade migration within our acceptance environment.

After the upgrade, the Citrix NetScaler restarted smoothly, but it was no longer possible to log in using our domain accounts (LDAPS). Fortunately, logging in with the local nsroot account still worked. Once logged in, it was immediately apparent that several load-balanced VIPs were down, causing the LDAPS load balancer to be inactive. Additionally, various NetScaler features were suddenly no longer visible.

Show Unlicensed Features

The navigation suddenly included an item labeled “Show Unlicensed Features,” which we hadn’t seen before. After clicking on it, all features became visible again. However, it became immediately apparent that many things seemed to be unlicensed all of a sudden. Features that we were using prior to the upgrade to build 14.1. While browsing through the NetScaler GUI, we navigated to System > License and discovered that we were running an Express edition instead of Platinum. Consequently, many of the commonly used features were indeed unlicensed.

ADC License

Next, we examined the existing license files located in the directory /nsconfig/license. What immediately caught our attention was the date present in the license file. In our case, the expiration date was older than the Eligibility Dates required for using the Citrix NetScaler 14.1 build, which is July 12, 2023 šŸ™

NetScaler License File

Citrix products and their Eligibility dates https://support.citrix.com/article/CTX111618/citrix-product-customer-success-services-eligibility-dates

Since this was a Citrix NetScaler VPX with a valid software subscription, the solution was fortunately quite simple. Simply redownload your license file via the MyCitrix license portal and upload it to the Citrix NetScaler VPX. The new license file will include a new SA Date, enabling you to run build 14.1. After restarting the Citrix NetScaler, all previously licensed features reappeared.

Check your product eligibility dates before you proceed with the upgrade!

Analyse your Citrix NetScaler config

by ⋅ Leave a Comment

In the past, it was possible to upload your NetScaler configuration file (ns.conf) to the Citrix Insight Service, which would then conduct an automated health check of the configuration. You would receive a report detailing any potential issues, best practices not followed, and so on. This was incredibly helpful during setup. Unfortunately, Citrix discontinued this self-diagnostic service some time ago.

During E2EVC 2022 Athens, I stumbled upon the “Arrow’s NetScaler config analyser” in one of the sessionsā€”a tool more than handy. After registration, it allows you to check your NetScaler configuration for free. However, in practice, I still regularly encounter NetScaler administrators who are unaware of its existence, so I thought I’d mention it again.

Arrow’s NetScaler config analyser https://app.xconfig.io

Although they offer more than just the free health check, in this case, I want to specifically mention the FREE “Online Config Analysis.”

Unless you choose to save your ns.config within your personal account, your ns.config is not uploaded to their website; instead, it is analyzed locally from your browser session.

For added security, it’s advisable to first mask any confidential data such as passwords, IP addresses, etc., ensuring they’re not usable.

Without registration, not all results are visible, so go ahead and register yourself.

After creating an account, you’ll have full visibility into all the issues discovered within your ns.conf. These issues are categorized into four categories:

  • Critical
  • Major
  • Medium
  • Low

If you ask me, your configuration shouldn’t contain any Critical, Major, or Medium findings! šŸ˜Š

An example of a Critical finding might be:

An example of a Medium finding might be:

What’s also very handy besides the analysis of your NetScaler configuration is the easy browsing through your configuration. By selecting an item on the left side (which looks identical in structure to a NetScaler), you’ll see the corresponding lines from your configuration on the right. This makes the configuration much more readable and understandable.

The tool is constantly evolving, with new recommendations being added regularly. For a comprehensive overview of the change log, you can navigate to the “What’s new” section. If you encounter false positives or have recommendations for improvements, don’t hesitate to let them know. In my experience, they are responsive to user feedback and often address issues or implement suggestions in subsequent releases!

Citrix Application Lingering in conjunction with Ivanti Workspace Control

by ⋅ Leave a Comment

Recently, I worked on a project where the workload needed to shift from using a Citrix Published Desktop to a physical laptop, with locally installed applications. As always, there are applications that, for various reasons, cannot be moved from the Citrix Published Desktop to the physical laptop. For these applications, we chose to offer them as Citrix Published Applications. Although this transition went well technically, end users reported that working with published applications was not considered very pleasant.

Scenario: The published apps were offered from a Citrix Virtual Apps en Desktops Farm, utilizing Ivanti Workspace Control. Ivanti Workspace Control is a workspace management solution provided by Ivanti, a company specializing in IT management software. It offers features for managing user workspaces across various devices and environments, including physical desktops, virtual desktop infrastructure (VDI), and application virtualization platforms. Unfortunately, Ivanti has announced that Ivanti Workspace Control will reach end of life on December 31, 2026, but at the moment, we are using it to our full satisfaction. When starting a Citrix Published Application, it takes some time due to, among other factors, the loading of the Windows profile and Ivanti Workspace Control settings before the application actually starts. When you subsequently start a second published application, it loads faster since the entire profile and UEM (User Environment Management) don’t need to be processed again. When you close the last Citrix Published Application, it also logs out the entire user session, resulting in the next Citrix Published application taking some time again, as your entire Citrix sessions needs to be loaded

Read More →