While upgrading our Citrix Provisioning Server (2203 LTSR) to the latest LTSR version 2402 CU1, we ran into some difficulties that we didn’t encounter with earlier versions. In addition to issues with the Citrix Provisioning Server Console, we also had some struggles getting both Citrix Studio and Citrix PVS Console to run from a single admin server. In this blog, I’ll share our findings.

Until now, our default upgrade procedure was to upgrade all Citrix Provisioning Server Consoles, test the console, and then upgrade the Citrix Provisioning Server itself. However, we quickly noticed that the new PVS Console (based on LTSR 2402 CU1) was unable to successfully connect to a Provisioning Server running LTSR 2203.

When connecting a 2402 PVS Console to a 2203 PVS Server, the following error messages appear:
FX:{d4362438-2b4a-4f95-91b1-4f813d7b18e7} Type is not resolver for member ‘EnterpriseAccess.EAException

For completeness, an older 2203 PVS Console was able to connect to a 2402 PVS Server without any error messages. The product documentation was recently updated with this information.

Upgrade the Citrix Provisioning console and server on every Citrix Provisioning server. After all servers are upgraded, upgrade the console on any other systems it is installed on. The console is not backward compatible.

Next, we upgraded the first Citrix Provisioning Server to 2024 CU1 LTSR. Although the upgrade went flawlessly and the configuration wizard succeeded without any errors, the upgraded server appeared as “down” in the PVS Console. The first error message that appeared in Event Viewer was Event ID 268: “StreamProcess – Cannot establish a connection to the database because the server cannot be found.” It seemed to be something related to the database. The issue was caused by the ‘Microsoft OLE DB Driver 19.x‘ driver, which, due to stricter security measures, requires certificate validation on the SQL Server. In our case, the Microsoft SQL Server was configured with the ‘force encryption’ option enabled, but did not provide a certificate. Regarding encryption, PVS DOES NOT use the default. It specifies Encrypt=Optional when it connects to the database.  This means the connection is encrypted if the SQL Server has Force Encryption=Yes, and otherwise it is unencrypted.  This is identical to the behavior with earlier versions.

CTX69229 : PVS Server Down In Console After Upgrade to 2402 CU1

Because Citrix Provisioning 2308 and later only supports Microsoft OLE DB Driver 19.3 or newer, downgrading to OLE DB 18.x (which doesn’t require a certificate) was not an option. As a result, we had to involve the database team and ask them to modify our SQL instance. After this change, we were able to successfully start the upgraded Citrix Provisioning Server.

The next issue we encountered was the compatibility with Citrix Studio (2402). As mentioned earlier, we like to have all Citrix consoles installed on our admin servers. Unfortunately, Citrix Studio was completely broken after installing the Citrix Provisioning Server console. We tried several installation sequences, but in the end, the only working sequence was as follows:

  1. Uninstall Citrix Studio
  2. Install Citrix Provisioning Server Console
  3. Uninstall Citrix DaaS Remote PowerShell SDK
  4. Reinstall Citrix Studio

After this sequence, we finally had both Citrix Studio and the Citrix Provisioning Server Console working side by side on the admin servers.

Uninstall Citrix Studio before installing the Citrix Provisioning Server Console.

Depending on your environment, the third step might not apply to your setup. However, in our case, we still use Ivanti Workspace Control to publish Citrix applications (on-prem, non-cloud). Unfortunately, this does not work well when the Citrix DaaS Remote PowerShell SDK is installed. One thing we found very frustrating is that the ‘Citrix DaaS Remote PowerShell SDK’ is installed as a requirement by the ‘Citrix Provisioning Server Console.’ There is no option to skip this component, even if another SDK is already in place. We discussed this issue with Citrix Support, and hopefully, they will make adjustments in the future. For now, manually uninstalling the Citrix DaaS Remote PowerShell SDK was our only option. 😞

Ivanti Workspace Control application publishing (on-prem) is not compatible with the “Citrix DaaS Remote PowerShell SDK”

That’s it for now! Hopefully, we’ve saved you some frustrations while upgrading your Citrix Provisioning Servers. πŸ˜‰ Finally, I would like to thank my colleague Hugo Koop for his research efforts and insightful discussions with Citrix Support.

Within a large environment where Ivanti Workspace Control is used in combination with a Microsoft SQL Database, the SQL Server has to handle numerous connections initiated by all the Ivanti Workspace Control Agents.To minimize the load and connections on the SQL Server, Ivanti introduced the Ivanti Relay Servers, which act as proxies for all agent connections to the SQL Server, thereby reducing the total number of SQL connections and the overall load.

While monitoring our Ivanti Relay Server, we noticed that a couple of times a day, the Ivanti Relay Servers started queuing all messages. The transactions folder filled up with transactions that weren’t forwarded to the SQL Server. The Ivanti Relay Servers have a built-in failsafe option, where they stop accepting new transactions when the queue exceeds 50,000 transactions. So, in a short period of time, all Ivanti Relay Servers went offline because they hit the 50,000 transaction threshold.

Read More →

Let’s start by stating I’m not a SQL DBA πŸ˜‰ when it comes to databases, I’m just a user who needs a database for my applications πŸ™‚ Lately, in various environments, we’ve been creating different Ivanti Workspace Control and Automation Manager databases. We simply request an empty database from the SQL department and are assigned DBO rights. Subsequently, we establish a connection to the database from Ivanti Automation Manager (for example) and handle the initial database setup within the application itself.

After initializing the SQL database, the user who performed the initialization can start Ivanti Workspace Control, for example, without encountering any issues. Another user, who has the same permissions (DBO) on the database, is unable to launch the application. The second user is able to access the SQL Database using SQL Management Studio, where they seem to have full control over the database.

Read More →

For managing several environments, we utilize Ivanti Automation Manager, leveraging Microsoft SQL Server as the database. According to the documentation, Ivanti Automation Manager does not support “SQL Server Always On availability groups,” and unfortunately, there is no mention of using a “SQL Server multi-subnet failover cluster.”

Supported database systems https://help.ivanti.com/res/help/en_US/IA/2024/Admin/Content/48735.htm

Within our environments, however, the use of a “SQL Server multi-subnet failover cluster” is the standard database configuration that we must use. Simply by adding the parameter “MultiSubnetFailover=True” to the database connection string, the SQL Client recognizes that it’s a MultiSubnetFailover cluster. However, since the database connection string is initiated by Ivanti Automation Manager, we don’t have the ability to add “MultiSubnetFailover=True” to it directly. This parameter will need to be included from within the Ivanti Automation Manager software.

SqlConnection.ConnectionString Property https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.connectionstring

Upon inquiry with Ivanti, it was indeed confirmed that there is no support for a “SQL Server multi-subnet failover cluster.” The request from Ivanti was to submit a Uservoice through the Ivanti Ideas Portal for this feature. We have duly submitted the request as requested. However, for unclear reasons, Ivanti has chosen not to implement this feature.

Uservoice: MultiSubnetFailover support (Microsoft OLE DB Driver for SQL Server) https://ivanti.ideas.aha.io/ideas/IA-I-44

MultiSubnetFailover Uservoice

Without the “MultiSubnetFailover=True” value in the connection string, for example, Ivanti Automation Manager may fail to start after the active SQL node is changed.

Connection error

Since we couldn’t avoid using a SQL Server multi-subnet failover cluster, we have temporarily resolved this by implementing a script. It may not be the most elegant solution, but it gets the job done!

We have created a scheduled task on all servers where the Ivanti Automation Manager Console and Ivanti Dispatchers are installed. This task runs every 5 minutes and executes a PowerShell script, which checks if the connection to the database is still possible. If not, it identifies the active SQL node and updates the hosts file accordingly, allowing the Consoles and Dispatchers to establish a connection with the database again.

<#
.SYNOPSIS
This PowerShell script updates the hosts file on a target machine with the current active SQL node IP address.
It checks if the specified target hostname is reachable. If not, it determines the active SQL node and updates the hosts file accordingly.

.DESCRIPTION
This script is designed to be run on a target machine to ensure that it always resolves a specific hostname to the active SQL node IP address.
It checks the availability of the target hostname and updates the hosts file with the IP address of the active SQL node if necessary.

.NOTES
- Script Name: Update-HostsFile.ps1
- Version: 1.0
- Authors: Rink Spies
- Date: 08-04-2024

.PARAMETER None
This script does not accept any parameters.

.EXAMPLE
.\Update-HostsFile.ps1
This command runs the script to update the hosts file with the current active SQL node IP address.

#>

# VARIABLES
$HostsFile = "$env:SystemRoot\System32\drivers\etc\hosts"
$TargetHostname = "MySqlServerName" # <<Update with SQL Server Instance name >>
$SQLNodes = @("1.2.3.4", "2.3.4.5", "3.4.5.6")  # << update with all SQL Node IP's >>
$LogFile = "C:\Windows\Temp\Update-hosts-file.log"

# FUNCTIONS

# Add-HostRecord function adds a record to the hosts file.
function Add-HostRecord {
    param(
        [string]$HostsFilePath,
        [string]$IP,
        [string]$Hostname
    )

    Add-Content -Path $HostsFilePath -Value "$IP`t`t$Hostname"
}

# Test-ActiveSQLNode function checks if a given SQL node is active.
function Test-ActiveSQLNode {
    param(
        [string]$SQLNode
    )

    return (Test-NetConnection -ComputerName $SQLNode -Port 1433 -InformationLevel Quiet -ErrorAction SilentlyContinue)
}

# Update-HostsFile function updates the hosts file with the IP address of the active SQL node.
function Update-HostsFile {
    foreach ($Node in $SQLNodes) {
        if (Test-ActiveSQLNode $Node) {
            Add-HostRecord -HostsFilePath $HostsFile -IP $Node -Hostname $TargetHostname
            return $Node
        }
    }
    return $null
}

# Log-Output function logs messages to the console and a log file.
function Log-Output {
    param(
        [string]$Message,
        [bool]$IncludeTimestamp = $true
    )

    $logEntry = if ($IncludeTimestamp) {
        "$(Get-Date -Format 'dd-MM-yyyy HH:mm:ss') $Message"
    } else {
        $Message
    }

    Write-Output $logEntry
    Add-Content -Path $LogFile -Value $logEntry
}

# SCRIPT

# Start the script
Log-Output "#############################################"
Log-Output "Starting update hosts file script."

# Check if the current IP for the target hostname is active
if (-not (Test-ActiveSQLNode $TargetHostname)) {
    Log-Output "Current IP for $TargetHostname is not active anymore."
    $activeNode = Update-HostsFile
    if ($activeNode) {
        Log-Output "Active IP $activeNode is online and configured in the hosts file."
    } else {
        Log-Output "None of the IPs are active."
    }
} else {
    Log-Output "Current IP for $TargetHostname is still active."
}

# End the script
Log-Output "Stopping update hosts file script."

As mentioned, not really the solution you’d ideally want to use, but hopefully Ivanti Automation Manager will still receive support for MultiSubnetFailover in the future.