Automating Let’s Encrypt cerificates lifecycle for Unified Access Gateway, Horizon and Workspace One services

The UAG is very important part of Workspace One platform, delivering secure access to services like Horizon (including Horizon Cloud on Azure and AWS), Content Gateway, Tunnel and recently even Secure Email Gateway.

If you deploy Unified Access Gateway in production, you probably put it behind a virtual IP (VIP) that will load balance the service, offload SSL and bridge with UAG. There are several use cases (PoC, Pilot, Lab, Demo, UAT etc.) where you will have your UAG exposed directly to the Internet. There are also scenarios where you just don’t want to or can’t offload SSL on the load balancer and the traffic is passed unaltered to the UAGs – client certificate authentication is one of this scenarios as it can’t work across the SSL bridge.
Production or not, you still need a trusted, public SSL certificate on the front. In many cases it is convenient to use a Wildcard or UCC certificate with many subject alternate names – and this is very expensive. Last year I’ve paid around 300 dollars for such certificate. This year when it came to renewal, I’ve decided there are better ways to spend this money, especially when you can have it for free!

Introducing Let’s Encrypt and ACME protocol for certificate lifecycle management

Let’s Encrypt is a free, automated, and open certificate authority (CA) provided by the non-profit Internet Security Research Group (ISRG). In a very quick summary, Let’s Encrypt provides the following:
– Single hostname, SAN with up to 100 hostnames and Wildcard certificates
– Certificates trusted by almost all modern browsers, operating systems and applications thanks to cross-signature from IdenTrust
– Certificates valid for 90 days
– Based on Automated Certificate Management Protocol for requests and renewals
– Automated installation of requested certificates in operating systems, web servers, load balancers and other services
– Possibility to save all the flavors of certificate formats as files (PEM, PFX etc)
– Has clients for Windows (I recommend Posh-ACME) and Linux/UNIX (for instance Certbot)
– Provides secure and automated domain ownership verification by HTTP-01 or DNS-01 challenge – easy to integrate with your DNS provider REST API.
– Certificate expiration notices over email and automatic renewals with scripts

Logical workflow of the automation

The scripts presented below are intended to automate the request, install/replace and renewal operations of the certificates on Unified Access Gateways and Horizon Connection Servers. They are designed to run on a Connection Server. In my example they request a wildcard certificate for the domain.com, perform a DNS-01 challenge against GoDaddy DNS server with REST API, put the certificate on the UAG through the REST API interface on port 9443, install the certificate in the Local Machine store on the Connection Server, replace add the ‘vdm’ Friendly Name (during renewals they rename the old certificate’s Friendly Name) and restarts the Horizon Connection Server service to pick up new cert.

At the moment, the scripts work only against Connection Servers they are run from. It is possible to install the certificate on additional Connection Servers remotely with some modifications to the code. Here is a code snippet to give you an idea on how to perform it. It still requires a mechanism to put a PFX file on the file share.

$conservers = connserver2.domain.com, connserver3.domain.com
ForEach ($conserver in $conservers){

    Enter-PSSession -ComputerName $conserver
       #Commands below this point will execute remotely
       Import-PFXCertificate -CertStoreLocation Cert:\LocalMachine\My -FilePath \\server\path\to\cert.pfx
    Exit-PSSession
}

You can use the above approach with ForEach loop also to make the script work with multiple UAGs.

Prerequisites

Install Posh-ACME from the PowerShell gallery:

Install-Module -Name Posh-ACME

Generate GoDaddy API Key and Secret:

1. Login to https://developer.godaddy.com/keys
2. Click “Create New API Key”
3. Change the envrionment to “Production”
4. Save the Key and Secret immediatly (Secret will not be displayed again once you close this window).

Create a dedicated admin account for REST API calls on UAG.

Request.ps1

This script is intended to run manually once on the Connection Server during first request of the certificate. Redirect the output to a log file for reference and potential troubleshooting:

.\MyScript.ps1 > certreq.log

<#
.DESCRIPTION
Script requests certificates from Letsencrypt with automatic DNS-01 challange and installs them on UAG and Connection Server
 
.NOTES
Author: Roch Norwa https://digitalworkspace.blog @rochnorwa https://linkedin.com/rochnorwa
 
#>
#go back to C:\ just in case the location i set to cert:
Set-Location C:

#information to be used in the cert request. use comma seperated domains for UCC/SAN certificate.
$domain = "*.domain.com","domain.com"
$uag_host = "uag.domain.com"
 
# information for letsncrypt POSH-ACME powershell script
$psMod = "Posh-ACME"
$dnsPlugin = "GoDaddy"
$pArgs = @{GDKey="9ZtodG7jWNC_84teqTR1231U3tpRV4PGKJ"; GDSecret="3TPk3B123rCyCAXhKEadQa"}
$email = "certadmin@domain.com"
 
# letsencrypt server address. Use LE_STAGE to test, if everything ok switch to LE_PROD.
Set-PAServer LE_STAGE
  
# Request the certificate
New-PACertificate $domain -AcceptTOS -DnsPlugin $dnsPlugin -PluginArgs $pArgs -Contact $email -Verbose -Force
 
# transform PEM key and certificate into single text lines
$PACert = Get-PACertificate
$key_multiline = [IO.File]::ReadAllText($PACert.KeyFile)
$key_oneline = $key_multiline.Replace("`n",'\n')
$cert_multiline = [IO.File]::ReadAllText($PACert.FullChainFile)
$cert_oneline = $cert_multiline.Replace("`n",'\n')

#replace cert on UAG with REST API call
#provide UAG credential and info for API access in the format username:password encoded with base64 https://www.base64encode.org/

$uagcred = "YWRWJDWDJDdsd13YXJlMSE="  
$API_Settings = @{
    Headers     = @{ "Authorization" = "Basic $uagcred" }
    Method      = "PUT"
    Body        = $json_uag
    ContentType = "application/json"
}
#Create json string and make api call. Refer to https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html
$json_uag = '{"privateKeyPem":"' + $key_oneline + '","certChainPem":"' + $cert_oneline + '"}'
$API_Endpoint = "https://" + $uag_host + ":9443/rest/v1/config/certs/ssl"
Invoke-RestMethod $API_Endpoint @API_Settings

#now lets replace the certificate on Connection Server. First we install the certificate in the local machine store
Install-PACertificate

#we will check if there are an existing certificates with a friendly name of "vdm"and rename it to "replaced" with the current date for the record

Set-Location cert:\LocalMachine\My
$old_certs = Get-Childitem |Where-Object {$_.FriendlyName -eq 'vdm'}
ForEach ($old_cert in $old_certs){
$old_cert.FriendlyName = "Replaced $(Get-Date -format 'u')"
write-host $old_cert.Thumbprint 
}

#now we will set the friendly name of this certificate to "vdm"

$cert_thumbprint_dirty = Get-PACertificate |fl Thumbprint
$cert_thumbprint = Out-String -InputObject $cert_thumbprint_dirty -Width 100
$cert = gci $cert_thumbprint.Substring(17)
$cert.FriendlyName = “vdm”

#and the last thing is to restart Hotizon Connection Server service to pickup the new certificate

Restart-Service -Name wsbroker
Set-Location C:

Renew.ps1

This script should be scheduled to run every 60-80 days (start automatic renewal at least 10 days prior to expiry of the current certificate to give yourself time for manual renewal in case of failure). This should be run under the account that was used to start the Request.ps1 script on the same Connection Server. Redirect the output to a log file for reference and potential troubleshooting:

.\MyScript.ps1 > output.txt

<#
.DESCRIPTION
Script requests certificates from Letsencrypt with automatic DNS-01 challange and installs them on UAG and Connection Server
 
.NOTES
Author: Roch Norwa https://digitalworkspace.blog @rochnorwa https://linkedin.com/rochnorwa
 
#>
#go back to C:\ just in case the location i set to cert:
Set-Location C:

#information to be used in the cert request. use comma seperated domains for UCC/SAN certificate.
$domain = "*.domain.com","domain.com"
$uag_host = "uag.domain.com"
 
# information for letsncrypt POSH-ACME powershell script
$psMod = "Posh-ACME"
$dnsPlugin = "GoDaddy"
$pArgs = @{GDKey="9ZtodG7jWNC_84teqTR1231U3tpRV4PGKJ"; GDSecret="3TPk3B123rCyCAXhKEadQa"}
$email = "certadmin@domain.com"
 
# letsencrypt server address. Use LE_STAGE to test, if everything ok switch to LE_PROD.
Set-PAServer LE_STAGE
  
# Request the certificate
#New-PACertificate $domain -AcceptTOS -DnsPlugin $dnsPlugin -PluginArgs $pArgs -Contact $email -Verbose -Force

# Renew the certificate. Keep the above New-PACertificate hashed out!
# -AllOrders means all certificates requested by a specific account (email address). -Force will renew even if the certificate is not at the renewal period (30 days before expiration)
Submit-Renewal -AllOrders -Verbose -Force
 
# transform PEM key and certificate into single text lines
$PACert = Get-PACertificate
$key_multiline = [IO.File]::ReadAllText($PACert.KeyFile)
$key_oneline = $key_multiline.Replace("`n",'\n')
$cert_multiline = [IO.File]::ReadAllText($PACert.FullChainFile)
$cert_oneline = $cert_multiline.Replace("`n",'\n')

#replace cert on UAG with REST API call
#provide UAG credential and info for API access in the format username:password encoded with base64 https://www.base64encode.org/

$uagcred = "YWRWJDWDJDdsd13YXJlMSE="  
$API_Settings = @{
    Headers     = @{ "Authorization" = "Basic $uagcred" }
    Method      = "PUT"
    Body        = $json_uag
    ContentType = "application/json"
}
#Create json string and make api call. Refer to https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html
$json_uag = '{"privateKeyPem":"' + $key_oneline + '","certChainPem":"' + $cert_oneline + '"}'
$API_Endpoint = "https://" + $uag_host + ":9443/rest/v1/config/certs/ssl"
Invoke-RestMethod $API_Endpoint @API_Settings

#now lets replace the certificate on Connection Server. First we install the certificate in the local machine store
Install-PACertificate

#we will check if there are an existing certificates with a friendly name of "vdm"and rename it to "replaced" with the current date for the record

Set-Location cert:\LocalMachine\My
$old_certs = Get-Childitem |Where-Object {$_.FriendlyName -eq 'vdm'}
ForEach ($old_cert in $old_certs){
$old_cert.FriendlyName = "Replaced $(Get-Date -format 'u')"
write-host $old_cert.Thumbprint 
}

#now we will set the friendly name of this certificate to "vdm"

$cert_thumbprint_dirty = Get-PACertificate |fl Thumbprint
$cert_thumbprint = Out-String -InputObject $cert_thumbprint_dirty -Width 100
$cert = gci $cert_thumbprint.Substring(17)
$cert.FriendlyName = “vdm”

#and the last thing is to restart Hotizon Connection Server service to pickup the new certificate

Restart-Service -Name wsbroker
Set-Location C:

Setting up parameters in the scripts

There is a number of of parameters you have to define in the script. Below is the description of all the options you have to tune.

Define domain names for Unified Communication Certificates with Subject Alternate Names:

$domain = "host1.domain.com","host2.domain.com",host3.domain.com"

For Wildard certificate use this pattern:

$domain = "*.domain.com","domain.com"

Define your Unified Access Server host-name with this parameter. Remember it needs to point to REST API interface on port 9443 (depending on you deployment model one-two-three NIC). Our script will change certificate only for Internet Facing interface, not for the management one.

$uag_host = "uag.domain.com"

Choose a dedicated plugin for your DNS provider. Available options are here.

$dnsPlugin = "GoDaddy"

Provide DNS API authentication. Most likely an API Key and API Secret, otherwise username and password depending on your provider. This could be done in a more secure way with SecureString or PSCredentails, consider this if you plan to use this approach in a production.

$pArgs= @{GDKey="9ZtodG7jWNC_84teqTR1231U3tpRV4PGKJ"; GDSecret="3TPk3B123rCyCAXhKEadQa"}

Certificate Administrator email adddress – this will identify your Account at Let’s Encrypt.

$email = "certadmin@domain.com"

Define Let’s Encrypt ACME server address. Start with test stating server to check if the script works ok, then switch to production (LE_PROD)

Set-PAServer LE_STAGE

Provide UAG admin credentials in the format username:password and encoded with Base64 algorithm (you can use https://www.base64encode.org/). This could be done in a more secure way with SecureString or PSCredentails, consider this if you plan to use this approach in a production.

$uagcred = "YWRWJDWDJDdsd13YXJlMSE="

Final tips for getting the logs, certificates and scheduling renewals in Windows Scheduler

After successful request and every renewal the Posh-ACME script will save all the variables of certificates and keys in the folder structure of the user that was executing the scripts C:\Users\Administrator\AppData\Local\Posh-ACME
As you can see, for each server (Production and Test/Staging) there is a seperate subfolder. The current-server is noted in the current-server.txt file – make sure this points to LE_PROD.

In the production server folder there is a number of subfolders with numeric names. They correspond to Let’s Encrypt Accounts created for the email address you have given in your requests. Also the current-account.txt lists the account number that is beeing used for default renewals.

In the specific Account folder you will see the actual order folder (order maps to a domain from the request). There is also a plugindata.xml file with cached GoDaddy API Key and Secret, acct.json file with cached request script and email address for the account and the

Lastly, all the certificates, keys, chains and csr’s are in the domain folder. Default password for the PFX file is poshacme – it can be also found in the order.json file along with other certificate parameters.

Using HTTP-01 challange instead of DNS-01

In some cases, you might now have the possibility to automate the DNS challenge creation using REST API from your DNS provider, or you might not have permissions to do that. In this case, you can put an acme challenge file to your web server:

http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN&gt;

In our case the web server is Tomcat that is running Horizon Connection Server web services. Therefore you would have to save the challenge file to the folder:

C:\Program Files\VMware\VMware View\Server\broker\webapps\.well-known\acme-challenge

To make it accessible from the Internet, we need to add additional proxy pattern on the UAG Horizon service config:

You will also have to re-write above Power Shell scripts to save the HTTP-01 challenge as a file with specific name and content. Some hints on how to do it can be found below. I might write a blog post about this method in the near term future.
https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation

Sources, tools and additional reading

https://letsencrypt.org/docs/ – official documentation

https://github.com/rmbolger/Posh-ACME – Power Shell based client for ACME

https://developer.godaddy.com/getstarted – playing with GoDaddy REST API

https://www.base64encode.org/ – encoder for username:password

https://sourceforge.net/projects/xca/ – Great vault for your certificates and keys with easy options of converting to different formats. Good alternative to OpenSSL if you prefer GUI. And it works on MacOS!

4 thoughts on “Automating Let’s Encrypt cerificates lifecycle for Unified Access Gateway, Horizon and Workspace One services

  1. Thanks for the EXCELENT writeup and scripts!!! these were exactly what I’ve been looking for and need. Everything appears to work except the upload to UAG. I was getting this error…

    Invoke-RestMethod : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

    …so I added some code to disable cert checking…

    #requires -Version 5
    #requires -PSEdition Desktop
    class TrustAllCertsPolicy : System.Net.ICertificatePolicy {
    [bool] CheckValidationResult([System.Net.ServicePoint] $a,
    [System.Security.Cryptography.X509Certificates.X509Certificate] $b,
    [System.Net.WebRequest] $c,
    [int] $d) {
    return $true
    }
    }
    [System.Net.ServicePointManager]::CertificatePolicy = [TrustAllCertsPolicy]::new()

    …but that only changed to a similar error in the same function and now Im getting the error below…

    VERBOSE: Updating cert expiration and renewal window
    VERBOSE: Successfully created certificate.
    Invoke-RestMethod : Error in uploading the certificate. See log for more details.
    At C:\Users\administrator.\Documents\UAGsslcertSCRIPTS\Renew.ps1:65 char:1
    + Invoke-RestMethod $API_Endpoint @API_Settings
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
    24xx_thumbprint_of_my_previous_cert_xxC0
    WARNING: Waiting for service ‘VMware Horizon View Connection Server (wsbroker)’ to stop…
    WARNING: Waiting for service ‘VMware Horizon View Connection Server (wsbroker)’ to stop…

    I am able to perform a GET rest api request with my same credentials using the Postman app. Not having much luck on where to go from here. This is whome new self-signed test environment of all the newest Horizon 7.11 and UAG.

    Like

  2. So after much searching and code changes and testing I finally got this working. I did add a section to disable cert checking but after many bangings of head->desk I believe the final fix was the order of the API_Settings…

    $API_Settings = @{
    Method = “PUT”
    ContentType = “application/json”
    Headers = @{ “Authorization” = “Basic $uagcred” }
    Body = $json_uag
    }

    Like

  3. OK, so something with the API_Settings section wasn’t working (Body= blank and Headers=System.Collections.DictionaryEntry)

    Here is the FINAL working UAG section where I replaced $API_Settings with $headers and rewrote the Invoke-RestMethod line……

    #replace cert on UAG with REST API call
    #provide UAG credential and info for API access in the format username:password encoded with base64 https://www.base64encode.org/

    $uagcred = “yourBASE64ENCODEDusername:password”

    $headers = New-Object “System.Collections.Generic.Dictionary[[String],[String]]”
    $headers.Add(“Content-Type”, “application/json”)
    $headers.Add(“Authorization”, “Basic $uagcred”)

    #Create json string and make api call. Refer to https://docs.vmware.com/en/Unified-Access-Gateway/2.8/com.vmware.access-point-28-deploy-config/GUID-EDC244DD-07AB-4841-A893-84ADF8D59838.html
    $json_uag = ‘{“privateKeyPem”:”‘ + $key_oneline + ‘”,”certChainPem”:”‘ + $cert_oneline + ‘”}’
    $API_Endpoint = “https://” + $uag_host + “:9443/rest/v1/config/certs/ssl”
    Invoke-RestMethod $API_Endpoint -Method ‘PUT’ -Headers $headers -Body $json_uag -Verbose

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s