Upgrading to VMware Cloud Director 10.1 – SSL Management Changes

This will be a quick post discussing SSL changes inside of VMware Cloud Director 10.1. I wanted to publish this as it’s important to provide further context around these changes. Furthermore, there could be errors around endpoint behavior post-upgrade when using self-signed certificates.

Update 2 June 2020 – there is a public knowledge base (KB) article on this behavior – https://kb.vmware.com/s/article/79091

Certificate management within VMware Cloud Director (VCD) is changing with 10.1. Our goal is to make this more intuitive but also provide SSL centralized management. Starting with 10.1, VCD must trust all underlying certificates when connected to vCenter or NSX instances, including self-signed certs.

Within the API, we have a new endpoint for centralized SSL management. In this release, vCenter, NSX, and LDAP have moved to this location:

/cloudapi/1.0.0/ssl/trustedCertificates

Post 10.1 release, providers should be using fully-signed (or well-signed) certificates. This will also impact any connected(system or org-level) LDAP instances and will require mandatory well-signed certificates in the future.

There is a new Cell Management Tool (CMT) function to trust all certs post-upgrade –

trust-infra-cert

I’ll explain later how one can use this.

Post 10.1 Upgrade Behavior

I recently upgraded one of my labs to 10.1 from 10.0.0.1 and noticed that all of my vCenter/NSX connections were failing:

From the vcloud-container-debug.log file, I was seeing SSL errors:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
	at 
	... 29 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)

However, I did not have any verification for verifying vCenter or NSX Manager certificates –

Initial Resolution

The fix I implemented (before I spoke to Engineering) was just to simply re-input the SSO credentials to each backed instance:

Once I did that, reconnection tasks were successful:

Options

After speaking with the team and verifying this was expected, there are a few options one can do in preparation for a VCD 10.1 upgrade.

Option 1 –

Post successful upgrade, utilize the CMT tool and input in “trust-infra-certs” to accept said certificates. The official documentation further reviews options within this new argument. However, one can utilize the command to accept them without further input:

/opt/vmware/vcloud-director/bin/cell-management-tool trust-infra-certs --vsphere --unattended

Once executed, all vCenter connections should reconnect successfully.

Option 2 –

Utilize the User Interface and just re-accept to each backed vCenter and accept the certificate. While I re-inputted my password, this was not necessary. This is what I did above and is a supported path for the new centralized SSL management.

Option 3 –

If you are using well-signed certificates, this should not be an issue. Therefore, no changes are necessary. Your certificates would have been imported prior to the 10.1 upgrade and one should see this change within the Administration UI:

Conclusion

While any upgrade requires further planning, this is a change on how VCD handles certificates going forward. This is analogous to how a browser prompts to trust a self-signed certificate. In summary, one should be utilizing well-signed certificates for production environments.

Happy upgrading!

-Daniel