Infrastructure Certificate Handling Changes in VMware Cloud Director 10.3

With the release of VMware Cloud Director 10.3, there are several new additions and enhancements to this self-service platform. I’d like to talk about some of the continued security and certificate changes that one must be aware of before upgrading to 10.3.

This is an ongoing effort to simplify certificate handling, but also conform to industry security standards.

Initial Issue

I’ve been recently working in the lab on Tanzu/TKG integration within VCD. I noticed after upgrading to our 10.3 GA build, I was receiving certificate errors related to my NSX-T Manager. This was popping up when attempting to apply a Kubernetes policy to a tenant –

I did not see any connectivity issues within Tasks and Events. However, I noticed that when attempting to add a new T0, I had nothing available from my NSX-T Manager!

I have plenty of T0s available within my NSX-T deployment –

Upon attempting to verify my NSX certificate, VCD was stating that the cert was verified already, but would not allow me to continue.

The NSX-T cert was already in my Trusted Certificates, so I was quite puzzled on this one. Moreover, there were no other infrastructure changes other than the upgrade to VCD 10.3 (from 10.2.2).

What happened? Let’s jump in.

Explanation

With the release of version 10.3, we have removed the ability to disable hostname verification for NSX Managers. This is to conform with industry standard security guidelines and our platform criteria.

One can continue to utilize self-signed certificates, however, VCD will reject any URL that does not match the values present in the certificate.

If we look at the Certificate settings within a VCD 10.2.2 instance, we see the following which shows the ability to Disable hostname verification for vCenter Server, vSphere, and NSX Managers –

Now with my newly upgraded 10.3 instance, we only see vCenter Server and vSphere –

Hence, with 10.3, we have no way of disabling hostname verification for NSX Managers.

So, what specifically happened in my lab environment?

First off, when my NSX-T Manager was deployed with the Subject Alternative Name as “nsxt” –

There was no other Subject Alternative Name (SAN) applied within this self-signed certificate. When we built these VCD sites, we added it with the fully qualified domain name (nsxt.atlanta.zpod.io). We can still see this present within our Denver site –

Therefore, to resolve my issue, I had to change the URL to just “https://nsxt” since short name resolution is available in my environment –

Once modified, all related NSX-T objects started to work again. I was able to add in a new T0 –

To summarize, the URL of any type of NSX Manager (V or T) must conform with the certificate’s SAN content to ensure the hostname explicitly matches. In theory, you could utilize whatever URL matches your SAN or certificate values. Just note that if it is not found, VCD will reject it.

It is also equally important that you are utilizing the same exact configuration for any NSX Advanced Load Balancer (Avi) integration!

LDAP SSL Handling

I was working with a partner in their lab environment and we noticed that their VCD 10.3 database upgrade failed due to an existing LDAP SSL configuration – “Found one or more organizations with an LDAP configuration that bypasses SSL certificate verification. This version of VMware Cloud Director cannot support such configurations. Ensure that each of the above organizations reconfigures their LDAP settings to import the certificate of their LDAP server, if necessary, and turns off “Accept All SSL Certificates”.

This is quite the conundrum (headache) as the binaries have been updated, but the database upgrade fails. Therefore, one must revert back to the snapshot to resolve the SSL issue.

However, after working with engineering and our support organization, we decided to publish a KB article – check it out here. We are hoping to make this easier for VCD providers in a future version.

Looking Forward and Preparing for vCenter Changes

While this would most likely impact lab or QA environments, this is something providers should be aware of when upgrading to 10.3.

After speaking further with our engineering team on this change, disabling vCenter and vSphere hostname verification will be removed in the next VCD release (including a patch release!). When reviewing vCenter certificates, ensure you trust your CA certificates. We have updated the documentation to show what options one has.

If you are using native vCenter certificate management, you can prepare for hostname verification. With VCD 10.3, we now can retrieve the VMCA directly from the respective vCenter. This only will work with VCD 10.3 and vSphere 7.0 or later.

To do this, follow the below steps. Note this will briefly disconnect vCenter, so please execute this in a change or maintenance window.

Delete the existing vCenter certificate under Administration -> Trusted Certificates

Reconnect the respective vCenter – you will be prompted with a window to trust the certificate, but this presents a new button – Retrieve

Select the CA and hit “Trust Selected”

Now within your Trusted Certificates panel, we can see the CA.

We are now ready to enable hostname verification for vCenter –

After forcing a vCenter reconnect, we can see a successful task –

Big thanks to Ankit Shah for his continued collaboration on this effort!

-Daniel