Infrastructure Certificate Handling Changes in VMware Cloud Director 10.3

With the release of VMware Cloud Director 10.3, there are several new additions and enhancements to this self-service platform. I’d like to talk about some of the continued security and certificate changes that one must be aware of before upgrading to 10.3.

This is an ongoing effort to simplify certificate handling, but also conform to industry security standards.

Initial Issue

I’ve been recently working in the lab on Tanzu/TKG integration within VCD. I noticed after upgrading to our 10.3 GA build, I was receiving certificate errors related to my NSX-T Manager. This was popping up when attempting to apply a Kubernetes policy to a tenant –

I did not see any connectivity issues within Tasks and Events. However, I noticed that when attempting to add a new T0, I had nothing available from my NSX-T Manager!

I have plenty of T0s available within my NSX-T deployment –

Upon attempting to verify my NSX certificate, VCD was stating that the cert was verified already, but would not allow me to continue.

The NSX-T cert was already in my Trusted Certificates, so I was quite puzzled on this one. Moreover, there were no other infrastructure changes other than the upgrade to VCD 10.3 (from 10.2.2).

What happened? Let’s jump in.

Explanation

With the release of version 10.3, we have removed the ability to disable hostname verification for NSX Managers. This is to conform with industry standard security guidelines and our platform criteria.

If we look at the Certificate settings within a VCD 10.2.2 instance, we see the following which shows the ability to Disable hostname verification for vCenter Server, vSphere, and NSX Managers –

Now with my newly upgraded 10.3 instance, we only see vCenter Server and vSphere –

Hence, with 10.3, we have no way of disabling hostname verification for NSX Managers.

So, what specifically happened in my lab environment?

First off, when my NSX-T Manager was deployed with the Subject Alternative Name as “nsxt” –

There was no other Subject Alternative Name (SAN) applied within this self-signed certificate. When we built these VCD sites, we added it with the fully qualified domain name (nsxt.atlanta.zpod.io). We can still see this present within our Denver site –

Therefore, to resolve my issue, I had to change the URL to just “https://nsxt” since short name resolution is available in my environment –

Once modified, all related NSX-T objects started to work again. I was able to add in a new T0 –

To summarize, the URL of any type of NSX Manager (V or T) must conform with the certificate’s SAN content to ensure the hostname explicitly matches. If it does not, you will see issues like the above.

Looking Forward and Preparing for vCenter Changes

While this would most likely impact lab or QA environments, this is something providers should be aware of when upgrading to 10.3.

After speaking further with our engineering team on this change, disabling vCenter and vSphere hostname verification will be removed in the next VCD release (including a patch release!). When reviewing vCenter certificates, ensure you trust your CA certificates. We have updated the documentation to show what options one has.

If you are using native vCenter certificate management, you can prepare for hostname verification. With VCD 10.3, we now can retrieve the VMCA directly from the respective vCenter. This only will work with VCD 10.3 and vSphere 7.0 or later.

To do this, follow the below steps. Note this will briefly disconnect vCenter, so please execute this in a change or maintenance window.

Delete the existing vCenter certificate under Administration -> Trusted Certificates

Reconnect the respective vCenter – you will be prompted with a window to trust the certificate, but this presents a new button – Retrieve

Select the CA and hit “Trust Selected”

Now within your Trusted Certificates panel, we can see the CA.

We are now ready to enable hostname verification for vCenter –

After forcing a vCenter reconnect, we can see a successful task –

Big thanks to Ankit Shah for his continued collaboration on this effort!

-Daniel

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.