VMware Cloud Director 10.2 – SSL Management and Public Address Configuration

With the release of VMware Cloud Director (VCD) 10.2, I wanted to review how SSL management has changed. In this blog post, I will highlight the new SSL functionality and securing a new VCD 10.2 instance.

Introduction

First, certificate management has always been a challenging process and not intuitive for newer VCD providers. Our engineering team has made a concerted effort to simplify certificate and endpoint management as we continue to evolve the platform.

This is the second phase of implementing SSL changes within VCD. Previously, I spoke about self-signed certificate behavior after upgrading to 10.1. There’s more to come here – this isn’t complete by any means.

After upgrading, to 10.2, a provider administrator is presented with a new section called Certificate Management –

Trusted Certificates

Trusted Certificates is the view of VCD’s certificates that are trusted at this point in time. Think of this section as “What does my VCD instance trust?” In my lab environment, I can see my connected vCenter instances, NSX managers, and so forth. We can also see that this is multi-site aware – I see my secondary site, Site-B.

One thing to note is these certificates are stored in a database, not the certificates.ks file. This is a rather interesting change.

When selecting a certificate, we are presented with several options. We can download, edit, delete, or copy this PEM certificate.

More importantly, we can expand this certificate to see the current status and when it expires –

When I upgraded one of my sites, the original self-signed certificate was showing within the Trusted Certificate window. As one can see, it is clearly marked as Expired. You might be wondering on why I see it twice – I will explain that further in my Multisite section.

Certificates Library

Next, the Certificates Library is a place where a provider can store certificates that can be used by tenants. This section is utilized for any tenant-facing services – so it’s a new addition. A great example of use would be for our newly added Avi Load Balancer functionality which Fojta recently wrote about. Again, since these certificates are stored in a database, we will be able to use this for future tenant-facing services.

In my example, below, I can add my wildcard cert for my domain (*.dp-pod.zpod.io)

Once imported, we see it visible within our library –

From there, I can use this certificate to secure my NSX (Avi) Advanced Load Balancer virtual services. In the future, we should see other services and components utilize this centralized certificate library.

At this time, this Certificate Library is not utilized for the VCD URL public address configuration. In the next section, we will dive into this configuration.

10.2 VCD Public Address Certificate Installation

I recently installed wildcard certificates for each of my lab sites so I can experience well-signed behavior. Thanks to Timo for getting Let’s Encrypt in our lab environment and issuing certificates. 😀

I have a net new site, vcd.vcpp.zpod.io, with a 10.2 VCD appliance instance. My plan is to install a well-signed wildcard certificate. Note this is a single cell appliance – so not your typical production state, but let’s review a typical production configuration.

Production Configuration

Typically, we see a minimum of three cells (due to Postgres clustering) that are behind a load balancer. The load balancer will handle and terminate all API and UI requests but passthrough console connectivity. There still needs a relationship of trust between the cells and LB, hence a certificate chain is established. This does not have to be the same Certificate Authority as the public (Internet) facing authority, but one must ensure all precautions are taken from a security perspective.

My Installation Process

Within this new environment, I have a few certs in my trusted store (Avi Controller, Default cert, NSX-T, VCSA) and nothing in my Certificate Library –

Currently, this single converged appliance was deployed with the self-signed cert and does not participate with any other cell or DB nodes. Also note this is not associated to any other site at this time.

Pre-10.2 Cell Certificate Installation for Public Address

Pre 10.2, there are five steps required per cell to install the certificate. We need to convert the pem files to p12 files, use the keytool for the conversion and converge into a temporary certificates.ks file, and then finally import it via the CMT process. Post-10.2, this has decreased slightly – and I will highlight that next.

1 – copy the fullchain.pem and privkey.pem to a temp directory on the cell

root@vcd [ /tmp ]# mkdir certs
root@vcd [ /tmp ]# cd certs/
root@vcd [ /tmp/certs ]# ls
fullchain.pem  privkey.pem

2 – Use openssl to export the HTTP and Console .p12 file

root@vcd [ /tmp/certs ]# openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out http.p12 -name http
Enter Export Password:
Verifying - Enter Export Password:
root@vcd [ /tmp/certs ]# openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out consoleproxy.p12 -name consoleproxy
Enter Export Password:
Verifying - Enter Export Password:

3 – use the keytool to convert it to JCEKS (or PKCS12) and converge it to a temp certificates.ks file

root@vcd [ /opt/vmware/vcloud-director/jre/bin ]# ./keytool -importkeystore -deststoretype JCEKS -deststorepass XXXXXX -destkeystore /tmp/certificate.ks -srckeystore /tmp/certs/http.p12 -srcstoretype PKCS12 -srcstorepass XXXXXX
Importing keystore /tmp/certs/http.p12 to /tmp/certificate.ks...
Entry for alias http successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

root@vcd [ /opt/vmware/vcloud-director/jre/bin ]# ./keytool -importkeystore -deststoretype JCEKS -deststorepass XXXXXX -destkeystore /tmp/certificate.ks -srckeystore /tmp/certs/consoleproxy.p12 -srcstoretype PKCS12 -srcstorepass XXXXXX
Importing keystore /tmp/certs/consoleproxy.p12 to /tmp/certificate.ks...
Entry for alias consoleproxy successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

4 – Change the permissions to the newly created certificate.ks file to vcloud:vcloud.

root@vcd [ /tmp/certs ]# chown vcloud:vcloud certificate.ks 
root@vcd [ /tmp/certs ]# ls -l
total 32
-rw------- 1 vcloud vcloud 7744 Oct 16 20:57 certificate.ks
-rw------- 1 root   root   4270 Oct 16 20:55 consoleproxy.p12
-rw------- 1 root   root   3554 Oct 16 20:43 fullchain.pem
-rw------- 1 root   root   4238 Oct 16 20:55 http.p12
-rw------- 1 root   root   1704 Oct 16 20:43 privkey.pem
root@vcd [ /tmp/certs ]# 

5 – Finally, use the CMT to import the new certificate keystore. Once successful, restart the services.

root@vcd [ /opt/vmware/vcloud-director/bin ]# ./cell-management-tool certificates -j -p --keystore /tmp/certificate.ks --keystore-password XXXXXX
Certificate replaced by user specified keystore at /tmp/certificate.ks. You will need to restart the cell for changes to take effect.
root@vcd [ /opt/vmware/vcloud-director/bin ]# service vmware-vcd restart

Update – I’ve been getting reports the command has been changed for importing the http and consoleproxy cert. You will need to run the above command separately. Therefore, like the following:

/opt/vmware/vcloud-director/bin ]# ./cell-management-tool certificates -j --keystore /tmp/certificate.ks --keystore-password XXXXXX

/opt/vmware/vcloud-director/bin ]# ./cell-management-tool certificates -p --keystore /tmp/certificate.ks --keystore-password XXXXXX

10.2 Cell Certificate Installation for Public Address

With VCD 10.2, we do not need to use keytool or change permissions to the temporary certificates.ks file. VCD 10.2 now allows for direct import of a p12 file while handling the conversion process. This is boiled down to three steps.

Therefore, our steps would be as follows.

1 – copy the fullchain.pem and privkey.pem to a temp directory on the cell

root@vcd [ /tmp ]# mkdir certs
root@vcd [ /tmp ]# cd certs/
root@vcd [ /tmp/certs ]# ls
fullchain.pem  privkey.pem

2 – Use openssl to export the HTTP and Console .p12 file

root@vcd [ /tmp/certs ]# openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out http.p12 -name http
Enter Export Password:
Verifying - Enter Export Password:
root@vcd [ /tmp/certs ]# openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out consoleproxy.p12 -name consoleproxy
Enter Export Password:
Verifying - Enter Export Password:

3 – Use CMT to import the new certificate keystore. Once successful, restart the services.

root@vcd [ /opt/vmware/vcloud-director/bin ]# ./cell-management-tool certificates -j -p --keystore /tmp/http.12 --keystore-password XXXXXX
root@vcd [ /opt/vmware/vcloud-director/bin ]# ./cell-management-tool certificates -j -p --keystore /tmp/consoleproxy.12 --keystore-password XXXXXX
root@vcd [ /opt/vmware/vcloud-director/bin ]# service vmware-vcd restart

As one can see, this is a multi-step process (not inclusive of getting the .pem) for installing it on my single cell environment. Moreover, this can vary based on the type of certificate and how you are acquiring such certificate. My peer, Sean Smith, wrote about his heavily involved process within his blog posts.

10.2 VCD UI Public Address Configuration

This next step has been enhanced within the VCD 10.2 UI.

Next, we need to modify the Public Addresses. Under Public Addresses, click the Edit button, which brings up this wizard dialog –

First, I am going to select my fullchain.pem file, which we now see reflected in the UI –

Once we click next, the API section comes up. By default, the “Use Web Portal Settings” is selected. Once can override this and select a specific base URL or certificate chain.

Last, we confirm our console proxy port –

Press save when completed.

Once completed, we can now see my newly secured VCD instance is now using our installed certificate –

Multisite

For VCD Multisite, the site association process has not changed. I have a previous blog post that discusses the instructions on executing the site association, but I am going to recap the process in the context of 10.2. I am also going to create a mesh between three sites: Site-A, Site-B, and Site-C.

My headers within Postman (note I am using API version 35):

Accept application/*+xml;version=35.0
Content-Type application/vnd.vmware.admin.siteAssociation+xml

Created a single document of all of my localSiteAssociation calls –

GET https://{{host}}/api/site/associations/localAssociationData

I am starting on my new site, Site-C, and creating the POST body to connect Site-A and Site-B.

POST https://{{host}}/api/site/associations

Accepted…

Now, onto Site-B –

For my existing sites, Site-A and Site-B, I am pasting the newly created siteAssociationMember payload from Site-C to append it. This is just another POST call like before.

After waiting a few moments, we can now see Site-C is active to Site-A –

Last of all, we can now see my three independent sites connected!

From a tenant perspective, I can now create my site “mesh” between the three sites.

Certificate Management with Multisite

Now, we can see that once we added in Site-C, our Trusted Certificates have expanded –

The reason is this is a multisite call when looking at Trusted Certificates. Now to discuss something that I learned post-VCD 10.2 upgrade. I was rather perplexed by this SITE_ENDPOINT and why it was duplicated across each site –

After discussing this with engineering, part of the upgrade procedure is to move each SiteAssociationMember entry into its respective, local database instance. The reason is two fold: 1) This allows us to put this into the centralized store, but 2) also use any level of trust established between two VCD sites. This will aid in ease of maintaining site to site association, but also managing certificates (as expirations are rather limited).

Summary and What’s Next

Centralized SSL management is a much needed addition for VCD. I’m looking forward to the evolution of SSL management and also multisite configuration to make this more intuitive.

-Daniel