Five Quick Tips on VMware vCloud Availability 3.0

I wanted to summarize a few things I’ve found when working with VMware vCloud Availability 3.0 (vCAv) over the past few months that will be helpful to providers and tenants. I’m sure there’s others, but these are the ones that come to mind.

Tunnel (Public API) Endpoint

This is a very important step for production deployments – setting the URL endpoint to ensure proper cloud access from tenants and other vCD instances.

The public API endpoint can be configured from the Cloud Replication Management (CRM) under Configuration:

Or can be configured directly on the tunnel:

When one configures it from the CRM, it does push this change to the tunnel appliance over port 8047 (internal communication port between CRM and the tunnel appliance).

Note that you will need a proper DNS or public IP address along with the DNAT port utilized (such as 443). Any configuration or reconfiguration requires a service restart on the CRM and Tunnel appliance –

Port Requirements

The above table summarizes all necessary ports for proper vCAv management. While this is ingress communication to the vCAv appliances, utilizing the combined appliance does present a different path for configuration. Below are the explicit ports for configuration of each role inside of a single appliance:

  • vApp Replication Manager + Portal – https://appliance-ip:8046/ui/admin
    • Provides the main interface for the cloud-to-cloud replication operations. It understands the vCloud Director level concepts and works with vApps and virtual machines.
  • Replication Manager – https://appliance-ip:8044/ui/admin
    • A management service operating on the vCenter Server level. It understands the vCenter Server level concepts for starting the replication workflow for the virtual machines.
  • Replicatorhttps://appliance-ip:8043/ui/admin
    • Exposes the low-level HBR primitives as REST APIs.
  • Tunnelhttps://appliance-ip:8047/ui/admin
    • Simplifies provider networking setup by channeling all incoming and outgoing traffic for a site through a single point

HBR Management

In my lab, I’ve rebuilt vCAv many times and in some scenarios, I’ve forgot that I had an active replication/protection on a VM. When re-enabling protection, one would receive an error stating “This VM or vApp is already replicated” and fails to protect.

This is due to the host-based replication process still enabled on this VM. To disable it, the provider will need to locate the VM and SSH to the ESXi host. From there, we can utilize the “vim-cmd” process and utilize the hbrsvc scope.

There’s two command you’ll need to know:

  • vim-cmd hbrsvc/getallvms
    • This will get the world ID of the VM’s so you can utilize this on the next command.
  • vim-cmd hbrsvc/vmreplica disable <VM_ID>
    • disables HBR on said VM

From there, one can successfully re-enable protection for this VM.

Plugin Management

There are two things I want to discuss:

  1. Plugin Visibility
  2. Changes to the vCAv Configuration and how it relates to the Availability Plugin

Plugin Visibility

When the provider configures vCAv for the first time and connects the vCD instance, it immediately makes an API call to push the Availability plugin. This is available to all by default –

You have two options on configuring visibility to the Availability plugin: 1) in vCD 9.7, utilize the Customize Portal plugin or 2) utilize the API calls to restrict access.

For 9.7 installs, it’s fairly easy -go to Customize Portal -> select vCloud Availability -> Publish and remove the selected tenants:

Pre 9.7, we will need to utilize the API to manage accessibility. My esteemed peer Chris Johnson did a writeup on managing access to vCAv 3.0 recently, but my older vCAv C2C 1.5 article also applies.

vCloud Availability tenants list

Changes to the vCAv Configuration

When the provider changes a vCAv service/system configuration, the plugin must also be updated with this new information. This is important as the change could have a new tunnel address.

This is very easy – all we need to do is re-register the vCD instance from vCAv. When we re-authenticate, vCAv will push the updated plugin to vCD.

Provider and Tenant Diagrams

The below are two diagrams that depict port communication for a provider and tenant environment. This is very helpful to understand what is required from an ingress and egress perspective –


Tenant – as discussed before, there is no need for a DNAT rule as all traffic originates from the on-premises tunnel to communicate to the provider vCAv environment.

That’s all for now.


Overview of VMware vCloud Availability 3.0 – Tenant Deployment, Protection Workflow, Resources

Once the provider site is operational, we are ready to bring the on-premises / Tenant site online for VMware vCloud Availability 3.0 (vCAv). Again, recap of the deployment steps:

  1. Deploy vCAv On-Premises Appliance
  2. Start configuration wizard
  3. Connect to vCAv Cloud Tunnel
  4. Configuration of local placement
  5. Validation and vSphere UI

Before we get started, let’s take a look at a port mapping diagram.

What’s interesting is one does not need a DNAT rule for tunnel traffic. The reason is any traffic is initiated from the on-prem site negating any ingress traffic (everything flows outbound), hence a standard SNAT (route) is sufficient. This is great as we do not need any network changes on the client side.

Deploy vCAv On-Premises Appliance

Deploying the appliance is very similar to the provider side. We have packaged up a standalone on-premises appliance that does not have the selection of the roles (and minimizes any client confusion). In the on-premises version, one does not have a dropdown of the service role selection, but just a acceptance and typical OVF deployment –

So again, very easy and similar to typical VMware OVF deployments.

Start Configuration Wizard

Let’s open a browser to https://onprem-fqdn/ui/admin and login –

You will be prompted to change the password to the appliance. From there, let’s hit the initial setup wizard –

Set your site name and any pertinent description. Click Next when complete.

As expected, we need to establish the lookup service along with SSO credentials.

On Cloud Details, this is where we pair with our vCloud/vCAv site. Configure the public API endpoint (for my lab, I am using 8048 but I showed earlier on utilizing 443) along with your organization administrative credentials.

Toggle the “Allow Access from Cloud” option if you want users from vCD to have the ability to browse and configure VMs locally from this site.

Accept (or remove) the CEIP and let’s take a look at the final completion screen –

Before hitting Finish, let’s toggle the “Configure local placement now” option to knock this out.

Local placement sets the vCenter/resource hierarchy for cloud to on-premises / failback protection.

Next, we will see a 5 step process for Local Placement – walk through the UI and select the hierarchy objects.

Validate and hit the Finish button.

Validation and vCenter UI

From our Cloud site, we can now see a new On-Prem Site and shows a status of OK.

Re-logging into the on-premises appliance, we can see the Manager and Cloud status as healthy also –

From the vSphere Client, we can also see vCloud Availability available –

Protection Workflow

We have two operations available: Protection and Migration. In the two below screenshots, our options change based on what button is selected.

One can establish incoming or outgoing replications between cloud or on-prem –

While I am not going to exhaustively go through every permutation, one can see how intuitive it is to protect or migrate workloads.

Protection from On-Premises

In my source site, I only have one choice as I select from On-Prem and have a single paired vCenter/Tenant site.

From here, I select the VMs I want to protect.

Select the Target oVDC –

If there’s a Seed VM available, select it.

Now I can specify my protection settings: my RPO, storage policy, retention policy for point in time instances, and if I want to quiesce/compress the instance and traffic.

Scheduling can be defined –

Finally, we get to see our validation.

Protection Settings – Viewing, Re-Addressing

Reviewing the current state, one can ascertain the health of the current protected workload with my source, destination, and RPO –

Clicking on the Networks button brings up our menu on what we want to do on Migrate/Failover or Test Failover –

This can be applied to all associated vApps or VMs, or explicitly broken down on per vNIC basis. One can also reset the MAC. Note that all of the same vCD guidelines apply – can’t set a manual IP outside of the CIDR block of that oVDC network, etc.

Clicking the sub-button Test Failover presents similar options, but one can copy from the Migrate/Failover menu to get started.

If we need to change the Replication Owner, we can click the Owner and select the new organization owner.

Migrate/Failover/Test Failover

Going through the Migrate, Failover, and Test Failover options are very intuitive.

For Migrate, we can select to power on the recovered vApps and apply the specific preconfigured network settings (or override that and select a specific network) –

For Failover, very similar to migrate, but we can drill into a Recovery Instance –

Lastly, Test Failover provides the ability to test a VM/workload without impacting production. This can be associated to a “bubble/fenced” network and tested by the application team to verify functionality.


As a final thought, I want to say how it’s been a pleasure working with the team to see this to fruition and public release. I believe this is going to be an extremely powerful platform and this is just the start.

After vCAv 3.0 is released, I will have more material along with many of my peers who will be discussing vCAv further. Below are some lightboard videos that introduces some of the concepts through these posts. Enjoy!


Overview of VMware vCloud Availability 3.0 – Provider Deployment

In this post, we will be reviewing the steps on setting up and operationalizing vCloud Availability 3.0 (vCAv) for a provider site.

There is a presumption that you will be deploying for production, so that is what I’ll be reviewing. The consolidated (combined) appliance would be an easier deployment, but still requires the below configurations post-deployment.

Recap of the Provider steps:

  1. Deployment of Cloud Replication Management (CRM) Instance
    1. Initial Replication Management Setup
    2. Initial Setup Wizard
  2. Deploy vCAv Replicator(s)
  3. Deploy vCAv Tunnel
  4. Configuration of CRM instance and start of site wizard
  5. Configuration of Replicator
    1. Pairing Replicator with Replication Manager
  6. Configuration of Tunnel
  7. Validation


  1. Available DNS and NTP server
  2. SSO Lookup Service Address
  3. Routing and Firewall Ports in place – see below for further insight
  4. vCenter and vCD on interoperability matrix
  5. Certificate Management – all certificates can be managed via the UI utilizing PKCS#12 certificates. Services must be restarted post-import.

Provider Port Mapping

Below is a diagram my esteemed peer, Chris Johnson, worked up for our upcoming EMPOWER presentation.


  1. Establishing a DNAT rule from 443 to 8048 is crucial for tunnel connectivity. This also has to be set as the API endpoint and will be pushed from the CRM instance.
  2. Ensure we can route and have direct port access between payload/resource vCenters, replicators, and Cloud Management.

Deployment of Cloud Replication Management (CRM) Instance

All of the roles we deploy for the provider will be coming from a single OVF – this is very similar to other VMware based virtual appliances. However, during the OVF deployment process, you will be prompted for the below role selection. For deployment of CRM, select Cloud Replication Management.

Initial Replication Manager Setup

Wait a few moments post-power on for initial configuration to take place, then open a browser to https://crm-fqdn:8441 so we can set the initial lookup service configuration.

We will be prompted for changing the default password. Note this is the same process for any newly deployed vCAv appliance and must be done on initial login –

From our initial screen, we can see that we have two issues: 1) missing Lookup Service settings and 2) Configured Replicators – there is none. The latter is fine for now, we will pair the replicator once we are done with the site wizard.

Let’s go over to Configuration and set the lookup service –

Accept the certificate…

As discussed prior, we will not see any replicators right now and will come back at a later time.

Initial Setup Wizard

Open a new tab to https://crm-fqdn/ui/admin and log in with your root account.

From here, we can see a link to run the initial setup wizard –

This is a very simple wizard that brings us through the site setup. From the beginning, we need to set a sitename. Note that you cannot utilize spaces and it is case-sensitive.

Second, set your public API endpoint address. Note this is where the traffic will ingress in from your tunnel node. In my lab environment, I will be directly connecting over 8048 (compared to traditional perimeter environment that would utilize 443 and DNAT rule to forward that traffic).

Here’s what I would if that was the case.

Next, lookup service address. You’ll be setting this quite a bit. 🙂

vCD configuration – note that you must include /api after the vCD FQDN. Also, during this initial setup, vCAv will take care of publishing the Availability plugin to your vCD instance. On boot of the CRM vCAv appliance (or during any upgrade), the plugin will refresh or push an update if required – very nice.

Apply your vCAv license key –

Consent or remove the check for the VMware Customer Experience Improvement Program (CEIP) –

Finally, we review our desired state. Verify everything looks to your specification, and hit complete.

This will take a few moments for the configuration. You will be prompted to log back in and you will be brought to the vApp Replication Manager Admin UI page. You can now utilize vCD administrative credentials too!

Let’s click on the Configuration link on the left side. As we can see, we still have some work to do for the Replicator and Tunnel configuration.

Deploy vCAv Replicator(s)

Next up, let’s configure the Replicator instance. Repeat this process for every required Replicator needed for your environment.

Open a tab to https://replicator-fqdn/ui/admin

After setting your password, you will be prompted to set the lookup service address –

That’s it for the replicator! Now, we are ready to pair this replicator with the Replication Manager.

Pairing Replicator with Replication Manager

Open your tab to https://crm-fqdn:8441 and browse to Replicators on the left side –

Let’s click the New button and open up the wizard –

We need to provide the fully qualified domain name along with port 8043 (this is what’s utilized for the Replication Manager to Replicator API connectivity) along with the appliance password and SSO administrator credentials.

Once paired, we will see it in the list. Repeat this process for any additional replicators.

Now, from the CRM Provider UI, we can see a newly added Replicator instance. Next up, Tunnel configuration.

Configuration of Tunnel

Final configuration – let’s configure the tunnel for inbound and outbound connectivity. Browse to https://tunnel-fqdn/ui/admin and login –

Once you set your password, you will be prompted to set two things: 1) lookup service address and 2) Public API endpoint

As discussed before, the public API endpoint will be based off of your network topology. For my lab, I am using direct 8048 access. However, if I was going to DNAT from a public IP/FQDN utilizing 443, I would have the following –

Once completed, we will see the two fields completed.

Let’s hop over to the CRM Provider UI configuration and configure the tunnel –

From here, we need to establish CRM to Tunnel API communication, which happens on port 8047 –

Type in the appliance password. Once applied, we will see a tunnel configuration (again, I was using 443 for a period of time, but you will see 8048 for future configurations).

After any port changes, we recommend doing a service restart. This can be achieved by going to System Monitoring and clicking Restart Service –


After a site deployment and configuration, I always walk through to see service health.

From the main provider UI page, I can see overall system health –

From my System Monitoring page, we can see everything is green and I see my Tunnel and associated Replicators –

From vCloud Director, my plugin is also available too for self-service management.

Final Thoughts:

  1. As depicted above, deployment is rather straight forward and pretty seamless.
  2. Site Deployment must be done on a per-vCD instance basis. So if you have four sites, expect to do this four times.

Next up, Tenant/On-Premises setup.


Utilizing Central Point of Management (CPoM) in VMware vCloud Director 9.7

One of the extremely exciting additions to VMware vCloud Director (vCD) is the ability to present vCenter instances securely to tenant organizations utilizing the vCD user interface – this is referred to as Central Point of Management, or what we abbreviate to as CPoM. Tom Fojta did a great job highlighting what’s new inside of vCD 9.7 here.

In this post, I am going to review the steps required to successfully deploy your first vCenter-SDDC to an organization inside of vCloud Director 9.7. You will need to utilize the CloudAPI, but do not be alarmed, I will walk you through these steps.

First off, what the heck is CPoM?

CPoM is the ability to present a dedicated vCenter instance to an organization, or tenant, via the vCloud Director user interface. In the past, we’ve had many providers providing hosted or dedicated private cloud services.

With the addition of this functionality, we can now utilize vCloud Director to manage both shared and dedicated cloud resources from a single portal.

In essence, the VMware Cloud Provider team is delivering on the promise of transforming vCloud Director into a services platform – and this additional functionality demonstrates that.

So, how do we get started?

High-Level Steps

  1. Add new vCenter instance to vCD Provider Management – UI
  2. Creating a new SDDC instance – API
  3. Publish to specific org/tenant – API
  4. Add Tenant permissions and CPoM Plugin -UI
  5. Proxy Configuration – UI
  6. Enjoy dedicated vCenter-SDDC access via vCD

Adding new vCenter Instance to vCD Provider Management

First up, we need to add this dedicated vCenter instance to the provider management.

My tenant is T1 and will be adding vcsa-01b.corp.local to their organization.

Before, we go into the steps, let’s review the documentation that talks about the high-level steps required.

Managing SDDCs and SDDC Proxies

One important note – you cannot attach an existing vCenter that’s utilized as a provider VDC (pVDC). This is pointed out in the documentation –

First, let’s navigate to vSphere Resources -> vCenters and click the Add button –

Let’s plug in our required connection data –

Next up is the configuration of the NSX-V Manager. I had a conversation with Engineering on this and the applicability for CPoM. While this is not mandatory for accessibility, a tenant can also proxy NSX commands via the proxy configuration. Therefore, depending on your use case, you may or may not want to set this up.

Once completed, we can now see the new vCenter (and what I called vCenter-T1 since it will be dedicated to organization T1) being added to the vCD Provider UI.

Once the operation is complete, we should see this as enabled, but not published to a SP or Tenant.

Creating a new SDDC instance

This has be done via the CloudAPI. First off, let’s explore the new API Explorer which provides some examples and how to parse this utilizing Postman, Curl, or whatever you choose. Navigate to https://vcd-fqdn/api-explorer/provider

Under the sddcs section, we have the following commands available –

Okay, let’s get our bearer-token first. If you are new on setting up Postman for connectivity, Fojta did a great blog post on utilizing the new bearer token with the CloudAPI functionality here.

Get our initial token and pass it through as a variable –

POST https://vcd-fqdn/api/sessions

Now, I want to view the current list of sddcs inside of this vCD instance. I realize we do not have any as of yet, but let’s see what it returns.

GET https://vcd-fdqn/cloudapi/1.0.0/sddcs

As expected, we do not have any SDDCs just yet. Let’s create our first one!

Our POST command will require a JSON body like the below. There are many more fields, but these are the required fields.

  "name":"<name of your new SDDC>",
  "description":"<description for your new SDDC>",
"vcId":"<ID of the VC recorded>"

But before we add it, we have a prerequisite – we need to get the vCenter ID of the newly added vCenter instance.

To do this, we need to run a GET command to vCenter ID. You will need to run the following:

GET https://vcd-fdqn/api/admin/extension/vimServerReferences

Okay, here’s my results –

And here is my “vCenter-T1” ID!

So for my vCenter-T1, my ID is the following:


We will add this to the body for when we add this as a new SDDC.

Creation of my JSON body –

"description":"T1 dedicated vCenter",

Alright, ready to hit the Send button…

And accepted…

I could use the Postman API to check the task, but I also see it running in the UI…


Let’s check and see what we have from the API now. Excellent, we now see vCenter-T1 as a new SDDC –

We also see the new ID for this SDDC:

This will be important as we will need this to publish it to the organization T1. First, let’s check and ensure it’s not published to anyone yet.

GET https://vcd-fdqdn/cloudapi/1.0.0/sddcs/urn:vcloud:sddc:29bf292b-07df-4d2f-9155-97dce2bad95d/tenants

As expected, no one has this from a tenant perspective –

Next pre-requisite – we need to get the orgID so we can create the body for the POST to publish this to T1. Our body will need to look like this:


Okay, let’s do a GET to /api/org –

Now, let’s crack open the T1 org further – /api/org/<org-ID> to get the URN –

So for my T1 organization, my ID is:


Assembling my body for the POST –


Let’s send it!

Got a 200 OK –

Now, if I do a GET to see what tenants/orgs have it..

GET https://vcd-fqdn/cloudapi/1.0.0/sddcs/urn:vcloud:sddc:29bf292b-07df-4d2f-9155-97dce2bad95d/tenants

We can see T1 has it published:

Behold, I can see the new vCenter-T1 from my T1 Org Admin! But why is it showing a red X?

Because by default, when you add the new SDDC, it defaults to false on “enabled”

So we have to enable it, here’s the body for my PUT – (changed it from false to true)

"name": "vCenter-T1",
"id": "urn:vcloud:sddc:29bf292b-07df-4d2f-9155-97dce2bad95d",
"enabled": true,
"vcId": "urn:vcloud:vimserver:419173a9-56bd-471f-929c-3f00c287d21b"


And green check mark in the T1 org admin view!

Add Tenant permissions and CPoM Plugin

By default, the rights bundles and the organization admin global role does not have SDDC access. Also, we need to publish the CPoM Plugin. Let’s do that first.

Under Provider UI, hit the hamburger menu -> Customize Portal, check the CPoM Extension and Publish –

Publish to selected tenants –

Let’s now provide the correct permissions.

On the Provider UI, toggle to Administration -> Tenant Access Control -> Rights Bundles – select the respective rights bundle you need to provide SDDC access and click Edit:

Under Infrastructure – SDDC, we can see the SDDC View and Manage option. Click View –

Ensure this rights bundle is published to your respective organizations –

Now, under Organization Administrator for the Global Role, add this new view permission –

Finally, publish –

Now, any organization admin can see the associated/published SDDC from their login. This can also apply to another role you might deem necessary.

Proxy Configuration and Logging In

There are three steps that need to take place for successful login:

  1. Checking out the default proxy to line up to the vCD VIP/cells
  2. Activating the proxy
  3. Downloading and configuring the proxy configuration file
  4. Importing the .PEM for certificate management

First, let’s GET the existing proxy configuration –


Now let’s get the explicit body for this default proxy –

GET https://vcd-fqdn/cloudapi/1.0.0/sddcProxies/urn:vcloud:sddcProxy:625ddc5a-66ee-4d1a-94a3-603a83604cd2

Now if I wanted to create a specific one, I can do a POST and do the following –

We can now see I have two proxies now. So, this is a great way to explicitly state what you’re trying to do.

Next, click on Manage Proxies –

Activate the default proxy –

Now activated, one can download the proxy configuration file, configure it to your browser and also download the proxy certificate (and upload it).

The proxy configuration file is pretty basic –

We can also see how it does the redirection from the cell to the T1-vCenter (check the bottom URL):

Again, before we can successfully login, ensure you’ve configured your browser for the PAC/proxy configuration. If you click the ? icon, there is a tutorial on configuring it for Chrome, Firefox, etc.

Once clicked, you are prompted for the org user along with the access token, and voila! Login.


I am extremely pleased with this new functionality inside of vCloud Director. While you need to know how to utilize the CloudAPI for initial configuration, it’s quite simple and elegant to utilize with Postman. The API-Explorer provides examples too. This is the first iteration of the CPoM service and will only get better as time progresses.

I believe this is a solid use case for dedicated private cloud alongside existing cloud organization VDCs – great way to provide distinct cloud services in a secure manner.

More to come, enjoy!