VMware vCloud Availability for Cloud-to-Cloud DR – Pairing and Usability (1 of 2)

As a continuation of my vCloud Availability for Cloud-to-Cloud DR (vCAv-C2C) series, we will be covering the initial pairing and usage of C2C in this blog post.

Blog Post – what is vCloud Availability for Cloud-to-Cloud DR? 

Blog Post – vCloud Availability for Cloud-to-Cloud DR – Installation and Setup

I will break this post into the following sections and into two blog posts:

  1. Part 1:
    1. The pairing of the two vCD/vCAv sites
    2. Logging into the vCAv Portal and Portal Overview
    3. Provider view of the vCAv Portal
  2. Part 2:
    1. Initial authentication between the two sites
    2. Migrating a workload in the same site
    3. Protecting a workload between two sites
    4. Testing the protected workload
    5. Edit Options
    6. Failing Over

The pairing of the two vCD/vCAv Sites

  1. First, the Provider must set up an association between the two vCAv sites before the tenant can authenticate within the vCAv portal.
  2. This is done in the Replication Manager portal – point your browser to https://<vcav-rep-mgr>:8046, authenticate, and click on Sites -> New site
  3. We then assign a Site name, the full URL of the respective peer replication manager, and the appliance password. Within my setup, I did this on both sites, SiteA and SiteB. 
  4. Once we hit OK, you will receive a Task succeeded message – 
  5. Now, if we click on the Sites subsection, we can now see our local site along with our newly peered site. From SiteA and SiteB, I can see each respective site. 
  6. Complete! Now we are ready to log into the vCAv Portal.

Logging into the vCAv Portal and Portal Overview

  1. Point your browser to the fully qualified domain name along with port 8443 – this is the default port the portal runs (can be changed). 
  2. Logging in is very similar to vCAv-DR2C – you utilize the username@org-name parameter. For my SiteA, I have an “Org1” while my SiteB is “Org2” – for SiteA, I am logging in as org1admin@org1 with the appropriate password.
  3. Once authenticated, I get the very clean interface that shows the Cloud Topology. We haven’t started any replications/migrations yet, so I don’t have any ingress/egress traffic yet.  
  4. On the home page, you get a very clear view of the topology, workload statistics, and even the Organization VDC status. I thought this was a very efficient use of current oVDC utilization along with what’s being used within each oVDC. 
  5. Under the DR Workloads tab, we would be able to see incoming and outgoing protected workloads. Since we haven’t set up any yet, nothing to show yet. 
  6. Paired Clouds shows available vCD/vCAv instances and their authentication status. Since the Provider paired with SiteB, we have it available but it shows unauthenticated. We still need to authenticate with the appropriate credentials that reside on SiteB. 
  7. Last of all, Tasks will show any previous or current tasks and the event log. This will show logs for any connected site, so very easy to see exactly is going on between the paired sites. 
  8. For those of you that have used vCAv-DR2C, this is very similar to that experience with a few minor enhancements. The learning curve is very minimal and we will go through a few test scenarios.

Provider view of the vCAv Portal

  1. The development team did a great job from a provider view on providing very useful information.
  2. First of all, we get to see a current status ticker that shows state of the vCAv environment. I thought this was extremely useful and intuitive to gain an operational understanding of the health of the current environment. 
  3. I didn’t mention this in the previous portal post, but this is also available there too. You can expand and see the details of the current replications/migrations between sites. This is done by clicking the carrot icon in the top right corner. 
  4. If you scroll down also, you get to see the pVDC view of the current environment with a resource utilization rollup. Very similar experience to what we see now in vCD 9.1 Multi-Site. 
  5. Under the Orgs tab, we get to see the current organization and any registered vApps and current status. 
  6. One very nice item is that shield/checkmark icon. If we click that, we can actually impersonate the organization admin and assist with any tasks that the organization may be having trouble with. All done through this single interface! 
  7. I won’t cover DR Workloads as it’s the same as the tenant view, but just a macro-rollup of all protected workloads. However, under Administration, we get to see Configuration and Registration. From here, we can actually register other components to monitor from this dashboard along with resource threshold display. Very nice! 

Onto the next post, where we will review migrations and protection operations. Thanks!


VMware vCloud Availability for Cloud-to-Cloud DR – Installation and Setup

As discussed in my introduction post about vCloud Availability for Cloud-to-Cloud DR (vCAv-C2C), I am going to do a high-level write-up of my installation/deployment process for this new vCD site-to-site replication and migration solution.

While this write-up does cover the deployment methodology, a production state will need to be rationalized inclusive of certificates, ports, and firewall rules that will be required.

Below are the high-level steps I took for deployment. VMware’s documentation does go in a different order, but this is the way I rationalized it along with a combined role (all-in-one appliance).

High-Level Steps for Installation and Setup:

  1. Download OVF and deploy at each respective site
  2. Configure vCAv-C2C Replication Manager
  3. Configure vCAv-C2C Replicator
  4. Pair Replication Manager with Replicator
  5. Configure vCAv-C2C Replication Service/Manager
  6. Configure vCAv-C2C Availability Portal

Again, my high-level architecture –

vCenter Deployment

  1. Let’s grab the OVF (remember, we only need one for vCAv-C2C), and deploy it to my management cluster – 
  2.  We can see it’s the new Cloud to Cloud appliance – 
  3. As we discussed before, the team has done a great job of simplifying the deployment of vCAv-C2C to a simple and single OVF for all of the roles required for Cloud to Cloud. We can see the drop-down for each respective role. For my deployment, we will be selecting a combined since this is a lab environment. 
  4. We now have to put in the required network configuration while enabling SSH (this is not a mandated requirement, but nice to have in my lab environment). One thing to note – the root password is temporary. When we get into the initial portal configuration, it will prompt us to change it to a new password from a security perspective. 
  5. Final screen to complete the deployment – 
  6. Alright, initial appliance deployment is done! Now off to the configuration.

Configure vCAv-C2C Replication Manager

  1. For the initial configuration, we will want to open a browser to “https://Appliance-IP-address:8044” or in my case, https://vcav-repmgr-01a.corp.local:8044 – 
  2. Let’s click on the Configuration Portal link and now set the new password after a successful login with the password we set in the OVF deployment – 
  3. We can see we have a red x where the lookup service is missing. This is for us to point to the resource vCenter (remember, resource and management vCenters must be in the same SSO domain at each site). 
  4. Let’s go to Configuration -> Set lookup service and put in the lookup service FQDN and we should see a successful message. 
  5. Back to the diagnostics tab, we can now see the lookup service is green!
  6. Now, the next step is correctly set up the replication instance and then come back to the Replication Manager for pairing.

Configure vCAv-C2C Replicator

  1. Open a browser to “https://Appliance-IP-address:8043” or in my case, https://vcav-repmgr-01a.corp.local:8043 – 
  2. Change password – 
  3. Set lookup service – again, I am using a consolidated vCenter instance for resource and management so this is pretty straightforward – 
  4. Success! 
  5. Now, back to the Replication Manager for pairing.

Pair Replication Manager with Replicator

  1. On the Replication Manager, let’s go to Replicators -> New replicator while giving the full FQDN along with the password (remember, new password!) and SSO credentials –
  2. Accept the cert and we should see the success message shortly.
  3. Now, let’s click on Show all managers and we can see the replication instance is now registered – 
  4. While from the replication instance, we can see the replication manager also configured – 
  5. Excellent! Now, off to the Replication Service Manager

Configure vCAv-C2C Replication Service/Manager

  1. Open a browser to “https://Appliance-IP-address:8046” or in my case, https://vcav-repmgr-01a.corp.local:8046 – this time, I used the new password as it seems to have propagated to the other roles. 
  2. Now, we get a nice clean wizard for the setup. 
  3. Let’s put in the Site information, this is going to be my SiteA. 
  4. Next, lookup service again with accepting the certificate – 
  5. Now, we are going to point it to the Replication Manager we previously setup. This is on port 8044, so let’s put that full FQDN along with port 8044 – 
  6. Now, we get to setup vCloud Director. I used the manual configuration here to verify everything was good to go. 
  7. Finally, we get a summary screen to show the stated configuration – 
  8. Excellent! All green and expanding out the Manager data shows our registered replication instance too. Very slick interface. 
  9. Now, off to the Availability Portal configuration as the last setup step.

Configure vCAv-C2C Availability Portal

  1. Home stretch for the first site installation and configuration – let’s open a browser to “https://Appliance-IP-address:5480” or in my case, https://vcav-repmgr-01a.corp.local:5480 
  2. I am prompted for vApp Replication Manager / vCD Connection information. By default, I did see a “” but decided to change it to the FQDN of my combined instance. Put in the vCD credentials (administrator@system) and hit connect to get a successful message – 
  3. Now, click Test to verify everything is operational from a vCD perspective – 
  4. We will be using the defaults for the database, but you are prompted to select a custom database if you so desire – 
  5. Alright, final step. Here’s where we can change the port for user access (remember, this is what will be publicly facing) along with the certificate. I am staying with the defaults for my installation. 
  6. Now, we are ready to hit the Start Service button. Running…
  7. Success! 
  8. Now, we get a nice, simple portal that’s used to verify all of our services are operational 

Wow, a very streamlined process that was very intuitive while using a sleek and simple interface.

I am going to check to verify my org user can log into the portal…

Success! Our vCD authentication was passed through and now I get the vCAv C2C interface. 

Before I can have the paired sites and start testing workload migration, I need to go set up my SiteB vCloud Director instance and deploy the combined appliance. On the next blog post, we will go through the Site pairing process and do a migration/replication.




What’s VMware vCloud Availability for Cloud-to-Cloud DR?

As of yesterday (May 17th, 2018), VMware announced the release of VMware vCloud Availability for Cloud-to-Cloud DR 1.0 – release notes here while my esteemed colleague, Tom Fojta, announced this on Twitter:

Many of you may be wondering, what is vCloud Availability for Cloud-to-Cloud and how may I use this in the VMware Cloud Provider Program?

To start off, vCloud Availability for Cloud-to-Cloud (vCAv-C2C) is VMware’s solution to vCloud Director instance to instance disaster recovery and migration. Here’s a nice summary of what vCAv-C2C provides:

  1. Replicate and recover vApps (VMs) between two vCD instances for migration, DR, and planned migration use cases.
  2. Complete self-serviceability for the provider and tenant administrator. A unified HTML5 portal that will be utilized alongside vCloud Director. Replication, migration, and failover can be managed completely by the tenant or provided as a managed service by the provider.
  3. A simplified and streamlined architecture to support vCloud Director 8.20, 9.0, and 9.1 while supporting vSphere 6.0U3 and 6.5U1.

vCloud Availability for Cloud to Cloud DR Installation Documentation

In my opinion, point #3 is one of the most critical benefits to both providers and tenants. When we discuss multi-tenant architecture, this does tend to add layers of complexity, but the VMware Cloud Service Provider Business Unit has done a great job of rationalizing the architecture and streamlining it for vCAv-C2C and future solutions. I will get to the architecture shortly.

Before we get into the details of vCAv-C2C, many of you have experienced our other migration or disaster recovery-based solutions. I made this simple chart to showcase each of our current VMware Cloud Provider (CSPBU) solutions and how they complement one another:

As you can see, vCAv-C2C will complement the traditional vCAv solution while vCD Extender can still be used for on-prem tenant migrations to a vCD instance. vCAv-C2C fills a void on migration between vCD instances, which is a much-needed capability for our Providers.

So let’s talk about the high-level architecture. As I mentioned before, a lot of thought and development went into vCAv-C2C to make the architecture simplified and seamless. With vCAv-C2C, everything is packaged into a simple OVA deployment – no need to manually/CLI configure a vCAv deployment anymore. I was fortunate enough to be part of the alpha testing team (along with Fojta and my other peer Fernando Escobar) and was very pleased with this capability – ease of deployment and configuration is something that is required for many of our Providers.

Furthermore, this single OVA has every role required for vCAv-C2C. Per the documentation, we have a few roles:

  1. Replication Manager
  2. Replicator Node (Large Replicator role available too)
  3. Tunnel Node

Best of all, there’s a Combined role now that can be utilized for smaller or proof of concept (PoC) deployments. This is what I’ll be using in my lab environment.

Let’s talk about a high-level architecture –

As you can see, this is an appliance-based architecture that will protect (or migrate) vApps between site to site. Moreover, we can simplify this for PoC/small deployments by using a combined vCAv-C2C appliance –

Cloud to Cloud tunneling is utilized if you are going over a public internet connection and do not have private (VPN or Direct Connect) connectivity between the two vCD instances. VMware’s documentation writeup is here along with a nice drawing that depicts the DNAT and port requirements.

As for scale and concurrency guidelines, the team did a great job with support a significant amount of replications/migrations. From the release notes –

  • Scale Limits
    • 300 active protections for a single tenant
    • 300 active protections using a single large vCloud Availability Replicator. For more information about the replicator types, see Deploy vCloud Availability for Cloud-to-Cloud DR Services by Using the vSphere Web Client.
    • 1300 active protections across 20 tenants
    • 20 tenants with active replications
    • 7 active vCloud Availability Replicator instances
    • Up to 2 TB size of protected workloads
  • Concurrency Limits
    • 60 concurrent Protect, Test Failover, Reverse Protect, Test Failback, and Failback operations
    • 110 Concurrent Failover operations

If you’re a provider, you might be wondering how do I download the bits so I can start testing it?! Well, reach out to your respective VMware Cloud Provider field team as this is going to be an initial release and we want to work with our providers on ensuring all vCAv-C2C requirements are met for a successful deployment. You can also reach out to me directly and I’ll be happy to put you in touch with your respective team.

Up next – high-level installation instructions for vCAv-C2C!


vCloud Director Extender 1.1 – Add Permissions Script for Organization Admin

15June2018 Update – note this script is ONLY for vCD Extender 1.1 release. There was a permissions change with version of vCD Extender. PLEASE review this new blog post for the latest script! 

We saw that vCloud Director Extender 1.1 was released a few weeks ago and I’m working with it in my lab. More to come with new features available in this release.

One of the nice things about this newest released is enhanced error state messages and ensuring the right permissions are applied to the organization user before the tenant can connect to the vCloud environment.

So, if we attempt to connect with the org account and it does not have the correct permissions, you will get an error like this:

We can see that the user does not have the exact permission required to successfully enroll with vCD Extender.

These are the VMware Documentation instructions on adding the correct permissions, but this adds ALL provider org admin permissions, which I’m not a fan of (nor are other Providers I support).

I’ve used Jon Waite’s script in the past here but I’ve updated and tested the exact permissions you need for vCloud Director and vCD Extender 1.1 –

Step 1 – Grab PowerShell Script

Here is the current PowerShell script based on my testing on what permissions are needed. I used this in two 9.1 environments and was successful for Extender connectivity. I’m sure this can be ported to other automation tools, but this is what I was comfortable with. Save this as a ps1 file while changing OrgToUpdate and APIendpoint lines below.

# vCloud Director Extender Permissions Setup - initially created by KiwiCloud.Ninja - modified by Daniel Paluszek - paluszek.com
# Creation Date: 2018-May-2nd
# Version 2.0 - for vCD Extender 1.1 and vCloud Director 9.1
# Adds specific permissions required for vCD Extender Org Admin to connect successfully to cloud instance.
# NOTE: These are tested on version vCD
# Note that Organization roles (e.g. Organizational Administrator) still need to be edited to add these rights once is executed
# NOTE: You must be connected to the vCloud API (Connect-CIServer) with a System administrative user prior to running the script for this to work.
# Add your Org name and vCD instance name below
$OrgToUpdate = '&lt;INSERT-ORG-NAME&gt;'
$APIendpoint = '&lt;INSERT-IP-OR-FQDN-OF-VCD&gt;'

Function vCloud-REST(
[string]$Method = 'Get',
[string]$ApiVersion = '27',
[int]$Timeout = 40
$mysessionid = ($global:DefaultCIServers | Where { $_.Name -eq $APIendpoint }).SessionId
$Headers = @{"x-vcloud-authorization" = $mysessionid; "Accept" = 'application/*+xml;version=' + $ApiVersion}
if (!$ContentType) { Remove-Variable ContentType }
if (!$Body) { Remove-Variable Body }
[xml]$response = Invoke-RestMethod -Method $Method -Uri $URI -Headers $headers -Body $Body -ContentType $ContentType -TimeoutSec $Timeout
Write-Host "Exception: " $_.Exception.Message
if ( $_.Exception.ItemName ) { Write-Host "Failed Item: " $_.Exception.ItemName }
Write-Host "Exiting."
return $response
} # Function vCloud-REST End

# Adds required permissions for vCD Extender connectivity - still require to apply permissions in the UI once executed!
$newrights = @{}
$newrights.Add("Organization vDC Gateway: View Firewall", "7fee6646-ec0c-34c9-9585-aff6f4d92473")
$newrights.Add("Organization vDC Gateway: View L2 VPN", "105191de-9e29-3495-a917-05fcb5ec1ad0")
$newrights.Add("Organization vDC Gateway: Configure L2 VPN", "eeb2b2a0-33a1-36d4-a121-6547ad992d59")
$newrights.Add("Organization vDC Gateway: Configure System Logging", "ff3fc70f-fd25-3c0a-9d90-e7ff82456be5")
$newrights.Add("Organization Network: Create or Delete", "60be4106-1f9f-325c-8ff4-8bf2c6d9bc0a")
$newrights.Add("Organization VDC: view metrics", "99bd8096-73d1-3abe-970f-82f08dfbd762")
$newrights.Add("Organization vDC: View", "f66d8e79-b584-3d79-a501-d71aaa2ebbf9")
$newrights.Add("Organization vDC: Extended View", "501166b9-0525-3c0e-87f5-46642c7c650f")
$newrights.Add("Right: View", "66b32e08-1eeb-37ac-9266-ffbd19b39dd8")
$newrights.Add("VCD Extension: View", "af1aa6c1-f693-39b0-baf5-455bd8f6e378")
$newrights.Add("VCD Extension: Register, Unregister, Refresh, Associate or Disassociate", "f41bed42-23e8-3422-abcb-cd78ee0a7cf8")
$newrights.Add("Provider vDC: View", "01e66fc7-c18e-3511-ac3e-89b1ede75585")
$newrights.Add("Catalog: Shadow VM View", "b2480b40-1a09-38e7-a01d-40f1bf76dd72")
$newrights.Add("Catalog: Import Media from vSphere", "fbf21e1d-55c9-30d7-9d21-6ce2e63d2d54")
$newrights.Add("vApp: Import Options", "10bb6775-87b6-327c-95ce-62090fc514cd")
$newrights.Add("vApp: VM Check Compliance", "efcdc465-ed11-3059-850c-2c1436754dca")
$newrights.Add("vApp: Shadow VM View", "7405e842-79f2-31d8-8401-360e2b7947d7")
$newrights.Add("vApp: Enter/Exit Maintenance Mode", "23e2452d-2a8a-3cf2-a95d-0d2cfbb9540d")
$newrights.Add("Datastore: View", "805f2957-f3cf-3faf-aa04-f10c4565c37e")
$newrights.Add("Datastore: Edit", "76a35a35-13fd-3702-9514-6c781c64d47c")
$newrights.Add("Host: Migrate VMs", "b853515b-398c-3658-b622-c45d98c5ecaa")
$newrights.Add("Host: View", "fa553cfc-ab48-349f-b4f3-b836b99717e7")
$newrights.Add("Resource Pool: Open", "3eae48e4-949c-3acf-99fb-eaf4896aefc9")
$newrights.Add("Resource Pool: View", "02d7a15f-2d71-3e23-854e-7a711754f87f")
$newrights.Add("Network Pool: View", "293f322e-c536-3e60-b062-08e9f2cf4efb")
$newrights.Add("Network Pool: Edit", "dca04a78-e9a5-36a3-aee7-a5f298acc34c")
$newrights.Add("Network Pool: Repair", "e323a73b-5140-3092-87aa-0ccb43412606")
$newrights.Add("Network Pool: Create or Delete", "404b4c7d-4ed3-39fa-b9a0-9389d43ec9cf")
$newrights.Add("Task: Update", "f4ac42f6-7458-3180-92f6-8ccf6140f698")
$newrights.Add("Task: Resume, Abort, or Fail", "d9515366-74dc-3e97-bebe-8b080e310549")

$myendpoint = $global:DefaultCIServers | Where { $_.Name -eq $APIendpoint }

if (!$myendpoint.IsConnected) {
Write-Host "Not connected to this vCloud endpoint, use 'Connect-CIServer' before running this script."

$org = Get-Org -Name $OrgToUpdate -Server $APIendpoint

if (!$org) {
Write-Host "Couldn't match organization with name $OrgToUpdate, exiting."

$rightsuri = 'https://' + $APIendpoint + "/api/admin/org/" + $org.Id.Substring($org.Id.LastIndexOf(':')+1) + "/rights"

[xml]$rights = vCloud-REST -URI $rightsuri -ContentType 'application/vnd.vmware.admin.org.rights+xml' -Method 'Get' -ApiVersion '27.0'

# Add the new API v27 'RightsReference' elements to the XML returned:
foreach($newrule in $newrights.Keys) {
$newright = $rights.CreateElement("RightReference", "http://www.vmware.com/vcloud/v1.5")

# Update the Organization with the ammended rights:
vCloud-REST -URI $rightsuri -ContentType 'application/vnd.vmware.admin.org.rights+xml' -Body $rights.InnerXml -Method 'Put' -ApiVersion '27.0'

Step 2 – Check out the existing permissions

Go to your organization’s orgadmin role and take a look at the current permissions. You should see what’s currently available.

From the UI, we would see the following:

One could also procure this through the API by issuing a GET command with


Step 3 – Execute PowerShell Script

I imported the script into PowerShell ISE (I was working on a few modifications so I just ran it there, you can always ./ it too) and executed it.

Remember, you need to fill out lines 8 and 9 with the Organization and vCD instance name respectively.

Let’s execute!

Let’s take a look at the org admin permissions now. We do see new boxes to check.

We can also see the specific permissions required under the vApp category that are required.

Step 4 – Connect via Extender

Based on my testing, this was the only permissions needed for an org-admin user. After testing this in two instances, I was successful with connectivity inside of vCloud Director Extender.

From here, you can continue with your vCloud Director Extender migrations and L2 VPN connections (or what’s called DC Extensions). Special thanks to Jon Waite for his initial script and Sean Smith for his assistance on testing. More to come here, I’ll be reviewing some of the new functionality of vCD Extender 1.1! Thanks.