Overview of VMware vCloud Availability 3.0 – Introduction, Roles, Deployment Process

In this series of blog posts, I will be discussing the new release of VMware vCloud Availability 3.0 (vCAv). This is a very exciting release for the VMware Cloud Provider team. vCAv 3.0 will be out shortly (by end of our fiscal quarter) and I want to provide my insight into this platform. I will be focusing on the following points –

  1. Introduction to vCAv 3.0
  2. High-Level Architecture
  3. vCAv 3.0 Service Roles
  4. Deployment Process
  5. Deployment Approach for Provider
  6. Deployment for Tenant (On-Premises)
  7. Protection Workflow
  8. Resources

Introduction to vCAv 3.0

First off, let’s discuss what vCAv provides from a functionality perspective. vCAv is what I like to call “functional convergence.” In the past, we had many different products that provided some level of availability or migration capability –

In my opinion, this was a duplication of appliances and could be confusing to customers. The team has done a great job of putting forth a significant investment into vCAv 3.0 to simplify the architecture. Therefore, here’s where we are today –

Therefore, no need for multiple tools for migration, DR from/to Cloud, or between Clouds. We now have a single solution that answers all of the above.

vCAv 3.0 Key Functionality

So what does vCAv 3.0 provide?

  1. Simple, Unified Architecture
    1. There is a single OVF for the Provider and the Tenant. On the Provider side, each role can be easily deployed used by the vSphere Client or CLI.
    2. Deployment is very intuitive and scalable – each role can be quickly deployed in a matter of minutes.
    3. On-Premises Appliance is unified and provides vCenter UI integration for management.
  2. On-Premises Migration and Protection
    1. On-Premises appliance provides the same UI experience as connected to the Cloud instance, but built into vCenter.
    2. Migration and/or protection of workloads can be done with a few clicks.
    3. Allows for protection/migration to and from vCloud instances.
  3. Cloud to Cloud Migration and Protection
    1. Very similar behavior to C2C 1.5, we can protect workloads (vApps and VM’s) between vCD instances.
  4. Network Mapping and Re-Addressing
    1. One of the new additions is the ability to re-address and map out protected workflows for faster recovery at the destination site.
    2. This maps to existing vCD oVDC Network constructs such as Static IP pools.
  5. Scale
    1. As discussed before, the team is aware of scalability requirements for Providers. For this version, here is the stated guidelines:
      1. 300 tenants with active protections paired to vCloud Director instance
      2. 20 vCAv Replicator instances per instance
      3. 500 active protections per vCAv Replicator instance
      4. 9,500 active protections across tenants to a Cloud
      5. 5TB protected VM size (contingent upon Cloud storage)

Business Value

Direct and native vCloud Director integration for providers and tenants – with the ability to provide self-service, this provides a unique experience that meets DRaaS requirements and migration functionality.

Ease of Operationalization – I’ve done several deployments during the development process and it’s one of the easiest VMware Cloud Provider solutions to deploy. Once we review the roles and concepts, anyone should be able to operationalize this with ease.

Cost-Effective Approach – vCAv 3.0 will be part of the VMware Cloud Provider Program (VCPP). This is based on the monthly consumption of points, which is a very cost-effective solution that can be modeled and productized for DRaaS and migration offerings.

High-Level Architecture

In the following diagram, we can see how this all comes together with a single, on-premises vCenter along with two vCD instances. One can pair this up with vCD multi-site federation capability.

Moreover, this pairs very well with existing vCD services such as CrossVDC Networking or L2VPN connectivity between on-premises and organization VDCs.

vCAv 3.0 Service Roles

Let’s review the distinct service roles of vCAv 3.0.


On the Provider side, we have the following –

  1. Cloud Replication Management
    1. This is a logical entity that consists of the core of vCAv 3.0.
    2. vCloud Availability Portal – User Interface for the tenant and provider. All UI configuration ingresses from this service and applies all to all necessary connected components.
    3. vCloud Availability vApp Replication Manager – communicates directly to vCD and understands tenancy constructs such as organizations, vApps, etc. Also responsible for enabling protections or migrations.
    4. vCloud Availability Replication Manager – understands vCenter and ESXi concepts and will interoperate between the replicators and protected vCenters.
  2. vCloud Availability Replicator – lightweight node responsible for executing on the host-based replication from a specific host. Typically, you deploy a replicator per vCenter.
  3. vCloud Availability Tunnel – this is the tunneling service that is responsible for providing secure connectivity between on-premises vCenter(s) and connected vCD instances.

Each of these roles can be deployed separately or in a combined virtual appliance. For production deployments (which we will review later), the recommendation is standalone deployments for each role.

Tenant / On-Premises

On the tenant side, we have a single appliance that has a combined appliance approach –

  1. vCloud Availability Replicator – just like on the Provider side, the Replicator is responsible for executing the host-based replication (HBR) process
  2. vCloud Availability Tunnel – provides secure connectivity between on-premises and vCloud environment. All traffic securely ingresses and egresses through this service.
  3. vCloud Availability Plugin – this plugin provides local vCenter UI management that is the same experience as connecting the vCAv Cloud environment.

Deployment Process

While this blog series will cover the Provider and On-Premises side in further detail, we will have the following steps to execute on for a successful deployment.


  1. Deployment of Cloud Replication Management (CRM) Instance
  2. Deploy vCAv Replicator(s)
  3. Deploy vCAv Tunnel
  4. Configuration of CRM instance and start of site wizard
  5. Configuration of Replicator
  6. Configuration of Tunnel
  7. Validation


  1. Deploy vCAv On-Premises Appliance
  2. Start configuration wizard
  3. Connect to vCAv Cloud Tunnel
  4. Configuration of local placement
  5. Validation

Next up, I will review the Provider deployment process in further detail while providing the step-by-step procedures. Stay tuned!


VMware vCloud Director – Installation of PostgreSQL and Migration from Oracle

In one of my lab instances, I currently have Oracle still running as my backend vCloud Director database. In this post, I am going to document the steps it takes to install Postgres10 and migrate away from Oracle.


First, taking a snapshot of my vCD instance – always back up before making any type of database changes! 🙂

Next, my system is a little dated, so I am running a yum update to get all of the latest binaries before we install PgSQL.

I am also running RHEL, so your steps may different based on your distribution.

Installing and Starting PostgreSQL 10

My esteemed colleague, Sean Smith, wrote a nice post on setting up an all-in-one vCD appliance here so I am going to borrow his steps on installing PgSQL 10.

Get the RPM and start the install –

rpm -Uvh https://yum.postgresql.org/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
yum install postgresql10-server postgresql10

Let’s initialize the database –

service postgresql-10 initdb

From there, using chkconfig to enable it to start on boot –

chkconfig postgresql-10 on

Ready to start!

service postgresql-10 start

Let’s change the password to the default postgres account (ignore my superweak password) –

Before we can make the authentication change, we need to set the default postgres user password.

Switch user to postgres and type in psql followed by a \password to set it –

su - postgres

\q and go back into the root console.

Finally, we need to allow authentication to the database. I am going to allow full access to local and remote logins to the database.

Edit the /var/lib/pgsql/10/data/pg_hba.conf file and modify this line –

local all all  peer


local all all md5

And adding this line below –

host all all md5

Now, edit the postgresql.conf file and remove the # from the ‘listen_addresses line –

Finally, restart the postgresql-10 service –

We are now ready for the next step which is creating the new vCloud database that we can move over to.

Setting up the new vCloud database on PostgreSQL 10

We are now ready to create our new database and prepare it for the migration.

First, let’s switch user over to the postgres account and enter psql –

su - postgres

We need to create the vcloud account with a password –

create user vcloud with password 'vcloudpass';

Now I’m ready to create my vcloud database. I already have my vcloud user account on the system, so no need to create that again. Following these instructions from the VMware master docs.

create database vcloud owner vcloud;

Finally, altering it so it enables the database owner on login:

alter role vcloud with login;

From here, one can setup SSL for secured communication. Since this is my lab, I’m going to skip over that configuration.

Stopping the vCD Instance and Migrating

Let’s stop the vCD service –

Now we can follow the instructions here on the documentation on using the cell-management-tool for dbmigrate

cell-management-tool is under /opt/vmware/vcloud-director/bin –

Now we are ready to run the cell-management-tool dbmigrate command. For me, this was my configuration – it will differ based on your setup.

./cell-management-tool dbmigrate -dbhost vcd-01a.corp.local -dbport 5432 -dbuser vcloud -dbname vcloud -dbpassword vcloudpass



Ready now to run the reconfigure-database command, and boom! Complete.

/opt/vmware/vcloud-director/bin/cell-management-tool reconfigure-database -dbhost vcd-01a.corp.local -dbport 5432 -dbuser vcloud -dbname vcloud -dbpassword vcloudpass -dbtype postgres

Let’s start back up vCD….

We are back up and running!

Lessons Learned

  1. While this was not a difficult task, every distribution is different, inclusive of Sean’s post where he did the installation and setup of PostgreSQL-10.
  2. The cell-management-tool works great for database migrations to PostgreSQL-10.
  3. Note that I did not setup SSL communication. This requires further steps to set it up. Sean did a great job on the steps here.
  4. Test, test, test before you do this in production.



VMware Cloud on AWS Management Exam 2019 5v0-31.19 – Exam Review and Tips

VMware and Amazon Web Services are making a significant investment in providing a vSphere experience in a hyperscaler environment and we continue to see adoption for Cloud Service Providers utilizing VMware Cloud on AWS as an offering for managed services.

In this post, I will review my approach, what to expect on the exam, and my raw study notes.

I decided to prepare for the VMware Cloud on AWS Management Exam 2019 (Exam 5v0-31.19) and sit for it. While there’s a lot of great material that I will review below, I wanted to provide my approach for others.

My Approach

  • Always start with the blueprint. I print this out and go through line to verify I have a base level understanding of the objective and what I need to focus on.
  1. First off, there is some great material already online that I reviewed:
    1. Paul McSharry‘s writeup on the material and his mindmap which is here.
    2. Manny Sidhu did a great writeup of his notes and approach here.
  2. I went through the VMware Cloud on AWS: Deploy and Manage – On Demand course on VMware Education.
    1. This was beneficial as it covered a lot of the fundamentals including some AWS concepts I was not aware of.
    2. I would highly recommend this for people even new to VMware vSphere as it reviews many of these concepts at a high-level.
  • Went through the VMware Hands on Lab for VMware Cloud on AWS (1987-01-HBD) – this is great to get your hands-on and experience setting up a SDDC. Walks through many of the same concepts I saw in the Deploy and Manage course.


It’s 30 questions and is done via the Pearson Vue site online. I thought it was very fair as a skill/badge exam and goes over many of the fundamental requirements. As expected, it’s true to the blueprint.

My Raw Notes

For me, I always prepare a final checklist of things I review before I take any examination. Below is my list of raw notes I used before I took the exam. As they are my raw notes, expect some abbreviations.

Anyway, enjoy the exam and I look forward to the expansion of VMware Cloud on AWS!

VMC on AWS Study Notes:

  1. Use Cases:
    1. Extension of onprem DC to the public cloud to expand resource capacity, increase disaster avoidance and recovery options, or localize application instances.
    2. Consolidation
    3. Peering the private and public cloud that allows for application mobility
  2. Global compliance – ISO, SOC1-3, GDPR, and HIPAA
  3. Many different AWS regions available plus GovCloud
  4. SDDC
    1. Minimum of 3 hosts and maximum of 32 hosts
    2. Up to 10 clusters can be added to a SDDC
    3. Stretched Cluster – between two AZ’s (min of 6 hosts and max of 28)
  5. Host Configuration
    1. 2 18 core sockets – Broadwell 
    2. 512 gibibytes of memory (550GB)
    3. 14.3TB of NVMe SSD’s – 3.6TB for flash, 10.7TB for capacity
    4. 1 AWS ENA – 25Gbps
  6. vSAN/Storage
    1. Two Disk Groups per host
    2. Dedupe/Compression is on by default
    3. Encryption is happening at the drive level
    4. Two different datastores – WorkloadDatastore (for workloads) and vsanDatastore (management VMs, cannot be modified)
    5. Default policy is PFTT 1, RAID1
    6. Can pick from PFTT of 1 to 3, RAID1/5/6 (if available hosts)
    7. Since All Flash, reads are from cap tier
  7. Stretched Cluster
    1. Sync writes between two AZ’s
    2. Witness host is added and not charged to customer
    3. Pre-reqs
      1. Requires a AWS VPC with two subnets, one subnet per AZ
      2. Smallest SC is 6, largest is 28
      3. Must grow in pairs
      4. Adding hosts trombones between AZs – first one is added to AZ1, next is AZ2, next is AZ1, and so on
    4. Site Disaster Tolerance – default is dual site mirroring 
  8. Network
    1. Traffic is separate between management and compute gateways
    2. Amazon Direct Connect allows for low-latency connections between on-prem and AWS. 
    3. ENA’s are highly redundant even though there’s a single pNIC per host
    4. Two types of VPCs
      1. One created and managed by VMware – underlying VPC that is created when the SDDC is created 
      2. Second – VPC that you create so you can peer with native AWS services 
    5. Firewall
      1. Default deny all
      2. Must add rules for vCenter access, IPsec, etc
      3. Firewall Rule Accelerator created a group of rules to accelerate successfully connecting a VPN tunnel
    6. Logical Networks
      1. Routed network – internal communication over a IPsec or Internet connection. Normal overlay network that we are used to. 
      2. External Network – utilized for L2VPN connectivity. Requires Tunnel ID. Think of this as a subint
    7. Inter-Networking Scenarios
      1. Compute GW – IPsec for guest OS connectivity
      2. Compute CW – L2VPN for vMotion, same L2 domain 
      3. Direct Connect with pub virtual interface – in conjunction with IPsec or L2VPN or Pub Internet. Used for AWS services
      4. Direct Connect with private virtual interface – secured to direct SDDC
  9. Hybrid Linked Mode
    1. Allows for a single management interface between on-prem vCenter and VMC
    2. Pre-req for migration from on-prem to VMC
    3. Same SSO is not needed 
    4. Configuration is only done from one of the vCenters to configure HLM. Will only be visible from this vCenter for future management. So, no bi-directional UI support. 
    5. Pre-reqs
      1. IPsec VPN connection between on-prem and SDDC management gateway
      2. Network connectivity between your VMC vCenter and on-prem vCenter server and identity source
      3. Same DNS 
      4. Less than 100ms RTT
      5. Misc ports needed for successful connectivity 
    6. vCenter Cloud Gateway Appliance configured HLM. 
  10. HCX
    1. Can do migrations between vSphere 5.1 to VMC
    2. No charge
  11. VPC
    1. Only one VPC can be connected to a SDDC
    2. VPC subnets can only reside in one AZ. 
    3. Elastic IP addresses are public IPv4 addresses mapped to the AWS account, not the resource.
    4. Connecting – 
      1. Must connect a Amazon VPC or if it’s a single node SDDC, can delay up to 14 days.
  12. Migrating VMs
    1. Cluster EVC and Per-VM EVC 
      1. In 6.7, can enable disable or change the EVC mode at the VM level. 
    2. Requirements for Hybrid Cold Migration
      1. vSphere 6.5 patch d or later, 6.0U3, vSphere 5.1/5.5
      2. IPsec VPN
      3. HLM but can use move-vm cmdlet 
    3. Hybrid Migration with vMotion
      1. Minimum bandwidth of 250Mbps and less than 100ms RTT
      2. vSphere 6.5 patch d / vSphere 6.0U3
      3. IPsec VPN
      4. vCSS/vDS 6.0 or 6.5
      5. AWS DC with a private virtual interface 
      6. HLM or move-vm cmdlet 
      7. L2VPN to extend VM networks between on-prem and VMC
      8. All FW rules in order. 
      9. VM hardware version 9, Cluster based EVC baseline on Broadwell
    4. Per-VM EVC
      1. Must be hardware version 14 or greater 
      2. VM must be powered off to change Per-VM EVC
  13. Permissions and Security
    1. CloudAdmin –
      1. Necessary privileges for creating/managing workloads in the SDDC
      2. Does not allow changing the configuration of management components that are supported by VMW
    2. CloudGlobalAdmin – 
      1. Associated with global privileges that allows you to create and manage content library objects and perform other global tasks. 
    3. cloudadmin@vmc.local is the default user generated during creation. 
    4. Other users cannot be created until HLM is configured. DO NOT modify solution users associated with the VMC created in an on-prem vSphere domain
  14. Elastic DRS
    1. Allows the SDDC to scale based on resource thresholds 
    2. Not supported for multi-AZ deployment or single host SDDC
    3. If a user adds or removes a host, current EDRS remediations are ignored 
  15. Licensing/Pricing
    1. On-Demand, One-Year and Three-Year Subscription models
    2. HLP discounts of up to 25%
    3. Site Recovery is an add-on cost
    4. All other AWS services are billed separately
  16. Cloud Services Roles
    1. Organization Owners – 
      1. Can have one or more
      2. Owners can invite additional owners and users, manage access
    2. Organization Users –
      1. Access VMware Cloud services
      2. Cannot invite users, change access, or remove
  17. Deployment
    1. Default Subnet CIDR is – reservations for other RFC1918 addresses
    2. is reserved for default compute
    3. Maximum hosts are dictated by the CIDR block you state
  18. Content Libraries 
    1. Onboarding Assistant is a java CLI tool for transferring to VMC
    2. Can still utilize subscribe functionality
    3. Utilize vSphere Client to upload files
  19. Site Recovery
    1. vSphere Replication based
    2. Supports Active-Active, Active-Passive, Bidirectional
    3. Pre-Reqs
      1. vCenter 6.7/6.5/6.0U3, ESXi 6.0U3 or later
      2. SRM 8.x on-prem

How does VMware vCloud Availability for Cloud-to-Cloud 1.5 interoperate with vCD Cross-VDC Networking?

I get this question quite a bit due to the new vCloud Director 9.5 Cross-VDC networking functionality – does vCloud Availability for Cloud-to-Cloud 1.5 (C2C) work with stretched networking inside of Cross-VDC networking?

The answer is: yes!

This is another great addition for recoverability considerations as one could fail over between vCloud Director instances without modifying the guest OS IP address. Furthermore, based on the application architecture, one could have active-active applications and ensure replication/failover in the event of a disaster.

Let’s go through my example high-level design I’ve worked up in my lab –

Example Cross-VDC setup with vCloud Availability for Cloud-to-Cloud 1.5

In the above diagram, we can see I have two active vCloud Director instances, Site-A and Site-B. I have two organizations, “Daniel” that resides in Site-A along with “Daniel-B” that resides in Site-B.

C2C is deployed on each site in the combined model and I have multi-site pairing completed so I can easily manage this between my two sites –

Within my Cross-VDC networking setup, I currently have my active egress setup to Site-A as depicted in the diagram above.

Last of all, I ran a protected workflow from Site-A to Site-B for my Stretched-vApp-VM –

From there, one can either migrate or failover the workload and without any guest OS IP changes. I am going to do a video shortly, but here’s a short GIF I created that shows the ease of use of failing over between my Site-A and Site-B –


After failover, I can then access Stretched-vApp-VM from the new NAT address on Site-B.

An Organization Administrator could also configure active/active or active/passive egress points for additional resiliency. This provides further capability inside of vCloud Director, especially with stretched networking and a complementary availability solution.