vCD Extender – Warm Migration Setup – 2 of 2

Continuation of vCD Extender – Warm Migration Setup – 1 of 2

3. Configure L2 Appliance configuration inside of tenant Extender Appliance

Alright, now we are ready to set up the DC Extension on the tenant side. First, we need to tell the tenant Extender appliance where to position the L2VPN-Client edge in the tenant environment.

First, let’s log into the Extender tenant appliance:

Click on DC Extensions and Add Appliance Configuration (top button) – remember, our tenant does not have NSX (but could as you see below!) so we will have Extender deploy a standalone L2VPN Client edge.

Now let’s put in the configuration parameters for the L2 Appliance. As you can see, this is for my “Company-Network” so I set its IP, default GW, and the prefix length. Make sure you hit the Add button before hitting Create:

Ok, great, now we have an L2 Appliance Configuration:

Next, let’s get into the Plugin and establish our DC Extension.

4. Establish DC Extension in Extender plugin

Last step before we can start migrating! This is assuming you have your provider cloud established – let’s now go to DC Extensions

Click on New Extension –

Put in a name, source info (you can see I selected my Company-Network) and Destination will populate your vDC along with the subinterface Network. If you do not see any Network, something went awry with the trunk interface, check it out.

Also, note that we added the local Egress default gateway IP to enable the L2VPN-Client to route any packets sent towards the Egress Optimization IP address locally, and send everything else over the tunnel. If the default gateway for the virtual machines (belonging to the subnets you’re stretching) is same across the two sites you will need this setting to ensure traffic will be locally routed on each site. Should you need to migrate VM from one site to the other, you can do so without changing the network configuration on the guest VM and hence the mobility.

Now click that Start button…

Now we will see the status as Connecting…

So what’s happening in the background? Well, Extender is now deploying the L2VPN standalone edge and also creating a Distributed Port Group for the sink traffic:

On the Provider side, what do we have here? We have a new site config under the L2VPN-Server:

Really slick! So Extender automatically creates a new config based on this configuration.

Going back to the tenant, I see the IP was set to the L2VPN-Client Edge and the sink port was configured completely:

Taking a closer look at the DVS port Extender created, we can see Promiscuous Mode and Forged Transmits were set to Accept:

Oh, and look at that – it’s Connected! Woohoo!!!

On the Provider NSX side, we can see the tunnel is up and kicking:

COMPLETE! You are now ready for your first warm migration.

Again, big thanks to my buddy Wissam for the collaboration here.

Next – let’s do a warm migration!

vCD Extender – Warm Migration Setup – 1 of 2

Hey everybody! I appreciate all of the interest in vCloud Director Extender – today I will be walking through setting up vCD Extender for warm migrations. This is going to be a lengthy post, but very straightforward once you go through the steps.

First off, what is a “warm migration?” Well, a warm migration in vCD Extender allows you to continue to replicate a VM until you’re ready to cut it over. The goal of a warm migration is to minimize the downtime required for this migration. This is NOT a cross-vCenter vMotion: Extender replicates it using the H4 replication system and then requests you to cut it over. Again, the goal is to minimize the downtime to minutes, if not seconds.

Big thank you to my peer Wissam Mahmassani – he and I worked on this together to demonstrate functionality.

To start, I’m going to list out the high-level steps on setting this up.

  1. Advanced Gateway Services enabled on vCD Tenant
  2. Create trunked network inside of org VDC and setup L2VPN-Server
  3. Setup DC Extension placement inside of tenant Extender Manager
  4. Establish DC Extension in Extender plugin
  5. Warm Migrate away!

Before we get started, here’s what we are going to build out. If you are looking for how Extender is logically laid out, please check out my initial Extender post here.


This is the same Extender/vCD Topology as before – pretty flat demo environment, but I’m using Company as my tenant now.

1. Advanced Gateway Services Permissions Setup

The organization administrator will require advanced entitlements for setting up the L2VPN connection and Extender functionality. This is a little painful to do manually, so after further consideration, I decided to take Jon Waite’s PowerShell script and modify it (Thank you so much, Jon!) – read more here on how Jon created this.

Note that I am doing this in a vCD 8.20 environment. Permissions are similar but I believe 9.0 may have these by default (need to verify this)DCP 17Jan2018: Wrong – per my testing, I modified the below script for vCD 9.x environments here. As of this blog post, 9.0 does not have warm migration functionality – this will be added in a patch release.

Below is the PowerShell script I modified and will demonstrate how this works.

# vCD Extender Permissions Setup - initially created by KiwiCloud.Ninja - modified by Daniel Paluszek -
# Script to add new OrgRights options for administering advanced Edge Gateway to a vCloud Director organisation.
# Note that Organisation roles (e.g. Organizational Administrator) still need to be edited to add these rights once
# this script has been run against their org.
# NOTE: You must be connected to the vCloud API (Connect-CIServer) with a System administrative user prior to running the script for this to work.
# Add your Org name in line 7 while the vCD instance name is added in line 8
$OrgToUpdate = ''
$APIendpoint = ''

Function vCloud-REST(
[string]$Method = 'Get',
[string]$ApiVersion = '27.0',
[int]$Timeout = 40
$mysessionid = ($global:DefaultCIServers | Where { $_.Name -eq $APIendpoint }).SessionId
$Headers = @{"x-vcloud-authorization" = $mysessionid; "Accept" = 'application/*+xml;version=' + $ApiVersion}
if (!$ContentType) { Remove-Variable ContentType }
if (!$Body) { Remove-Variable Body }
[xml]$response = Invoke-RestMethod -Method $Method -Uri $URI -Headers $headers -Body $Body -ContentType $ContentType -TimeoutSec $Timeout
Write-Host "Exception: " $_.Exception.Message
if ( $_.Exception.ItemName ) { Write-Host "Failed Item: " $_.Exception.ItemName }
Write-Host "Exiting."
return $response
} # Function vCloud-REST End

# The new vCloud Director API v27.0 OrgRights for vCD Extender Preparation and Advanced Networking:
$newrights = @{}
$newrights.Add("Hybrid Cloud Operations: View from-the-cloud tunnel", "629c90fd-78a4-3929-98bd-57e4747d067b")
$newrights.Add("Organization vDC Distributed Firewall: Enable/Disable", "a100f6a0-2c81-3b61-90c3-c4dbd721b3a8")
$newrights.Add("Organization vDC Gateway: Configure BGP Routing", "2c4eb5ac-15f5-33f0-8b4a-680b3a1d3707")
$newrights.Add("Organization vDC Gateway: Configure DHCP", "be1abe9a-7ddc-38f6-bdf3-94affb01e46b")
$newrights.Add("Organization vDC Gateway: Configure Firewall", "b755b050-772e-3c9c-9197-111c286f563d")
$newrights.Add("Organization vDC Gateway: Configure IPSec VPN", "209cde55-55db-33f1-8357-b27bba6898ed")
$newrights.Add("Organization vDC Gateway: Configure L2 VPN", "eeb2b2a0-33a1-36d4-a121-6547ad992d59")
$newrights.Add("Organization vDC Gateway: Configure Load Balancer", "27be9828-4ce4-353e-8f68-5cd69260d94c")
$newrights.Add("Organization vDC Gateway: Configure NAT", "c9e19573-3d54-3d4a-98f2-f56e446a8ef9")
$newrights.Add("Organization vDC Gateway: Configure OSPF Routing", "3b337aef-42a8-3ed1-8616-341152bc5790")
$newrights.Add("Organization vDC Gateway: Configure Remote Access", "72c5e652-c8d7-3f19-ab83-283d30cb679f")
$newrights.Add("Organization vDC Gateway: Configure SSL VPN", "92b7d500-6bb6-3176-b9eb-d1fda4ce444d")
$newrights.Add("Organization vDC Gateway: Configure Static Routing", "f72af304-97b0-379e-9d6d-68eb89bdc6cf")
$newrights.Add("Organization vDC Gateway: View BGP Routing", "d9dabcab-579e-33c5-807b-dc9232bf7eff")
$newrights.Add("Organization vDC Gateway: View DHCP", "8e16d30d-1ae3-3fff-8d4b-64c342b186a9")
$newrights.Add("Organization vDC Gateway: View Firewall", "7fee6646-ec0c-34c9-9585-aff6f4d92473")
$newrights.Add("Organization vDC Gateway: View IPSec VPN", "82beb471-ab7f-3e2b-a615-136ba6645525")
$newrights.Add("Organization vDC Gateway: View L2 VPN", "105191de-9e29-3495-a917-05fcb5ec1ad0")
$newrights.Add("Organization vDC Gateway: View Load Balancer", "2a097e48-f4c4-3714-8b24-552b2d573754")
$newrights.Add("Organization vDC Gateway: View NAT", "fb860afe-2e15-3ca9-96d8-4435d1447732")
$newrights.Add("Organization vDC Gateway: View OSPF Routing", "eb525145-08e5-3934-91ef-ec80837c9177")
$newrights.Add("Organization vDC Gateway: View Remote Access", "65439584-6aad-3c2c-916f-794099ee85bf")
$newrights.Add("Organization vDC Gateway: View SSL VPN", "cdb0edb0-9623-30a8-89de-b133db7cfeab")
$newrights.Add("Organization vDC Gateway: Configure Services", "b080bb50-cff1-3258-9683-842d34255a95")
$newrights.Add("Organization vDC Gateway: Configure Syslog", "84ddb40f-a49a-35e1-918e-3f11507825d7")
$newrights.Add("Organization vDC Gateway: Configure System Logging", "ff3fc70f-fd25-3c0a-9d90-e7ff82456be5")
$newrights.Add("Organization vDC Gateway: Convert to Advanced Networking", "9dc33fcb-346d-30e1-8ffa-cf25e05ba801")

$myendpoint = $global:DefaultCIServers | Where { $_.Name -eq $APIendpoint }

if (!$myendpoint.IsConnected) {
Write-Host "Not connected to this vCloud endpoint, use 'Connect-CIServer' before running this script."

$org = Get-Org -Name $OrgToUpdate -Server $APIendpoint

if (!$org) {
Write-Host "Couldn't match organization with name $OrgToUpdate, exiting."

$rightsuri = 'https://' + $APIendpoint + "/api/admin/org/" + $org.Id.Substring($org.Id.LastIndexOf(':')+1) + "/rights"

[xml]$rights = vCloud-REST -URI $rightsuri -ContentType 'application/' -Method 'Get' -ApiVersion '27.0'

# Add the new API v27 'RightsReference' elements to the XML returned:
foreach($newrule in $newrights.Keys) {
$newright = $rights.CreateElement("RightReference", "")

# Update the Organization with the ammended rights:
vCloud-REST -URI $rightsuri -ContentType 'application/' -Body $rights.InnerXml -Method 'Put' -ApiVersion '27.0'

Well, why are we doing this? By default, the org admin does not have these permissions added to the default role. So we need to add them. In this example, I will walk through what you’ll see before and after.

In this example, I’ll be adding the required permissions to org ‘T1’

Before we run this PowerShell script, here are the current permissions of the Organization Administrator. As you can see, there’s no section for Gateway Advanced Services settings.

Let’s go ahead and execute the PowerShell Script. Just make sure you have the latest PowerCLI pack installed – you will need to connect using “Connect-CIServer” first before execution.

Now login into your vCD cell –

Change the PS script to your specific parameters, as I pointed in the arrows below:

Now execute!

Now I see a whole new mess of rights! Go ahead and grant the rights to the Org Admin by checking the boxes. 

2. Create trunked network inside Org VDC and Setup L2VPN-Server

On to Step 2 – let’s go ahead and create an Org VDC network, convert it to a subinterface, and set up our L2VPN-Server.

Let’s log into our Company and create a new Org VDC Network

Setting the gateway for my 172.16.10.x Web Network I will be using.

Naming it ‘company-network’

Go ahead and right click – Convert to subinterface:

Continue on…

Now we see company-network as a subinterface:

Now we are ready to configure our L2VPN-Server, right click and Configure Services:

Go to the VPN tab, L2 VPN – then I start with Server Sites:

Add your peer site information and ensure you select your stretched interface:

Ensure you are setting this as the Server, select your Algorithm, Enabled is green, and Save Changes!

Ok, great! We have 2 of the 5 steps complete. We can also see from the Provider vCenter the VPN is set up, but down right now.

On the next post, we will finish up the setup.

Next Post -> vCD Extender Warm Migration Setup – 2 of 2


vCloud Director Extender – Installation Review

I’m happy to announce VMware has released vCloud Extender for vCloud Director. With that said, I was given the opportunity to provide feedback to our talented engineering and product management teams on installing and reviewing this new valuable solution set.

To start – what is vCloud Extender? vCloud Extender allows tenants to seamlessly migrate workloads to a vCloud Director environment. This is without any net new infrastructure or software purchase: the tenant just needs to add two vApps while the provider would add three – very streamlined. For a provider to get started with Extender, you just need vCloud Director for Service Providers.

Check out VMware’s Introduction to vCloud Director Extender video:

The installation of Extender is extremely streamlined for the tenant and provider environments. This post will go through the installation steps and requirements for an initial successful Extender installation.

Link for vCloud Director Extender download here 

vCloud Director Extender Documentation Link

First off, let’s talk about architecture. I covered this in a previous blog post, but it wouldn’t hurt to reiterate a few of the points. If you haven’t seen the previous blog post on vCloud Extender, check it out here.

On the provider side, we have the following workflow:

  1. Management vCenter deploys Extender Manager appliance. This provides the functionality for managing Extender.
  2. Extender Manager does the following…
    1. Registers to vCD and management vCenter instance
    2. Deploys and activates Replication Manager
    3. Deploys and activates Replication Instance
      1. Replication Instance is then pointed to Resource vCenter (think the Consumption environment where the tenant resides)
  3. A few points of interest:
    1. Proxy Server – you can access the Replication Manager and Replication Instance through a proxy server or a gateway. One of the requirements are to provide a proxy with a public endpoint and configure rules to route the network traffic to the replication components.
      1. This would allow you to have vCD, Extender Manager, and a Reverse Proxy in the DMZ while the replication instances and Replication Manager are in a private management network behind an Edge/FW.
    2. Control traffic – traffic between the Extender instances and replicator instances – all run over HTTPs / 443 traffic. It’s important to note that the on-prem replication instance and the Replication Manager must have bi-directional 443 communication. This is something to ensure your tenants are aware of when planning for installation.
    3. Replication Traffic goes over encrypted TCP which is on port 44045.

On the tenant side, it’s pretty straight forward. We have two appliances to deploy:

  1. Extender Appliance – manages the association with the tenant vCenter instance along with replication instance deployments.
  2. Replication Appliance – is deployed from Extender Appliance and controls the migration (warm or cold) of VM’s. This is based on our new H4 engine – next generation vSphere Replication engine.

From the tenant perspective, we would have something like this:

High Level Steps for installation:

  1. Provider
    1. Deploy Extender appliance in SP management vCenter environment.
    2. Bring up the Extender Manager UI (HTML access)
    3. Start the Configuration Wizard
      1. Associate it with the Management vCenter (where your other management appliances will reside)
      2. Register with vCD
      3. Register with Resource vCenter instance
      4. Deploy Replication Manager – then activate
      5. Deploy Replicator Instance – then activate
    4. Complete – verify Extender sees all connected resources.
    5. For L2 VPN Network Span, this is done by having administrator privileges to the L2 VPN configuration in vCD. From there, we need to establish a L2 VPN Server and L2 VPN Client (Client would be on the tenant environment). A future blog post will cover this in further detail.
  2. Tenant
    1. Deploy Extender appliance in tenant vCenter environment.
    2. Bring up the Extender UI (HTML)
    3. Start the Configuration Wizard
      1. Register tenant vCenter
      2. Register Plugin with vCenter instance
      3. Deploy Replicator instance – then activate
    4. Complete – from here, we could deploy a Network Stretch function (this will be covered in another blog post)

Demonstration Environment for Extender Installation

For the sake of this post and video, I wanted to create a streamlined installation process for vCD Extender. Therefore, my demonstration environment architecture looks like the following:

As you probably can imagine, this is not built for production state, but just to demonstrate the Extender installation. Initial recommendations:

  1. Separation of management/resource vCenter instances
  2. Separation of compute/management clusters for Provider environment
  3. I will be using internal IP’s/FQDN’s for this demonstration – a production environment would have externally facing resources and/or DNS addresses.
  4. Utilization of a Reverse Proxy to segment DMZ / Private Management connectivity. I will point out these options during the installation video.
  5. L2 VPN connectivity / warm migrations will be covered in a future blog post. L2 VPN is not required for cold migrations.

Provider Installation Video:

Tenant Installation Video:

Tenant Connection to Provider and Cold Migration:


Any feedback would be much appreciated.


vCloud Director 9 Released! My 3 Favorite Things.

So I’m sure many of you saw the release of vCD9 – in my opinion, one of the largest releases in some time now. Many exciting features to talk about, but I’m going to highlight a few very valuable things that will change how providers utilize vCD9 in the future.

Oh, before I forget – release notes:

Multi-Site Federation Capability

This is a phased approach, but the first phase of this providing the ability to view two distinct vCD sites from a single console and provide a trust relationship between them.

More info here – Configuring and Managing Multisite Deployment – vCD9 Documentation

Tenant Metrics built into vCD + vROps Tenant App

Big addition here. Natively, some level of performance data has been in the Cassandra DB for some time – it was up to the SP’s to utilize the data if they choose.

Part of vCD9 is the ability to see basic monitoring data of their vApps/VM’s from the tenant UI.

There is no additional charge for this capability inside of vCD9 – which is fantastic: new capabilities for your end-users.

Now if you want advanced monitoring and analytical capabilities, we have the vRealize Operations tenant plugin for vCD. This complements the built-in tenant metrics and one could run this side by side based on the tenant use case.

So how does the vROps integration work – it collects data from vCenter backing vCD and there’s a management pack for vCD that filters data into vCD constructs like org vDCs and so on.

For multi-tenancy, one can provide two distinct views: Provider and Tenant. A user configured as Service Provider admin can see both views while a Tenant can only see the Org views. User and Role creation is a manual process at this time.

Please note that for vCD9 integration, you must run vROps 6.6 with the 4.5 Management Pack. Get the management pack here:

As expected, for the vROps integration, you will need to be licensed for Advanced or Enterprise for the vCD9 integration.

NSX Networking Additions

Always love this. Some awesome additions:

  1. Ability to spin up DLR’s between org networks and edge gateway from the UI
    1. Notes:
      1. Existing org network can be converted to use a DLR
      2. Org network can either be connected to an Edge or DLR
      3. Only DNS and DHCP from the edge can be relayed to org networks
    2. Read more here in the What’s New Whitepaper – Link to vCD9 What’s New Whitepaper
  2. Trunking Support for VLANs and External Networks
    1. Notes:
      1. Support only for router and external networks
  3. Security Groups for Distributed Firewalls
    1. Documentation for Managing Security Groups

Gosh, there’s so much more to cover. But these are my three top things I wanted to cover this morning.

Last of all, I want to thank our Product Management, Marketing, and Engineering teams at VMware – without them, this could not of happened. I’m just highlighting THEIR efforts while I provided feedback.

Enjoy and I look forward to seeing vCD9 in action with many of our providers.