Setting up VMware vCloud Director 9.5 for Cross-VDC Networking

In this post, I will be reviewing the necessary steps to support Cross-VDC Networking inside of VMware vCloud Director 9.5. These are fairly straightforward since it aligns to the standard requirements set forth from Cross-vCenter NSX.


  1. Multi-Site management must be configured between the vCloud Director instances. I will try to add a post on establishing this at a later time.
  2. Ensure you have a unique vCloud Director installation ID. If you have duplicate IDs, this can lead to MAC address conflicts. Fojta did a blog post on updating your ID – please accomplish this before continuing.

Cross-vCenter NSX Configuration

vCD 9.5 does require a standard Cross-vCenter NSX configuration implemented between the resource/payload vCenters before we can do any configuration at the vCloud Director level.

There are many guides out there, but here’s a link to the official VMware documentation on setting up cross-vCenter NSX. 

This can be a single or multi-SSO domain topology. In my environment, here’s what I’ve configured between my two sites: Site-A and Site-B.

  1. From the Networking and Security plugin, I’ve assigned my Site-A NSX Manager while linking Site-B NSX Manager as the secondary instance – 
  2. From there, I need to establish my Universal Segment ID pool and Transport Zone.
  3. Keep in mind you do not want to overlap with an existing Segment ID pool, so pick a number that’s high enough (or out of reach from other pools) – 
  4. From the Transport Zone screen, I’ve created my new Transport Zone named “Universal-TZ” – 
  5. Now I’m ready to connect it to my respective clusters on Site-A and Site-B. Keep in mind I need to hit the drop down for the NSX Manager and attach the respective cluster at your secondary (or additional) location.
  6.  That’s it! Onto the next configuration which is at the vCloud Director level.

vCloud Director Initial Provider Setup

In this step, we need to assign the correlated NSX Manager to each vCenter instance that’s participating in the Cross-VDC networking solution. I will be showing how I’ve done this for my two sites, Site-A and Site-B, while establishing a fault domain.

  1. From my Site-A, navigate to System -> Manage & Monitor -> vSphere Resources -> vCenters – 
  2. We are going to right click, go to Properties of this vCenter – 
  3. From there, we need to go the NSX Manager tab. This is where we populate the following:
    1. Host/IP of NSX Manager
    2. Admin username
    3. Admin password
    4. Control VM Resource Pool vCenter Path – this can be either the MOref object id OR the ‘Cluster/RP’ path – I chose the former.
    5. Control VM Datastore Name – full name
    6. Control VM Management Interface Name – again, full name
    7. Network Provider Scope – now this is where we establish a fault domain. This Network Provider Scope could cover one or many vCenters in a single vCloud Instance. However, when we establish the vdc-Group, we must have a minimum of two different/unique fault domains (or network provider scope) inside of the created vdc-Group.
  4. Now, on my Site-B, I will configure my respective properties along with a Network Provider Scope of “region-b” – 
  5. Great! Next step is to add the Universal Transport Zone as a new network pool on each vCD instance. This is purely importing the created Universal-TZ and moving on, so very easy – 
  6. That’s it – now we are ready to enable a specific orgVDC for cross-VDC networking.

Enabling an orgVDC for Cross-VDC Networking

This is a very simple process – really just enable it on a per orgVDC basis.

  1. Go to your orgVDCs and right click on the orgVDC you want to enable cross-VDC Networking on. For example, I am enabling this on my Daniel oVDC’s – 
  2. Click on the Network Pool and Services sub-tab and you’ll see a new box below the Network Pool that states ‘Enable Cross VDC Networking (Using Network Pool “<Universal-TZ-Pool>”‘ Check this box.
    1. This still allows for local oVDC network creation using the traditional network pool as stated in the screenshot above – this is not a complete conversion to the Universal Transport Zone.
  3. Now, enabling this on my Site-B – 

Permissions/Rights required for Cross-VDC Networking

As discussed in the previous blog post, there are specific rights and roles required for Cross-VDC networking that are not enabled by default for the organization administrator. Please review these before the tenant utilizes Cross-VDC networking.

Cross-VDC Networking Permissions Review

Creation of the initial Cross-VDC Group

Now we are ready to test the creation of a new Cross-VDC group.

  1. Let’s log into the Tenant UI and we should see the Datacenter Groups from the context switching menu – 
  2. Now, I can create my first Cross-VDC group and start establishing my egress points. Awesome! 

More to come here on the Cross-VDC networking capabilities within vCD 9.5 from myself, Wissam Mahmassani, and Abhinav Mishra. Thanks!


Migrate VMs and Networking to vCloud Director – Video Walkthrough

I wanted to provide a quick walkthrough on how easy it is to import a VM (or adopt a VM) into a tenant organization for vCloud Director.

Tomas Fojta covers a lot of great detail on when this was introduced in vCD 8.20 here. 

In this video, I go through and show how I moved a tenant workload (DanielApp) along with a network to vCloud Director and NSX.

While this does require a stepped process, it’s a pretty seamless process.

Migration Steps:

  1. Move the routing interface from the current physical underlay to the NSX Edge inside of the vCloud Director tenant organization (DCP-Edge-01)
  2. Switch over the VMNIC for the workloads to the logical switch presented by DCP-Edge-01
  3. Drag the VM to the orgVDC resource pool that’s provisioned by vCloud Director. Done!

Note 1 and 2 do require some level of coordination with your network team with a brief maintenance window (route changes and validation). Moreover, the important distinction is we are allowing a tenant to utilize NSX functionality alongside vCloud Director.

Step 3 is the easiest. vCloud Director does all of the work and shows it in the UI without any further intervention. This is a great feature that demonstrates vCD can be utilized for existing tenant workloads that may be in a “naked” vCenter environment (or utilizing an existing CMP they are moving away from).

Anyway, here’s the video I created that shows me moving DanielApp to vCloud Director under my “Daniel” organization.


Last bit I’ll leave you with – while it’s great to migrate both the network and to vCD, this may not be possible based on use case. Other migration method could be exposing the existing distributed virtual portgroup as a vCD External Network to the pVDC, then the oVDC. Then it’s as simple as just dragging in the VM(s) to the resource pool.

However, I do lose any self-service and NSX functionality, which could include overlapping networks when I scale out tenants.

Happy migrating!


Security Compliance – Pre-configure your vApp firewall rules inside vCloud Director using NSX DFW (Part 1 of 2)

This is a joint blog series with Wissam Mahmassani 

Today, we will be discussing pre-configured firewall rules for vApps inside of vCloud Director (vCD).

Recently, I received a request from a new provider that wants to deploy a specific vApp when they onboard a new tenant (or organization). This specific vApp will need to have NSX Distributed Firewall rules in place – the vApp will be the same for every tenant and will need to be secured accordingly.

While this is a Provider managed approach, this is a very simple way of “stamping” out vApps or other required virtual machines that need specific policies applied to them. Moreover, we would like an automatic way to pick up the associated DFW rules so it’s one less step for the Provider.

Overall Steps:

  1. Creation of a Security Group with Dynamic Membership
  2. Creation of a Security Policy
  3. Activating Security Policy against Security Group
  4. Creating vApp that meets dynamic membership criteria

Creation of a Security Group with Dynamic Membership

  1. Navigate to Menu -> Networking and Security -> Service Composer
  2. The first thing we are going to do is create a Security Group that will associate the VMs based on criteria.
  3. The easiest way to do this is by using a dynamic membership policy. We want to apply this group to any VM’s that meet a specific name. In my example, I’m going to be utilizing “mgmt-pod” as my criteria – 
  4. Click Finish, and we are off to the next step.

Creation of a Security Policy

  1. Let’s click on the Security Policies tab inside of Service Composer and create a new Security Policy – 
  2. Let’s give it a name – I am using Standard vApp DFW Rules. 
  3. From here, we can click on Firewall Rules and create our rules. In my example, I am going to let HTTPS traffic in and block everything else. Typically, for micro-seg rules, we would create granular rules to secure all types of traffic. I am using these just as an example. 
  4. Creating DFW policies is fairly straightforward in the Service Composer – 

Activating Security Policy against Security Group

  1. Now, we are ready to apply our newly created policy to our group. Click the Apply button while your newly created policy is selected – 
  2. From the pop-up window, we will select our Standard vApp Rule group as a Selected Object – 
  3. Success – now we can see it has been applied –
  4. From the DFW view, we can see a new section created with associated DFW rules – 
    1. Solid note from Tom Fojta“Do not forget the order of DFW sections is important. If tenant’s DFW VDC section is above and he creates any-any-allow rule it will nullify provider rules. Tenant sections are created by default on top unless forced with API to be created at the bottom.”

Creating vApp that meets dynamic membership criteria

  1. Now from my Provider UI, I am going to go ahead and create my Management vApp for a new tenant. Again, the context is this would be managed by the Provider initially while we are inheriting the Security Group set forth above. 
  2. Once my vApp is up, I can verify that I am unable to access via ICMP which meets my criteria. We can see the Standard vApp Rule group was associated with the vApp and I am unable to ping it. 

While this is not the only path of securing Provider-managed VM’s for a tenant. Check out Wissam’s approach here by utilizing Resource Pools!


A deeper look at vCloud Director and NSX Distributed Logical Router and Firewall Services with Usage Meter feature detection

Need to work on my titles, but that’s what I have for now. I want to spend some time reviewing how NSX Distributed, Edge, and DLR Firewall services behave in respect to vCloud Director and vCloud Usage Meter.

Some may know vCloud Usage Meter provides automated feature detection on a per-VM basis – essentially allowing granular-level billing for tenants. For example, tenant ACME may have 5 VM’s that need Distributed Firewall out of 20 – why charge them for all if they aren’t using it? Since version 3.6 of Usage Meter, this feature has been applied on a per VM level, for the most part.

There are three different versions of NSX available in the Cloud Provider Program –

We can see that your basic fundamentals are available in SP Base: your distributed switching and routing, Edge Firewall services, NAT’ing, etc. Advanced adds in Distributed Firewalling, Service Insertion and other advanced functionality while Enterprise is adding in X-vCenter NSX, HW VTEP integration.

Second, let’s take a look at the chart for NSX Editions by Feature. This comes from our vCloud Usage Meter Feature Detection whitepaper. Don’t have it? Get on VMware Partner Central and grab a copy.

Take note at the statement of “All VMs on all networks serviced by the edge” – this means even if you have a VM on that edge that does not use that feature, it will charge for it since it still has access to it. For example, if you set up a L2VPN on an Edge, be aware that any other connected networks will be charged for NSX Enterprise even though they may not be able to traverse to that tunnel!

Lab Setup

I wanted to design a DLR environment within vCD along with using Edge and Distributed Firewall rules. I came up the following design in my 9.x environment:

  1. Creating an NSX Distributed Logical Router (DLR) is pretty easy. Select the Advanced Gateway box and check Enable Distributed Routing. You can also enable Distributed Routing on any existing Advanced Edge – 
  2. The great thing about how vCloud Director creates Distributed Routing is it’s actually two Edges – a standard perimeter edge along with the DLR southbound. The transit interface is created as a /30 while applying static routes for any connected DLR interface. DLR’s default gateway is the Edge transit interface. Below is what I can see inside of NSX. 

Using the NSX Distributed Logical Router is in the NSX Base bundle and covered by Advanced SP Bundle, which is great for providers. However, Edge Firewall services are only available in the NSX Base bundle, where some providers may want to provide “high-powered” firewalling services.

The key difference between Edge and Distributed Firewall services is we do not hairpin for forwarding decisions in the DFW model: the decision is made inside of the hypervisor before the packet ever egresses anywhere. Hence, provides better throughput and capability compared to Edge or traditional firewall services – check out Wissam’s discussion on DFW capabilities here. 

vCD Access to Edge Firewall vs Distributed Firewall

I think Tomas did a great job of summarizing vCD Firewall capabilities and pointing out some of the architecture differences between the two firewall technologies.

From a vCD UI access point of view, you have to access the firewall services in two different places.

  1. Edge Firewall services are accessed by right-clicking on the Edge Gateway and selecting Edge Services. Makes sense, right? 
  2. Now Distributed Firewall management is accessed by right-clicking on the organization VDC and clicking on Manage Firewall. Not my favorite and doesn’t state DFW, but this is how it’s accessed today in the Flex UI. 

NSX Distributed Firewall Changes

Okay, let’s get into a few scenarios and test out how Usage Meter will put up the changes.

  1. I freshly created my three vApps: T1-App-1, T1-DB-01, T1-DLR-tclinux-01/02a (Web Servers). We can run a VM History Report and pick them up and see they all have been associated with the Advanced Bundle initially. I will try to point out the vROps and NSX column in the following discussion, but we will be looking for a checkmark and an “A” that refers to Advanced usage. 
  2. So let’s go ahead and crack open the DFW and start carving up some rules. This is a simple 3-tier architecture so I’m going to lay out allowing web, web to app traffic, app to DB, allowing SSH to App and DB, and blocking everything else in Web, App, and DB Tiers. 
  3. As you can see, the Applied To for my deny rule is only applied to my three Distributed Routing tiers (or routed Org VDC Networks). Not only is this a good NSX practice, this cuts down on who gets charged for NSX Advanced!
  4. Now looking at the DFW rules from vCenter, I can see vCD created the tenant folder and applied the policies in order. Very streamlined. 
  5. Now let’s check to see if my new rules work. Yep, I can still SSH to my VM’s but do not receive any ICMP replies. Perfect! 

How did Usage Meter detect the changes?

  1. So let’s now take a look at the Usage Meter VM History Report. What’s fascinating is Usage Meter tracks every move called state change. We can see in my tclinux-vApp (or Web) that it tracked when it started using Base NSX usage, then switched to Advanced, then my vROPs Enterprise instance picked it up and added it to its inventory. All in a very short timespan. Very useful! 
  2. With the DB and App layers, we see something very similar – vROps Enterprise, however, picked up these VM’s first and then we see Advanced NSX usage. 
  3. Now, I can see a few new line items populated on my Customer Monthly Usage and Monthly Usage report. I now see Advanced Bundle with Networking, Advanced Bundle with Management, Advanced Bundle with Networking and Management.
  4. This is to be expected since my VM’s went through a few iterations – my Web VM’s initially went with NSX Advanced without vROps which falls in the Advanced with Management while DB/App started using NSX Advanced without vROps registration (Advanced with Networking bundle). Last of all, all three are now assigned to the Advanced w/Networking and Management since we are using NSX Advanced features along with vROps Enterprise.
  5. While I see the new line items, I don’t see any units to be reported. Why? Well, I have some very small VM’s (256MB in vRAM) and they just ran a few hours during this testing. Keep in mind Usage Meter averages out the usage over the entire calendar month (720 hours for a 30 day period or 744 hours in a 31 day period). Again, to be expected but was pleased that new categories were added immediately.

In summary, Usage Meter does a great job of detecting NSX feature usage and bills it accordingly based on a specific service – Distributed Firewall being one of them. Thanks!