This was something I ran into a week or so ago in an NSX design – obviously not thinking right!
As a friendly reminder, disable the Edge firewall if you will be using ECMP mode on VMware NSX! There isn’t any message or warning if you enable ECMP mode with the Edge Firewall still on.
Here’s my understanding – since the firewall is a stateful service (this also applies to NAT/Load Balancing), it cannot work with asymmetric routing. For example, the 2nd Edge cannot be aware of a session that was started on the 1st edge (no SYN), so the traffic is dropped.
In my testing, it seems this impacts traffic traversing North to South, but routing South to North seems to work.
I did a quick video of my testing with my current lab environment to depict the results I see – which is the loss of network connectivity and pings from another routed segment.
On new deployments of VMware vCloud Usage Meter, sometimes the wrong vRealize Operations Instance (vROps) is propagated to Usage Meter on initial connection. Today, you cannot change or modify this vROps instance. This is frustrating for new users of Usage Meter.
Well, Usage Meter relies on the Managed Object Browser (MOB) to correlate to the connected vROps instance. In some cases, a VM Administrator may have a stale or older vROps instance still registered to the vCenter environment. For correct reporting and to remove any errors from Usage Meter, this needs to be resolved.
How do I resolve this?
Three steps to solving this:
If applicable, remove the vCenter extension/registration from vRealize Operations.
Remove the incorrect vROps instance from the vCenter MOB.
Register the correct vROps instance and synchronize the inventory in Usage Meter.
Before, any of this is done, I suggest snapshotting your vCenter and Usage Meter appliance. Don’t forget to remove the snaps post-completion!
Step 1 – If applicable, remove the vCenter extension/registration from vRealize Operations
If you have an operating vROps instance that needs to be removed/modified, let’s remove this registration from the vCenter(s).
In the vROps UI, navigate to Administration -> Solutions -> Select VMware vSphere -> click Configure (the gear wheel) and click Manage Registrations –
From there, we need to put in the credentials and click the Unregister button to complete this request.
Again, if your vROps instance is not available, skip ahead.
Step 2 – Remove the incorrect vROps instance from the vCenter MOB.
Open your browser to “https://<vCenter-FQDN>/mob/?moid=ExtensionManager” – this is the direct link to the Extension Manager section. Type in administrative credentials to log in.
Verify you see the extensionList[“com.vmware.vcops”] extension in the list. Click on it. We are going to verify that we see the incorrect vROps instance before we remove it.
From here, click on “server.” We are going to verify that the incorrect vROps instance is showing up in the extension.
Verify you see the incorrect vROps instance.
OK, press the back button twice and back to the original URL. Now we will unregister the incorrect vROps instance.
Click on the UnregisterExtension Method at the bottom –
Now you’ll get a popup requesting the extension name that we will unregister. Type in “com.vmware.vcops” in the box and press the Invoke Method button.
This may take a few seconds to run. However, you will see a void message. Close the popup and refresh the main browser tab that has the MOB information. We should see that the vcops extension has been successfully removed.
Step 3 – Register the correct vROps instance and synchronize the inventory in Usage Meter.
We are now ready to register the right vROps instance to vCenter.
Log into the vROps web console and navigate to Administration -> Solutions -> select the VMware vSphere name. You might see it collecting, but we need to register the plugin into vCenter.
Click on the wheel icon right under Solutions to open up the Configuration section. We will now click on Manage Registrations to re-register it to vCenter.
Check the box to “Use collection credentials” and click the Register button.
This might take a moment to register the plugin inside of vCenter, but a successful message will look like the following –
Going back to my vCenter MOB tab, I can now see vcops under the ExtensionManager section.
Almost complete! Hop over to the Usage Meter console and click “Synchronize All vCenter Inventories” button right under the vCenter Server section. Then, click the Rebuild button under vRealize Operations Manager.
There we go! We now see our vROps instance. Click on Edit to put in the correct credentials. Remember, you need to utilize a LOCAL vROps read-only or administrator account. AD/LDAP accounts do NOT work!
Once you put in the credentials, you will see a message stating the credentials are correct. Complete! On the next hourly run, we should see it fully activated and the “Not yet discovered” message will be removed.
Complete! Now, you’ll be able to monitor your vROps instance and bill based on the usage.
Verifying and Configuring vCloud Usage Meter Data Collection
Alright, home stretch!
Click Licenses at the top right corner – let’s verify the connected vCenter licenses are shown as VCPP. By default, they will be imported as VCPP. If you have any Perpetual licenses connected to the monitored vCenter, it is important you classify it as such. Any VCPP MUST be classified as VCPP under the licenses.
Let’s go to the Monitor tab – this shows how many collections have happened – you want to see 24 on a daily basis (1 an hour). If there’s not, there’s an issue. (Ignore my Day 10 – that’s when I changed my time from UTC to EST 🙂 ).
Now we are going to create two new Customers (think tenants) that will be part of my itemized billing. What’s great about Usage Meter is we can provide granular reporting based on tenant usage – a roll-up of how many points a Customer uses on a monthly basis. Usage Meter is not intended for customer-facing chargeback, but more for compliance between the Provider and VMware – this is a nice addition that definitely helps out the provider.
Note – if you are setting up Usage Meter for entire data center collection, Customer setup is optional. By default, Usage Meter will pull the entire environment into the Monthly Reports. The following steps can be skipped.
Customer/Rules Setup and Walkthrough
Click on the Customers link at the top right corner. By default, no customers are added by vCenter connections – however, vCloud Director will import any Orgs and assign them based on their respective org VDC (think Resource Pool) – very nice option!
I’m going to go ahead and establish a new customer called “Tenant” – click on the Add button and provide the name, Country (note to my fellow US citizens – US is fifteen down from the top, not in alphabetical order), and zip code. If you want to anonymize this customer data, click the box Restricted.
We can now see Tenant in the Customer list, but no rules or VM’s attached to it. Do you also see how now we have Active VM’s attached to the Public, T1, and T2 Customers? This is because UM detected those Org, assigned the rule, and is collecting data – slick!
OK – let’s click on Rules now on the top right corner.
Before we create a rule for Tenant, let’s check out the existing vCD rules that were automatically created. Click on Rules List –
From here, we can see how the Rule was applied – there’s an exact string applied for each Org/Customer.
Now let’s check out which VM’s are under Public – click the Customer list at the top and select Public
You can also see the list has changed – I see the only Public rule applied. From here, I can click Display Mapped Virtual Machines
Now we can see the mapped VM’s to Public – this is a great way of verifying if you rule is setup the way you want it.
Okay, back to setting up Tenant for collection. Press Back or Click on Rules at the top and select the Customer drop-down – Tenant. From here, we will be creating a rule that maps to my Management cluster for this demonstration. Remember, you can create rules that are associated with an entire Data Center, Cluster, Resource Pool, Folder, prefix, etc – many options are available.
Note – if you look at my screenshot below, I deployed Usage Meter to my Management Cluster. Where is it? 🙂 We automatically remove Usage Meter from any data collection – you do not pay for any resource usage for the UM appliance.
I selected the RegionA01-MGMT01 object and we can see the Rule Creator it automatically changed the Object Type Cluster and has a specific value. This value matches to how vCenter sees the cluster in the MOB browser. In my example, I have “domain-c71”
When you’re ready to create the rule – click Create on the Rule Creator section. Once it’s created, the Rules List sublink will blink Yellow, and let’s go ahead and click that.
Excellent, we see our new rule populated. Note – Active VM’s will not show until the next monitoring interval. By default, it runs 5 minutes past the hour.
Automatic Reporting Tab
Here’s where you can set up automatic monthly reporting to a specific email address outside of your point of contact. Currently, this DOES NOT send your usage results to VMware or the iAsset portal. You still need to input your monthly usage by the 5th of every month.
I configured where I wanted to send this when to send this – pretty straightforward.
Monthly Report Walkthrough
Let’s go ahead and click on Reports in the top right corner.
By default, the monthly reports will be emailed to the point of contact identified in the setup. There are five different reports:
Customer Monthly Usage
Horizon DaaS Tenants
Virtual Machine History
Basic functions – you can either Browse or Export the reports. Browse will show on the screen while Export will turn the report into a TSV file for you to download. The latter is the required item for compliance.
This report shows the entire aggregate of what Usage Meter sees. How many points on each bundle Usage Meter has collected, vSAN usage, VM’s by Product Server, and vCD Summary. If we had Horizon DaaS, it would show up here.
Customer Monthly Usage
This shows what each Customer uses on a monthly basis along with the respective bundle. Since we just started the collection, I don’t have any data here but we can see the Customers inside of the Monthly Usage section.
Virtual Machine History
This is a great report for getting initial estimates of what an environment could utilize in the VCPP program – this is the report that I use to provide estimates.
The VM History report shows the VM state, allocated vRAM, and then Billed vRAM – that’s where sizing estimates come into play.
This report is used for vSAN – shows usage based on vSAN storage on disk along with any features that may be applied. For example, my COMP01 cluster has DeDuplication turned on. This requires an Enterprise licensed so we see it applied.
Monthly Report Collection
Okay, let’s go ahead and export all of the reports – this is a good example of what you would send to your aggregator for monthly billing.
Click on Reports at the top and then hit the Export button for each report.
From here, we will see the reports under the Download Completed Reports
They are TSV files – so can be opened with Excel or a notepad editor. As you can see, they are all tab separated, so the data can be reviewed as such.
Note – any changes to this file will change the MD5 hash – DO NOT make any modifications!
That’s it! I realize this series was lengthy, but it’s good for our existing and up and coming providers to understand Usage Meter setup and what I’ve learned in my time working with it. More to come!