14Jan2018 – Updated with latest VMware Public KB and Intel Sightings info
I’m sure many of you have heard of the Intel CPU vulnerabilities and how they can impact x86 architectures – I have a feeling this is just the start of something larger too.
I’ve had many Providers reach out to me, very concerned about how this can impact a SP design, especially with virtualization.
To step back, I found this very simple depiction that was shared on Twitter that summarizes the two vulnerabilities – thank you Daniel Miessler (link to his blog article):
So, what does this mean for VMware Cloud Providers? Well, official statements are underway, but here’s what I know so far – again, this is NOT an official statement:
There are three CVEs:
The patches made available by VMware should cover CVE-2017-5753 and CVE-2017-5715. Considering the patches were released earlier, you may very well be already patched.
CVE-2017-5754 (or Meltdown) does not seem to affect ESXi, Workstation, and Fusion because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides. Thus, the underlying OS that should probably be patched for the vulnerability. Therefore, the following three links are for the first two CVEs only.
This was something I ran into a week or so ago in an NSX design – obviously not thinking right!
As a friendly reminder, disable the Edge firewall if you will be using ECMP mode on VMware NSX! There isn’t any message or warning if you enable ECMP mode with the Edge Firewall still on.
Here’s my understanding – since the firewall is a stateful service (this also applies to NAT/Load Balancing), it cannot work with asymmetric routing. For example, the 2nd Edge cannot be aware of a session that was started on the 1st edge (no SYN), so the traffic is dropped.
In my testing, it seems this impacts traffic traversing North to South, but routing South to North seems to work.
I did a quick video of my testing with my current lab environment to depict the results I see – which is the loss of network connectivity and pings from another routed segment.
On new deployments of VMware vCloud Usage Meter, sometimes the wrong vRealize Operations Instance (vROps) is propagated to Usage Meter on initial connection. Today, you cannot change or modify this vROps instance. This is frustrating for new users of Usage Meter.
Well, Usage Meter relies on the Managed Object Browser (MOB) to correlate to the connected vROps instance. In some cases, a VM Administrator may have a stale or older vROps instance still registered to the vCenter environment. For correct reporting and to remove any errors from Usage Meter, this needs to be resolved.
How do I resolve this?
Three steps to solving this:
Remove the incorrect vROps instance from the vCenter MOB.
Remove the incorrect vROps instance from the Usage Meter database.
Register the correct vROps instance to vCenter.
Before, any of this is done, I suggest snapshotting your vCenter and Usage Meter appliance. Don’t forget to remove the snaps post-completion!
Step 1 – Remove the incorrect vROps instance from the vCenter MOB.
Open your browser to “https://<vCenter-FQDN>/mob/?moid=ExtensionManager” – this is the direct link to the Extension Manager section. Type in administrative credentials to log in.
Verify you see the extensionList[“com.vmware.vcops”] extension in the list. Click on it. We are going to verify that we see the incorrect vROps instance before we remove it.
From here, click on “server.” We are going to verify that the incorrect vROps instance is showing up in the extension.
Verify you see the incorrect vROps instance.
OK, press the back button twice and back to the original URL. Now we will unregister the incorrect vROps instance.
Click on the UnregisterExtension Method at the bottom –
Now you’ll get a popup requesting the extension name that we will unregister. Type in “com.vmware.vcops” in the box and press the Invoke Method button.
This may take a few seconds to run. However, you will see a void message. Close the popup and refresh the main browser tab that has the MOB information. We should see that the vcops extension has been successfully removed.
Step 1 is complete.
Step 2 – Remove the incorrect vROps instance from the Usage Meter database.
Before you start this, please take a snapshot of the Usage Meter appliance. Don’t forget to remove it after you verified the change was successful!
Open up the Console (or SSH) to the Usage Meter appliance. Log into the console.
Note – If you attempt to SSH, ensure you utilize the “usgmtr” account as remote root logins are NOT permitted by default.
Run the “sql” command to enter the database.
Query vCOPS table to verify the record we need to delete – type in select * from “VcopsServer”; We can see I only have one instance here, so it’s ID 1.
Now we will delete ID 1. Delete entry using ID, replace [ID] with the ID gathered from above statement – so type the following two commands:
delete from “VcVcops” where “vcopsServerId” = [ID];
delete from “VcopsServer” where id = [ID];
In my case, I am using
delete from “VcVcops” where “vcopsServerId” = 1;
delete from “VcopsServer” where id = 1;
Now let’s check and make sure the vROps instance is removed from UM. Press the up arrow three times and run the select command again. As we can see, there’s 0 rows for “VcopsServer”
Now type “\q” to quit out of the database.
Going back to the Usage Meter console, we can now see the vROps instance is now removed from Usage Meter.
Now let’s move to re-adding the right vROps instance to Usage Meter!
Step 3 – Register the correct vROps instance to vCenter.
We are now ready to register the right vROps instance to vCenter.
Log into the vROps web console and navigate to Administration -> Solutions -> select the VMware vSphere name. You might see it collecting, but we need to register the plugin into vCenter.
Click on the wheel icon right under Solutions to open up the Configuration section. We will now click on Manage Registrations to re-register it to vCenter.
Check the box to “Use collection credentials” and click the Register button.
This might take a moment to register the plugin inside of vCenter, but a successful message will look like the following –
Going back to my vCenter MOB tab, I can now see vcops under the ExtensionManager section.
Almost complete! Hop over to the Usage Meter console and click “Synchronize All vCenter Inventories” button right under the vCenter Server section. Then, click the Rebuild button under vRealize Operations Manager.
There we go! We now see our vROps instance. Click on Edit to put in the correct credentials.
Once you put in the credentials, you will see a message stating the credentials are correct. Complete! On the next hourly run, we should see it fully activated and the “Not yet discovered” message will be removed.
Complete! Now, you’ll be able to monitor your vROps instance and bill based on the usage.