Upgrading a Minimal Installation of VCF 9.0.1 to 9.0.2

Smaller environments that have VCF 9 deployed might have a subset of the VCF components/appliances deployed. This minimizes the virtual infrastructure management footprint and resource consumption. For example, a three-host environment might have VCF Operations 9 + vCenter 9 + ESX 9. The downside to these types of deployments is you lose some of the VCF functionality such as fleet management. This begs the question, "How do I update the components I have?" You can upgrade version 9 components such as VCF Operations, vCenter, and ESX similar how you upgraded version 8 components.

We'll upgrade a small environment that consists of VCF Operations 9.0.1, vCenter 9.0.1, and ESX 9.0.1 hosts running a few workloads.

We'll update each component in the order below as recommended here:

• VCF Operations
• vCenter
• ESX

Download the .pak file for the VCF Operations version you are upgrading to. In this case, we are upgrading 9.0.1 to 9.0.2 so I need this file -- Operations-Upgrade-9.0.2.0.25137843.pak -- from the My Downloads section of the Broadcom Support Portal.

Next, I log into the admin console of VCF Operations. Add /admin to the VCF Operations URL, e.g., https://<vcf-ops-ip-address>/admin

In the left column, click on Software Update. Click the Install A Software Update button. Browse for the downloaded .pak file and click the Upload button.

The upload and staging of the file will likely take a few minutes. You might see the following warning:

"The update will restart the cluster for the entirety of the update."

This is referring to the VCF Operations cluster (even if it is a single-node cluster), not the ESX cluster. Continue through the upgrade wizard and click the Install button. The upgrade will naturally take some time.

The next component to upgrade is vCenter. Start by reading this knowledge base (KB) article:

https://knowledge.broadcom.com/external/article?legacyId=92659

We'll download and use the VMware-VCSA-all-9.0.2.0.25148086.iso file to upgrade from 9.0.1 to 9.0.2. You will need to mount that .iso file to the vCenter appliance similar to what is shown below.

Select vCenter in your vSphere Client inventory and click Updates. The UI will walk you through the vCenter upgrade.

You should back up vCenter before performing the upgrade. If you are not familiar with how to do this, start here:

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-0/vcenter-configuration/configuring-vcenter-server-using-the-management-interface/configure-and-schedule-backups.html

You might see a warning, "Plugin should be upgraded to proceed further." Simply click the Upgrade Plug-in link before clicking Next. This will take a few minutes.

You will likely have to run the pre-checks after the plug-in upgrade (shown below). Then click Next assuming the source pre-checks ran successfully.

Click the Configure Target Appliance link. The selections in this wizard should be familiar from previous vCenter upgrades. I used the default settings.

In the final step you'll click the Start Upgrade link and set the Switchover Execution. I chose Automated from the drop-down menu. After that, I started the upgrade and waited patiently. Behind the scenes, a new vCenter appliance is deployed, the settings and data from the original appliance are copied over, and a switch-over occurs from the original appliance to the new one. As you can probably imagine, this takes a while. Ensure you have sufficient storage and compute resources for this upgrade.

You can delete the old vCenter VM after confirming the upgrade/migration completed successfully.

Finally, we upgrade the ESX hosts. This is done using Lifecycle Manager in the vSphere Client. Click on the cluster or host if you have standalone hosts. Click on Updates. Edit the image to use the ESX version you want to upgrade to along with any vendor, firmware, and driver add-ons. Click Validate and then Save assuming the image is valid.

Lifecycle Manager should automatically check image compliance after you click Save. In the example below, we see that the single host in my lab environment is out of compliance with the new image.

Run the Pre-Check and then Remediate assuming the pre-check passed. Many upgrades still require a reboot although this is getting better with live patching. I recommend reading through the Lifecycle Manager documentation prior to using it. If you have a single host, you will need to shutdown all of the VMs on the host, put the host into maintenance mode, and perform an interactive upgrade using a CD, DVD, USB device, .iso file, ESXCLI, or script--reference this documentation for guidance.

VCF 9 Installer Validation Found Less Than Two Uplinks For The DVS

Installing VMware Cloud Foundation (VCF) 9 in a lab often means using less than ideal hardware configurations. In my case, I have ASUS Intel NUCs that have only one physical NIC for an uplink. Recommendations dictate you should have at least two for redundancy and the VCF Installer checks this. If you have less than two uplinks, VCF Installer will block the deployment. There is a way around this and it is simple.

I received the following details when the VCF Installer was validating existing components:

VcManager <vCenter FQDN>: Found less than two uplinks for the DVS <distributed virtual switch name>

Remediation: Please ensure a minimum of 2 configured uplinks for each DVS.

This is because I had only one uplink configured for the DVS. Getting around this was easier than I thought. I assumed I needed two physical NICs on each host. It turns out I only need two uplinks configured--even if one of the uplinks does not have a physical NIC backing.

I proceeded to the Network section of the vSphere Client, right-clicked the DVS, clicked Settings, and then Edit Settings.

I clicked Uplinks in the Distributed Switch - Edit Settings popup window, added an uplink, and clicked OK.

I re-ran the VCF Installer validations and everything succeeded.

Cross vCenter vMotion with Different Distributed Switch Versions

Cross vCenter vMotion enables live VM migrations from one vCenter environment to another. This prevents downtime when migrating between these two separate environments. An example use case is moving workloads from a vCenter 8U3 environment to a new VCF (vCenter) 9 deployment. However, by default, vMotion does not allow live migration of VMs across different virtual distributed switch (VDS) versions. There is a workaround.

In our example above, this would be migrating VMs from a VDS 8.0.3 to a VDS 9.0.0. When attempting this cross vCenter vMotion, you get this error:

The target host does not support the virtual machine's current hardware requirements. The destination virtual switch version or type (VDS 9.0.0) is different than the minimum required version or type (VDS 8.0.3) necessary to migrate VM from source virtual switch.

The workaround: Add the following advanced setting to the source and destination vCenter instances. Follow these steps:

1. In the vSphere Client, go to Inventory and click on the vCenter instance.
2. Click Configure.
3. Select Advanced Settings.
4. Click the Edit Settings button.
5. Scroll to the bottom of the popup window.
6. Enter config.vmprov.enableHybridMode in the Name field.
7. Enter true in the Value field.
8. Click the Add button.
9. Click the Save button.

IMPORTANT: Test a few migrations using non-critical workloads first. This is an advanced setting and there have been a few reports of networking issues after a VM was migrated, e.g., loss of static IP settings.

Optional: Revert the setting by changing Value to false after the migrations are complete.

Introduction to VMware vSAN Data Protection

VMware vSAN Data Protection, introduced in vSphere 8.0 Update 3, is a powerful capability included in the VMware Cloud Foundation (VCF) license for customers running vSAN ESA. This solution provides local protection for virtual machine workloads at no additional licensing cost. Key capabilities include:

• Define protection groups and retention schedules.
• Schedule crash-consistent snapshots of VMs at regular intervals.
• Create immutable snapshots to help recover from ransomware.
• Retain up to 200 snapshots per VM.
• Per-VM replication locally and/or to remote clusters.

The simplicity and tight integration of the tool make it a terrific way to augment existing data protection strategies or serve as a basic, but effective solution for protecting virtual machine workloads. The images below show the UI and provide some context. You can click the images to get a better look.

vSAN Data Protection simplifies common recovery tasks, giving administrators the ability to restore one or more VMs directly within the vSphere Client, even if they have been deleted from inventory—a capability not supported by historical VMware snapshots. Getting started involves deploying the VMware Live Recovery (VLR) virtual appliance and configuring protection groups, which can also be set to create immutable snapshots for basic ransomware resilience.

For extended functionality, VCF 9.0 introduces a unified virtual appliance and the option for vSAN-to-vSAN replication to other clusters for disaster recovery, which is available with an add-on license. More information about VMware Live Recovery: https://www.vmware.com/products/cloud-infrastructure/live-recovery 

VMware vSAN Daemon Liveness - EPD Status Abnormal

I reinstalled ESXi and vSAN ESA 8.0 on hosts that were previously running ESX and vSAN ESA 9.0. It's a lab environment. I did not bother to delete the previous VMFS volume on a local drive in each host. This is where ESXi was storing scratch data, logs, etc. After reinstalling 8.0 and turning on vSAN, I noticed the following error in vSAN Skyline Health: vSAN daemon liveness.

I expanded the health check to get more information. There I could see Overall Health, CLOMD Status, EPD Status, etc. EPD Status showed Abnormal on all of the hosts. I naturally did some searching online and came up with some clues including the knowledge base (KB) articles below.

https://knowledge.broadcom.com/external/article?articleNumber=318410

https://knowledge.broadcom.com/external/article?articleNumber=326596

They were helpful, but did not get me to resolution. The post below from a few years ago got me closer.

https://community.broadcom.com/vmware-cloud-foundation/discussion/vsan-epd-status-abnormal

I browsed the existing VMFS volume and noticed the  epd-storeV2.db file in the .locker folder.

Since it is a lab environment, I figured why not just delete the file and see if vSAN heals itself (recreates the necessary files). I put the host in maintenance mode as a precaution, deleted the file, and rebooted the host. This resolved the issue. In addition to the .db file, I noticed the addition of a new file,  epd-storeV2.db-journal.

I checked vSAN Skyline Health and the error was gone for that host. You can see the status of each host if you click the Troubleshoot button for the vSAN daemon liveness health check. I repeated that effort for each host in the cluster.

The vSAN Skyline Health error was gone after completing the process on all of the hosts in the cluster.

I probably could've restarted services on the hosts as detailed in the KB article above, but I chose to reboot them since they were already in maintenance mode.

VMware VCF Installer Warning - Evacuate Offline VMs Upgrade Policy

I ran across this warning in the VMware Cloud Foundation (VCF) Installer when deploying VCF 9:

cluster <cluster name>: Evacuate Offline VMs upgrade policy configured for cluster <cluster name> on vCenter does not match default SDDC Manager ESXi upgrade policy. It has value false in vCenter, and default value true in SDDC Manager.

The fix is simple...

In the vSphere Client, click the three horizontal lines in the top left corner.

Click Lifecycle Manager.

Click Settings.

Click Images.

Click the Edit button for Images on the right side of the window.

Click the checkbox next to "Migrate powered off and suspended VMs to other hosts in the cluster, if a host must enter maintenance mode" to check the box.

Click Save.

Re-run the VCF Installer validations.

VMware Live Recovery Not Accessible Error

• VMware Live Recovery 9.0.3.0 and later combine the control plane, replication, and data protection into a single appliance.
• Ensure DNS--forward and reverse lookup--are functioning correctly.
• A "Not accessible" error could indicate an issue with certificates.

I recently had the opportunity to set up the latest VMware Live Recovery appliance, which simplifies deployment and configuration into a single virtual appliance. More details about this new model can be found in this blog article.

I encountered an issue where the UI displayed vSphere Replication and VMware Live Site Recovery, formerly Site Recovery Manager (SRM), was marked as "Not accessible."

I double-checked that DNS was correctly configured, and I was able to ping all directions between the vCenter instances and the VMware Live Recovery appliances. It was not a name resolution or network connectivity issue. Pro tip: Always verify DNS is configured correctly. I've seen this cause issues in countless scenarios.

After some searching and trial and error, I determined the issue was a mismatch between the server name and what was in the certificate. Specifically, the Common Name (CN) in the Issued to section. I logged into the appliance management UI by appending port 5480 to the fully-qualified domain name (FQDN) of the server. Using the example FQDN in the screenshot, the URL would be https://vlr01.vmware.lab:5480.

After logging in with the "admin" username and the password I specified for that account during deployment, I selected Certificates in the left menu bar. Then, I clicked Change near the top right corner to generate a new self-signed certificate.

I completed the fields in the Change certificate UI and clicked the Change button to generate the new certificate. I verified the CN matched the FQDN and rebooted the VMware Live Recovery virtual appliance just for good measure. When the server came back up, the error was gone, and I had the green checkmark OK indicator.