Upgrading a Minimal Installation of VCF 9.0.1 to 9.0.2

Smaller environments running VCF 9 might have a subset of the VCF components/appliances deployed. This minimizes the virtual infrastructure management footprint and resource consumption. For example, a three-host environment might have VCF Operations 9 + vCenter 9 + ESX 9. The downside to these types of deployments is you lose some of the VCF functionality such as fleet management. This begs the question, "How do I update the components I have?" You can upgrade version 9 components such as VCF Operations, vCenter, and ESX similar how you upgraded version 8.

In this example, we'll upgrade a small environment that consists of VCF Operations 9.0.1, vCenter 9.0.1, and ESX 9.0.1 to 9.0.2.

We'll upgrade the components in the order below as recommended here:

• VCF Operations
• vCenter
• ESX

VCF Operations

Download the .pak file for the VCF Operations version you are upgrading to. In this case, we are upgrading 9.0.1 to 9.0.2 so I need this file -- Operations-Upgrade-9.0.2.0.25137843.pak -- from the My Downloads section of the Broadcom Support Portal.

Next, I log into the admin console of VCF Operations. Add /admin to the VCF Operations URL, e.g., https://<vcf-ops-ip-address>/admin

In the left column, click on Software Update. Click the Install A Software Update button. Browse for the downloaded .pak file and click the Upload button.

The upload and staging of the file will likely take a few minutes. You might see the following warning:

"The update will restart the cluster for the entirety of the update."

This is referring to the VCF Operations cluster (even if it is a single-node cluster), not the ESX cluster. Continue through the upgrade wizard and click the Install button. The upgrade will naturally take some time.

vCenter

The next component to upgrade is vCenter. Start by reading this knowledge base (KB) article:

https://knowledge.broadcom.com/external/article?legacyId=92659

We'll download and use the VMware-VCSA-all-9.0.2.0.25148086.iso file to upgrade from 9.0.1 to 9.0.2. You will need to mount that .iso file to the vCenter appliance similar to what is shown below.

Select vCenter in your vSphere Client inventory and click Updates. The UI will walk you through the vCenter upgrade.

You should back up vCenter before performing the upgrade. If you are not familiar with how to do this, start here:

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-0/vcenter-configuration/configuring-vcenter-server-using-the-management-interface/configure-and-schedule-backups.html

You might see a warning, "Plugin should be upgraded to proceed further." Simply click the Upgrade Plug-in link before clicking Next. This will take a few minutes.

You will likely have to run the pre-checks after the plug-in upgrade (shown below). Then click Next assuming the source pre-checks ran successfully.

Click the Configure Target Appliance link. The selections in this wizard should be familiar from previous vCenter upgrades. I used the default settings.

In the final step you'll click the Start Upgrade link and set the Switchover Execution. I chose Automated from the drop-down menu. After that, I started the upgrade and waited patiently. Behind the scenes, a new vCenter appliance is deployed, the settings and data from the original appliance are copied over, and a switch-over occurs from the original appliance to the new one. As you can probably imagine, this takes a while. Ensure you have sufficient storage and compute resources for this upgrade.

You can delete the old vCenter VM after confirming the upgrade/migration completed successfully.

ESX

Finally, we upgrade the hosts. This is done using Lifecycle Manager in the vSphere Client. Click on the cluster or host if you have standalone hosts. Click on Updates. Edit the image to use the ESX version you want to upgrade to along with any vendor, firmware, and driver add-ons. Click Validate and then Save assuming the image is valid.

Lifecycle Manager should automatically check image compliance after you click Save. In the example below, we see that the single host in my lab environment is out of compliance with the new image.

Run the Pre-Check and then Remediate assuming the pre-check passed. Many upgrades still require a reboot although this is getting better with live patching. I recommend reading through the Lifecycle Manager documentation prior to using it. If you have a single host, you will need to shutdown all of the VMs on the host, put the host into maintenance mode, and perform an interactive upgrade using a CD, DVD, USB device, .iso file, ESXCLI, or script--reference this documentation for guidance.

VMware Cloud Foundation 9.0 Minimal Deployment

In the vSphere 8 era, many small environments relied on the lean duo of ESXi and vCenter for simplicity and low overhead. VMware Cloud Foundation (VCF) 9 evolves this model into a full-stack private cloud, offering automated lifecycle management, advanced networking, native container support, application blueprint provisioning, private AI, and much more--albeit with higher resource demands.

Fortunately, a minimal VCF installation enables smaller shops to use vCenter and ESX 9 without overcommitting their hardware. It is important to understand that you will not get all of the full-stack VCF 9 private cloud features and benefits when you perform a minimal installation or upgrade.

The main difference between a vSphere 8 environment and a minimal deployment of VCF 9 is the addition of VCF Operations (formerly Aria Operations). The following image shows an example of a minimal deployment--a 2-node cluster running vCenter, ESX, and VCF Operations 9.0.

Licensing was moved from vCenter to VCF Operations. In other words, you will need to deploy vCenter and VCF Operations to license and manage a 9 environment. There are numerous other benefits to running VCF Operations in a small environment such as...

• Analytics: Predictive issue detection to help prevent downtime.
• Security: Single dashboard for security posture.
• Capacity: Forecasting and recommendations to eliminate resource waste and optimize performance.
• Compliance: Monitors the environment against industry benchmarks (PCI-DSS, HIPAA, etc.).

Another key difference is full-stack VCF is deployed using the VCF Installer. If you want to keep the deployment to a minimum, you must install or upgrade the vCenter and ESX components manually and deploy VCF operations from an .ova file.

The remainder of this article provides an overview of an upgrade from vCenter and ESX 8 to a minimal installation of vCenter, ESX, and VCF Operations 9. This article does not provide detailed, step-by-step instructions. See the VCF 9.0 documentation for comprehensive coverage of deploying, upgrading, and running VCF 9.

Hardware Compatibility

Before starting any upgrade, you should verify compatibility and support of your hardware with the software version you are upgrading to. See the Broadcom Compatibility Guide. Note that there have been some changes to what is supported with VCF 9.0. Some older hardware is supported with 9.0, but might require a Request for Product Qualification/Technical Qualification (RPQ/TQ) as discussed in Broadcom Knowledge Base (KB) article 428874.

vCenter

Ensure your environment is healthy and properly configured before starting an upgrade. This includes items such as consistent NTP settings across the environment, proper DNS configuration, allocation of IP addresses for the VCF components, a current backup of your vCenter virtual appliance, and making sure all of your certificates are valid.

The upgrade of vCenter deploys a new vCenter virtual appliance and copies the settings and data from the original vCenter to the new vCenter. The original vCenter appliance is shut down when the new one is ready, but the original is not deleted. You can delete the original appliance after verifying the migration to the new appliance went well.

Next, you will need to update the download sources in Lifecycle Manager (if you haven't already) to enable the upgrade of ESX. See Broadcom KB articles 390098 and 390121. The URLs you add to Lifecycle Manager in vCenter will look similar to this:

Be sure to restart the Update Manager service in vCenter for the changes take effect.

ESX

Ensure the version of ESX you are currently running can be upgraded directly to the version you are upgrading to. Use the Product Interoperability Matrix > Upgrade Path to verify. The image below shows an example of what this upgrade path matrix looks like. In this example, you can upgrade to 9.01 or 9.0.2 if you are running 8.0U3g, but you must upgrade to 9.0.2 if you are running 8.0U3h.

I encourage you to read the documentation for using Lifecycle Manager to upgrade hosts. This documentation provides guidance on creating and using Lifecycle Manager images in vCenter.

VCF Operations

Now it is time to add VCF Operations to your vCenter 9 and ESX 9 environment. Locate the VCF Cloud Foundation Operations downloads on the Broadcom Support site.

Download the .ova file for initial deployment. The .pak file is for upgrading an existing instance of VCF Operations.

Below are the high-level steps for deploying and configuring VCF Operations:

Reserve a static IP address and add a DNS record for VCF Operations.

Select the ideal configuration size for your environment. Extra Small is good for very small environments to minimize the footprint of VCF Operations.

Power on the appliance and go to the URL on the console of the appliance. The Express Installation option is fine for most small environments.

The Cluster Status will show Not Started on the VCF Operations Administration page. Click the Start VCF Operations button.

On the Home > Overview screen, select vCenter and add an account.

Enter the necessary information to connect to vCenter. Also configure vSAN in this window if you are running vSAN.

Go to Integrations in the left column of the VCF Operations window. Notice the Status is Stopped. Place a check in the box and click Activate Management. This is necessary for license management.

Start the VCF Operations collection of data from vCenter. Click the three dots and click Start Collecting All.

Register and license your environment by following the documentation here.

Again, this article is not meant to be a step-by-step guide on deploying a minimal installation of VCF 9 or upgrading from vSphere 8 to VCF 9. The product documentation should be consulted for details and guidance on all aspects of deploying, upgrading, and running VCF.

VCF 9 Installer Validation Found Less Than Two Uplinks For The DVS

Installing VMware Cloud Foundation (VCF) 9 in a lab often means using less than ideal hardware configurations. In my case, I have ASUS Intel NUCs that have only one physical NIC for an uplink. Recommendations dictate you should have at least two for redundancy and the VCF Installer checks this. If you have less than two uplinks, VCF Installer will block the deployment. There is a way around this and it is simple.

I received the following details when the VCF Installer was validating existing components:

VcManager <vCenter FQDN>: Found less than two uplinks for the DVS <distributed virtual switch name>

Remediation: Please ensure a minimum of 2 configured uplinks for each DVS.

This is because I had only one uplink configured for the DVS. Getting around this was easier than I thought. I assumed I needed two physical NICs on each host. It turns out I only need two uplinks configured--even if one of the uplinks does not have a physical NIC backing.

I proceeded to the Network section of the vSphere Client, right-clicked the DVS, clicked Settings, and then Edit Settings.

I clicked Uplinks in the Distributed Switch - Edit Settings popup window, added an uplink, and clicked OK.

I re-ran the VCF Installer validations and everything succeeded.

Cross vCenter vMotion with Different Distributed Switch Versions

Cross vCenter vMotion enables live VM migrations from one vCenter environment to another. This prevents downtime when migrating between these two separate environments. An example use case is moving workloads from a vCenter 8U3 environment to a new VCF (vCenter) 9 deployment. However, by default, vMotion does not allow live migration of VMs across different virtual distributed switch (VDS) versions. There is a workaround.

In our example above, this would be migrating VMs from a VDS 8.0.3 to a VDS 9.0.0. When attempting this cross vCenter vMotion, you get this error:

The target host does not support the virtual machine's current hardware requirements. The destination virtual switch version or type (VDS 9.0.0) is different than the minimum required version or type (VDS 8.0.3) necessary to migrate VM from source virtual switch.

The workaround: Add the following advanced setting to the source and destination vCenter instances. Follow these steps:

1. In the vSphere Client, go to Inventory and click on the vCenter instance.
2. Click Configure.
3. Select Advanced Settings.
4. Click the Edit Settings button.
5. Scroll to the bottom of the popup window.
6. Enter config.vmprov.enableHybridMode in the Name field.
7. Enter true in the Value field.
8. Click the Add button.
9. Click the Save button.

IMPORTANT: Test a few migrations using non-critical workloads first. This is an advanced setting and I did hear of one case where there was a networking issues after a VM was migrated: loss of static IP settings on a Windows VM.

See this Broadcom Knowledge Base (KB) article: 318582

Optional: Revert the setting by changing Value to false after the migrations are complete.

Introduction to VMware vSAN Data Protection

VMware vSAN Data Protection, introduced in vSphere 8.0 Update 3, is a powerful capability included in the VMware Cloud Foundation (VCF) license for customers running vSAN ESA. This solution provides local protection for virtual machine workloads at no additional licensing cost. Key capabilities include:

• Define protection groups and retention schedules.
• Schedule crash-consistent snapshots of VMs at regular intervals.
• Create immutable snapshots to help recover from ransomware.
• Retain up to 200 snapshots per VM.
• Per-VM replication locally and/or to remote clusters.

The simplicity and tight integration of the tool make it a terrific way to augment existing data protection strategies or serve as a basic, but effective solution for protecting virtual machine workloads. The images below show the UI and provide some context. You can click the images to get a better look.

vSAN Data Protection simplifies common recovery tasks, giving administrators the ability to restore one or more VMs directly within the vSphere Client, even if they have been deleted from inventory—a capability not supported by historical VMware snapshots. Getting started involves deploying the VMware Live Recovery (VLR) virtual appliance and configuring protection groups, which can also be set to create immutable snapshots for basic ransomware resilience.

For extended functionality, VCF 9.0 introduces a unified virtual appliance and the option for vSAN-to-vSAN replication to other clusters for disaster recovery, which is available with an add-on license. More information about VMware Live Recovery: https://www.vmware.com/products/cloud-infrastructure/live-recovery 

VMware vSAN Daemon Liveness - EPD Status Abnormal

I reinstalled ESXi and vSAN ESA 8.0 on hosts that were previously running ESX and vSAN ESA 9.0. It's a lab environment. I did not bother to delete the previous VMFS volume on a local drive in each host. This is where ESXi was storing scratch data, logs, etc. After reinstalling 8.0 and turning on vSAN, I noticed the following error in vSAN Skyline Health: vSAN daemon liveness.

I expanded the health check to get more information. There I could see Overall Health, CLOMD Status, EPD Status, etc. EPD Status showed Abnormal on all of the hosts. I naturally did some searching online and came up with some clues including the knowledge base (KB) articles below.

https://knowledge.broadcom.com/external/article?articleNumber=318410

https://knowledge.broadcom.com/external/article?articleNumber=326596

They were helpful, but did not get me to resolution. The post below from a few years ago got me closer.

https://community.broadcom.com/vmware-cloud-foundation/discussion/vsan-epd-status-abnormal

I browsed the existing VMFS volume and noticed the  epd-storeV2.db file in the .locker folder.

Since it is a lab environment, I figured why not just delete the file and see if vSAN heals itself (recreates the necessary files). I put the host in maintenance mode as a precaution, deleted the file, and rebooted the host. This resolved the issue. In addition to the .db file, I noticed the addition of a new file,  epd-storeV2.db-journal.

I checked vSAN Skyline Health and the error was gone for that host. You can see the status of each host if you click the Troubleshoot button for the vSAN daemon liveness health check. I repeated that effort for each host in the cluster.

The vSAN Skyline Health error was gone after completing the process on all of the hosts in the cluster.

I probably could've restarted services on the hosts as detailed in the KB article above, but I chose to reboot them since they were already in maintenance mode.

VMware VCF Installer Warning - Evacuate Offline VMs Upgrade Policy

I ran across this warning in the VMware Cloud Foundation (VCF) Installer when deploying VCF 9:

cluster <cluster name>: Evacuate Offline VMs upgrade policy configured for cluster <cluster name> on vCenter does not match default SDDC Manager ESXi upgrade policy. It has value false in vCenter, and default value true in SDDC Manager.

The fix is simple...

In the vSphere Client, click the three horizontal lines in the top left corner.

Click Lifecycle Manager.

Click Settings.

Click Images.

Click the Edit button for Images on the right side of the window.

Click the checkbox next to "Migrate powered off and suspended VMs to other hosts in the cluster, if a host must enter maintenance mode" to check the box.

Click Save.

Re-run the VCF Installer validations.