VCF 9 Installer Validation Found Less Than Two Uplinks For The DVS

Installing VMware Cloud Foundation (VCF) 9 in a lab often means using less than ideal hardware configurations. In my case, I have ASUS Intel NUCs that have only one physical NIC for an uplink. Recommendations dictate you should have at least two for redundancy and the VCF Installer checks this. If you have less than two uplinks, VCF Installer will block the deployment. There is a way around this and it is simple.

I received the following details when the VCF Installer was validating existing components:

VcManager <vCenter FQDN>: Found less than two uplinks for the DVS <distributed virtual switch name>

Remediation: Please ensure a minimum of 2 configured uplinks for each DVS.

This is because I had only one uplink configured for the DVS. Getting around this was easier than I thought. I assumed I needed two physical NICs on each host. It turns out I only need two uplinks configured--even if one of the uplinks does not have a physical NIC backing.

I proceeded to the Network section of the vSphere Client, right-clicked the DVS, clicked Settings, and then Edit Settings.

I clicked Uplinks in the Distributed Switch - Edit Settings popup window, added an uplink, and clicked OK.

I re-ran the VCF Installer validations and everything succeeded.

Cross vCenter vMotion with Different Distributed Switch Versions

Cross vCenter vMotion enables live VM migrations from one vCenter environment to another. This prevents downtime when migrating between these two separate environments. An example use case is moving workloads from a vCenter 8U3 environment to a new VCF (vCenter) 9 deployment. However, by default, vMotion does not allow live migration of VMs across different virtual distributed switch (VDS) versions. There is a workaround.

In our example above, this would be migrating VMs from a VDS 8.0.3 to a VDS 9.0.0. When attempting this cross vCenter vMotion, you get this error:

The target host does not support the virtual machine's current hardware requirements. The destination virtual switch version or type (VDS 9.0.0) is different than the minimum required version or type (VDS 8.0.3) necessary to migrate VM from source virtual switch.

The workaround: Add the following advanced setting to the source and destination vCenter instances. Follow these steps:

1. In the vSphere Client, go to Inventory and click on the vCenter instance.
2. Click Configure.
3. Select Advanced Settings.
4. Click the Edit Settings button.
5. Scroll to the bottom of the popup window.
6. Enter config.vmprov.enableHybridMode in the Name field.
7. Enter true in the Value field.
8. Click the Add button.
9. Click the Save button.

IMPORTANT: Test a few migrations using non-critical workloads first. This is an advanced setting and there have been a few reports of networking issues after a VM was migrated, e.g., loss of static IP settings.

Optional: Revert the setting by changing Value to false after the migrations are complete.

Introduction to VMware vSAN Data Protection

VMware vSAN Data Protection, introduced in vSphere 8.0 Update 3, is a powerful capability included in the VMware Cloud Foundation (VCF) license for customers running vSAN ESA. This solution provides local protection for virtual machine workloads at no additional licensing cost. Key capabilities include:

• Define protection groups and retention schedules.
• Schedule crash-consistent snapshots of VMs at regular intervals.
• Create immutable snapshots to help recover from ransomware.
• Retain up to 200 snapshots per VM.
• Per-VM replication locally and/or to remote clusters.

The simplicity and tight integration of the tool make it a terrific way to augment existing data protection strategies or serve as a basic, but effective solution for protecting virtual machine workloads. The images below show the UI and provide some context. You can click the images to get a better look.

vSAN Data Protection simplifies common recovery tasks, giving administrators the ability to restore one or more VMs directly within the vSphere Client, even if they have been deleted from inventory—a capability not supported by historical VMware snapshots. Getting started involves deploying the VMware Live Recovery (VLR) virtual appliance and configuring protection groups, which can also be set to create immutable snapshots for basic ransomware resilience.

For extended functionality, VCF 9.0 introduces a unified virtual appliance and the option for vSAN-to-vSAN replication to other clusters for disaster recovery, which is available with an add-on license. More information about VMware Live Recovery: https://www.vmware.com/products/cloud-infrastructure/live-recovery 

VMware vSAN Daemon Liveness - EPD Status Abnormal

I reinstalled ESXi and vSAN ESA 8.0 on hosts that were previously running ESX and vSAN ESA 9.0. It's a lab environment. I did not bother to delete the previous VMFS volume on a local drive in each host. This is where ESXi was storing scratch data, logs, etc. After reinstalling 8.0 and turning on vSAN, I noticed the following error in vSAN Skyline Health: vSAN daemon liveness.

I expanded the health check to get more information. There I could see Overall Health, CLOMD Status, EPD Status, etc. EPD Status showed Abnormal on all of the hosts. I naturally did some searching online and came up with some clues including the knowledge base (KB) articles below.

https://knowledge.broadcom.com/external/article?articleNumber=318410

https://knowledge.broadcom.com/external/article?articleNumber=326596

They were helpful, but did not get me to resolution. The post below from a few years ago got me closer.

https://community.broadcom.com/vmware-cloud-foundation/discussion/vsan-epd-status-abnormal

I browsed the existing VMFS volume and noticed the  epd-storeV2.db file in the .locker folder.

Since it is a lab environment, I figured why not just delete the file and see if vSAN heals itself (recreates the necessary files). I put the host in maintenance mode as a precaution, deleted the file, and rebooted the host. This resolved the issue. In addition to the .db file, I noticed the addition of a new file,  epd-storeV2.db-journal.

I checked vSAN Skyline Health and the error was gone for that host. You can see the status of each host if you click the Troubleshoot button for the vSAN daemon liveness health check. I repeated that effort for each host in the cluster.

The vSAN Skyline Health error was gone after completing the process on all of the hosts in the cluster.

I probably could've restarted services on the hosts as detailed in the KB article above, but I chose to reboot them since they were already in maintenance mode.

VMware VCF Installer Warning - Evacuate Offline VMs Upgrade Policy

I ran across this warning in the VMware Cloud Foundation (VCF) Installer when deploying VCF 9:

cluster <cluster name>: Evacuate Offline VMs upgrade policy configured for cluster <cluster name> on vCenter does not match default SDDC Manager ESXi upgrade policy. It has value false in vCenter, and default value true in SDDC Manager.

The fix is simple...

In the vSphere Client, click the three horizontal lines in the top left corner.

Click Lifecycle Manager.

Click Settings.

Click Images.

Click the Edit button for Images on the right side of the window.

Click the checkbox next to "Migrate powered off and suspended VMs to other hosts in the cluster, if a host must enter maintenance mode" to check the box.

Click Save.

Re-run the VCF Installer validations.

VMware Live Recovery Not Accessible Error

• VMware Live Recovery 9.0.3.0 and later combine the control plane, replication, and data protection into a single appliance.
• Ensure DNS--forward and reverse lookup--are functioning correctly.
• A "Not accessible" error could indicate an issue with certificates.

I recently had the opportunity to set up the latest VMware Live Recovery appliance, which simplifies deployment and configuration into a single virtual appliance. More details about this new model can be found in this blog article.

I encountered an issue where the UI displayed vSphere Replication and VMware Live Site Recovery, formerly Site Recovery Manager (SRM), was marked as "Not accessible."

I double-checked that DNS was correctly configured, and I was able to ping all directions between the vCenter instances and the VMware Live Recovery appliances. It was not a name resolution or network connectivity issue. Pro tip: Always verify DNS is configured correctly. I've seen this cause issues in countless scenarios.

After some searching and trial and error, I determined the issue was a mismatch between the server name and what was in the certificate. Specifically, the Common Name (CN) in the Issued to section. I logged into the appliance management UI by appending port 5480 to the fully-qualified domain name (FQDN) of the server. Using the example FQDN in the screenshot, the URL would be https://vlr01.vmware.lab:5480.

After logging in with the "admin" username and the password I specified for that account during deployment, I selected Certificates in the left menu bar. Then, I clicked Change near the top right corner to generate a new self-signed certificate.

I completed the fields in the Change certificate UI and clicked the Change button to generate the new certificate. I verified the CN matched the FQDN and rebooted the VMware Live Recovery virtual appliance just for good measure. When the server came back up, the error was gone, and I had the green checkmark OK indicator.

Technical Introduction to vSAN ESA 2-Node Clusters

• VMware vSAN can be deployed in a 2-node cluster.

• vSAN ESA 2-node clusters provide excellent performance and availability in a small form factor.

• A vSAN Witness Host virtual appliance enables protection against "split-brain."

Introduction

VMware's vSAN Express Storage Architecture (ESA) has streamlined hyperconverged infrastructure (HCI) by optimizing performance and efficiency. A particularly compelling deployment model within this architecture is the 2-node cluster. This setup offers a high-availability solution ideal for small sites and edge computing environments where space and hardware resources are limited.

Core Architecture and Requirements

A vSAN ESA 2-node cluster is a specialized configuration that provides data redundancy and high availability with just two physical hosts at the primary site. Unlike larger clusters, it doesn't require a minimum of three nodes. Instead, it relies on a virtual machine appliance called a vSAN Witness Host located on a host other than the two physical nodes.

Key Components

1. Two Physical Nodes: These are two physical ESXi hosts that run virtual machines and store the data associated with those virtual machines. In ESA, these nodes must be all-NVMe, utilizing high-performance, enterprise-class NVMe storage devices for both caching and capacity. There are no traditional vSAN disk groups; instead, there's a single, flexible storage pool.

2. Witness Host: The witness is a virtual machine appliance that resides on a third host, typically at a primary data center, when deploying a 2-node cluster in a remote site. Its primary role is to act as a tiebreaker in the event of a failure or network partition between the two data nodes. The witness host stores only metadata in the form of witness components. It does not run virtual machines or store virtual machine data (VMDK files).

Networking Essentials

Proper networking is critical for a stable 2-node cluster.

• Data Node Interconnect: The two data nodes need a high-speed, low-latency network connection, usually a direct link or dedicated switch. At least 10 Gbps is required to support vSAN traffic and vMotion effectively. 25Gbps or higher is recommended.

• vSAN Witness Host Traffic: The connection between the physical hosts and the vSAN Witness Host has specific requirements. While the latency requirements are less strict than for stretched clusters, they are still important.

• Latency: The round-trip time (RTT) to the witness should be less than 500 ms.
• Bandwidth: A 2 Mbps connection is generally sufficient, as only metadata is transmitted.
• Traffic: It's best practice to tag the witness traffic on a separate VMkernel adapter to keep it isolated from other network traffic.

Data Placement and Protection

In a vSAN ESA 2-node cluster, data protection is achieved through mirroring. The default storage policy for a 2-node cluster is RAID 1 (Mirroring).

Here's how it works:

1. A virtual machine's disk object (VMDK) is created on the vSAN datastore.

2. The object has two complete copies, or replicas.

3. One replica is placed on Node A, and the second replica is placed on Node B.

4. A small witness component is created and placed on the witness host.

This creates a data layout of Replica 1, Replica 2, and Witness. This configuration ensures that if one data node fails, a complete copy of the data is still available on the surviving node. The witness component ensures that there is a quorum ($greater\; than50\%$) of components available to keep the VM object online.

For example, if Node A fails, Node B still has a complete data replica, and the witness host provides the third component vote. The cluster recognizes that a valid, complete copy of the data exists and keeps the VM running (after a vSphere HA restart) on Node B.

Handling Failures: Split-Brain Scenario

The primary function of the witness is to prevent a "split-brain" scenario. Imagine the direct network link between Node A and Node B fails, but both nodes can still communicate with the witness host.

• Without a witness, both nodes would think the other is offline and would attempt to take sole ownership of the virtual machines, leading to data corruption.

• With a witness, both Node A and Node B will attempt to place a lock on the witness components for their VMs. The node that successfully acquires the lock gains ownership of the VM objects and continues writing to its replica, preventing a split-brain condition and ensuring data integrity.

Summary

vSAN ESA 2-node clusters offer a robust and efficient high-availability solution for small environments and edge solutions such as remote offices. By utilizing the power of NVMe hardware and a simple yet effective witness architecture, it delivers enterprise-grade speed and resilience in a compact form factor.