"Fault domain" is a term that comes up fairly often in availability discussions. In IT, a fault domain usually refers to a group of servers, storage, and/or networking components that would be impacted collectively by an outage. A very common example of this is a server rack. If a top-of-rack (TOR) switch or the power distribution unit (PDU) for a server rack would fail, it would take all of the servers in that rack offline even though the servers themselves are functioning properly. That server rack is considered a fault domain.
Virtual SAN (VSAN) includes a feature called "Rack Awareness", which enables an administrator to configure fault domains in the context of a Virtual SAN cluster. Before we get into the details of this feature, let's briefly revisit the default behavior of VSAN.
Host is Implicit Fault Domain
Each host in a VSAN cluster is an implicit fault domain. Virtual SAN automatically distributes components of a VSAN object, e.g., virtual disk (VMDK), across hosts in a cluster based on the Number of Failures to Tolerate (FTT) rule assigned to that virtual machine (VM).
For example, a VM has the default setting of FTT=1 - the object can withstand the failure of one disk or host that contains a component and the object is still accessible. These objects are spread across multiple hosts to provide that resiliency. The following diagram shows this basic example. The dark blue dots are data components and the smaller, light blue dot is the witness component for this object. We learned in Part 2 of this blog series that the object remains accessible as long as greater than 50% (2 out of 3, in this case) are available.
While the failure of a disk or entire host can be tolerated, what if all of these servers are in the same rack and the TOR switch goes offline? Answer: All hosts are isolated from each other, as described in Part 3 of this series, and none of the objects are accessible.
To mitigate this risk, the servers in a VSAN cluster should be spread across server racks and fault domains must be configured in the VSAN user interface. After fault domains are configured, VSAN will redistribute the components across server racks to eliminate the risk of a rack failure taking multiple objects offline. The diagram below shows what this might look like with a 12-node VSAN cluster spread across four server racks.
It is very simple to configure fault domains in VSAN. This 84-second video (no audio) below demonstrates how easy it is to configure rack awareness using fault domains in the VSAN user interface.
Regardless of whether fault domains are configured or not, here are some numbers to keep in mind when designing a VSAN cluster:
RAID-1 mirroring requires 2N+1 fault domains
FTT=1 requires at least three fault domains
FTT=2 ... five fault domains
FTT=3 ... seven fault domains
RAID-5/6 erasure coding requires 2N+2 fault domains
FTT=1 ... four fault domains
FTT=2 ... six fault domains
FTT=3 is not supported with erasure coding
If you attempt to assign a policy that requires more hosts than what are available, you will see an error message like this one:
Consider an Additional Host
Another scenario to consider is enough fault domains for rebuilds when there is a host or disk failure. As discussed in Part 4 of this series, VSAN will either attempt to rebuild objects with reduced availability immediately or after 60 minutes. However, there must be enough fault domains to satisfy the availability rules in a storage policy. Consider a 3-node cluster. If a host is offline, that leaves only two hosts. VSAN objects will still be available, but the cluster will not be able to tolerate another failure. Furthermore, VSAN is unable to rebuild components to restore redundancy while that host is offline. Adding a fourth host to the cluster enables the rebuild of components in cases where a disk or host has failed or must be in maintenance mode for an extended period of time.
Recommendation: Add at least one more host for rebuilds. Example: RAID-1, FTT=1, at least 4 hosts/fault domains.
In Part 6, we move the conversation from unplanned downtime to planned downtime and cover vSphere Maintenance Mode in a VSAN cluster environment.