From ec985cce17fa6ea79342ffddeea14f55bdd52f99 Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Mon, 7 Jun 2021 11:16:43 +0200 Subject: [PATCH 01/11] Cleanup --- source/adminguide/accounts.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/adminguide/accounts.rst b/source/adminguide/accounts.rst index a83bcd105d..cd6edc5360 100644 --- a/source/adminguide/accounts.rst +++ b/source/adminguide/accounts.rst @@ -337,7 +337,7 @@ the user are used. #. CloudStack searches for it in LDAP by the configured ``ldap.username.attribute``. - #. If an LDAP user is found is found, CloudStack does a bind + #. If an LDAP user is found, CloudStack does a bind request with the returned principal for that LDAP user and the entered password. @@ -345,7 +345,7 @@ the user are used. authenticated user exists in the domain it is trying to log on to. - #. If the user exists in CloudStack, it is ensured to be enabled + #. If the user exists in CloudStack, it is ensured to be enabled. #. If it doesn't exist it is created in a new account with the username as names for both account and user. From 5d2694d2986606b439d678e70787b38f77edff7b Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Fri, 11 Jun 2021 11:40:53 +0200 Subject: [PATCH 02/11] Added host HA detail --- source/adminguide/reliability.rst | 161 ++++++++++++++++++++++++++---- 1 file changed, 143 insertions(+), 18 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 44647cbac8..830fc0c373 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -78,25 +78,8 @@ HA features work with iSCSI or NFS primary storage. HA with local storage is not supported. -HA for Hosts ------------- - -The user can specify a virtual machine as HA-enabled. By default, all -virtual router VMs and Elastic Load Balancing VMs are automatically -configured as HA-enabled. When an HA-enabled VM crashes, CloudStack -detects the crash and restarts the VM automatically within the same -Availability Zone. HA is never performed across different Availability -Zones. CloudStack has a conservative policy towards restarting VMs and -ensures that there will never be two instances of the same VM running at -the same time. The Management Server attempts to start the VM on another -Host in the same cluster. - -HA features work with iSCSI or NFS primary storage. HA with local -storage is not supported. - - Dedicated HA Hosts -~~~~~~~~~~~~~~~~~~ +------------------ One or more hosts can be designated for use only by HA-enabled VMs that are restarting due to a host failure. Setting up a pool of such @@ -126,6 +109,148 @@ that you want to dedicate to HA-enabled VMs. a crash. +HA-Enabled Hosts +---------------- + +The user can specify a host as HA-enabled, In the event of a host +failure, attemps will be made to recover the failed host by first +issuing some OOBM commands. If the host recovery fails the host will be +fenced and placed into maintenance mode. To restore the host to normal +operation, manual intervention would then be required. + +Out of band management is a requirement of HA-Enabled hosts and has to be +confiured on all intended participating hosts. +(see `“Out of band management” `_). + +Host-HA has granular configuration on a host/cluster/zone level. In a large +environment, some hosts from a cluster can be HA-enabled and some not, + +Host-HA uses a state machine design to manage the operations of recovering +and fencing hosts. The current status of a host is reported when quering a +specific host. + +Timely health investigations are done on HA-Enabled hosts to monitor for +any failures. Specific thersholds can be set for failed investigations, +only when it’s exceeded, will the host transition to a different state. + +Host-HA uses both health checks and activity checks to make decisions on +recovering and fencing actions. Once determined that the host is in faulty +state (health checks failed) it runs activity checks to figure out if there is +any disk activity on the VMs running on the specific host. + +HA Resource Management Service +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The HA Resource Management Service manages the check/recovery cycle including +periodic execution, concurrency management, persistence, back pressure and +clustering operations. Administrators associate a provider with a partition +type (e.g. KVM HA Host provider to clusters) and may override the provider on a +per-partition (i.e. zone, cluster, or pod) basis. The service operates on all +resources of the type supported by the provider contained in a partition. +Administrators can also enable or disable HA operations globally or on a +per-partition basis. + +Only one (1) HA provider per resource type may be specified for a partition. +Nested HA providers by resource type is not supported (e.g. a pod +specifying an HA resource provider for hosts and a containing cluster +specifying a HA resource provider for hosts). The service is designed to be +opt-in where by only resources with a defined provider and HA enabled will be +managed. + +For each resource in an HA partition, the HA Resource Management Service +maintains and persists an FSM composed of the following states: + +DISABLED: +The resource is part of a partition where HA operations have been disabled. + +AVAILABLE: +The initial health and eligibility of the resource for HA management is +currently found to be fine. The resource stays in the AVAILABLE state based on +the passage of the most recent health check and it's containing partition has +an HA state of ACTIVE and all the eligibility conditions are met. When +transitioning to this state, the number of retry attempts is reset. + +INELIGIBLE: +The resource's enclosing partition has an HA state of ACTIVE but its current +state does not support HA check and/or recovery operations. If it is a single +host in the cluster for a KVM provider the host will become ineligible as the +KVM provider requires a neighbouring host to carry on its investigations. Any +resource in maintenance mode is automatically transitioned to INELIGIBLE. + +SUSPECT: +The resource pending an activity check due to failing its most recent health +check. If the maximum recovery attempts has been exceeded, the HA state is +transitioned to FENCED. Otherwise, the node will be scheduled for an activity +check. When a node fails multiple activity checks/recovery attempts, the +duration between re-attempts will decay to the maximum interval specified by +the provider (e.g. first check after 10 seconds, second check after 20 seconds, +third check after 40 seconds to a maximum interval of 250 seconds). + +DEGRADED: +The resource cannot be managed by the control plane but passed its +most recent activity check indicating that the resource is still servicing +end-user requests + +CHECKING: +An activity check is currently being performed on the resource. The HA provider +defines the number of activity checks must be performed and number of failed +activity checks required to trigger recovery. If the number of activity checks +is greater than or equal to the total number of acceptable failures, the HA +state of the resource is transitioned to RECOVERING causing a recovery attempt. +If the total number of activity checks has been attempted and number of failure +is less than the number of acceptable failures, the HA state of the resource +will be transitioned to DEGRADED. If the number of activity checks is less than +the total number of required and the number of failures is +less than the acceptable number of failures, then the HA state of the resource +is transitioned to SUSPECT – triggering another activity check. + +RECOVERING: +Recovery operations are in-progress to bring the resource back to +a healthy state. If the recovery operation succeeds, the HA state of the +resource will be transitioned to INITIALIZING. If the recovery operation fails, +the HA state of the resource is transitioned to FENCED. Since Recovering is not +idempotent it is further split into ‘Recovering’ and ‘Recovered’. + +FENCED: +The resource is not operating normally and automated attempts to +recover it failed. Manual operator intervention is required to recover the +resource. Since Fenced operation is not idempotent it is further split into +‘Fencing’ and ‘Fenced’. + +When HA is enabled for a partition, the HA state of all contained resources +will be transitioned from DISABLED to AVAILABLE. Based on the state models, the +following failure scenarios and their responses will be handled by the HA +resource management service: + +- Activity check operation fails on the resource: Provide a semantic in the + activity check protocol to express that an error while performing the + activity check and a reason for the failure (e.g. unable to access the NFS + mount). If the maximum number of activity check attempts has not been + exceeded, the activity check will be retried. + +- Slow activity check operation: After a configurable timeout, the HA resource + management service abandons the check. The response to this condition would + be the same as a failure to recover the resource. + +- Traffic flood due to a large number of resource recoveries: The HA resource + management service must limit the number of concurrent recovery operations + permitted to avoid overwhelming the management server with resource status + updates as recovery operations complete. + +- Processor/memory starvation due to large number of activity check + operations: The HA resource management service must limit the number of + concurrent activity check operations permitted per management server to + prevent checks from starving other management server activities of scarce + processor and/or memory resources. + +- A SUSPECT, CHECKING, or RECOVERING resource passes a health check before the + state action completes: The HA resource management service refreshes the HA + state of the resource before transition. If it does not match the expected + current state, the result of state action is ignored. + +For further information around the inner workings of Host HA, please refer +to the design document at `https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA` + Primary Storage Outage and Data Loss ------------------------------------ From a4bc3a7bcf15662af9cf867618bd6fa0edef0e1e Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Tue, 22 Jun 2021 11:42:14 +0200 Subject: [PATCH 03/11] Added multiple management servers internal load balancing --- source/adminguide/reliability.rst | 84 ++++++++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 2 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 830fc0c373..001e9d3189 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -61,6 +61,84 @@ still available but the system VMs will not be able to contact the management server. +Multiple Management Servers Support on agents +--------------------------------------------- + +In a Cloudstack environment with multiple management servers, an agent can be +configured, based on an algorithm, to which management server to connect to. +This can be useful as an internal loadbalancer or for high availability. +An administrator is responsible for setting the list of management servers and +choosing a sorting algorithm using global settings. +The management server is responsible for propagating the settings to the +connected agents. + +Examples of an agent includes, the process responsible for communication to the +management server, running inside of the Secondary Storage Virtual Machine +(SSVM), Console Proxy Virtual Machine (CPVM) or the cloudstack-agent running on +a KVM host. + +The three global settings that need to be configured are the following: + +- hosts: a comma seperated list of management server IP addresses +- indirect.agent.lb.algorithm: The algorithm for the indirect agent LB +- indirect.agent.lb.check.interval: The preferred host check interval + for the agent's background task that checks and switches to an agent's + preferred host. + +These settings can be configured from the global settings page in the UI or +using the updateConfiguration API call. + +The indirect.agent.lb.algorithm setting supports following algorithm options: + +- static: Use the list of management server IP addresses as provided. +- roundrobin: Evenly spread hosts across management servers, based on the + host's id. +- shuffle: Pseudo Randomly sort the list (this is not recommended for + production). + +Any changes to the global settings - `indirect.agent.lb.algorithm` and +`host` does not require restarting of the management server(s) and the +agents. A change in these global settings will be propagated to all connected +agents. + +The comma-separated management server list is propagated to agents in +following cases: +- An addition of an agent (including ssvm, cpvm system VMs). +- Connection or reconnection of an agent to a management server. +- After an administrator changes the 'host' and/or the +'indirect.agent.lb.algorithm' global settings. + +On the agent side, the 'host' setting is saved in its properties file as: +`host=@`. + +From the agent's perspective, the first address in the propagated list +will be considered the preferred host. A new background task can be +activated by configuring the `indirect.agent.lb.check.interval` which is +a cluster level global setting from CloudStack and adminitrators can also +override this by configuring the 'host.lb.check.interval' in the +`agent.properties` file. + +When an agent gets a host and algorithm combination, the host specific +background check interval is also sent and is dynamically reconfigured +in the background task without need to restart agents. + +Note: The 'static' and 'roundrobin' algorithms, strictly checks for the +order as expected by them, however, the 'shuffle' algorithm just checks +for content and not the order of the comma separate management server +host addresses. + +To make things more clear, consider this example: +Suppose an environment which has 3 management servers: A, B and C and +3 KVM agents. + +Setting 'host' = 'A,B,C', agents will receive lists depending on +'direct.agent.lb' value: + +'static': Each agent will receive the list: 'A,B,C' +'roundrobin': First agent receives: 'A,B,C', second agent +receives: 'B,C,A', third agent receives: 'C,B,A' +'shuffle': Each agent will receive a list in random order. + HA-Enabled Virtual Machines --------------------------- @@ -248,8 +326,10 @@ resource management service: state of the resource before transition. If it does not match the expected current state, the result of state action is ignored. -For further information around the inner workings of Host HA, please refer -to the design document at `https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA` +For further information around the inner workings of Host HA, refer +to the design document at +`https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA +`_ Primary Storage Outage and Data Loss ------------------------------------ From 7aa7d1794fc9c35ef5d90af48e43fbcb0415e58c Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Wed, 30 Jun 2021 21:02:43 +0200 Subject: [PATCH 04/11] Shortened FSM state descriptions --- source/adminguide/reliability.rst | 76 ++++++++----------------------- 1 file changed, 18 insertions(+), 58 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 001e9d3189..7075771059 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -236,64 +236,24 @@ opt-in where by only resources with a defined provider and HA enabled will be managed. For each resource in an HA partition, the HA Resource Management Service -maintains and persists an FSM composed of the following states: - -DISABLED: -The resource is part of a partition where HA operations have been disabled. - -AVAILABLE: -The initial health and eligibility of the resource for HA management is -currently found to be fine. The resource stays in the AVAILABLE state based on -the passage of the most recent health check and it's containing partition has -an HA state of ACTIVE and all the eligibility conditions are met. When -transitioning to this state, the number of retry attempts is reset. - -INELIGIBLE: -The resource's enclosing partition has an HA state of ACTIVE but its current -state does not support HA check and/or recovery operations. If it is a single -host in the cluster for a KVM provider the host will become ineligible as the -KVM provider requires a neighbouring host to carry on its investigations. Any -resource in maintenance mode is automatically transitioned to INELIGIBLE. - -SUSPECT: -The resource pending an activity check due to failing its most recent health -check. If the maximum recovery attempts has been exceeded, the HA state is -transitioned to FENCED. Otherwise, the node will be scheduled for an activity -check. When a node fails multiple activity checks/recovery attempts, the -duration between re-attempts will decay to the maximum interval specified by -the provider (e.g. first check after 10 seconds, second check after 20 seconds, -third check after 40 seconds to a maximum interval of 250 seconds). - -DEGRADED: -The resource cannot be managed by the control plane but passed its -most recent activity check indicating that the resource is still servicing -end-user requests - -CHECKING: -An activity check is currently being performed on the resource. The HA provider -defines the number of activity checks must be performed and number of failed -activity checks required to trigger recovery. If the number of activity checks -is greater than or equal to the total number of acceptable failures, the HA -state of the resource is transitioned to RECOVERING causing a recovery attempt. -If the total number of activity checks has been attempted and number of failure -is less than the number of acceptable failures, the HA state of the resource -will be transitioned to DEGRADED. If the number of activity checks is less than -the total number of required and the number of failures is -less than the acceptable number of failures, then the HA state of the resource -is transitioned to SUSPECT – triggering another activity check. - -RECOVERING: -Recovery operations are in-progress to bring the resource back to -a healthy state. If the recovery operation succeeds, the HA state of the -resource will be transitioned to INITIALIZING. If the recovery operation fails, -the HA state of the resource is transitioned to FENCED. Since Recovering is not -idempotent it is further split into ‘Recovering’ and ‘Recovered’. - -FENCED: -The resource is not operating normally and automated attempts to -recover it failed. Manual operator intervention is required to recover the -resource. Since Fenced operation is not idempotent it is further split into -‘Fencing’ and ‘Fenced’. +maintains and persists an "Finite State Machine" composed of the following +states: + +- Available - The feature is enabled and Host-HA is available. +- Suspect - There are health checks failing with the host. +- Checking - Activity checks are being performed. +- Degraded - The host is passing the activity check ratio and still providing + service to the end user, but it cannot be managed from the CloudStack + management server. +- Recovering - The Host-HA framework is trying to recover the host by issuing + OOBM jobs. +- Recovered - The Host-HA framework has recovered the host successfully. +- Fencing - The Host-HA framework is trying to fence the host by issuing OOBM + jobs. +- Fenced - The Host-HA framework has fenced the host successfully. +- Disabled - The feature is disabled for the host. +- Ineligible - The feature is enabled, but it cannot be managed successfully by + the Host-HA framework. (OOBM is possibly not configured properly) When HA is enabled for a partition, the HA state of all contained resources will be transitioned from DISABLED to AVAILABLE. Based on the state models, the From 933909a429daeb122b96f7272b5b3bf5b2f99acb Mon Sep 17 00:00:00 2001 From: Spaceman1984 <49917670+Spaceman1984@users.noreply.github.com> Date: Thu, 1 Jul 2021 10:28:25 +0200 Subject: [PATCH 05/11] spelling error Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com> --- source/adminguide/reliability.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 7075771059..8c2a679db5 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -114,7 +114,7 @@ On the agent side, the 'host' setting is saved in its properties file as: From the agent's perspective, the first address in the propagated list will be considered the preferred host. A new background task can be activated by configuring the `indirect.agent.lb.check.interval` which is -a cluster level global setting from CloudStack and adminitrators can also +a cluster level global setting from CloudStack and administrators can also override this by configuring the 'host.lb.check.interval' in the `agent.properties` file. From d77da1216c67a49cfbe85df80d335791005743f3 Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Fri, 2 Jul 2021 08:29:15 +0200 Subject: [PATCH 06/11] Moved note about algorithms --- source/adminguide/reliability.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 7075771059..5fd8a8917d 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -96,6 +96,11 @@ The indirect.agent.lb.algorithm setting supports following algorithm options: - shuffle: Pseudo Randomly sort the list (this is not recommended for production). +.. note:: + The 'static' and 'roundrobin' algorithms, strictly checks for the order as + expected by them, however, the 'shuffle' algorithm just checks for content + and not the order of the comma separate management server host addresses. + Any changes to the global settings - `indirect.agent.lb.algorithm` and `host` does not require restarting of the management server(s) and the agents. A change in these global settings will be propagated to all connected @@ -122,11 +127,6 @@ When an agent gets a host and algorithm combination, the host specific background check interval is also sent and is dynamically reconfigured in the background task without need to restart agents. -Note: The 'static' and 'roundrobin' algorithms, strictly checks for the -order as expected by them, however, the 'shuffle' algorithm just checks -for content and not the order of the comma separate management server -host addresses. - To make things more clear, consider this example: Suppose an environment which has 3 management servers: A, B and C and 3 KVM agents. From bff89742bfdbb6beb80278bee84bd06696f670dc Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Fri, 2 Jul 2021 08:31:21 +0200 Subject: [PATCH 07/11] Removed heading --- source/adminguide/reliability.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 5fd8a8917d..84d76807e7 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -216,9 +216,6 @@ recovering and fencing actions. Once determined that the host is in faulty state (health checks failed) it runs activity checks to figure out if there is any disk activity on the VMs running on the specific host. -HA Resource Management Service -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - The HA Resource Management Service manages the check/recovery cycle including periodic execution, concurrency management, persistence, back pressure and clustering operations. Administrators associate a provider with a partition From afaa11d2f44c8a1001482d14932d956f6e818f3c Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Fri, 2 Jul 2021 10:32:47 +0200 Subject: [PATCH 08/11] Added agent name --- source/adminguide/reliability.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 52be14c6a4..049f280807 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -73,9 +73,9 @@ The management server is responsible for propagating the settings to the connected agents. Examples of an agent includes, the process responsible for communication to the -management server, running inside of the Secondary Storage Virtual Machine -(SSVM), Console Proxy Virtual Machine (CPVM) or the cloudstack-agent running on -a KVM host. +management server (cloud.service), running inside of the Secondary Storage +Virtual Machine (SSVM), Console Proxy Virtual Machine (CPVM) or the +cloudstack-agent running on a KVM host. The three global settings that need to be configured are the following: From 82cb0204dda1a9a993a2c19ed427cc4565a033f7 Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Fri, 2 Jul 2021 12:55:40 +0200 Subject: [PATCH 09/11] Simplified explanation --- source/adminguide/reliability.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 049f280807..896dfc2974 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -70,12 +70,8 @@ This can be useful as an internal loadbalancer or for high availability. An administrator is responsible for setting the list of management servers and choosing a sorting algorithm using global settings. The management server is responsible for propagating the settings to the -connected agents. - -Examples of an agent includes, the process responsible for communication to the -management server (cloud.service), running inside of the Secondary Storage -Virtual Machine (SSVM), Console Proxy Virtual Machine (CPVM) or the -cloudstack-agent running on a KVM host. +connected agents (running inside of the Secondary Storage +Virtual Machine, Console Proxy Virtual Machine or the KVM hosts). The three global settings that need to be configured are the following: From 3b98a4044be5276a4bed9b596a386e0ee357c55d Mon Sep 17 00:00:00 2001 From: spaceman1984 Date: Mon, 5 Jul 2021 11:33:58 +0200 Subject: [PATCH 10/11] Changes states to uppercase --- source/adminguide/reliability.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index 896dfc2974..d51369d2f1 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -232,20 +232,20 @@ For each resource in an HA partition, the HA Resource Management Service maintains and persists an "Finite State Machine" composed of the following states: -- Available - The feature is enabled and Host-HA is available. -- Suspect - There are health checks failing with the host. -- Checking - Activity checks are being performed. -- Degraded - The host is passing the activity check ratio and still providing +- AVAILABLE - The feature is enabled and Host-HA is available. +- SUSPECT - There are health checks failing with the host. +- CHECKING - Activity checks are being performed. +- DEGRADED - The host is passing the activity check ratio and still providing service to the end user, but it cannot be managed from the CloudStack management server. -- Recovering - The Host-HA framework is trying to recover the host by issuing +- RECOVERING - The Host-HA framework is trying to recover the host by issuing OOBM jobs. -- Recovered - The Host-HA framework has recovered the host successfully. -- Fencing - The Host-HA framework is trying to fence the host by issuing OOBM +- RECOVERED - The Host-HA framework has recovered the host successfully. +- FENCING - The Host-HA framework is trying to fence the host by issuing OOBM jobs. -- Fenced - The Host-HA framework has fenced the host successfully. -- Disabled - The feature is disabled for the host. -- Ineligible - The feature is enabled, but it cannot be managed successfully by +- FENCED - The Host-HA framework has fenced the host successfully. +- DISABLED - The feature is disabled for the host. +- INELIGIBLE - The feature is enabled, but it cannot be managed successfully by the Host-HA framework. (OOBM is possibly not configured properly) When HA is enabled for a partition, the HA state of all contained resources From d664b479723979ae04e433c4320e6cc7d193f349 Mon Sep 17 00:00:00 2001 From: Spaceman1984 <49917670+Spaceman1984@users.noreply.github.com> Date: Mon, 19 Jul 2021 13:20:48 +0200 Subject: [PATCH 11/11] Update source/adminguide/reliability.rst Co-authored-by: Abhishek Kumar --- source/adminguide/reliability.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/adminguide/reliability.rst b/source/adminguide/reliability.rst index d51369d2f1..859b971614 100644 --- a/source/adminguide/reliability.rst +++ b/source/adminguide/reliability.rst @@ -204,7 +204,7 @@ and fencing hosts. The current status of a host is reported when quering a specific host. Timely health investigations are done on HA-Enabled hosts to monitor for -any failures. Specific thersholds can be set for failed investigations, +any failures. Specific thresholds can be set for failed investigations, only when it’s exceeded, will the host transition to a different state. Host-HA uses both health checks and activity checks to make decisions on