Detect offline clusters #2933
Draft
Detect offline clusters #2933
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0xavi0
reviewed
Oct 8, 2024
weyfonk
force-pushed
the
594-detect-offline-cluster
branch
2 times, most recently
from
38a697b
to
452e56e
Compare
weyfonk
force-pushed
the
594-detect-offline-cluster
branch
2 times, most recently
from
fd85993
to
a1c1b94
Compare
This allows the Fleet controller to detect offline clusters and update statuses of bundle deployments targeting offline clusters. Next to do: * understand how bundle deployment status updates should be propagated to bundles (which currently simply appear as `Modified`) and further up * write tests (eg. integration tests, updating cluster status by hand?) * set sensible defaults (eg. monitoring interval higher, and threshold higher than agent's cluster status reporting interval)
This enables that state to be reflected upwards in bundle, GitRepo, cluster and cluster group statuses.
This also provides unit tests for detecting offline clusters.
The frequency of cluster status checks is currently hard-coded to 1 minute, but could be made configurable. The threshold for considering a cluster offline now explicitly depends on how often agents report their statuses to the management cluster. Changes to that configured interval should impact the cluster status monitor, which would take the new value into account from its next run onwards.
This fixes an error and ignores a few others to make the linter happy.
This adds checks ensuring that for offline clusters, for which calls to update bundle deployment statuses are expected, those statuses contain `Ready` and `Monitored` conditions with status `False` and reasons reflecting the cluster's offline status.
This ensures that creating a new agent bundle fails with an agent check-in interval set to 0.
This adds a check on the agent check-in interval to cluster import, for consistency with agent bundle updates.
This enables users to determine how often the Fleet controller will check for offline clusters, and based on which threshold. If the configured threshold is below the triple of the check-in interval, that tripled value will be used instead.
This optimises updates to bundle deployments, running them only against clusters which bundle deployments are not yet marked as offline.
This better reflects what is then known about workloads running in such clusters than `False`.
This should fix Fleet controller deployments complaining about the interval being 0 when it should never be.
weyfonk
force-pushed
the
594-detect-offline-cluster
branch
from
a1c1b94
to
8a32810
Compare
Running one cluster status monitor per Fleet controller pod is not necessary and may cause conflicts in sharded setups.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a cluster status monitor to the Fleet controller, which checks when each cluster last saw its agent online. If more than the expected interval elapses, that cluster is considered offline, and the monitor updates its bundle deployments' statuses to reflect that. This will trigger status updates to bundles,
GitRepos, clusters and cluster groups.Refers to #594.
Open points:
ReadyandMonitoredconditions while clearing modified and non-ready statuses, to prevent outdated messages from appearing in a bundle deployment's display status and further up the chain of status updates (to bundles, then upwards toGitRepos, etc)