Introduce addons cache #1239
Merged
Introduce addons cache #1239
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We see too many random failures on the lab running the e2e job. The most
common error is a git fetch error when applying ocm-controller
kustomization. Search shows that there is no way to avoid this error.
This change introduces a cache for kustomization directories, keeping
the output of `kustomize build kustomization_dir` locally, and applying
the cached resource when starting the addon.
Changes:
- The start hook can now fetch the cahe and apply the cached resource.
See ocm-controller for example.
- Add new "fetch" hook, called using the `drenv fetch` command. This can
be used to create the cache for all addons implementing this hook
periodically (e.g in a cron daily job). See ocm-controller fetch hook
for example.
- Add new "clear" command to clear the cache. There is no automatic
invalidation based on cache age.
To cache builds itself automatically, but it has to be invalidated
manually. To keep the cache fresh, you can run `drenv clear` and `drenv
fetch` periodically from a cron job.
The following examples demonstrate how to cache works.
Clearing the cache:
$ drenv clear envs/regional-dr.yaml
2024-03-12 22:51:37,001 INFO [rdr] Clearing cache
2024-03-12 22:51:37,002 INFO [rdr] Fetching finishied in 0.00 seconds
Pre-fetching all addons resources implementing the fetch hook:
$ drenv fetch envs/regional-dr.yaml -v
2024-03-12 22:51:48,884 INFO [rdr] Fetching
2024-03-12 22:51:48,892 INFO [rdr] Running addons/ocm-controller/fetch
2024-03-12 22:51:49,041 DEBUG [rdr] Fetching /home/github/.cache/drenv/addons/ocm-controller.yaml
2024-03-12 22:52:22,202 INFO [rdr] addons/ocm-controller/fetch completed in 33.31 seconds
2024-03-12 22:52:22,203 INFO [rdr] Fetching finishied in 33.32 seconds
Fetching again does nothing since the resource already exists:
$ drenv fetch envs/regional-dr.yaml -v
2024-03-12 22:52:27,423 INFO [rdr] Fetching
2024-03-12 22:52:27,427 INFO [rdr] Running addons/ocm-controller/fetch
2024-03-12 22:52:27,591 INFO [rdr] addons/ocm-controller/fetch completed in 0.16 seconds
2024-03-12 22:52:27,592 INFO [rdr] Fetching finishied in 0.17 seconds
Staring an addon uses the cached resources without accessing the
network:
$ addons/ocm-controller/start hub
Deploying ocm controller
customresourcedefinition.apiextensions.k8s.io/clusterclaims.hive.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterdeployments.hive.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterpools.hive.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusteractions.action.open-cluster-management.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusterimageregistries.imageregistry.open-cluster-management.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusterinfos.internal.open-cluster-management.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusterviews.view.open-cluster-management.io configured
serviceaccount/ocm-foundation-sa unchanged
clusterrole.rbac.authorization.k8s.io/managed-cluster-workmgr unchanged
clusterrole.rbac.authorization.k8s.io/open-cluster-management:ocm:foundation unchanged
clusterrolebinding.rbac.authorization.k8s.io/open-cluster-management:ocm:foundation unchanged
deployment.apps/ocm-controller unchanged
clustermanagementaddon.addon.open-cluster-management.io/work-manager unchanged
Waiting for ocm controller rollout
deployment "ocm-controller" successfully rolled out
Drop the cache:
$ drenv clear envs/regional-dr.yaml -v
2024-03-12 22:52:50,418 INFO [rdr] Clearing cache
2024-03-12 22:52:50,419 INFO [rdr] Fetching finishied in 0.00 seconds
Staring an addon fetch the resources again transparently:
$ addons/ocm-controller/start hub
Deploying ocm controller
Fetching /home/github/.cache/drenv/addons/ocm-controller.yaml
customresourcedefinition.apiextensions.k8s.io/clusterclaims.hive.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterdeployments.hive.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterpools.hive.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusteractions.action.open-cluster-management.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusterimageregistries.imageregistry.open-cluster-management.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusterinfos.internal.open-cluster-management.io configured
customresourcedefinition.apiextensions.k8s.io/managedclusterviews.view.open-cluster-management.io configured
serviceaccount/ocm-foundation-sa unchanged
clusterrole.rbac.authorization.k8s.io/managed-cluster-workmgr unchanged
clusterrole.rbac.authorization.k8s.io/open-cluster-management:ocm:foundation unchanged
clusterrolebinding.rbac.authorization.k8s.io/open-cluster-management:ocm:foundation unchanged
deployment.apps/ocm-controller unchanged
clustermanagementaddon.addon.open-cluster-management.io/work-manager unchanged
Waiting for ocm controller rollout
deployment "ocm-controller" successfully rolled out
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
| return os.path.expanduser(f"~/{cache_home}/drenv/{key}") | ||
|
|
||
|
|
||
| def fetch(kustomization_dir, dest, log=print): |
There was a problem hiding this comment.
I would prefer if fetch took a key and returned a path. If you have a strong reason to have it this way, I am willing to merge it as it is.
There was a problem hiding this comment.
We can do this, but we sill need to expose cache.path(). I was not sure about this change, so I kept this as 2 simple operations.
Lets add cache for few more addons and evaluate again.
|
Example run show ocm-controller finishing in 31 seconds (the cached already exists): And we failed later waiting for the managed clusters, need to debug this on the host. |
|
For the failure of ocm-cluster My simple solution that is working well for me is to change this section of the regional-dr.yaml to Running ocm-cluster addon in the second worker of the profile means it will start as soon as the cluster starts and the hub doesn't get time to finish ocm hub setup. |
It should work because ocm-cluster waits for the hub before starting. I guess we don't wait correctly for the hub. This change - using only 2 workers instead of 3 will make start slower on machines with good network, so we should understand better this failure before changing the envs. |
|
Nest run ocm-controller finished in 18 seconds: |
|
@raghavendra-talur also with current code ocm-cluster starts long time after the hub completed everything: hub starts here: hub completed last addon: ocm-cluster started more than 30 seconds later: But this is a successful run, we need to compare to failed run. |
nirs
commented
Mar 12, 2024
| try: | ||
| shutil.rmtree(cache_dir) | ||
| except FileNotFoundError: | ||
| pass |
There was a problem hiding this comment.
Clear implementation should move to drenv.cache.clear().
| for worker in env["workers"]: | ||
| for addon in worker["addons"]: | ||
| found[addon["name"]] = addon | ||
| return found.values() |
|
When we timeout waiting for managedcluster, we see this: Looks like the auto-accept feature is flaky. And the managed cluster does not have status - looks like it is not reconciled: |
Oh, good debugging. So may be a bug in ocm then. As a workaround, you can probably delete the pod and see if it reconciles when it starts again. |
raghavendra-talur
approved these changes
Mar 13, 2024
Sure, I need to sped more time debugging this. The last detail I found is that when this happens, the klusterlet-work-agent deployment on the affected managed cluster is not ready (0/1) when this happens. So the broken flow is this:
So clusteradm join is broken, it returns with zero exit code when the managed cluster is not accepted. Never seen on local environment, so this is probably related to the networking issues. I'll try to debug this more next week and open ocm issue. Related ocm issue: open-cluster-management-io/clusteradm#334 |
raghavendra-talur
approved these changes
Mar 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We see too many random failures on the lab running the e2e job. The most common error is a git fetch error when applying ocm-controller kustomization. Search shows that there is no way to avoid this error.
This change introduces a cache for kustomization directories, keeping the output of
kustomize build kustomization_dirlocally, and applying the cached resource when starting the addon.Changes:
The start hook can now fetch the cahe and apply the cached resource. See ocm-controller for example.
Add new "fetch" hook, called using the
drenv fetchcommand. This can be used to create the cache for all addons implementing this hook periodically (e.g in a cron daily job). See ocm-controller fetch hook for example.Add new "clear" command to clear the cache. There is no automatic invalidation based on cache age.
To cache builds itself automatically, but it has to be invalidated manually. To keep the cache fresh, you can run
drenv clearanddrenv fetchperiodically from a cron job.The following examples demonstrate how to cache works.
Clearing the cache:
Pre-fetching all addons resources implementing the fetch hook:
Fetching again does nothing since the resource already exists:
Staring an addon uses the cached resources without accessing the network:
Drop the cache:
Staring an addon fetch the resources again transparently: