Failed to apply manifest: manifestworks.work.open-cluster-management.io "example-manifestwork" is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa"

[Ghostwritten]

I follow the official documents: Deploy Kubernetes resources to the managed clusters
I want to create a deployment, I run:
$ clusteradm create work my-first-work -f manifest-work2.yaml --clusters cluster1

kind: ManifestWork
metadata:
  namespace: cluster1
  name: example-manifestwork
spec:
  workload:
    manifests:
      - apiVersion: v1
        kind: ServiceAccount
        metadata:
          namespace: default
          name: my-sa
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          namespace: default
          name: nginx-deployment
          labels:
            app: nginx
        spec:
          replicas: 3
          selector:
            matchLabels:
              app: nginx
          template:
            metadata:
              labels:
                app: nginx
            spec:
              serviceAccountName: my-sa
              containers:
                - name: nginx
                  image: nginx:1.14.2
                  ports:
                    - containerPort: 80

but output:

$ clusteradm get works my-first-work --cluster cluster1
Error: field label not supported: name

$ kubectl get  manifestwork my-first-work -n cluster1 -oyaml
apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  creationTimestamp: "2023-03-17T08:26:09Z"
  finalizers:
  - cluster.open-cluster-management.io/manifest-work-cleanup
  generation: 1
  managedFields:
  - apiVersion: work.open-cluster-management.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:workload:
          .: {}
          f:manifests: {}
    manager: clusteradm
    operation: Update
    time: "2023-03-17T08:26:09Z"
  - apiVersion: work.open-cluster-management.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"cluster.open-cluster-management.io/manifest-work-cleanup": {}
    manager: work
    operation: Update
    time: "2023-03-17T08:26:09Z"
  - apiVersion: work.open-cluster-management.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:conditions: {}
        f:resourceStatus:
          .: {}
          f:manifests: {}
    manager: work
    operation: Update
    subresource: status
    time: "2023-03-17T08:26:09Z"
  name: my-first-work
  namespace: cluster1
  resourceVersion: "257910"
  uid: a5e8acda-5832-424d-b9e9-65988d1b1357
spec:
  workload:
    manifests:
    - apiVersion: work.open-cluster-management.io/v1
      kind: ManifestWork
      metadata:
        name: example-manifestwork
        namespace: cluster1
      spec:
        workload:
          manifests:
          - apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: my-sa
              namespace: default
          - apiVersion: apps/v1
            kind: Deployment
            metadata:
              labels:
                app: nginx
              name: nginx-deployment
              namespace: default
            spec:
              replicas: 3
              selector:
                matchLabels:
                  app: nginx
              template:
                metadata:
                  labels:
                    app: nginx
                spec:
                  containers:
                  - image: nginx:1.14.2
                    name: nginx
                    ports:
                    - containerPort: 80
                  serviceAccountName: my-sa
status:
  conditions:
  - lastTransitionTime: "2023-03-17T08:26:09Z"
    message: 1 of 1 resources have unknown status
    observedGeneration: 1
    reason: ResourcesStatusUnknown
    status: Unknown
    type: Available
  - lastTransitionTime: "2023-03-17T08:26:09Z"
    message: Failed to apply manifest work
    observedGeneration: 1
    reason: AppliedManifestWorkFailed
    status: "False"
    type: Applied
  resourceStatus:
    manifests:
    - conditions:
      - lastTransitionTime: "2023-03-17T08:26:09Z"
        message: 'Failed to apply manifest: manifestworks.work.open-cluster-management.io
          "example-manifestwork" is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa"
          cannot get resource "manifestworks" in API group "work.open-cluster-management.io"
          in the namespace "cluster1"'
        reason: AppliedManifestFailed
        status: "False"
        type: Applied
      - lastTransitionTime: "2023-03-17T08:26:09Z"
        message: 'Failed to fetch resource: manifestworks.work.open-cluster-management.io
          "example-manifestwork" is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa"
          cannot get resource "manifestworks" in API group "work.open-cluster-management.io"
          in the namespace "cluster1"'
        reason: FetchingResourceFailed
        status: Unknown
        type: Available
      resourceMeta:
        group: work.open-cluster-management.io
        kind: ManifestWork
        name: example-manifestwork
        namespace: cluster1
        ordinal: 0
        resource: manifestworks
        version: v1
      statusFeedback: {}

$ for i in `k get clusterrolebinding -A | grep open | awk '{print $1}'`;do kubectl get clusterrolebinding $i -oyaml |grep -C 10 klusterlet-work-sa ;done
    time: "2023-03-17T07:57:03Z"
  name: open-cluster-management:klusterlet-work:agent
  resourceVersion: "252611"
  uid: 4918add0-5e05-4bcf-8900-c29aee3e5905
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: open-cluster-management:klusterlet-work:agent
subjects:
- kind: ServiceAccount
  name: klusterlet-work-sa
  namespace: open-cluster-management-agent
    time: "2023-03-17T07:57:04Z"
  name: open-cluster-management:klusterlet-work:execution
  resourceVersion: "252616"
  uid: 314f8194-f089-455b-8fb9-9c7063ecf91b
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: open-cluster-management:klusterlet-work:execution
subjects:
- kind: ServiceAccount
  name: klusterlet-work-sa
  namespace: open-cluster-management-agent
    time: "2023-03-17T07:57:04Z"
  name: open-cluster-management:klusterlet-work:execution-admin
  resourceVersion: "252620"
  uid: c7638908-63e2-4203-b14a-4880f4dfa2fd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
- kind: ServiceAccount
  name: klusterlet-work-sa
  namespace: open-cluster-management-agent

 k get clusterrole open-cluster-management:klusterlet-work:agent  -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2023-03-17T07:57:03Z"
  managedFields:
  - apiVersion: rbac.authorization.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:rules: {}
    manager: registration-operator
    operation: Update
    time: "2023-03-17T07:57:03Z"
  name: open-cluster-management:klusterlet-work:agent
  resourceVersion: "252600"
  uid: 001b9b66-ae70-4452-af31-f3a5c3d46db1
rules:
- apiGroups:
  - work.open-cluster-management.io
  resources:
  - appliedmanifestworks
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - work.open-cluster-management.io
  resources:
  - appliedmanifestworks/status
  verbs:
  - patch
  - update
- apiGroups:
  - work.open-cluster-management.io
  resources:
  - appliedmanifestworks/finalizers
  verbs:
  - update
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - impersonate

"appliedmanifestworks " in API group "work.open-cluster-management.io", it is not "manifestworks", why?

question:

• Do I need to give the User "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa" a new role?
• Is there anything I need to change about my yaml manifest-work2.yaml ?
• How do I normalize an application in a managed cluster?

[mikeshng]

Do I need to give the User "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa" a new role?

No, you shouldn't need to do that if your managed cluster is successfully registered to the hub cluster.
On the managed cluster, cluster1 could you do oc get klusterlet -o yaml and see if there are anything wrong.

Is there anything I need to change about my yaml manifest-work2.yaml ?

I am unable to reproduce the problem using the same example as https://open-cluster-management.io/scenarios/deploy-kubernetes-resources/
What version of clusteradm you are using?

How do I normalize an application in a managed cluster?

ManifestWork API can be used as a primitive/core/lowlevel way to deliver applications to managed clusters.
You can also try using ArgoCD https://github.com/open-cluster-management-io/OCM/tree/main/solutions/deploy-argocd-apps or the newer approach https://github.com/open-cluster-management-io/argocd-pull-integration

[qiujian16]

the manifest input of clusteradm create work -f should not be a ManifestWork resource, but a common kubernetes resource (e.g. Deployment). The clusteradm will wrap the input resource to a manifestwork. See the example here https://github.com/open-cluster-management-io/OCM/tree/main/solutions/deploy-manifests