Error installing monitoring in eks

[Santhosh-Radhakrishna]

Getting the below error while installing monitoring

    "playbook": "/kubesphere/playbooks/monitoring.yaml",
    "playbook_uuid": "4cb4531a-6779-4b70-ad5d-9a0f77d0b1c2",
    "remote_addr": "127.0.0.1",
    "res": {
      "_ansible_no_log": false,
      "attempts": 3,
      "changed": true,
      "cmd": "/usr/local/bin/helm upgrade --install notification-manager /kubesphere/kubesphere/notification-manager -f /kubesphere/kubesphere/notification-manager/custom-values-notification.yaml -n kubesphere-monitoring-system --force\n",
      "delta": "0:05:02.935234",
      "end": "2024-07-10 10:46:47.705398",
      "invocation": {
        "module_args": {
          "_raw_params": "/usr/local/bin/helm upgrade --install notification-manager /kubesphere/kubesphere/notification-manager -f /kubesphere/kubesphere/notification-manager/custom-values-notification.yaml -n kubesphere-monitoring-system --force\n",
          "_uses_shell": true,
          "argv": null,
          "chdir": null,
          "creates": null,
          "executable": null,
          "removes": null,
          "stdin": null,
          "stdin_add_newline": true,
          "strip_empty_ends": true,
          "warn": true
        }
      },
      "msg": "non-zero return code",
      "rc": 1,
      "start": "2024-07-10 10:41:44.770164",
      "stderr": "Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition",
      "stderr_lines": [
        "Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition"
      ],
      "stdout": "",
      "stdout_lines": []
    },

[Santhosh-Radhakrishna]

looking further into the issue, found the below error in replica set in the namespace "kubesphere-monitoring-system"

│ Message:             Error creating: Internal error occurred: failed calling webhook "rev.object.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod-1-14-6.istio-system.svc:443/inject?tim │
│ eout=10s": context deadline exceeded

this was a good hint something wrong in connecting as the istio pods and services were healthy.

Atlast found the the AWS SG group between nodes had the limitations to connect. Changing the SG to allow pods came up