Failing or missing response from https://172.17.245.148:6443/apis/external.metrics.k8s.io/v1beta1...Address is not allowed

[marksouletheprogrammer]

Installed KEDA 2.14.2 via helm chart to EKS cluster. Install seemed to go smoothly. When I installed a scaled object, the underlying HPA failed. When I looked into it, I saw that the APIService v1beta1.external.metrics.k8s.io is in a bad state.

Error from Rancher console
|Failing or missing response from https://172.17.245.148:6443/apis/external.metrics.k8s.io/v1beta1: Get "https://172.17.245.148:6443/apis/external.metrics.k8s.io/v1beta1": Address is not allowed| | | | | |
|||

Notice the IPs from my KEDA services

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE     SELECTOR

keda-admission-webhooks           ClusterIP   172.18.101.2     <none>        443/TCP            6d10h   app=keda-admission-webhooks

keda-operator                     ClusterIP   172.18.2.151     <none>        9666/TCP           6d10h   app=keda-operator

keda-operator-metrics-apiserver   ClusterIP   172.18.156.146   <none>        443/TCP,8080/TCP   6d10h   app=keda-operator-metrics-apiserver

I can reach it from a random pod:

# curl -v keda-operator-metrics-apiserver.keda.svc.cluster.local:443
*   Trying 172.18.156.146:443...
* TCP_NODELAY set
* Connected to keda-operator-metrics-apiserver.keda.svc.cluster.local (172.18.156.146) port 443 (#0)
> GET / HTTP/1.1
> Host: keda-operator-metrics-apiserver.keda.svc.cluster.local:443
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
<
Client sent an HTTP request to an HTTPS server.
* Closing connection 0

Expected Behavior

Expect v1beta1.external.metrics.k8s.io to be able to reach metrics API server pod.

Actual Behavior

v1beta1.external.metrics.k8s.io seems to be trying to contact a completely different IP.

Steps to Reproduce the Problem

1. Install KEDA chart 2.14.2 with default values.yaml to EKS.
2. Observe failed install of v1beta1.external.metrics.k8s.io

Specifications

• KEDA Version: https://github.com/kedacore/charts/releases/tag/v2.14.2
• Platform & Version: ???
• Kubernetes Version: v1.26.15-eks-db838b0
• Scaler(s): https://keda.sh/docs/2.14/scalers/apache-kafka/

[marksouletheprogrammer]

We discussed this on Kubernetes slack here https://kubernetes.slack.com/archives/CKZJ36A5D/p1722520142406699
I solved this issue, posting here for posterity.

The solution that worked for me was to add

metricsServer:
    useHostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet

To the values.yaml of the KEDA Helm chart.
I also had to fully uninstall and reinstall the KEDA Helm chart. That was probably needed because I mangled by own environment somehow.

However, depending on your configuration the host network property might not work. A better solution is to make sure that the firewall between the K8s worker nodes and the control plane is open the for port that KEDA needs (6443). Here are KEDA docs on this: https://keda.sh/docs/2.15/troubleshooting/#why-is-kubernetes-unable-to-get-metrics-from-keda

Thanks for your help!