Cannot run proxy sidecar in an environment with hard ResourceQuotas enabled

[jchandler-bhg]

Describe the bug
The sidecar is entirely unusable with a namespace where a ResourceQuota with hard limits has been enabled. The webhook does not set or have any capabilities to set such limits as far as I can see from the code, making it impossible to start a process where quotas are enforced.

The code I'm referring to here:

• https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L190
• https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L220

Steps To Reproduce
Create a namespace and apply a ResourceQuota to it, for example:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resource-quota-name
  namespace: resource-quota-namespace
spec:
  hard:
    count/configmaps: "100"
    count/cronjobs.batch: "10"
    count/deployments.apps: "100"
    count/jobs.batch: "100"
    count/persistentvolumeclaims: "10"
    count/pods: "200"
    count/services: "100"
    limits.memory: 8Gi
    requests.cpu: "6"
    requests.memory: 8Gi

Finally, deploy a workload identity-enabled pod with resources set to this namespace, with the proxy enable using azure.workload.identity/inject-proxy-sidecar: 'true'. It should fail to start and complain about quota errors as mentioned below.

Expected behavior
The proxy and init container will start without issues.

Logs

message: 'pods "<podName>" is forbidden: failed quota:
      <resourceQuotaName>: must specify limits.memory for: azwi-proxy,azwi-proxy-init; requests.cpu
      for: azwi-proxy,azwi-proxy-init; requests.memory for: azwi-proxy,azwi-proxy-init'

Environment

• Kubernetes version (use kubectl version): v1.29.2
• Cloud provider or hardware configuration: AKS
• OS (e.g: cat /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel (e.g. uname -a): Linux <nodeName> 5.15.0-1068-azure #77-Ubuntu SMP Fri Jun 21 21:21:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Terraform, ArgoCD, custom operators
• Network plugin and version (if this is a network-related bug): n/a
• Others: n/a

Additional context
Istio accomplishes a similar end using the proxyCPU and proxyMemory settings via annotations passed in on the pod - perhaps such an approach is applicable here?

• https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPU
• https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPULimit
• https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemory
• https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemoryLimit