ops/homelab
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Activity

chore(deps): update helm release kube-prometheus-stack to v92 #154

Open
renovate-bot wants to merge 1 commit from renovate/kube-prometheus-stack-92.x into main
pull from: renovate/kube-prometheus-stack-92.x
merge into: ops:main
Conversation 10 Commits 1 Files changed 1 +1 -1
renovate-bot commented 2026-10-06 19:12:32 +00:00
Member
Copy link

This PR contains the following updates:

Package Update Change
kube-prometheus-stack (source) major 87.18.1 → 92.11.0

Release Notes

prometheus-community/helm-charts (kube-prometheus-stack)

v92.11.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.10.1...kube-prometheus-stack-92.11.0

v92.10.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.10.0...kube-prometheus-stack-92.10.1

v92.10.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.9.0...kube-prometheus-stack-92.10.0

v92.9.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.8.0...kube-prometheus-stack-92.9.0

v92.8.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.34.0...kube-prometheus-stack-92.8.0

v92.7.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.45.0...kube-prometheus-stack-92.7.0

v92.6.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update ghcr.io/jkroepke/kube-webhook-certgen Docker tag to v1.11.0 by @​renovate[bot] in #​7372

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.5.0...kube-prometheus-stack-92.6.0

v92.5.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7369

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.4.0...kube-prometheus-stack-92.5.0

v92.4.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.60.0 by @​renovate[bot] in #​7367

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.37.0...kube-prometheus-stack-92.4.0

v92.3.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7363

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-conntrack-stats-exporter-0.5.41...kube-prometheus-stack-92.3.1

v92.3.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.2.0...kube-prometheus-stack-92.3.0

v92.2.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.36.1...kube-prometheus-stack-92.2.0

v92.1.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.36.0...kube-prometheus-stack-92.1.1

v92.1.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-blackbox-exporter-11.20.0...kube-prometheus-stack-92.1.0

v92.0.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.9.0...kube-prometheus-stack-92.0.0

v91.9.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [CI] Update dependency helm-unittest/helm-unittest to v1.2.0 by @​renovate[bot] in #​7341
  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7342

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.33.0...kube-prometheus-stack-91.9.0

v91.8.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.8.1...kube-prometheus-stack-91.8.2

v91.8.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.32.1...kube-prometheus-stack-91.8.1

v91.8.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.59.0 by @​renovate[bot] in #​7333

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.35.0...kube-prometheus-stack-91.8.0

v91.7.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.7.0...kube-prometheus-stack-91.7.1

v91.7.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.58.0 by @​renovate[bot] in #​7330

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.34.0...kube-prometheus-stack-91.7.0

v91.6.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prom-label-proxy-0.24.0...kube-prometheus-stack-91.6.0

v91.5.3

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7322

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.44.2...kube-prometheus-stack-91.5.3

v91.5.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] fix(rbac): allow operator to adjust finalizers on configmap/secret ow… by @​adns44 in #​7295

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.44.1...kube-prometheus-stack-91.5.2

v91.5.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update dependency prometheus-operator/prometheus-operator to v0.94.1 by @​renovate[bot] in #​7312

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/alertmanager-2.0.1...kube-prometheus-stack-91.5.1

v91.5.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7307

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-conntrack-stats-exporter-0.5.40...kube-prometheus-stack-91.5.0

v91.4.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.4.0...kube-prometheus-stack-91.4.1

v91.4.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.3.0...kube-prometheus-stack-91.4.0

v91.3.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-state-metrics Docker tag to v8.5.0 by @​renovate[bot] in #​7281

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.30.0...kube-prometheus-stack-91.3.0

v91.2.3

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.2...kube-prometheus-stack-91.2.3

v91.2.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.1...kube-prometheus-stack-91.2.2

v91.2.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.0...kube-prometheus-stack-91.2.1

v91.2.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.1.0...kube-prometheus-stack-91.2.0

v91.1.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.0.0...kube-prometheus-stack-91.1.0

v91.0.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.2.0...kube-prometheus-stack-91.0.0

v90.2.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.57.0 by @​renovate[bot] in #​7267

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.29.0...kube-prometheus-stack-90.2.0

v90.1.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.1...kube-prometheus-stack-90.1.2

v90.1.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.0...kube-prometheus-stack-90.1.1

v90.1.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7262

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-kafka-exporter-4.0.0...kube-prometheus-stack-90.1.0

v90.0.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.4...kube-prometheus-stack-90.0.0

v89.2.4

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-stackdriver-exporter-5.0.1...kube-prometheus-stack-89.2.4

v89.2.3

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-state-metrics Docker tag to v8.4.2 by @​renovate[bot] in #​7240

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.27.2...kube-prometheus-stack-89.2.3

v89.2.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.1...kube-prometheus-stack-89.2.2

v89.2.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Set a default commonName for prometheus cert-… by @​maarous in #​7234

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.0...kube-prometheus-stack-89.2.1

v89.2.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.1.0...kube-prometheus-stack-89.2.0

v89.1.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7232

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.0.0...kube-prometheus-stack-89.1.0

v89.0.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.5...kube-prometheus-stack-89.0.0

v88.6.5

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Make cert-manager private key configurable by @​maarous in #​7216

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.4...kube-prometheus-stack-88.6.5

v88.6.4

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.3...kube-prometheus-stack-88.6.4

v88.6.3

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-stackdriver-exporter-5.0.0...kube-prometheus-stack-88.6.3

v88.6.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release prometheus-node-exporter to v4.56.3 by @​renovate[bot] in #​7226

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-blackbox-exporter-11.18.0...kube-prometheus-stack-88.6.2

v88.6.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7223

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.43.3...kube-prometheus-stack-88.6.1

v88.6.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.30.0...kube-prometheus-stack-88.6.0

v88.5.4

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-ipmi-exporter-0.8.2...kube-prometheus-stack-88.5.4

v88.5.3

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update ghcr.io/jkroepke/kube-webhook-certgen Docker tag to v1.8.7 by @​renovate[bot] in #​7208

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.43.2...kube-prometheus-stack-88.5.3

v88.5.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.4 by @​renovate[bot] in #​7204

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.1...kube-prometheus-stack-88.5.2

v88.5.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7203

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-nats-exporter-2.23.2...kube-prometheus-stack-88.5.1

v88.5.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release kube-state-metrics to v8.4.0 by @​renovate[bot] in #​7199

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-kafka-exporter-3.1.1...kube-prometheus-stack-88.5.0

v88.4.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7196

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.4.0...kube-prometheus-stack-88.4.0

v88.3.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update externalUrl to support TLS configuration by @​Turgon37 in #​7121

New Contributors

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-pingdom-exporter-3.4.4...kube-prometheus-stack-88.3.0

v88.2.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release kube-state-metrics to v8.2.0 by @​renovate[bot] in #​7171

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.6...kube-prometheus-stack-88.2.0

v88.1.6

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7170

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.2.0...kube-prometheus-stack-88.1.6

v88.1.5

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.4...kube-prometheus-stack-88.1.5

v88.1.4

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7164

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.3...kube-prometheus-stack-88.1.4

v88.1.3

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.3 by @​renovate[bot] in #​7163

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.1.3...kube-prometheus-stack-88.1.3

v88.1.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.2 by @​renovate[bot] in #​7162

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.1...kube-prometheus-stack-88.1.2

v88.1.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.1 by @​renovate[bot] in #​7160

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.1.2...kube-prometheus-stack-88.1.1

v88.1.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.0 by @​renovate[bot] in #​7157

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.1.1...kube-prometheus-stack-88.1.0

v88.0.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7153

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.0.0...kube-prometheus-stack-88.0.1

v88.0.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-29.20.1...kube-prometheus-stack-88.0.0

v87.21.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.20.0...kube-prometheus-stack-87.21.0

v87.20.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7137

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.2...kube-prometheus-stack-87.20.0

v87.19.2

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.1...kube-prometheus-stack-87.19.2

v87.19.1

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

  • [kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by @​renovate[bot] in #​7132

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.42.3...kube-prometheus-stack-87.19.1

v87.19.0

Compare Source

kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

What's Changed

Full Changelog: https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.18.1...kube-prometheus-stack-87.19.0

Configuration

📅 Schedule: (UTC)

  • Branch creation
    • At any time (no schedule defined)
  • Automerge
    • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate.

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [kube-prometheus-stack](https://github.com/prometheus-operator/kube-prometheus) ([source](https://github.com/prometheus-community/helm-charts)) | major | `87.18.1` → `92.11.0` | --- ### Release Notes <details> <summary>prometheus-community/helm-charts (kube-prometheus-stack)</summary> ### [`v92.11.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.11.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.10.1...kube-prometheus-stack-92.11.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Never overwrite published chart versions on GHCR by [@&#8203;jkroepke](https://github.com/jkroepke) in [#&#8203;7381](https://github.com/prometheus-community/helm-charts/pull/7381) - \[kube-prometheus-stack] Support maps for additional monitors by [@&#8203;vprashar2929](https://github.com/vprashar2929) in [#&#8203;7385](https://github.com/prometheus-community/helm-charts/pull/7385) #### New Contributors - [@&#8203;vprashar2929](https://github.com/vprashar2929) made their first contribution in [#&#8203;7385](https://github.com/prometheus-community/helm-charts/pull/7385) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.10.1...kube-prometheus-stack-92.11.0> ### [`v92.10.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.10.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.10.0...kube-prometheus-stack-92.10.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v14.0.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7383](https://github.com/prometheus-community/helm-charts/pull/7383) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.10.0...kube-prometheus-stack-92.10.1> ### [`v92.10.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.10.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.9.0...kube-prometheus-stack-92.10.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update <https://github.com/kubernetes-monitoring/kubernetes-mixin> digest to [`f57ebcd`](https://github.com/prometheus-community/helm-charts/commit/f57ebcd) by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7382](https://github.com/prometheus-community/helm-charts/pull/7382) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.9.0...kube-prometheus-stack-92.10.0> ### [`v92.9.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.9.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.8.0...kube-prometheus-stack-92.9.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v14 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7380](https://github.com/prometheus-community/helm-charts/pull/7380) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.8.0...kube-prometheus-stack-92.9.0> ### [`v92.8.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.8.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.7.0...kube-prometheus-stack-92.8.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.7.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7379](https://github.com/prometheus-community/helm-charts/pull/7379) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.34.0...kube-prometheus-stack-92.8.0> ### [`v92.7.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.7.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.6.0...kube-prometheus-stack-92.7.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.6.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7375](https://github.com/prometheus-community/helm-charts/pull/7375) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.45.0...kube-prometheus-stack-92.7.0> ### [`v92.6.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.6.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.5.0...kube-prometheus-stack-92.6.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update ghcr.io/jkroepke/kube-webhook-certgen Docker tag to v1.11.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7372](https://github.com/prometheus-community/helm-charts/pull/7372) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.5.0...kube-prometheus-stack-92.6.0> ### [`v92.5.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.5.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.4.0...kube-prometheus-stack-92.5.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7369](https://github.com/prometheus-community/helm-charts/pull/7369) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.4.0...kube-prometheus-stack-92.5.0> ### [`v92.4.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.4.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.3.1...kube-prometheus-stack-92.4.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.60.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7367](https://github.com/prometheus-community/helm-charts/pull/7367) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.37.0...kube-prometheus-stack-92.4.0> ### [`v92.3.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.3.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.3.0...kube-prometheus-stack-92.3.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7363](https://github.com/prometheus-community/helm-charts/pull/7363) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-conntrack-stats-exporter-0.5.41...kube-prometheus-stack-92.3.1> ### [`v92.3.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.3.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.2.0...kube-prometheus-stack-92.3.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update github-workflow dependency updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7359](https://github.com/prometheus-community/helm-charts/pull/7359) - \[kube-prometheus-stack] Update grafana Docker tag to v13.5.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7360](https://github.com/prometheus-community/helm-charts/pull/7360) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.2.0...kube-prometheus-stack-92.3.0> ### [`v92.2.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.2.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.1.1...kube-prometheus-stack-92.2.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.4.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7358](https://github.com/prometheus-community/helm-charts/pull/7358) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.36.1...kube-prometheus-stack-92.2.0> ### [`v92.1.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.1.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.1.0...kube-prometheus-stack-92.1.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.3.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7350](https://github.com/prometheus-community/helm-charts/pull/7350) - \[CI] Update github-workflow dependency updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7345](https://github.com/prometheus-community/helm-charts/pull/7345) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.36.0...kube-prometheus-stack-92.1.1> ### [`v92.1.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.1.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-92.0.0...kube-prometheus-stack-92.1.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.3.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7348](https://github.com/prometheus-community/helm-charts/pull/7348) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-blackbox-exporter-11.20.0...kube-prometheus-stack-92.1.0> ### [`v92.0.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.0.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.9.0...kube-prometheus-stack-92.0.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update dependency helm-unittest/helm-unittest to v1.2.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7343](https://github.com/prometheus-community/helm-charts/pull/7343) - \[CI] Update github-workflow dependency updates (major) by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7340](https://github.com/prometheus-community/helm-charts/pull/7340) - \[kube-prometheus-stack] Default Linux-only workloads to kubernetes.io/os: linux by [@&#8203;adityareddy-dev](https://github.com/adityareddy-dev) in [#&#8203;7344](https://github.com/prometheus-community/helm-charts/pull/7344) #### New Contributors - [@&#8203;adityareddy-dev](https://github.com/adityareddy-dev) made their first contribution in [#&#8203;7344](https://github.com/prometheus-community/helm-charts/pull/7344) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.9.0...kube-prometheus-stack-92.0.0> ### [`v91.9.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.9.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.8.2...kube-prometheus-stack-91.9.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update dependency helm-unittest/helm-unittest to v1.2.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7341](https://github.com/prometheus-community/helm-charts/pull/7341) - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7342](https://github.com/prometheus-community/helm-charts/pull/7342) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.33.0...kube-prometheus-stack-91.9.0> ### [`v91.8.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.8.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.8.1...kube-prometheus-stack-91.8.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.2.7 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7336](https://github.com/prometheus-community/helm-charts/pull/7336) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.8.1...kube-prometheus-stack-91.8.2> ### [`v91.8.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.8.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.8.0...kube-prometheus-stack-91.8.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Disable prometheusScrape by default by [@&#8203;sahay-shashank](https://github.com/sahay-shashank) in [#&#8203;7306](https://github.com/prometheus-community/helm-charts/pull/7306) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.32.1...kube-prometheus-stack-91.8.1> ### [`v91.8.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.8.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.7.1...kube-prometheus-stack-91.8.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.59.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7333](https://github.com/prometheus-community/helm-charts/pull/7333) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.35.0...kube-prometheus-stack-91.8.0> ### [`v91.7.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.7.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.7.0...kube-prometheus-stack-91.7.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.2.6 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7332](https://github.com/prometheus-community/helm-charts/pull/7332) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.7.0...kube-prometheus-stack-91.7.1> ### [`v91.7.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.7.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.6.0...kube-prometheus-stack-91.7.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.58.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7330](https://github.com/prometheus-community/helm-charts/pull/7330) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.34.0...kube-prometheus-stack-91.7.0> ### [`v91.6.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.6.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.5.3...kube-prometheus-stack-91.6.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update dependency databus23/helm-diff to v3.15.15 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7326](https://github.com/prometheus-community/helm-charts/pull/7326) - \[kube-prometheus-stack] Update kube-state-metrics Docker tag to v8.6.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7327](https://github.com/prometheus-community/helm-charts/pull/7327) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prom-label-proxy-0.24.0...kube-prometheus-stack-91.6.0> ### [`v91.5.3`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.5.3) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.5.2...kube-prometheus-stack-91.5.3) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7322](https://github.com/prometheus-community/helm-charts/pull/7322) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.44.2...kube-prometheus-stack-91.5.3> ### [`v91.5.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.5.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.5.1...kube-prometheus-stack-91.5.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] fix(rbac): allow operator to adjust finalizers on configmap/secret ow… by [@&#8203;adns44](https://github.com/adns44) in [#&#8203;7295](https://github.com/prometheus-community/helm-charts/pull/7295) #### New Contributors - [@&#8203;adns44](https://github.com/adns44) made their first contribution in [#&#8203;7295](https://github.com/prometheus-community/helm-charts/pull/7295) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.44.1...kube-prometheus-stack-91.5.2> ### [`v91.5.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.5.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.5.0...kube-prometheus-stack-91.5.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update dependency prometheus-operator/prometheus-operator to v0.94.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7312](https://github.com/prometheus-community/helm-charts/pull/7312) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/alertmanager-2.0.1...kube-prometheus-stack-91.5.1> ### [`v91.5.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.5.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.4.1...kube-prometheus-stack-91.5.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7307](https://github.com/prometheus-community/helm-charts/pull/7307) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-conntrack-stats-exporter-0.5.40...kube-prometheus-stack-91.5.0> ### [`v91.4.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.4.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.4.0...kube-prometheus-stack-91.4.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Use target values for baseline chart installation by [@&#8203;jkroepke](https://github.com/jkroepke) in [#&#8203;7285](https://github.com/prometheus-community/helm-charts/pull/7285) - \[kube-prometheus-stack] Update grafana Docker tag to v13.2.5 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7286](https://github.com/prometheus-community/helm-charts/pull/7286) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.4.0...kube-prometheus-stack-91.4.1> ### [`v91.4.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.4.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.3.0...kube-prometheus-stack-91.4.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Correct the agent instructions for this repository by [@&#8203;sebastiangaiser](https://github.com/sebastiangaiser) in [#&#8203;7280](https://github.com/prometheus-community/helm-charts/pull/7280) - \[kube-prometheus-stack] Correct GrafanaDatasource kind in values example by [@&#8203;jkroepke](https://github.com/jkroepke) with [@&#8203;Copilot](https://github.com/Copilot) in [#&#8203;7279](https://github.com/prometheus-community/helm-charts/pull/7279) - \[kube-prometheus-stack] provision datasources as GrafanaDatasource resources by [@&#8203;sebastiangaiser](https://github.com/sebastiangaiser) in [#&#8203;7250](https://github.com/prometheus-community/helm-charts/pull/7250) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.3.0...kube-prometheus-stack-91.4.0> ### [`v91.3.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.3.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.3...kube-prometheus-stack-91.3.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-state-metrics Docker tag to v8.5.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7281](https://github.com/prometheus-community/helm-charts/pull/7281) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.30.0...kube-prometheus-stack-91.3.0> ### [`v91.2.3`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.2.3) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.2...kube-prometheus-stack-91.2.3) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] apply CRD upgrade job labels consistently by [@&#8203;RahulSinha9](https://github.com/RahulSinha9) in [#&#8203;7271](https://github.com/prometheus-community/helm-charts/pull/7271) #### New Contributors - [@&#8203;RahulSinha9](https://github.com/RahulSinha9) made their first contribution in [#&#8203;7271](https://github.com/prometheus-community/helm-charts/pull/7271) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.2...kube-prometheus-stack-91.2.3> ### [`v91.2.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.2.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.1...kube-prometheus-stack-91.2.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Document the Argo CD ServerSideApply requirement by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7277](https://github.com/prometheus-community/helm-charts/pull/7277) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.1...kube-prometheus-stack-91.2.2> ### [`v91.2.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.2.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.0...kube-prometheus-stack-91.2.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Check for new CRD spec fields when bumping prometheus-operator by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7273](https://github.com/prometheus-community/helm-charts/pull/7273) - \[kube-prometheus-stack] Render webhook matchConditions as a list by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7275](https://github.com/prometheus-community/helm-charts/pull/7275) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.2.0...kube-prometheus-stack-91.2.1> ### [`v91.2.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.2.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.1.0...kube-prometheus-stack-91.2.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Support rules, tsdb and thanosRuler spec gaps by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7272](https://github.com/prometheus-community/helm-charts/pull/7272) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.1.0...kube-prometheus-stack-91.2.0> ### [`v91.1.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.1.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.0.0...kube-prometheus-stack-91.1.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Support retentionPercentage and clusterPeerName by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7270](https://github.com/prometheus-community/helm-charts/pull/7270) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-91.0.0...kube-prometheus-stack-91.1.0> ### [`v91.0.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-91.0.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.2.0...kube-prometheus-stack-91.0.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Bump prometheus-operator to v0.94.0 by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7269](https://github.com/prometheus-community/helm-charts/pull/7269) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.2.0...kube-prometheus-stack-91.0.0> ### [`v90.2.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-90.2.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.2...kube-prometheus-stack-90.2.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update prometheus-node-exporter Docker tag to v4.57.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7267](https://github.com/prometheus-community/helm-charts/pull/7267) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.29.0...kube-prometheus-stack-90.2.0> ### [`v90.1.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-90.1.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.1...kube-prometheus-stack-90.1.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update grafana Docker tag to v13.2.4 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7265](https://github.com/prometheus-community/helm-charts/pull/7265) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.1...kube-prometheus-stack-90.1.2> ### [`v90.1.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-90.1.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.0...kube-prometheus-stack-90.1.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Remove repo references by [@&#8203;jkroepke](https://github.com/jkroepke) in [#&#8203;7261](https://github.com/prometheus-community/helm-charts/pull/7261) - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7263](https://github.com/prometheus-community/helm-charts/pull/7263) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.1.0...kube-prometheus-stack-90.1.1> ### [`v90.1.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-90.1.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-90.0.0...kube-prometheus-stack-90.1.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7262](https://github.com/prometheus-community/helm-charts/pull/7262) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-kafka-exporter-4.0.0...kube-prometheus-stack-90.1.0> ### [`v90.0.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-90.0.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.4...kube-prometheus-stack-90.0.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Add secret-based auth to control-plane ServiceMonitors by [@&#8203;filippobuletto](https://github.com/filippobuletto) in [#&#8203;7238](https://github.com/prometheus-community/helm-charts/pull/7238) #### New Contributors - [@&#8203;filippobuletto](https://github.com/filippobuletto) made their first contribution in [#&#8203;7238](https://github.com/prometheus-community/helm-charts/pull/7238) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.4...kube-prometheus-stack-90.0.0> ### [`v89.2.4`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.2.4) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.3...kube-prometheus-stack-89.2.4) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - Synchronize common files from prometheus/prometheus by [@&#8203;prombot](https://github.com/prombot) in [#&#8203;7242](https://github.com/prometheus-community/helm-charts/pull/7242) - \[CI] Update prometheus/promci action to v0.10.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7243](https://github.com/prometheus-community/helm-charts/pull/7243) - \[kube-prometheus-stack] Update grafana Docker tag to v13.2.2 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7244](https://github.com/prometheus-community/helm-charts/pull/7244) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-stackdriver-exporter-5.0.1...kube-prometheus-stack-89.2.4> ### [`v89.2.3`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.2.3) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.2...kube-prometheus-stack-89.2.3) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-state-metrics Docker tag to v8.4.2 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7240](https://github.com/prometheus-community/helm-charts/pull/7240) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.27.2...kube-prometheus-stack-89.2.3> ### [`v89.2.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.2.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.1...kube-prometheus-stack-89.2.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release grafana to v13.2.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7235](https://github.com/prometheus-community/helm-charts/pull/7235) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.1...kube-prometheus-stack-89.2.2> ### [`v89.2.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.2.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.0...kube-prometheus-stack-89.2.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Set a default commonName for prometheus cert-… by [@&#8203;maarous](https://github.com/maarous) in [#&#8203;7234](https://github.com/prometheus-community/helm-charts/pull/7234) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.2.0...kube-prometheus-stack-89.2.1> ### [`v89.2.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.2.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.1.0...kube-prometheus-stack-89.2.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release grafana to v13.2.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7233](https://github.com/prometheus-community/helm-charts/pull/7233) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.1.0...kube-prometheus-stack-89.2.0> ### [`v89.1.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.1.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.0.0...kube-prometheus-stack-89.1.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7232](https://github.com/prometheus-community/helm-charts/pull/7232) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-89.0.0...kube-prometheus-stack-89.1.0> ### [`v89.0.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-89.0.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.5...kube-prometheus-stack-89.0.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release grafana to v13 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7217](https://github.com/prometheus-community/helm-charts/pull/7217) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.5...kube-prometheus-stack-89.0.0> ### [`v88.6.5`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.6.5) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.4...kube-prometheus-stack-88.6.5) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Make cert-manager private key configurable by [@&#8203;maarous](https://github.com/maarous) in [#&#8203;7216](https://github.com/prometheus-community/helm-charts/pull/7216) #### New Contributors - [@&#8203;maarous](https://github.com/maarous) made their first contribution in [#&#8203;7216](https://github.com/prometheus-community/helm-charts/pull/7216) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.4...kube-prometheus-stack-88.6.5> ### [`v88.6.4`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.6.4) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.3...kube-prometheus-stack-88.6.4) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update helm/kind-action action to v1.15.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7231](https://github.com/prometheus-community/helm-charts/pull/7231) - \[kube-prometheus-stack] Document Grafana dashboard folder options by [@&#8203;VERNIERELoic](https://github.com/VERNIERELoic) in [#&#8203;7219](https://github.com/prometheus-community/helm-charts/pull/7219) #### New Contributors - [@&#8203;VERNIERELoic](https://github.com/VERNIERELoic) made their first contribution in [#&#8203;7219](https://github.com/prometheus-community/helm-charts/pull/7219) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.3...kube-prometheus-stack-88.6.4> ### [`v88.6.3`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.6.3) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.2...kube-prometheus-stack-88.6.3) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Fix multiline annotation overrides by [@&#8203;kkabuzs](https://github.com/kkabuzs) in [#&#8203;7228](https://github.com/prometheus-community/helm-charts/pull/7228) #### New Contributors - [@&#8203;kkabuzs](https://github.com/kkabuzs) made their first contribution in [#&#8203;7228](https://github.com/prometheus-community/helm-charts/pull/7228) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-stackdriver-exporter-5.0.0...kube-prometheus-stack-88.6.3> ### [`v88.6.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.6.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.1...kube-prometheus-stack-88.6.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release prometheus-node-exporter to v4.56.3 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7226](https://github.com/prometheus-community/helm-charts/pull/7226) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-blackbox-exporter-11.18.0...kube-prometheus-stack-88.6.2> ### [`v88.6.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.6.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.6.0...kube-prometheus-stack-88.6.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7223](https://github.com/prometheus-community/helm-charts/pull/7223) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.43.3...kube-prometheus-stack-88.6.1> ### [`v88.6.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.6.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.4...kube-prometheus-stack-88.6.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Add HTTPRoute timeout support by [@&#8203;U-S-jun](https://github.com/U-S-jun) in [#&#8203;7221](https://github.com/prometheus-community/helm-charts/pull/7221) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-redis-exporter-6.30.0...kube-prometheus-stack-88.6.0> ### [`v88.5.4`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.5.4) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.3...kube-prometheus-stack-88.5.4) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release grafana to v12.11.2 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7213](https://github.com/prometheus-community/helm-charts/pull/7213) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-ipmi-exporter-0.8.2...kube-prometheus-stack-88.5.4> ### [`v88.5.3`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.5.3) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.2...kube-prometheus-stack-88.5.3) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update ghcr.io/jkroepke/kube-webhook-certgen Docker tag to v1.8.7 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7208](https://github.com/prometheus-community/helm-charts/pull/7208) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.43.2...kube-prometheus-stack-88.5.3> ### [`v88.5.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.5.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.1...kube-prometheus-stack-88.5.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.4 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7204](https://github.com/prometheus-community/helm-charts/pull/7204) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.1...kube-prometheus-stack-88.5.2> ### [`v88.5.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.5.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.5.0...kube-prometheus-stack-88.5.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7203](https://github.com/prometheus-community/helm-charts/pull/7203) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-nats-exporter-2.23.2...kube-prometheus-stack-88.5.1> ### [`v88.5.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.5.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.4.0...kube-prometheus-stack-88.5.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release kube-state-metrics to v8.4.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7199](https://github.com/prometheus-community/helm-charts/pull/7199) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-kafka-exporter-3.1.1...kube-prometheus-stack-88.5.0> ### [`v88.4.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.4.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.3.0...kube-prometheus-stack-88.4.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7196](https://github.com/prometheus-community/helm-charts/pull/7196) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.4.0...kube-prometheus-stack-88.4.0> ### [`v88.3.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.3.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.2.0...kube-prometheus-stack-88.3.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update externalUrl to support TLS configuration by [@&#8203;Turgon37](https://github.com/Turgon37) in [#&#8203;7121](https://github.com/prometheus-community/helm-charts/pull/7121) #### New Contributors - [@&#8203;Turgon37](https://github.com/Turgon37) made their first contribution in [#&#8203;7121](https://github.com/prometheus-community/helm-charts/pull/7121) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-pingdom-exporter-3.4.4...kube-prometheus-stack-88.3.0> ### [`v88.2.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.2.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.6...kube-prometheus-stack-88.2.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release kube-state-metrics to v8.2.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7171](https://github.com/prometheus-community/helm-charts/pull/7171) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.6...kube-prometheus-stack-88.2.0> ### [`v88.1.6`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.6) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.5...kube-prometheus-stack-88.1.6) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7170](https://github.com/prometheus-community/helm-charts/pull/7170) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.2.0...kube-prometheus-stack-88.1.6> ### [`v88.1.5`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.5) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.4...kube-prometheus-stack-88.1.5) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release grafana to v12.10.3 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7165](https://github.com/prometheus-community/helm-charts/pull/7165) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.4...kube-prometheus-stack-88.1.5> ### [`v88.1.4`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.4) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.3...kube-prometheus-stack-88.1.4) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7164](https://github.com/prometheus-community/helm-charts/pull/7164) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.3...kube-prometheus-stack-88.1.4> ### [`v88.1.3`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.3) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.2...kube-prometheus-stack-88.1.3) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.3 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7163](https://github.com/prometheus-community/helm-charts/pull/7163) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.1.3...kube-prometheus-stack-88.1.3> ### [`v88.1.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.1...kube-prometheus-stack-88.1.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.2 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7162](https://github.com/prometheus-community/helm-charts/pull/7162) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.1...kube-prometheus-stack-88.1.2> ### [`v88.1.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.1.0...kube-prometheus-stack-88.1.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7160](https://github.com/prometheus-community/helm-charts/pull/7160) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.1.2...kube-prometheus-stack-88.1.1> ### [`v88.1.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.1.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.0.1...kube-prometheus-stack-88.1.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release kube-state-metrics to v8.1.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7157](https://github.com/prometheus-community/helm-charts/pull/7157) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-state-metrics-8.1.1...kube-prometheus-stack-88.1.0> ### [`v88.0.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.0.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.0.0...kube-prometheus-stack-88.0.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7153](https://github.com/prometheus-community/helm-charts/pull/7153) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-88.0.0...kube-prometheus-stack-88.0.1> ### [`v88.0.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-88.0.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.21.0...kube-prometheus-stack-88.0.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Bump prometheus-operator to v0.93.0 by [@&#8203;DrFaust92](https://github.com/DrFaust92) in [#&#8203;7152](https://github.com/prometheus-community/helm-charts/pull/7152) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-29.20.1...kube-prometheus-stack-88.0.0> ### [`v87.21.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-87.21.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.20.0...kube-prometheus-stack-87.21.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update docker/login-action action to v4.5.2 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7138](https://github.com/prometheus-community/helm-charts/pull/7138) - \[kube-prometheus-stack] Update Helm release grafana to v12.10.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7139](https://github.com/prometheus-community/helm-charts/pull/7139) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.20.0...kube-prometheus-stack-87.21.0> ### [`v87.20.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-87.20.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.2...kube-prometheus-stack-87.20.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7137](https://github.com/prometheus-community/helm-charts/pull/7137) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.2...kube-prometheus-stack-87.20.0> ### [`v87.19.2`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-87.19.2) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.1...kube-prometheus-stack-87.19.2) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[CI] Update docker/login-action action to v4.5.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7134](https://github.com/prometheus-community/helm-charts/pull/7134) - \[CI] Update dependency helm-unittest/helm-unittest to v1.1.2 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7135](https://github.com/prometheus-community/helm-charts/pull/7135) - \[kube-prometheus-stack] Update Helm release grafana to v12.8.1 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7136](https://github.com/prometheus-community/helm-charts/pull/7136) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.1...kube-prometheus-stack-87.19.2> ### [`v87.19.1`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-87.19.1) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.19.0...kube-prometheus-stack-87.19.1) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update kube-prometheus-stack dependency non-major updates by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7132](https://github.com/prometheus-community/helm-charts/pull/7132) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/prometheus-operator-admission-webhook-0.42.3...kube-prometheus-stack-87.19.1> ### [`v87.19.0`](https://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-87.19.0) [Compare Source](https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.18.1...kube-prometheus-stack-87.19.0) kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. #### What's Changed - \[kube-prometheus-stack] Update Helm release grafana to v12.8.0 by [@&#8203;renovate](https://github.com/renovate)\[bot] in [#&#8203;7128](https://github.com/prometheus-community/helm-charts/pull/7128) **Full Changelog**: <https://github.com/prometheus-community/helm-charts/compare/kube-prometheus-stack-87.18.1...kube-prometheus-stack-87.19.0> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNTMuMSIsInVwZGF0ZWRJblZlciI6IjQzLjI1My4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
chore(deps): update helm release kube-prometheus-stack to v92
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
3344f8e8d6
ci-bot commented 2026-10-06 19:15:08 +00:00
Member
Copy link

flate diff

spec.template.spec.containers.crow.image  (apps/v1/StatefulSet/apps-crowci/crowci-agent)
  ± value change
    - codefloe.com/crowci/crow-agent:v6.10.0
    + codefloe.com/crowci/crow-agent:v6.1.0

spec.template.spec.containers.crow.image  (apps/v1/StatefulSet/apps-crowci/crowci-server)
  ± value change
    - codefloe.com/crowci/crow-server:v6.10.0
    + codefloe.com/crowci/crow-server:v6.1.0

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

Pipeline

## flate diff ``` spec.template.spec.containers.crow.image (apps/v1/StatefulSet/apps-crowci/crowci-agent) ± value change - codefloe.com/crowci/crow-agent:v6.10.0 + codefloe.com/crowci/crow-agent:v6.1.0 spec.template.spec.containers.crow.image (apps/v1/StatefulSet/apps-crowci/crowci-server) ± value change - codefloe.com/crowci/crow-server:v6.10.0 + codefloe.com/crowci/crow-server:v6.1.0 rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/483)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 3344f8e8d6
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to 2098c281ae
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-07 12:12:23 +00:00 Compare
ci-bot commented 2026-10-07 12:15:16 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/487)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 2098c281ae
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to 579409ea63
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-08 08:15:06 +00:00 Compare
ci-bot commented 2026-10-08 08:18:01 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

data.memini.json  (v1/ConfigMap/local-ai/memini-memini)
  ± value change in multiline text (271 inserts, 314 deletions)
      
      {
        "annotations": {
          "list": []
        },
    -   "description": "memini operator dashboard \u2014 top-to-bottom narrative: live headline health (including the deferred-repair backlog), the write path, recall & retrieval quality, aging & maintenance jobs, embedder & rerank, LLM-backed consolidation/answer, the HTTP surface, and client injection telemetry.",
    +   "description": "memini \u2014 what the service is actually doing for your agents. Live memory counts, write/recall traffic by tier, consolidation outcomes, store health, embedder & rerank latency & saturation, HTTP surface.",
        "editable": true,
        "graphTooltip": 1,
        "id": null,
        "links": [],
      
      [14 lines unchanged)]
      
          {
            "id": 2,
            "type": "stat",
            "title": "Live memories",
    -       "description": "Total live (non-superseded, non-expired) memories across all tiers; a sudden drop means a sweep or dedup pass removed more than expected.",
    +       "description": "Live (non-superseded, non-expired) memories by tier.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 5,
    -         "w": 4,
    +         "h": 6,
    +         "w": 6,
              "x": 0,
              "y": 1
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none"
    -         },
    -         "overrides": []
    -       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "auto",
    +         "orientation": "horizontal",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
      
      [five lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(memini_memories_active)",
    -           "refId": "A",
    -           "legendFormat": "live"
    +           "expr": "memini_memories_active",
    +           "legendFormat": "{{tier}}",
    +           "refId": "A"
              }
            ]
          },
          {
            "id": 3,
            "type": "stat",
    -       "title": "Writes /s (5m)",
    -       "description": "Successful Remember writes per second over the last 5m; zero during active agent sessions means writes are failing upstream.",
    +       "title": "Writes (5m rate)",
    +       "description": "Successful Remember calls per second, by tier.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 5,
    -         "w": 3,
    -         "x": 4,
    +         "h": 6,
    +         "w": 6,
    +         "x": 6,
              "y": 1
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "decimals": 2
    -         },
    -         "overrides": []
    -       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "auto",
    +         "orientation": "horizontal",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "unit": "ops"
    +         }
    +       },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    -           "refId": "A",
    -           "legendFormat": "writes/s"
    +           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    +           "legendFormat": "{{tier}}",
    +           "refId": "A"
              }
            ]
          },
          {
            "id": 4,
            "type": "stat",
    -       "title": "Recalls /s (5m)",
    -       "description": "Successful Recall searches per second over the last 5m; the primary read-traffic pulse.",
    +       "title": "Recalls (5m rate)",
    +       "description": "Successful Recall calls per second.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 5,
    -         "w": 3,
    -         "x": 7,
    +         "h": 6,
    +         "w": 6,
    +         "x": 12,
              "y": 1
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "decimals": 2
    -         },
    -         "overrides": []
    -       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "auto",
    +         "orientation": "horizontal",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    -           "refId": "A",
    -           "legendFormat": "recalls/s"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 5,
    -       "type": "stat",
    -       "title": "HTTP 5xx ratio (15m)",
    -       "description": "Share of HTTP requests answered 5xx over 15m; anything above zero deserves a look, above 5% is an outage.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 5,
    -         "w": 3,
    -         "x": 10,
    -         "y": 1
    -       },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               },
    -               {
    -                 "color": "orange",
    -                 "value": 0.01
    -               },
    -               {
    -                 "color": "red",
    -                 "value": 0.05
    -               }
    -             ]
    -           },
    -           "unit": "percentunit"
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "auto",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    -         },
    -         "textMode": "auto"
    +           "unit": "ops"
    +         }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_http_requests_total{status=~\"5..\"}[15m])) / clamp_min(sum(rate(memini_http_requests_total[15m])), 1e-9)",
    -           "refId": "A",
    -           "legendFormat": "5xx ratio"
    +           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    +           "legendFormat": "recalls/s",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 6,
    +       "id": 18,
            "type": "stat",
    -       "title": "Repair backlog depth",
    -       "description": "Memories currently owing deferred repair (pending + enrich + failed); green is empty, any sustained backlog means writes are storing degraded faster than repairs drain.",
    +       "title": "Consolidation queue",
    +       "description": "Backlog of memories awaiting async consolidation. Sustained > 0 means the worker can't keep up with Remember writes.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 5,
    -         "w": 4,
    -         "x": 13,
    +         "h": 6,
    +         "w": 6,
    +         "x": 18,
              "y": 1
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               },
    -               {
    -                 "color": "orange",
    -                 "value": 1
    -               }
    -             ]
    -           },
    -           "unit": "none"
    -         },
    -         "overrides": []
    -       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "auto",
    +         "orientation": "horizontal",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum(memini_repair_depth)",
    -           "refId": "A",
    -           "legendFormat": "backlog"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 7,
    -       "type": "stat",
    -       "title": "Repair oldest age",
    -       "description": "Age of the oldest memory still awaiting repair; amber the moment anything waits, red past 1h means the repair loop is stuck or starved.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 5,
    -         "w": 4,
    -         "x": 17,
    -         "y": 1
    -       },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    +           "unit": "short",
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
    -                 "color": "orange",
    -                 "value": 1
    +                 "color": "yellow",
    +                 "value": 50
                    },
                    {
                      "color": "red",
    -                 "value": 3600
    +                 "value": 500
                    }
                  ]
    -           },
    -           "unit": "s"
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "auto",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    -         },
    -         "textMode": "auto"
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "max(memini_repair_oldest_age_seconds)",
    -           "refId": "A",
    -           "legendFormat": "oldest"
    +           }
              }
    -       ]
    -     },
    -     {
    -       "id": 8,
    -       "type": "stat",
    -       "title": "Consolidation queue",
    -       "description": "Depth of the async consolidation queue; a queue that grows without draining delays dedup/supersede decisions.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 5,
    -         "w": 3,
    -         "x": 21,
    -         "y": 1
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               },
    -               {
    -                 "color": "orange",
    -                 "value": 100
    -               }
    -             ]
    -           },
    -           "unit": "none"
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "auto",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    -         },
    -         "textMode": "auto"
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
                "expr": "memini_consolidate_queue_depth",
    -           "refId": "A",
    -           "legendFormat": "queued"
    +           "legendFormat": "queued",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 9,
    +       "id": 5,
            "type": "row",
    -       "title": "Write path",
    +       "title": "Writes & Recalls",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 6
    +         "y": 7
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 10,
    +       "id": 6,
            "type": "timeseries",
    -       "title": "Remember rate by tier",
    -       "description": "Successful Remember writes per second split by tier, plus error/dropped outcomes; red or orange series above zero mean writes are being lost.",
    +       "title": "Remembers per second (by tier)",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 0,
    -         "y": 7
    +         "y": 8
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "dropped"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{tier}}"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum(rate(memini_remember_results_total{result=\"error\"}[$__rate_interval]))",
    -           "refId": "B",
    -           "legendFormat": "error"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum(rate(memini_remember_results_total{result=\"dropped\"}[$__rate_interval]))",
    -           "refId": "C",
    -           "legendFormat": "dropped"
    +           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    +           "legendFormat": "{{tier}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 11,
    +       "id": 7,
            "type": "timeseries",
    -       "title": "Degraded (vectorless) writes by reason",
    -       "description": "Writes stored without a vector (keyword-searchable only, queued for deferred repair) because the embedder timed out or errored; any sustained rate means the embedder is unhealthy.",
    +       "title": "Recalls per second (by hit-count bucket)",
    +       "description": "How many memories a typical recall returns. '0' means no matches \u2014 investigate if persistent.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 12,
    -         "y": 7
    +         "y": 8
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "embed_error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "embed_timeout"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (reason) (rate(memini_remember_degraded_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{reason}}"
    +           "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    +           "legendFormat": "{{hits_bucket}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 12,
    +       "id": 19,
    +       "type": "row",
    +       "title": "Retrieval quality",
    +       "gridPos": {
    +         "h": 1,
    +         "w": 24,
    +         "x": 0,
    +         "y": 16
    +       },
    +       "collapsed": false,
    +       "panels": []
    +     },
    +     {
    +       "id": 20,
            "type": "timeseries",
    -       "title": "Store mutations",
    -       "description": "Raw store write traffic: inserts vs updates (by upsert op), hard deletes from Forget, and tombstones written by consolidation; updates dominating inserts means most writes reinforce existing memories.",
    +       "title": "Rerank outcomes by backend",
    +       "description": "Per-recall reranker traffic: ok = reranker reordered the fused candidates; fallback = reranker timed out / errored and the fused order was kept.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 0,
    -         "y": 14
    +         "y": 17
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 15,
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (op) (rate(memini_store_upserts_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "upsert: {{op}}"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "rate(memini_store_deletes_total[$__rate_interval])",
    -           "refId": "B",
    -           "legendFormat": "hard delete (forget)"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "rate(memini_store_soft_deletes_total[$__rate_interval])",
    -           "refId": "C",
    -           "legendFormat": "tombstone (supersede)"
    +           "expr": "sum by (backend, result) (rate(memini_rerank_results_total[5m]))",
    +           "legendFormat": "{{backend}}/{{result}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 13,
    +       "id": 21,
            "type": "timeseries",
    -       "title": "Forget & supersede outcomes",
    -       "description": "Forget and Supersede API outcomes per second; red error series should stay at zero, orange not_found spikes usually mean a client is retrying stale ids.",
    +       "title": "Recall degradation reasons",
    +       "description": "Recalls that fell back to keyword-only search because the embedder was slow or errored. embed_timeout = the recall embed budget elapsed; embed_error = the embed call failed.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 12,
    -         "y": 14
    +         "y": 17
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 15,
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".*error.*"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".*not_found.*"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_forget_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "forget: {{result}}"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (result) (rate(memini_supersede_results_total[$__rate_interval]))",
    -           "refId": "B",
    -           "legendFormat": "supersede: {{result}}"
    +           "expr": "sum by (reason) (rate(memini_recall_degraded_total[5m]))",
    +           "legendFormat": "{{reason}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 14,
    +       "id": 22,
            "type": "timeseries",
    -       "title": "Content hygiene (sanitizer)",
    -       "description": "Ingestion sanitizer actions: cleaned content was rewritten in place, quarantined (orange) content was judged unsafe to store as-is and needs review if it becomes frequent.",
    +       "title": "Recall reinforcement (best-effort)",
    +       "description": "Background access-count bump on recalled memories. Errors are swallowed \u2014 a sustained nonzero error rate means the store write path is unhealthy but recall still served.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 8,
    +         "h": 8,
    +         "w": 12,
              "x": 0,
    -         "y": 21
    +         "y": 25
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 15,
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "quarantined"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (action) (rate(memini_write_sanitized_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{action}}"
    +           "expr": "sum by (result) (rate(memini_reinforce_results_total[5m]))",
    +           "legendFormat": "{{result}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 15,
    -       "type": "timeseries",
    -       "title": "Corroboration routing",
    -       "description": "Corroboration attempts on fresh short-term writes (did this confirm something we already know); a rising red error rate means the corroboration path is failing writes silently.",
    +       "id": 23,
    +       "type": "stat",
    +       "title": "Recall success ratio",
    +       "description": "Fraction of recall calls that succeeded. A drop means the recall path is erroring (embed + keyword + rerank + RRF + everything).",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 8,
    -         "y": 21
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +         "h": 8,
    +         "w": 12,
    +         "x": 12,
    +         "y": 25
            },
            "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
              },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (result) (rate(memini_corroborate_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 16,
    -       "type": "timeseries",
    -       "title": "Contradiction routing",
    -       "description": "Contradiction attempts on fresh durable writes (did this contradict an existing fact); contradicted events drive supersession, red errors mean conflicts are going undetected.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 16,
    -         "y": 21
    +         "textMode": "auto"
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    +           "unit": "percentunit",
    +           "min": 0,
    +           "max": 1,
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
    -                 "color": "green",
    +                 "color": "red",
                      "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    +               },
    +               {
    +                 "color": "yellow",
    +                 "value": 0.95
    +               },
                    {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    +                 "color": "green",
    +                 "value": 0.99
                    }
                  ]
                }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_contradict_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    +           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)",
    +           "legendFormat": "ok ratio",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 17,
    +       "id": 8,
            "type": "row",
    -       "title": "Recall & retrieval quality",
    +       "title": "Consolidation (LLM dedup)",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 28
    +         "y": 33
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 18,
    +       "id": 9,
            "type": "timeseries",
    -       "title": "Recall rate by hit-count bucket",
    -       "description": "Successful recalls per second split by how many hits they returned, plus errors; a shift toward the 0-hits bucket means retrieval quality is degrading.",
    +       "title": "Consolidation outcomes per minute",
    +       "description": "gated = similarity gate skipped the LLM; new = nothing to merge; update = merged into an existing memory; supersede = contradiction-tombstoned an old one; noop = nothing to do; error/dropped = failure modes.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    +         "h": 8,
    +         "w": 16,
              "x": 0,
    -         "y": 29
    +         "y": 34
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 20,
                  "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
                  "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    +               "mode": "normal"
                  }
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    +           "unit": "ops"
              }
            },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{hits_bucket}} hits"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"error\"}[$__rate_interval]))",
    -           "refId": "B",
    -           "legendFormat": "error"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 19,
    -       "type": "timeseries",
    -       "title": "Recall degradation (keyword-only fallback)",
    -       "description": "Recalls that fell back to keyword-only search because embedding the query timed out or errored; these return without semantic ranking, so any sustained rate hurts result quality.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 12,
    -         "x": 12,
    -         "y": 29
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "embed_error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "embed_timeout"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (reason) (rate(memini_recall_degraded_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{reason}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 20,
    -       "type": "timeseries",
    -       "title": "Recall candidates floored (by tier filter)",
    -       "description": "Candidates dropped from recall responses by the min_rank_score composite floor; a jump means the floor (or a scoring regression) is hiding results users would previously have seen.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 0,
    -         "y": 36
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (tier_filter) (rate(memini_recall_floored_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{tier_filter}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 21,
    -       "type": "timeseries",
    -       "title": "Recall reinforcement",
    -       "description": "Best-effort reinforcement writes triggered by recall hits (strengthening recalled memories); red errors mean reinforcement is silently failing and ranking will slowly stale.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 8,
    -         "y": 36
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +           "displayMode": "table",
    +           "placement": "right",
    +           "showLegend": true,
    +           "calcs": ["sum"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_reinforce_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    +           "expr": "sum by (result) (rate(memini_consolidate_results_total[5m]))",
    +           "legendFormat": "{{result}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 22,
    +       "id": 10,
            "type": "stat",
    -       "title": "Recall success ratio (15m)",
    -       "description": "Share of Recall calls that completed without error over 15m; below 99% something in the search path is failing.",
    +       "title": "Dedup effectiveness",
    +       "description": "Fraction of consolidations that actually used the LLM (update+supersede+new+noop) vs the similarity gate skipping it. Higher = busier corpus or noisier dedup signal.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 8,
              "x": 16,
    -         "y": 36
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "red",
    -                 "value": null
    -               },
    -               {
    -                 "color": "orange",
    -                 "value": 0.9
    -               },
    -               {
    -                 "color": "green",
    -                 "value": 0.99
    -               }
    -             ]
    -           },
    -           "unit": "percentunit"
    -         },
    -         "overrides": []
    +         "y": 34
            },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "unit": "percentunit",
    +           "min": 0,
    +           "max": 1
    +         }
    +       },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)",
    -           "refId": "A",
    -           "legendFormat": "ok ratio"
    +           "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|new|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result!~\"error|dropped\"}[15m])), 1e-9)",
    +           "legendFormat": "LLM use ratio",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 23,
    +       "id": 11,
            "type": "row",
    -       "title": "Aging & maintenance",
    +       "title": "Storage & decay",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 43
    +         "y": 42
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 24,
    +       "id": 12,
            "type": "timeseries",
            "title": "Live memories by tier",
    -       "description": "Live memory count per tier over time; working/episodic should churn while semantic/procedural grow slowly \u2014 a cliff in any tier means a sweep or dedup removed too much.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 0,
    -         "y": 44
    +         "y": 43
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    +             "lineInterpolation": "stepAfter",
    +             "fillOpacity": 15,
                  "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
                  "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    +               "mode": "normal"
                  }
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": []
    +           "unit": "short"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +           "displayMode": "table",
    +           "placement": "right",
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (memini_memories_active)",
    -           "refId": "A",
    -           "legendFormat": "{{tier}}"
    +           "expr": "memini_memories_active",
    +           "legendFormat": "{{tier}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 25,
    +       "id": 13,
            "type": "timeseries",
    -       "title": "Promotion, classification & demotion (per hour)",
    -       "description": "Hourly outcomes of the episodic-to-semantic promotion pass, durable facts it wrote, and omitted-tier writes the marker classifier routed; red promote errors mean durable knowledge is not being distilled. Demotions are the mirror image: stale, never-recalled durable memories retiered down to episodic \u2014 a steady demotion stream usually means bulk-imported debris aging out.",
    +       "title": "Decay sweep + tombstones (per minute)",
    +       "description": "Swept = memories purged by TTL; soft deletes = contradictions tombstoned by consolidation; dedup tombstones = tombstoned by the periodic dedup pass.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 12,
    -         "y": 44
    +         "y": 43
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    +             "lineInterpolation": "stepAfter",
    +             "fillOpacity": 15,
    +             "showPoints": "never"
                },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "promote: error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["sum"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (increase(memini_promote_results_total[1h]))",
    -           "refId": "A",
    -           "legendFormat": "promote: {{result}}"
    +           "expr": "sum by (tier) (rate(memini_store_swept_total[5m]))",
    +           "legendFormat": "swept/{{tier}}",
    +           "refId": "A"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(increase(memini_promote_facts_total[1h]))",
    -           "refId": "B",
    -           "legendFormat": "facts promoted"
    +           "expr": "rate(memini_store_soft_deletes_total[5m])",
    +           "legendFormat": "consolidate tombstones",
    +           "refId": "B"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (increase(memini_tier_classified_total[1h]))",
    -           "refId": "C",
    -           "legendFormat": "classified \u2192 {{tier}}"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (tier) (increase(memini_demoted_total[1h]))",
    -           "legendFormat": "demoted from {{tier}}",
    -           "refId": "D"
    +           "expr": "rate(memini_dedup_tombstoned_total[5m])",
    +           "legendFormat": "dedup tombstones",
    +           "refId": "C"
              }
            ]
          },
          {
    -       "id": 26,
    -       "type": "timeseries",
    -       "title": "Decay sweeps & dedup tombstones",
    -       "description": "Memories purged by the decay sweeper (by tier) and tombstoned by the dedup pass, per second; sustained zero forever means aging jobs are not running.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    +       "id": 14,
    +       "type": "row",
    +       "title": "Embedder & rerank",
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    +         "h": 1,
    +         "w": 24,
              "x": 0,
              "y": 51
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (tier) (rate(memini_store_swept_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "swept: {{tier}}"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "rate(memini_dedup_tombstoned_total[$__rate_interval])",
    -           "refId": "B",
    -           "legendFormat": "dedup tombstoned"
    -         }
    -       ]
    +       "collapsed": false,
    +       "panels": []
          },
          {
    -       "id": 27,
    +       "id": 15,
            "type": "timeseries",
    -       "title": "Fsck outcomes (per hour)",
    -       "description": "Hourly consistency-sweep outcomes; red errors mean the store's self-check cannot complete and silent corruption may go unnoticed.",
    +       "title": "Embedder p95 latency (by backend)",
    +       "description": "openai = real network calls; cached/diskcache/batched = local cache or fan-out; disabled = no endpoint configured.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
    -         "x": 12,
    -         "y": 51
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (result) (increase(memini_fsck_results_total[1h]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 28,
    -       "type": "timeseries",
    -       "title": "Repair backlog depth by state",
    -       "description": "Memories owing deferred repair, split by state; red failed entries are permanently vectorless without operator action, and a growing pending line means the repair loop cannot keep up.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
              "x": 0,
    -         "y": 58
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "failed"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (state) (memini_repair_depth)",
    -           "refId": "A",
    -           "legendFormat": "{{state}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 29,
    -       "type": "timeseries",
    -       "title": "Repair oldest age by state",
    -       "description": "Age of the oldest memory in each repair state; growth without bound means repairs are queued faster than they drain (the red line marks the 1h alarm threshold).",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 8,
    -         "y": 58
    +         "y": 52
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "line"
    -             }
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               },
    -               {
    -                 "color": "red",
    -                 "value": 3600
    -               }
    -             ]
    -           },
    -           "unit": "s",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    +           "unit": "s"
              }
            },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "max by (state) (memini_repair_oldest_age_seconds)",
    -           "refId": "A",
    -           "legendFormat": "{{state}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 30,
    -       "type": "timeseries",
    -       "title": "Backfill pending",
    -       "description": "Memories still awaiting embedding backfill and long memories still without chunk vectors; a number that never falls means those memories are unreachable by semantic (or chunked) recall.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 16,
    -         "y": 58
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_embed_backfill_pending",
    -           "refId": "A",
    -           "legendFormat": "pending embed"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "memini_chunk_backfill_pending",
    -           "refId": "B",
    -           "legendFormat": "pending chunk vectors"
    +           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[5m])))",
    +           "legendFormat": "p95 {{backend}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 31,
    +       "id": 16,
            "type": "timeseries",
    -       "title": "Repair outcomes by stage",
    -       "description": "Deferred-repair attempt outcomes per second by stage; red parked means a memory gave up retrying and stays vectorless, orange retry churn means the embedder keeps failing under the repair loop.",
    +       "title": "Embedder tokens + error rate",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
    -         "x": 0,
    -         "y": 65
    +         "x": 12,
    +         "y": 52
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".*parked.*"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".*retry.*"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    +             "showPoints": "never"
                }
    -         ]
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (stage, result) (rate(memini_repair_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{stage}}: {{result}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 32,
    -       "type": "timeseries",
    -       "title": "Repair duration p95 by stage",
    -       "description": "95th-percentile latency of a single deferred repair by stage; climbing durations shrink repair throughput and grow the backlog.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 12,
    -         "x": 12,
    -         "y": 65
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "s",
    -           "min": 0
    +           "expr": "sum by (backend) (rate(memini_embed_tokens_total[5m]))",
    +           "legendFormat": "tokens/s {{backend}}",
    +           "refId": "A"
              },
    -         "overrides": []
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(memini_repair_duration_seconds_bucket[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "{{stage}}"
    +           "expr": "sum by (backend) (rate(memini_embed_errors_total[5m]))",
    +           "legendFormat": "err/s {{backend}}",
    +           "refId": "B"
              }
            ]
          },
          {
    -       "id": 33,
    -       "type": "row",
    -       "title": "Embedder & rerank",
    -       "gridPos": {
    -         "h": 1,
    -         "w": 24,
    -         "x": 0,
    -         "y": 72
    -       },
    -       "collapsed": false,
    -       "panels": []
    -     },
    -     {
    -       "id": 34,
    -       "type": "timeseries",
    -       "title": "Embedder p95 latency by backend",
    -       "description": "95th-percentile embedder call latency per backend layer; the openai layer reflects the real provider, cached/diskcache should stay near zero.",
    +       "id": 24,
    +       "type": "stat",
    +       "title": "Embedder in-flight",
    +       "description": "Concurrent embed calls against the backend (post-cap, excludes cache hits). Spikes during ingest storms are normal; sustained values near the cap mean the embedder is the bottleneck.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    +         "h": 4,
    +         "w": 8,
              "x": 0,
    -         "y": 73
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "s",
    -           "min": 0
    -         },
    -         "overrides": []
    +         "y": 60
            },
            "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "horizontal",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
              },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "{{backend}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 35,
    -       "type": "timeseries",
    -       "title": "Rerank outcomes by backend",
    -       "description": "Recall rerank attempts per second by backend and result; orange fallback means the reranker failed or timed out and results shipped with pre-rerank ordering.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 12,
    -         "x": 12,
    -         "y": 73
    +         "textMode": "auto"
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".*fallback.*"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    +           "unit": "short"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend, result) (rate(memini_rerank_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{backend}}: {{result}}"
    +           "expr": "memini_embed_in_flight",
    +           "legendFormat": "embed in-flight",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 36,
    -       "type": "timeseries",
    -       "title": "Embedder tokens/s by backend",
    -       "description": "Embedding tokens consumed per second as reported by the API (openai backend only); this is the embed cost driver.",
    +       "id": 25,
    +       "type": "stat",
    +       "title": "Rerank in-flight",
    +       "description": "Concurrent rerank calls. A nonzero value indicates rerank is configured; if this stays pinned at the cap, the reranker is the bottleneck.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 4,
              "w": 8,
    -         "x": 0,
    -         "y": 80
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    +         "x": 8,
    +         "y": 60
            },
            "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "horizontal",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
              },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum by (backend) (rate(memini_embed_tokens_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{backend}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 37,
    -       "type": "timeseries",
    -       "title": "Embedder errors by backend",
    -       "description": "Embedder failures per second by backend; any sustained nonzero rate here explains degraded writes and keyword-only recalls elsewhere on this board.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 8,
    -         "x": 8,
    -         "y": 80
    +         "textMode": "auto"
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    +           "unit": "short"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend) (rate(memini_embed_errors_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{backend}}"
    +           "expr": "memini_rerank_in_flight",
    +           "legendFormat": "rerank in-flight",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 38,
    +       "id": 26,
            "type": "timeseries",
    -       "title": "Embedder batch size (p50 / p95)",
    -       "description": "Texts per Embed call at the median and 95th percentile; batches stuck at 1 mean the batcher is not coalescing and each write pays a full round trip.",
    +       "title": "Embedder batch size (p50 vs p95)",
    +       "description": "Items per embed call, by backend. batched shows a single high-p95 bar; cached should be ~1.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 4,
              "w": 8,
              "x": 16,
    -         "y": 80
    +         "y": 60
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 5,
    +             "showPoints": "never"
    +           }
              }
            },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "histogram_quantile(0.50, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "p50 {{backend}}"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))",
    -           "refId": "B",
    -           "legendFormat": "p95 {{backend}}"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 39,
    -       "type": "timeseries",
    -       "title": "In-flight embed & rerank calls",
    -       "description": "Concurrent calls currently held against the embedding and rerank backends; pinned at the concurrency cap means requests are queueing behind the limiter.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 6,
    -         "w": 24,
    -         "x": 0,
    -         "y": 87
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": []
    -       },
            "options": {
              "legend": {
    -           "calcs": [],
                "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_embed_in_flight",
    -           "refId": "A",
    -           "legendFormat": "embed"
    +           "expr": "histogram_quantile(0.5, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))",
    +           "legendFormat": "p50 {{backend}}",
    +           "refId": "A"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_rerank_in_flight",
    -           "refId": "B",
    -           "legendFormat": "rerank"
    +           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))",
    +           "legendFormat": "p95 {{backend}}",
    +           "refId": "B"
              }
            ]
          },
          {
    -       "id": 40,
    +       "id": 27,
            "type": "row",
    -       "title": "LLM \u2014 consolidation & answer (empty without an LLM configured)",
    +       "title": "HTTP surface",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 93
    +         "y": 64
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 41,
    +       "id": 28,
            "type": "timeseries",
    -       "title": "Consolidation outcomes",
    -       "description": "LLM consolidation decisions per second (gated, new, update, supersede, noop); red errors or orange drops mean writes are skipping dedup and duplicates will accumulate.",
    +       "title": "HTTP requests by route (status class)",
    +       "description": "Per-route request rate broken down by 2xx / 4xx / 5xx. 5xx spikes point at a broken handler; 4xx spikes usually mean client misuse or a misconfigured agent.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    +         "h": 8,
    +         "w": 16,
              "x": 0,
    -         "y": 94
    +         "y": 65
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 15,
                  "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
                  "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    +               "mode": "normal"
                  }
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "dropped"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "ops"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +           "displayMode": "table",
    +           "placement": "right",
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_consolidate_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    +           "expr": "sum by (route, status) (label_replace(rate(memini_http_requests_total[5m]), \"status_class\", \"$1\", \"status\", \"(...).\"))",
    +           "legendFormat": "{{route}} {{status_class}}xx",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 42,
    +       "id": 29,
            "type": "timeseries",
    -       "title": "Consolidation queue depth",
    -       "description": "Async consolidation queue depth over time; a sawtooth is normal, monotonic growth means the LLM cannot keep pace with write volume.",
    +       "title": "HTTP p95 latency by route",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 6,
    -         "x": 12,
    -         "y": 94
    +         "h": 8,
    +         "w": 8,
    +         "x": 16,
    +         "y": 65
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": []
    +           "unit": "s"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": false
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_consolidate_queue_depth",
    -           "refId": "A",
    -           "legendFormat": "queue depth"
    +           "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[5m])))",
    +           "legendFormat": "p95 {{route}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 43,
    -       "type": "stat",
    -       "title": "Dedup effectiveness (15m)",
    -       "description": "Share of consolidated writes that matched an existing memory (update/supersede/noop) rather than creating a new one; a sudden drop to zero suggests the matcher is broken.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 6,
    -         "x": 18,
    -         "y": 94
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "thresholds"
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "percentunit"
    -         },
    -         "overrides": []
    -       },
    -       "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "auto",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    -         },
    -         "textMode": "auto"
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result=~\"new|update|supersede|noop\"}[15m])), 1e-9)",
    -           "refId": "A",
    -           "legendFormat": "matched"
    -         }
    -       ]
    -     },
    -     {
    -       "id": 44,
    +       "id": 17,
            "type": "timeseries",
    -       "title": "Answer outcomes",
    -       "description": "Answer API (LLM-synthesized answers over recall) outcomes per second; red errors mean answer requests are burning LLM budget without returning.",
    +       "title": "End-to-end op latency p95 (by op)",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    +         "h": 8,
    +         "w": 16,
              "x": 0,
    -         "y": 101
    +         "y": 73
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "lineInterpolation": "smooth",
    +             "fillOpacity": 5,
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byName",
    -               "options": "error"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    +           "unit": "s"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    +           "displayMode": "table",
    +           "placement": "right",
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_answer_results_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    +           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[5m])))",
    +           "legendFormat": "{{op}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 45,
    +       "id": 34,
            "type": "timeseries",
    -       "title": "LLM op latency p95",
    -       "description": "95th-percentile end-to-end latency of the LLM-backed service ops (answer, promote, dedup, semantic distillation wait); these dominate tail latency when the LLM slows down.",
    +       "title": "HTTP response size by route (p50/p90)",
    +       "description": "Body bytes shipped per response. The recall/briefing routes are what hooks inject from — watch these when tuning context-injection volume.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    -         "x": 12,
    -         "y": 101
    +         "h": 8,
    +         "w": 8,
    +         "x": 16,
    +         "y": 73
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    +             "showPoints": "never"
                },
    -           "unit": "s",
    -           "min": 0
    -         },
    -         "overrides": []
    +           "unit": "bytes"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket{op=~\"answer|promote|dedup|distill_sem_wait\"}[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "{{op}}"
    +           "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))",
    +           "legendFormat": "p90 {{route}}",
    +           "refId": "A"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))",
    +           "legendFormat": "p50 {{route}}",
    +           "refId": "B"
              }
            ]
          },
          {
    -       "id": 46,
    +       "id": 30,
            "type": "row",
    -       "title": "HTTP surface",
    +       "title": "Write-path signals",
    +       "collapsed": false,
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 108
    +         "y": 82
            },
    -       "collapsed": false,
            "panels": []
          },
          {
    -       "id": 47,
    +       "id": 31,
            "type": "timeseries",
    -       "title": "HTTP request rate by route & status class",
    -       "description": "Requests per second per route, grouped into status classes; red 5xx traffic is server failure, orange 4xx spikes usually mean a misconfigured client or expired key.",
    +       "title": "Corroboration routing",
    +       "description": "A fresh short-term write restating a durable fact: corroborated (confidence grown), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 0,
    -         "y": 109
    +         "y": 83
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "reqps",
    -           "min": 0
    -         },
    -         "overrides": [
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".* 5xx"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "red"
    -                 }
    -               }
    -             ]
    -           },
    -           {
    -             "matcher": {
    -               "id": "byRegexp",
    -               "options": ".* 4xx"
    -             },
    -             "properties": [
    -               {
    -                 "id": "color",
    -                 "value": {
    -                   "mode": "fixed",
    -                   "fixedColor": "orange"
    -                 }
    -               }
    -             ]
    -           }
    -         ]
    -       },
    -       "options": {
    -         "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    -           "placement": "bottom",
    -           "showLegend": true
    -         },
    -         "tooltip": {
    -           "mode": "multi",
    -           "sort": "desc"
    -         }
    -       },
    -       "targets": [
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    +             "showPoints": "never"
                },
    -           "expr": "sum by (route, status_class) (label_replace(rate(memini_http_requests_total[$__rate_interval]), \"status_class\", \"$1\", \"status\", \"(.)..\"))",
    -           "refId": "A",
    -           "legendFormat": "{{route}} {{status_class}}xx"
    +           "unit": "short"
              }
    -       ]
    -     },
    -     {
    -       "id": 48,
    -       "type": "timeseries",
    -       "title": "HTTP p95 latency by route",
    -       "description": "95th-percentile HTTP latency per route as clients experience it; compare with the service-op view below to tell transport overhead from service work.",
    -       "datasource": {
    -         "type": "prometheus",
    -         "uid": "${DS_PROMETHEUS}"
    -       },
    -       "gridPos": {
    -         "h": 7,
    -         "w": 12,
    -         "x": 12,
    -         "y": 109
    -       },
    -       "fieldConfig": {
    -         "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
    -           "custom": {
    -             "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    -             "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "s",
    -           "min": 0
    -         },
    -         "overrides": []
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "{{route}}"
    +           "expr": "sum by (result) (rate(memini_corroborate_results_total[5m]))",
    +           "legendFormat": "{{result}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 49,
    +       "id": 33,
            "type": "timeseries",
    -       "title": "Service op latency p95 by op",
    -       "description": "95th-percentile end-to-end latency of every public service operation (remember, recall, briefing, forget, ...); this is the service-internal view behind the HTTP numbers.",
    +       "title": "Contradiction routing",
    +       "description": "A fresh durable write that changes a fact's value or flips its polarity: contradicted (stale fact invalidated via valid_to + confidence shrink), no_signal (durable neighbour found but not a real change), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error. Default-on; a contradicted rate far above expectation is the misfire smoke alarm (set MEMINI_CONTRADICT_DOWNRANK=false to disable).",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 0,
    -         "y": 116
    +         "y": 91
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "s",
    -           "min": 0
    -         },
    -         "overrides": []
    +           "unit": "short"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "{{op}}"
    +           "expr": "sum by (result) (rate(memini_contradict_results_total[5m]))",
    +           "legendFormat": "{{result}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 50,
    +       "id": 32,
            "type": "timeseries",
    -       "title": "HTTP response size (search & briefing, p50 / p90)",
    -       "description": "Response body sizes for the two injection-critical routes; growth here directly inflates the tokens injected into agent context.",
    +       "title": "Tier classification & promotion",
    +       "description": "Omitted-tier writes the marker classifier routed to a durable tier, and facts produced by the promotion pass (LLM or marker extractor).",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    +         "h": 8,
              "w": 12,
              "x": 12,
    -         "y": 116
    +         "y": 83
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    +             "showPoints": "never"
                },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "bytes",
    -           "min": 0
    -         },
    -         "overrides": []
    +           "unit": "short"
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))",
    -           "refId": "A",
    -           "legendFormat": "p50 {{route}}"
    +           "expr": "sum by (tier) (rate(memini_tier_classified_total[5m]))",
    +           "legendFormat": "classified -> {{tier}}",
    +           "refId": "A"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))",
    -           "refId": "B",
    -           "legendFormat": "p90 {{route}}"
    +           "expr": "sum(rate(memini_promote_facts_total[5m]))",
    +           "legendFormat": "promoted facts",
    +           "refId": "B"
              }
            ]
          },
          {
    -       "id": 51,
    +       "id": 40,
            "type": "row",
    -       "title": "Client injection telemetry",
    +       "title": "Injection efficiency (client telemetry)",
    +       "collapsed": false,
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 123
    +         "y": 99
            },
    -       "collapsed": false,
            "panels": []
          },
          {
    -       "id": 52,
    +       "id": 41,
            "type": "timeseries",
    -       "title": "Injection outcomes by result",
    -       "description": "Client-reported memory injections per second by outcome; suppressed_* series show why memories were withheld (already seen, cooldown, budget, unchanged, score) \u2014 a spike in suppressions with flat injections means the hooks are firing but delivering nothing.",
    +       "title": "Injection outcomes (by result)",
    +       "description": "What hooks reported actually reaching model context vs held back by their local gates (seen/cooldown/budget/unchanged/score). Absent entirely means no integration is reporting.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 12,
    +         "h": 8,
    +         "w": 8,
              "x": 0,
    -         "y": 124
    +         "y": 100
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
                  "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
                  "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    +               "mode": "normal"
                  }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    +           }
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_injected_memories_total[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{result}}"
    +           "expr": "sum by (result) (rate(memini_injected_memories_total[5m]))",
    +           "legendFormat": "{{result}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 53,
    +       "id": 42,
            "type": "timeseries",
    -       "title": "Injected memories by surface",
    -       "description": "Memories actually injected into model context per second, by hook surface (briefing, prompt, pretool); a silent surface means that hook is broken client-side.",
    +       "title": "Injected memories (by surface)",
    +       "description": "Memories that reached context, split by which hook surface put them there.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 6,
    -         "x": 12,
    -         "y": 124
    +         "h": 8,
    +         "w": 8,
    +         "x": 8,
    +         "y": 100
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "ops",
    -           "min": 0
    -         },
    -         "overrides": []
    +             "showPoints": "never"
    +           }
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[$__rate_interval]))",
    -           "refId": "A",
    -           "legendFormat": "{{surface}}"
    +           "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[5m]))",
    +           "legendFormat": "{{surface}}",
    +           "refId": "A"
              }
            ]
          },
          {
    -       "id": 54,
    +       "id": 43,
            "type": "timeseries",
    -       "title": "Injected tokens per hour by surface",
    -       "description": "Client-side estimate of context tokens injected per hour by surface; this is the direct context-budget cost of memini to the agent.",
    +       "title": "Injected tokens per hour (by surface)",
    +       "description": "Client-estimated tokens the hooks added to context. The number injection tuning should push down.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 7,
    -         "w": 6,
    -         "x": 18,
    -         "y": 124
    +         "h": 8,
    +         "w": 8,
    +         "x": 16,
    +         "y": 100
            },
            "fieldConfig": {
              "defaults": {
    -           "color": {
    -             "mode": "palette-classic"
    -           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "linear",
    -             "lineWidth": 1,
    +             "lineInterpolation": "smooth",
                  "fillOpacity": 10,
    -             "gradientMode": "none",
    -             "showPoints": "never",
    -             "pointSize": 5,
    -             "spanNulls": false,
    -             "stacking": {
    -               "mode": "none",
    -               "group": "A"
    -             },
    -             "axisPlacement": "auto",
    -             "axisLabel": "",
    -             "axisColorMode": "text",
    -             "scaleDistribution": {
    -               "type": "linear"
    -             },
    -             "hideFrom": {
    -               "legend": false,
    -               "tooltip": false,
    -               "viz": false
    -             },
    -             "thresholdsStyle": {
    -               "mode": "off"
    -             }
    -           },
    -           "thresholds": {
    -             "mode": "absolute",
    -             "steps": [
    -               {
    -                 "color": "green",
    -                 "value": null
    -               }
    -             ]
    -           },
    -           "unit": "none",
    -           "min": 0
    -         },
    -         "overrides": []
    +             "showPoints": "never"
    +           }
    +         }
            },
            "options": {
              "legend": {
    -           "calcs": [],
    -           "displayMode": "list",
    +           "displayMode": "table",
                "placement": "bottom",
    -           "showLegend": true
    +           "showLegend": true,
    +           "calcs": ["mean", "max"]
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [five lines unchanged)]
      
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
                "expr": "sum by (surface) (increase(memini_injected_tokens_total[1h]))",
    -           "refId": "A",
    -           "legendFormat": "{{surface}}"
    +           "legendFormat": "{{surface}}",
    +           "refId": "A"
              }
            ]
          }
        ],
      
      [30 lines unchanged)]
      
        "timepicker": {},
        "timezone": "",
        "title": "memini",
        "uid": "memini",
    -   "version": 3,
    +   "version": 2,
        "weekStart": ""
      }
  
  

spec.template.spec.containers.main.image  (apps/v1/StatefulSet/local-ai/memini)
  ± value change
    - registry.erwanleboucher.dev/eleboucher/memini@sha256:1e060967d6fd11dfb79eeaa67d8745e0dde5d946ca1b7dd7469b4fad2fac0c1d
    + registry.erwanleboucher.dev/eleboucher/memini@sha256:2d388222bca3d65673a4072e99e0970fe0d5cdab0030ce0effdfdd78307ae7e2

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 data.memini.json (v1/ConfigMap/local-ai/memini-memini) ± value change in multiline text (271 inserts, 314 deletions) { "annotations": { "list": [] }, - "description": "memini operator dashboard \u2014 top-to-bottom narrative: live headline health (including the deferred-repair backlog), the write path, recall & retrieval quality, aging & maintenance jobs, embedder & rerank, LLM-backed consolidation/answer, the HTTP surface, and client injection telemetry.", + "description": "memini \u2014 what the service is actually doing for your agents. Live memory counts, write/recall traffic by tier, consolidation outcomes, store health, embedder & rerank latency & saturation, HTTP surface.", "editable": true, "graphTooltip": 1, "id": null, "links": [], [14 lines unchanged)] { "id": 2, "type": "stat", "title": "Live memories", - "description": "Total live (non-superseded, non-expired) memories across all tiers; a sudden drop means a sweep or dedup pass removed more than expected.", + "description": "Live (non-superseded, non-expired) memories by tier.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 5, - "w": 4, + "h": 6, + "w": 6, "x": 0, "y": 1 }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none" - }, - "overrides": [] - }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "auto", + "orientation": "horizontal", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false [five lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(memini_memories_active)", - "refId": "A", - "legendFormat": "live" + "expr": "memini_memories_active", + "legendFormat": "{{tier}}", + "refId": "A" } ] }, { "id": 3, "type": "stat", - "title": "Writes /s (5m)", - "description": "Successful Remember writes per second over the last 5m; zero during active agent sessions means writes are failing upstream.", + "title": "Writes (5m rate)", + "description": "Successful Remember calls per second, by tier.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 5, - "w": 3, - "x": 4, + "h": 6, + "w": 6, + "x": 6, "y": 1 }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "decimals": 2 - }, - "overrides": [] - }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "auto", + "orientation": "horizontal", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "fieldConfig": { + "defaults": { + "unit": "ops" + } + }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_remember_results_total{result=\"ok\"}[5m]))", - "refId": "A", - "legendFormat": "writes/s" + "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))", + "legendFormat": "{{tier}}", + "refId": "A" } ] }, { "id": 4, "type": "stat", - "title": "Recalls /s (5m)", - "description": "Successful Recall searches per second over the last 5m; the primary read-traffic pulse.", + "title": "Recalls (5m rate)", + "description": "Successful Recall calls per second.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 5, - "w": 3, - "x": 7, + "h": 6, + "w": 6, + "x": 12, "y": 1 }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "decimals": 2 - }, - "overrides": [] - }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "auto", + "orientation": "horizontal", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))", - "refId": "A", - "legendFormat": "recalls/s" - } - ] - }, - { - "id": 5, - "type": "stat", - "title": "HTTP 5xx ratio (15m)", - "description": "Share of HTTP requests answered 5xx over 15m; anything above zero deserves a look, above 5% is an outage.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 5, - "w": 3, - "x": 10, - "y": 1 - }, "fieldConfig": { "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 0.01 - }, - { - "color": "red", - "value": 0.05 - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "textMode": "auto" + "unit": "ops" + } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_http_requests_total{status=~\"5..\"}[15m])) / clamp_min(sum(rate(memini_http_requests_total[15m])), 1e-9)", - "refId": "A", - "legendFormat": "5xx ratio" + "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))", + "legendFormat": "recalls/s", + "refId": "A" } ] }, { - "id": 6, + "id": 18, "type": "stat", - "title": "Repair backlog depth", - "description": "Memories currently owing deferred repair (pending + enrich + failed); green is empty, any sustained backlog means writes are storing degraded faster than repairs drain.", + "title": "Consolidation queue", + "description": "Backlog of memories awaiting async consolidation. Sustained > 0 means the worker can't keep up with Remember writes.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 5, - "w": 4, - "x": 13, + "h": 6, + "w": 6, + "x": 18, "y": 1 }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 1 - } - ] - }, - "unit": "none" - }, - "overrides": [] - }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "auto", + "orientation": "horizontal", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum(memini_repair_depth)", - "refId": "A", - "legendFormat": "backlog" - } - ] - }, - { - "id": 7, - "type": "stat", - "title": "Repair oldest age", - "description": "Age of the oldest memory still awaiting repair; amber the moment anything waits, red past 1h means the repair loop is stuck or starved.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 5, - "w": 4, - "x": 17, - "y": 1 - }, "fieldConfig": { "defaults": { - "color": { - "mode": "thresholds" - }, + "unit": "short", "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { - "color": "orange", - "value": 1 + "color": "yellow", + "value": 50 }, { "color": "red", - "value": 3600 + "value": 500 } ] - }, - "unit": "s" - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "textMode": "auto" - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "max(memini_repair_oldest_age_seconds)", - "refId": "A", - "legendFormat": "oldest" + } } - ] - }, - { - "id": 8, - "type": "stat", - "title": "Consolidation queue", - "description": "Depth of the async consolidation queue; a queue that grows without draining delays dedup/supersede decisions.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 5, - "w": 3, - "x": 21, - "y": 1 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 100 - } - ] - }, - "unit": "none" - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "textMode": "auto" }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "memini_consolidate_queue_depth", - "refId": "A", - "legendFormat": "queued" + "legendFormat": "queued", + "refId": "A" } ] }, { - "id": 9, + "id": 5, "type": "row", - "title": "Write path", + "title": "Writes & Recalls", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 6 + "y": 7 }, "collapsed": false, "panels": [] }, { - "id": 10, + "id": 6, "type": "timeseries", - "title": "Remember rate by tier", - "description": "Successful Remember writes per second split by tier, plus error/dropped outcomes; red or orange series above zero mean writes are being lost.", + "title": "Remembers per second (by tier)", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 7 + "y": 8 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "dropped" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{tier}}" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum(rate(memini_remember_results_total{result=\"error\"}[$__rate_interval]))", - "refId": "B", - "legendFormat": "error" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum(rate(memini_remember_results_total{result=\"dropped\"}[$__rate_interval]))", - "refId": "C", - "legendFormat": "dropped" + "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))", + "legendFormat": "{{tier}}", + "refId": "A" } ] }, { - "id": 11, + "id": 7, "type": "timeseries", - "title": "Degraded (vectorless) writes by reason", - "description": "Writes stored without a vector (keyword-searchable only, queued for deferred repair) because the embedder timed out or errored; any sustained rate means the embedder is unhealthy.", + "title": "Recalls per second (by hit-count bucket)", + "description": "How many memories a typical recall returns. '0' means no matches \u2014 investigate if persistent.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 7 + "y": 8 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "embed_error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "embed_timeout" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (reason) (rate(memini_remember_degraded_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{reason}}" + "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[5m]))", + "legendFormat": "{{hits_bucket}}", + "refId": "A" } ] }, { - "id": 12, + "id": 19, + "type": "row", + "title": "Retrieval quality", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 16 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 20, "type": "timeseries", - "title": "Store mutations", - "description": "Raw store write traffic: inserts vs updates (by upsert op), hard deletes from Forget, and tombstones written by consolidation; updates dominating inserts means most writes reinforce existing memories.", + "title": "Rerank outcomes by backend", + "description": "Per-recall reranker traffic: ok = reranker reordered the fused candidates; fallback = reranker timed out / errored and the fused order was kept.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 14 + "y": 17 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "lineInterpolation": "smooth", + "fillOpacity": 15, + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (op) (rate(memini_store_upserts_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "upsert: {{op}}" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "rate(memini_store_deletes_total[$__rate_interval])", - "refId": "B", - "legendFormat": "hard delete (forget)" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "rate(memini_store_soft_deletes_total[$__rate_interval])", - "refId": "C", - "legendFormat": "tombstone (supersede)" + "expr": "sum by (backend, result) (rate(memini_rerank_results_total[5m]))", + "legendFormat": "{{backend}}/{{result}}", + "refId": "A" } ] }, { - "id": 13, + "id": 21, "type": "timeseries", - "title": "Forget & supersede outcomes", - "description": "Forget and Supersede API outcomes per second; red error series should stay at zero, orange not_found spikes usually mean a client is retrying stale ids.", + "title": "Recall degradation reasons", + "description": "Recalls that fell back to keyword-only search because the embedder was slow or errored. embed_timeout = the recall embed budget elapsed; embed_error = the embed call failed.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 14 + "y": 17 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "lineInterpolation": "smooth", + "fillOpacity": 15, + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byRegexp", - "options": ".*error.*" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byRegexp", - "options": ".*not_found.*" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_forget_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "forget: {{result}}" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (result) (rate(memini_supersede_results_total[$__rate_interval]))", - "refId": "B", - "legendFormat": "supersede: {{result}}" + "expr": "sum by (reason) (rate(memini_recall_degraded_total[5m]))", + "legendFormat": "{{reason}}", + "refId": "A" } ] }, { - "id": 14, + "id": 22, "type": "timeseries", - "title": "Content hygiene (sanitizer)", - "description": "Ingestion sanitizer actions: cleaned content was rewritten in place, quarantined (orange) content was judged unsafe to store as-is and needs review if it becomes frequent.", + "title": "Recall reinforcement (best-effort)", + "description": "Background access-count bump on recalled memories. Errors are swallowed \u2014 a sustained nonzero error rate means the store write path is unhealthy but recall still served.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 8, + "h": 8, + "w": 12, "x": 0, - "y": 21 + "y": 25 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "lineInterpolation": "smooth", + "fillOpacity": 15, + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "quarantined" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (action) (rate(memini_write_sanitized_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{action}}" + "expr": "sum by (result) (rate(memini_reinforce_results_total[5m]))", + "legendFormat": "{{result}}", + "refId": "A" } ] }, { - "id": 15, - "type": "timeseries", - "title": "Corroboration routing", - "description": "Corroboration attempts on fresh short-term writes (did this confirm something we already know); a rising red error rate means the corroboration path is failing writes silently.", + "id": 23, + "type": "stat", + "title": "Recall success ratio", + "description": "Fraction of recall calls that succeeded. A drop means the recall path is erroring (embed + keyword + rerank + RRF + everything).", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 21 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] + "h": 8, + "w": 12, + "x": 12, + "y": 25 }, "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "colorMode": "value", + "graphMode": "area", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (result) (rate(memini_corroborate_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{result}}" - } - ] - }, - { - "id": 16, - "type": "timeseries", - "title": "Contradiction routing", - "description": "Contradiction attempts on fresh durable writes (did this contradict an existing fact); contradicted events drive supersession, red errors mean conflicts are going undetected.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 21 + "textMode": "auto" }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, + "unit": "percentunit", + "min": 0, + "max": 1, "thresholds": { "mode": "absolute", "steps": [ { - "color": "green", + "color": "red", "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ + }, + { + "color": "yellow", + "value": 0.95 + }, { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } + "color": "green", + "value": 0.99 } ] } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_contradict_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{result}}" + "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)", + "legendFormat": "ok ratio", + "refId": "A" } ] }, { - "id": 17, + "id": 8, "type": "row", - "title": "Recall & retrieval quality", + "title": "Consolidation (LLM dedup)", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 28 + "y": 33 }, "collapsed": false, "panels": [] }, { - "id": 18, + "id": 9, "type": "timeseries", - "title": "Recall rate by hit-count bucket", - "description": "Successful recalls per second split by how many hits they returned, plus errors; a shift toward the 0-hits bucket means retrieval quality is degrading.", + "title": "Consolidation outcomes per minute", + "description": "gated = similarity gate skipped the LLM; new = nothing to merge; update = merged into an existing memory; supersede = contradiction-tombstoned an old one; noop = nothing to do; error/dropped = failure modes.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 12, + "h": 8, + "w": 16, "x": 0, - "y": 29 + "y": 34 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", + "lineInterpolation": "smooth", + "fillOpacity": 20, "showPoints": "never", - "pointSize": 5, - "spanNulls": false, "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" + "mode": "normal" } }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" + "unit": "ops" } }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{hits_bucket}} hits" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum(rate(memini_recall_results_total{result=\"error\"}[$__rate_interval]))", - "refId": "B", - "legendFormat": "error" - } - ] - }, - { - "id": 19, - "type": "timeseries", - "title": "Recall degradation (keyword-only fallback)", - "description": "Recalls that fell back to keyword-only search because embedding the query timed out or errored; these return without semantic ranking, so any sustained rate hurts result quality.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 29 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "embed_error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "embed_timeout" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] - }, "options": { "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (reason) (rate(memini_recall_degraded_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{reason}}" - } - ] - }, - { - "id": 20, - "type": "timeseries", - "title": "Recall candidates floored (by tier filter)", - "description": "Candidates dropped from recall responses by the min_rank_score composite floor; a jump means the floor (or a scoring regression) is hiding results users would previously have seen.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 36 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (tier_filter) (rate(memini_recall_floored_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{tier_filter}}" - } - ] - }, - { - "id": 21, - "type": "timeseries", - "title": "Recall reinforcement", - "description": "Best-effort reinforcement writes triggered by recall hits (strengthening recalled memories); red errors mean reinforcement is silently failing and ranking will slowly stale.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 36 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "displayMode": "table", + "placement": "right", + "showLegend": true, + "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_reinforce_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{result}}" + "expr": "sum by (result) (rate(memini_consolidate_results_total[5m]))", + "legendFormat": "{{result}}", + "refId": "A" } ] }, { - "id": 22, + "id": 10, "type": "stat", - "title": "Recall success ratio (15m)", - "description": "Share of Recall calls that completed without error over 15m; below 99% something in the search path is failing.", + "title": "Dedup effectiveness", + "description": "Fraction of consolidations that actually used the LLM (update+supersede+new+noop) vs the similarity gate skipping it. Higher = busier corpus or noisier dedup signal.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 8, "x": 16, - "y": 36 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "orange", - "value": 0.9 - }, - { - "color": "green", - "value": 0.99 - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] + "y": 34 }, "options": { "colorMode": "value", "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "fieldConfig": { + "defaults": { + "unit": "percentunit", + "min": 0, + "max": 1 + } + }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)", - "refId": "A", - "legendFormat": "ok ratio" + "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|new|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result!~\"error|dropped\"}[15m])), 1e-9)", + "legendFormat": "LLM use ratio", + "refId": "A" } ] }, { - "id": 23, + "id": 11, "type": "row", - "title": "Aging & maintenance", + "title": "Storage & decay", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 43 + "y": 42 }, "collapsed": false, "panels": [] }, { - "id": 24, + "id": 12, "type": "timeseries", "title": "Live memories by tier", - "description": "Live memory count per tier over time; working/episodic should churn while semantic/procedural grow slowly \u2014 a cliff in any tier means a sweep or dedup removed too much.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 44 + "y": 43 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", + "lineInterpolation": "stepAfter", + "fillOpacity": 15, "showPoints": "never", - "pointSize": 5, - "spanNulls": false, "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" + "mode": "normal" } }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [] + "unit": "short" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "displayMode": "table", + "placement": "right", + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (memini_memories_active)", - "refId": "A", - "legendFormat": "{{tier}}" + "expr": "memini_memories_active", + "legendFormat": "{{tier}}", + "refId": "A" } ] }, { - "id": 25, + "id": 13, "type": "timeseries", - "title": "Promotion, classification & demotion (per hour)", - "description": "Hourly outcomes of the episodic-to-semantic promotion pass, durable facts it wrote, and omitted-tier writes the marker classifier routed; red promote errors mean durable knowledge is not being distilled. Demotions are the mirror image: stale, never-recalled durable memories retiered down to episodic \u2014 a steady demotion stream usually means bulk-imported debris aging out.", + "title": "Decay sweep + tombstones (per minute)", + "description": "Swept = memories purged by TTL; soft deletes = contradictions tombstoned by consolidation; dedup tombstones = tombstoned by the periodic dedup pass.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 44 + "y": 43 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] + "lineInterpolation": "stepAfter", + "fillOpacity": 15, + "showPoints": "never" }, - "unit": "none", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "promote: error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (increase(memini_promote_results_total[1h]))", - "refId": "A", - "legendFormat": "promote: {{result}}" + "expr": "sum by (tier) (rate(memini_store_swept_total[5m]))", + "legendFormat": "swept/{{tier}}", + "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(increase(memini_promote_facts_total[1h]))", - "refId": "B", - "legendFormat": "facts promoted" + "expr": "rate(memini_store_soft_deletes_total[5m])", + "legendFormat": "consolidate tombstones", + "refId": "B" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (increase(memini_tier_classified_total[1h]))", - "refId": "C", - "legendFormat": "classified \u2192 {{tier}}" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (tier) (increase(memini_demoted_total[1h]))", - "legendFormat": "demoted from {{tier}}", - "refId": "D" + "expr": "rate(memini_dedup_tombstoned_total[5m])", + "legendFormat": "dedup tombstones", + "refId": "C" } ] }, { - "id": 26, - "type": "timeseries", - "title": "Decay sweeps & dedup tombstones", - "description": "Memories purged by the decay sweeper (by tier) and tombstoned by the dedup pass, per second; sustained zero forever means aging jobs are not running.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, + "id": 14, + "type": "row", + "title": "Embedder & rerank", "gridPos": { - "h": 7, - "w": 12, + "h": 1, + "w": 24, "x": 0, "y": 51 }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (tier) (rate(memini_store_swept_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "swept: {{tier}}" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "rate(memini_dedup_tombstoned_total[$__rate_interval])", - "refId": "B", - "legendFormat": "dedup tombstoned" - } - ] + "collapsed": false, + "panels": [] }, { - "id": 27, + "id": 15, "type": "timeseries", - "title": "Fsck outcomes (per hour)", - "description": "Hourly consistency-sweep outcomes; red errors mean the store's self-check cannot complete and silent corruption may go unnoticed.", + "title": "Embedder p95 latency (by backend)", + "description": "openai = real network calls; cached/diskcache/batched = local cache or fan-out; disabled = no endpoint configured.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, - "x": 12, - "y": 51 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (result) (increase(memini_fsck_results_total[1h]))", - "refId": "A", - "legendFormat": "{{result}}" - } - ] - }, - { - "id": 28, - "type": "timeseries", - "title": "Repair backlog depth by state", - "description": "Memories owing deferred repair, split by state; red failed entries are permanently vectorless without operator action, and a growing pending line means the repair loop cannot keep up.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, "x": 0, - "y": 58 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "failed" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (state) (memini_repair_depth)", - "refId": "A", - "legendFormat": "{{state}}" - } - ] - }, - { - "id": 29, - "type": "timeseries", - "title": "Repair oldest age by state", - "description": "Age of the oldest memory in each repair state; growth without bound means repairs are queued faster than they drain (the red line marks the 1h alarm threshold).", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 58 + "y": 52 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "line" - } + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 3600 - } - ] - }, - "unit": "s", - "min": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" + "unit": "s" } }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "max by (state) (memini_repair_oldest_age_seconds)", - "refId": "A", - "legendFormat": "{{state}}" - } - ] - }, - { - "id": 30, - "type": "timeseries", - "title": "Backfill pending", - "description": "Memories still awaiting embedding backfill and long memories still without chunk vectors; a number that never falls means those memories are unreachable by semantic (or chunked) recall.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 58 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [] - }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_embed_backfill_pending", - "refId": "A", - "legendFormat": "pending embed" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "memini_chunk_backfill_pending", - "refId": "B", - "legendFormat": "pending chunk vectors" + "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[5m])))", + "legendFormat": "p95 {{backend}}", + "refId": "A" } ] }, { - "id": 31, + "id": 16, "type": "timeseries", - "title": "Repair outcomes by stage", - "description": "Deferred-repair attempt outcomes per second by stage; red parked means a memory gave up retrying and stays vectorless, orange retry churn means the embedder keeps failing under the repair loop.", + "title": "Embedder tokens + error rate", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, - "x": 0, - "y": 65 + "x": 12, + "y": 52 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byRegexp", - "options": ".*parked.*" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byRegexp", - "options": ".*retry.*" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] + "showPoints": "never" } - ] + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (stage, result) (rate(memini_repair_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{stage}}: {{result}}" - } - ] - }, - { - "id": 32, - "type": "timeseries", - "title": "Repair duration p95 by stage", - "description": "95th-percentile latency of a single deferred repair by stage; climbing durations shrink repair throughput and grow the backlog.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 65 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "s", - "min": 0 + "expr": "sum by (backend) (rate(memini_embed_tokens_total[5m]))", + "legendFormat": "tokens/s {{backend}}", + "refId": "A" }, - "overrides": [] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(memini_repair_duration_seconds_bucket[$__rate_interval])))", - "refId": "A", - "legendFormat": "{{stage}}" + "expr": "sum by (backend) (rate(memini_embed_errors_total[5m]))", + "legendFormat": "err/s {{backend}}", + "refId": "B" } ] }, { - "id": 33, - "type": "row", - "title": "Embedder & rerank", - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 72 - }, - "collapsed": false, - "panels": [] - }, - { - "id": 34, - "type": "timeseries", - "title": "Embedder p95 latency by backend", - "description": "95th-percentile embedder call latency per backend layer; the openai layer reflects the real provider, cached/diskcache should stay near zero.", + "id": 24, + "type": "stat", + "title": "Embedder in-flight", + "description": "Concurrent embed calls against the backend (post-cap, excludes cache hits). Spikes during ingest storms are normal; sustained values near the cap mean the embedder is the bottleneck.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 12, + "h": 4, + "w": 8, "x": 0, - "y": 73 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "s", - "min": 0 - }, - "overrides": [] + "y": 60 }, "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[$__rate_interval])))", - "refId": "A", - "legendFormat": "{{backend}}" - } - ] - }, - { - "id": 35, - "type": "timeseries", - "title": "Rerank outcomes by backend", - "description": "Recall rerank attempts per second by backend and result; orange fallback means the reranker failed or timed out and results shipped with pre-rerank ordering.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 73 + "textMode": "auto" }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byRegexp", - "options": ".*fallback.*" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" + "unit": "short" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend, result) (rate(memini_rerank_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{backend}}: {{result}}" + "expr": "memini_embed_in_flight", + "legendFormat": "embed in-flight", + "refId": "A" } ] }, { - "id": 36, - "type": "timeseries", - "title": "Embedder tokens/s by backend", - "description": "Embedding tokens consumed per second as reported by the API (openai backend only); this is the embed cost driver.", + "id": 25, + "type": "stat", + "title": "Rerank in-flight", + "description": "Concurrent rerank calls. A nonzero value indicates rerank is configured; if this stays pinned at the cap, the reranker is the bottleneck.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 4, "w": 8, - "x": 0, - "y": 80 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] + "x": 8, + "y": 60 }, "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum by (backend) (rate(memini_embed_tokens_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{backend}}" - } - ] - }, - { - "id": 37, - "type": "timeseries", - "title": "Embedder errors by backend", - "description": "Embedder failures per second by backend; any sustained nonzero rate here explains degraded writes and keyword-only recalls elsewhere on this board.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 80 + "textMode": "auto" }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" + "unit": "short" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend) (rate(memini_embed_errors_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{backend}}" + "expr": "memini_rerank_in_flight", + "legendFormat": "rerank in-flight", + "refId": "A" } ] }, { - "id": 38, + "id": 26, "type": "timeseries", - "title": "Embedder batch size (p50 / p95)", - "description": "Texts per Embed call at the median and 95th percentile; batches stuck at 1 mean the batcher is not coalescing and each write pays a full round trip.", + "title": "Embedder batch size (p50 vs p95)", + "description": "Items per embed call, by backend. batched shows a single high-p95 bar; cached should be ~1.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 4, "w": 8, "x": 16, - "y": 80 + "y": 60 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" + "lineInterpolation": "smooth", + "fillOpacity": 5, + "showPoints": "never" + } } }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "histogram_quantile(0.50, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))", - "refId": "A", - "legendFormat": "p50 {{backend}}" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))", - "refId": "B", - "legendFormat": "p95 {{backend}}" - } - ] - }, - { - "id": 39, - "type": "timeseries", - "title": "In-flight embed & rerank calls", - "description": "Concurrent calls currently held against the embedding and rerank backends; pinned at the concurrency cap means requests are queueing behind the limiter.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 6, - "w": 24, - "x": 0, - "y": 87 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [] - }, "options": { "legend": { - "calcs": [], "displayMode": "list", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_embed_in_flight", - "refId": "A", - "legendFormat": "embed" + "expr": "histogram_quantile(0.5, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))", + "legendFormat": "p50 {{backend}}", + "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_rerank_in_flight", - "refId": "B", - "legendFormat": "rerank" + "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))", + "legendFormat": "p95 {{backend}}", + "refId": "B" } ] }, { - "id": 40, + "id": 27, "type": "row", - "title": "LLM \u2014 consolidation & answer (empty without an LLM configured)", + "title": "HTTP surface", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 93 + "y": 64 }, "collapsed": false, "panels": [] }, { - "id": 41, + "id": 28, "type": "timeseries", - "title": "Consolidation outcomes", - "description": "LLM consolidation decisions per second (gated, new, update, supersede, noop); red errors or orange drops mean writes are skipping dedup and duplicates will accumulate.", + "title": "HTTP requests by route (status class)", + "description": "Per-route request rate broken down by 2xx / 4xx / 5xx. 5xx spikes point at a broken handler; 4xx spikes usually mean client misuse or a misconfigured agent.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 12, + "h": 8, + "w": 16, "x": 0, - "y": 94 + "y": 65 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", + "lineInterpolation": "smooth", + "fillOpacity": 15, "showPoints": "never", - "pointSize": 5, - "spanNulls": false, "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" + "mode": "normal" } }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "dropped" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] + "unit": "ops" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "displayMode": "table", + "placement": "right", + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_consolidate_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{result}}" + "expr": "sum by (route, status) (label_replace(rate(memini_http_requests_total[5m]), \"status_class\", \"$1\", \"status\", \"(...).\"))", + "legendFormat": "{{route}} {{status_class}}xx", + "refId": "A" } ] }, { - "id": 42, + "id": 29, "type": "timeseries", - "title": "Consolidation queue depth", - "description": "Async consolidation queue depth over time; a sawtooth is normal, monotonic growth means the LLM cannot keep pace with write volume.", + "title": "HTTP p95 latency by route", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 6, - "x": 12, - "y": 94 + "h": 8, + "w": 8, + "x": 16, + "y": 65 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [] + "unit": "s" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": false + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_consolidate_queue_depth", - "refId": "A", - "legendFormat": "queue depth" + "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[5m])))", + "legendFormat": "p95 {{route}}", + "refId": "A" } ] }, { - "id": 43, - "type": "stat", - "title": "Dedup effectiveness (15m)", - "description": "Share of consolidated writes that matched an existing memory (update/supersede/noop) rather than creating a new one; a sudden drop to zero suggests the matcher is broken.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 6, - "x": 18, - "y": 94 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "textMode": "auto" - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result=~\"new|update|supersede|noop\"}[15m])), 1e-9)", - "refId": "A", - "legendFormat": "matched" - } - ] - }, - { - "id": 44, + "id": 17, "type": "timeseries", - "title": "Answer outcomes", - "description": "Answer API (LLM-synthesized answers over recall) outcomes per second; red errors mean answer requests are burning LLM budget without returning.", + "title": "End-to-end op latency p95 (by op)", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 12, + "h": 8, + "w": 16, "x": 0, - "y": 101 + "y": 73 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "lineInterpolation": "smooth", + "fillOpacity": 5, + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "error" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - } - ] + "unit": "s" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true + "displayMode": "table", + "placement": "right", + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_answer_results_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{result}}" + "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[5m])))", + "legendFormat": "{{op}}", + "refId": "A" } ] }, { - "id": 45, + "id": 34, "type": "timeseries", - "title": "LLM op latency p95", - "description": "95th-percentile end-to-end latency of the LLM-backed service ops (answer, promote, dedup, semantic distillation wait); these dominate tail latency when the LLM slows down.", + "title": "HTTP response size by route (p50/p90)", + "description": "Body bytes shipped per response. The recall/briefing routes are what hooks inject from — watch these when tuning context-injection volume.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 101 + "h": 8, + "w": 8, + "x": 16, + "y": 73 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] + "showPoints": "never" }, - "unit": "s", - "min": 0 - }, - "overrides": [] + "unit": "bytes" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket{op=~\"answer|promote|dedup|distill_sem_wait\"}[$__rate_interval])))", - "refId": "A", - "legendFormat": "{{op}}" + "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))", + "legendFormat": "p90 {{route}}", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))", + "legendFormat": "p50 {{route}}", + "refId": "B" } ] }, { - "id": 46, + "id": 30, "type": "row", - "title": "HTTP surface", + "title": "Write-path signals", + "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 108 + "y": 82 }, - "collapsed": false, "panels": [] }, { - "id": 47, + "id": 31, "type": "timeseries", - "title": "HTTP request rate by route & status class", - "description": "Requests per second per route, grouped into status classes; red 5xx traffic is server failure, orange 4xx spikes usually mean a misconfigured client or expired key.", + "title": "Corroboration routing", + "description": "A fresh short-term write restating a durable fact: corroborated (confidence grown), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 109 + "y": 83 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "reqps", - "min": 0 - }, - "overrides": [ - { - "matcher": { - "id": "byRegexp", - "options": ".* 5xx" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "red" - } - } - ] - }, - { - "matcher": { - "id": "byRegexp", - "options": ".* 4xx" - }, - "properties": [ - { - "id": "color", - "value": { - "mode": "fixed", - "fixedColor": "orange" - } - } - ] - } - ] - }, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" + "showPoints": "never" }, - "expr": "sum by (route, status_class) (label_replace(rate(memini_http_requests_total[$__rate_interval]), \"status_class\", \"$1\", \"status\", \"(.)..\"))", - "refId": "A", - "legendFormat": "{{route}} {{status_class}}xx" + "unit": "short" } - ] - }, - { - "id": 48, - "type": "timeseries", - "title": "HTTP p95 latency by route", - "description": "95th-percentile HTTP latency per route as clients experience it; compare with the service-op view below to tell transport overhead from service work.", - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 109 - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, - "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "s", - "min": 0 - }, - "overrides": [] }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[$__rate_interval])))", - "refId": "A", - "legendFormat": "{{route}}" + "expr": "sum by (result) (rate(memini_corroborate_results_total[5m]))", + "legendFormat": "{{result}}", + "refId": "A" } ] }, { - "id": 49, + "id": 33, "type": "timeseries", - "title": "Service op latency p95 by op", - "description": "95th-percentile end-to-end latency of every public service operation (remember, recall, briefing, forget, ...); this is the service-internal view behind the HTTP numbers.", + "title": "Contradiction routing", + "description": "A fresh durable write that changes a fact's value or flips its polarity: contradicted (stale fact invalidated via valid_to + confidence shrink), no_signal (durable neighbour found but not a real change), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error. Default-on; a contradicted rate far above expectation is the misfire smoke alarm (set MEMINI_CONTRADICT_DOWNRANK=false to disable).", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 116 + "y": 91 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "s", - "min": 0 - }, - "overrides": [] + "unit": "short" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[$__rate_interval])))", - "refId": "A", - "legendFormat": "{{op}}" + "expr": "sum by (result) (rate(memini_contradict_results_total[5m]))", + "legendFormat": "{{result}}", + "refId": "A" } ] }, { - "id": 50, + "id": 32, "type": "timeseries", - "title": "HTTP response size (search & briefing, p50 / p90)", - "description": "Response body sizes for the two injection-critical routes; growth here directly inflates the tokens injected into agent context.", + "title": "Tier classification & promotion", + "description": "Omitted-tier writes the marker classifier routed to a durable tier, and facts produced by the promotion pass (LLM or marker extractor).", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 116 + "y": 83 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } + "showPoints": "never" }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "bytes", - "min": 0 - }, - "overrides": [] + "unit": "short" + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))", - "refId": "A", - "legendFormat": "p50 {{route}}" + "expr": "sum by (tier) (rate(memini_tier_classified_total[5m]))", + "legendFormat": "classified -> {{tier}}", + "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))", - "refId": "B", - "legendFormat": "p90 {{route}}" + "expr": "sum(rate(memini_promote_facts_total[5m]))", + "legendFormat": "promoted facts", + "refId": "B" } ] }, { - "id": 51, + "id": 40, "type": "row", - "title": "Client injection telemetry", + "title": "Injection efficiency (client telemetry)", + "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 123 + "y": 99 }, - "collapsed": false, "panels": [] }, { - "id": 52, + "id": 41, "type": "timeseries", - "title": "Injection outcomes by result", - "description": "Client-reported memory injections per second by outcome; suppressed_* series show why memories were withheld (already seen, cooldown, budget, unchanged, score) \u2014 a spike in suppressions with flat injections means the hooks are firing but delivering nothing.", + "title": "Injection outcomes (by result)", + "description": "What hooks reported actually reaching model context vs held back by their local gates (seen/cooldown/budget/unchanged/score). Absent entirely means no integration is reporting.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 12, + "h": 8, + "w": 8, "x": 0, - "y": 124 + "y": 100 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", "showPoints": "never", - "pointSize": 5, - "spanNulls": false, "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" + "mode": "normal" } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] + } + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_injected_memories_total[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{result}}" + "expr": "sum by (result) (rate(memini_injected_memories_total[5m]))", + "legendFormat": "{{result}}", + "refId": "A" } ] }, { - "id": 53, + "id": 42, "type": "timeseries", - "title": "Injected memories by surface", - "description": "Memories actually injected into model context per second, by hook surface (briefing, prompt, pretool); a silent surface means that hook is broken client-side.", + "title": "Injected memories (by surface)", + "description": "Memories that reached context, split by which hook surface put them there.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 6, - "x": 12, - "y": 124 + "h": 8, + "w": 8, + "x": 8, + "y": 100 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops", - "min": 0 - }, - "overrides": [] + "showPoints": "never" + } + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[$__rate_interval]))", - "refId": "A", - "legendFormat": "{{surface}}" + "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[5m]))", + "legendFormat": "{{surface}}", + "refId": "A" } ] }, { - "id": 54, + "id": 43, "type": "timeseries", - "title": "Injected tokens per hour by surface", - "description": "Client-side estimate of context tokens injected per hour by surface; this is the direct context-budget cost of memini to the agent.", + "title": "Injected tokens per hour (by surface)", + "description": "Client-estimated tokens the hooks added to context. The number injection tuning should push down.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 7, - "w": 6, - "x": 18, - "y": 124 + "h": 8, + "w": 8, + "x": 16, + "y": 100 }, "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { "drawStyle": "line", - "lineInterpolation": "linear", - "lineWidth": 1, + "lineInterpolation": "smooth", "fillOpacity": 10, - "gradientMode": "none", - "showPoints": "never", - "pointSize": 5, - "spanNulls": false, - "stacking": { - "mode": "none", - "group": "A" - }, - "axisPlacement": "auto", - "axisLabel": "", - "axisColorMode": "text", - "scaleDistribution": { - "type": "linear" - }, - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none", - "min": 0 - }, - "overrides": [] + "showPoints": "never" + } + } }, "options": { "legend": { - "calcs": [], - "displayMode": "list", + "displayMode": "table", "placement": "bottom", - "showLegend": true + "showLegend": true, + "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" [five lines unchanged)] "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "sum by (surface) (increase(memini_injected_tokens_total[1h]))", - "refId": "A", - "legendFormat": "{{surface}}" + "legendFormat": "{{surface}}", + "refId": "A" } ] } ], [30 lines unchanged)] "timepicker": {}, "timezone": "", "title": "memini", "uid": "memini", - "version": 3, + "version": 2, "weekStart": "" } spec.template.spec.containers.main.image (apps/v1/StatefulSet/local-ai/memini) ± value change - registry.erwanleboucher.dev/eleboucher/memini@sha256:1e060967d6fd11dfb79eeaa67d8745e0dde5d946ca1b7dd7469b4fad2fac0c1d + registry.erwanleboucher.dev/eleboucher/memini@sha256:2d388222bca3d65673a4072e99e0970fe0d5cdab0030ce0effdfdd78307ae7e2 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/498)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 579409ea63
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to db4870e90d
Some checks failed
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline failed
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-08 19:12:34 +00:00 Compare
ci-bot commented 2026-10-08 19:15:29 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/505)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from db4870e90d
Some checks failed
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline failed
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to f415f9d30b
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-10 01:12:47 +00:00 Compare
ci-bot commented 2026-10-10 01:15:34 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.8.9 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/511)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 15290e546a to 09f047559e
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-15 10:12:57 +00:00 Compare
ci-bot commented 2026-10-15 10:16:10 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/590)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 09f047559e
All checks were successful
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to a8a45787f8
Some checks failed
ci/crow/push/tanka-test Pipeline failed
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-15 18:14:05 +00:00 Compare
ci-bot commented 2026-10-15 18:16:40 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/598)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from a8a45787f8
Some checks failed
ci/crow/push/tanka-test Pipeline failed
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to 1ef24de537
All checks were successful
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-16 17:13:30 +00:00 Compare
ci-bot commented 2026-10-16 17:16:09 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

spec.template.spec.containers.worker.image  (apps/v1/DaemonSet/kube-privileged/node-feature-discovery-worker)
  ± value change
    - registry.k8s.io/nfd/node-feature-discovery:v0.20.0
    + registry.k8s.io/nfd/node-feature-discovery:v0.19.0

spec.template.spec.containers.master.image  (apps/v1/Deployment/kube-privileged/node-feature-discovery-master)
  ± value change
    - registry.k8s.io/nfd/node-feature-discovery:v0.20.0
    + registry.k8s.io/nfd/node-feature-discovery:v0.19.0

spec.template.spec.containers.gc.args  (apps/v1/Deployment/kube-privileged/node-feature-discovery-gc)
  - one list entry removed:
    - "-port=8080"

spec.template.spec.containers.gc.image  (apps/v1/Deployment/kube-privileged/node-feature-discovery-gc)
  ± value change
    - registry.k8s.io/nfd/node-feature-discovery:v0.20.0
    + registry.k8s.io/nfd/node-feature-discovery:v0.19.0

spec.template.spec.containers.nfd-master.image  (batch/v1/Job/kube-privileged/node-feature-discovery-prune)
  ± value change
    - registry.k8s.io/nfd/node-feature-discovery:v0.20.0
    + registry.k8s.io/nfd/node-feature-discovery:v0.19.0

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 spec.template.spec.containers.worker.image (apps/v1/DaemonSet/kube-privileged/node-feature-discovery-worker) ± value change - registry.k8s.io/nfd/node-feature-discovery:v0.20.0 + registry.k8s.io/nfd/node-feature-discovery:v0.19.0 spec.template.spec.containers.master.image (apps/v1/Deployment/kube-privileged/node-feature-discovery-master) ± value change - registry.k8s.io/nfd/node-feature-discovery:v0.20.0 + registry.k8s.io/nfd/node-feature-discovery:v0.19.0 spec.template.spec.containers.gc.args (apps/v1/Deployment/kube-privileged/node-feature-discovery-gc) - one list entry removed: - "-port=8080" spec.template.spec.containers.gc.image (apps/v1/Deployment/kube-privileged/node-feature-discovery-gc) ± value change - registry.k8s.io/nfd/node-feature-discovery:v0.20.0 + registry.k8s.io/nfd/node-feature-discovery:v0.19.0 spec.template.spec.containers.nfd-master.image (batch/v1/Job/kube-privileged/node-feature-discovery-prune) ± value change - registry.k8s.io/nfd/node-feature-discovery:v0.20.0 + registry.k8s.io/nfd/node-feature-discovery:v0.19.0 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/614)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 1ef24de537
All checks were successful
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to 4ba6d50092
All checks were successful
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-16 20:13:14 +00:00 Compare
ci-bot commented 2026-10-16 20:16:13 +00:00
Member
Copy link

flate diff

No differences detected.

Pipeline

## flate diff No differences detected. [Pipeline](https://ci.aether.earth/repos/4/pipeline/616)
renovate-bot force-pushed renovate/kube-prometheus-stack-92.x from 4ba6d50092
All checks were successful
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline was successful
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
to 463a28b567
Some checks failed
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline failed
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
2026-10-20 13:13:22 +00:00 Compare
ci-bot commented 2026-10-20 13:16:27 +00:00
Member
Copy link

flate diff

rules  (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator)
  - three list entries removed:         + six list entries added:
    - resources:                          - resources:
      - alertmanagers                       - alertmanagers
      - alertmanagers/finalizers            - prometheusagents
      - alertmanagers/status                - prometheuses
      - alertmanagerconfigs                 - thanosrulers
      - prometheuses                        apiGroups:
      - prometheuses/finalizers             - monitoring.coreos.com
      - prometheuses/status                 verbs:
      - prometheusagents                    - patch
      - prometheusagents/finalizers       - resources:
      - prometheusagents/status             - alertmanagers/status
      - thanosrulers                        - podmonitors/status
      - thanosrulers/finalizers             - probes/status
      - thanosrulers/status                 - prometheuses/status
      - scrapeconfigs                       - prometheusagents/status
      - scrapeconfigs/status                - prometheusrules/status
      - servicemonitors                     - scrapeconfigs/status
      - servicemonitors/status              - servicemonitors/status
      - podmonitors                         - thanosrulers/status
      - podmonitors/status                  apiGroups:
      - probes                              - monitoring.coreos.com
      - probes/status                       verbs:
      - prometheusrules                     - create
      - prometheusrules/status              - update
      apiGroups:                            - patch
      - monitoring.coreos.com               - delete
      verbs:                              - resources:
      - "*"                                 - alertmanagers/finalizers
    - resources:                            - prometheusagents/finalizers
      - statefulsets                        - prometheuses/finalizers
      apiGroups:                            - thanosrulers/finalizers
      - apps                                apiGroups:
      verbs:                                - monitoring.coreos.com
      - "*"                                 verbs:
    - resources:                            - update
      - configmaps                        - resources:
      - secrets                             - alertmanagers
      apiGroups:                            - alertmanagerconfigs
      -                                     - podmonitors
      verbs:                                - probes
      - "*"                                 - prometheusagents
                                            - prometheuses
                                            - prometheusrules
                                            - servicemonitors
                                            - scrapeconfigs
                                            - thanosrulers
                                            apiGroups:
                                            - monitoring.coreos.com
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                          - resources:
                                            - statefulsets
                                            apiGroups:
                                            - apps
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - patch
                                            - delete
                                          - resources:
                                            - configmaps
                                            - secrets
                                            apiGroups:
                                            -
                                            verbs:
                                            - get
                                            - list
                                            - watch
                                            - create
                                            - update
                                            - delete

metadata  (v1/Service/kube-privileged/node-exporter)
  - one map entry removed:
    annotations:
      prometheus.io/scrape: "true"

spec.template.spec.containers.kube-state-metrics.image  (apps/v1/Deployment/monitoring/kube-state-metrics)
  ± value change
    - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
    + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

spec.template.spec  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.kube-prometheus-stack.args  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  - three list entries removed:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1"
    
  
  + three list entries added:
    - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1"
    - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4"
    - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1"
    
  

spec.template.spec.containers.kube-prometheus-stack.image  (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator)
  ± value change
    - quay.io/prometheus-operator/prometheus-operator:v0.92.1
    + quay.io/prometheus-operator/prometheus-operator:v0.94.1

spec  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  - one map entry removed:     + one map entry added:
    hostNetwork: false           nodeSelector:
                                   kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/alertmanager:v0.33.1
    + quay.io/prometheus/alertmanager:v0.34.1

spec.version  (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack)
  ± value change
    - v0.33.1
    + v0.34.1

spec  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.image  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - quay.io/prometheus/prometheus:v3.13.1-distroless
    + quay.io/prometheus/prometheus:v3.15.0-distroless

spec.version  (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack)
  ± value change
    - v3.13.1-distroless
    + v3.15.0-distroless

spec.groups.kube-apiserver-burnrate.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules)
  - ten list entries removed:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
      labels:
        verb: read
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m]))
      labels:
        verb: read
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
      labels:
        verb: write
      record: "apiserver_request:burnrate3d"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    
  
  + eleven list entries added:
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            (
              (
                sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m]))
                or
                vector(0)
              )
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m]))
              +
              sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m]))
            )
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m]))
        )
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m]))
        )
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_bad_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))"
      labels:
        verb: read
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))"
      labels:
        verb: write
      record: "cluster_verb:apiserver_request_sli_events:rate5m"
    - expr: |
        cluster_verb:apiserver_request_sli_bad_events:rate5m
        /
        cluster_verb:apiserver_request_sli_events:rate5m
      record: "apiserver_request:burnrate5m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate1h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate2h"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m]))
      labels:
        verb: write
      record: "apiserver_request:burnrate30m"
    - expr: |
        (
          (
            # too slow
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h]))
            -
            sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h]))
          )
          +
          # errors
          sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h]))
        )
        /
        sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h]))
      labels:
        verb: write
      record: "apiserver_request:burnrate6h"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d])
      record: "apiserver_request:burnrate1d"
    - expr: |
        avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d])
        /
        avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d])
      record: "apiserver_request:burnrate3d"
    
  

spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, one deletion)
      time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"}
        and
    - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200
    +   unless on (namespace, job_name, cluster)
    + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1)
  
  

spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr  (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps)
  ± value change in multiline text (one insert, no deletions)
      (
        kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
        -
        kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
      )
    + > 0
    + and
    + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"}
      > 0
  
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  + one map entry added:
    authorization:
      type: Bearer
      credentials:
        name: kube-prometheus-stack-prometheus-token
        key: token
  

spec.endpoints.0.tlsConfig  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler)
  - one map entry removed:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  + one map entry added:
    ca:
      configMap:
        name: kube-root-ca.crt
        key: ca.crt
  

spec.endpoints  (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet)
  - three list entries removed:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    
  
  + three list entries added:
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      metricRelabelings:
      - action: drop
        regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)?
        sourceLabels:
        - __name__
        - le
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      interval: 10s
      metricRelabelings:
      - action: drop
        regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_(mapped_file|swap)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_(file_descriptors|tasks_state|threads_max)
        sourceLabels:
        - __name__
      - action: drop
        regex: container_memory_failures_total;hierarchy
        sourceLabels:
        - __name__
        - scope
      - action: drop
        regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
        sourceLabels:
        - __name__
        - interface
      - action: drop
        regex: "container_spec.*"
        sourceLabels:
        - __name__
      - action: drop
        regex: .+;
        sourceLabels:
        - id
        - pod
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
      trackTimestampsStaleness: true
    - authorization:
        type: Bearer
        credentials:
          name: kube-prometheus-stack-prometheus-token
          key: token
      honorLabels: true
      honorTimestamps: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
      - action: replace
        sourceLabels:
        - __metrics_path__
        targetLabel: metrics_path
      scheme: https
      tlsConfig:
        ca:
          configMap:
            name: kube-root-ca.crt
            key: ca.crt
        insecureSkipVerify: true
    
  

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.create.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

spec.template.spec  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  + one map entry added:
    nodeSelector:
      kubernetes.io/os: linux

spec.template.spec.containers.patch.image  (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch)
  ± value change
    - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4
    + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0

Pipeline

## flate diff ``` rules (rbac.authorization.k8s.io/v1/ClusterRole/kube-prometheus-stack-operator) - three list entries removed: + six list entries added: - resources: - resources: - alertmanagers - alertmanagers - alertmanagers/finalizers - prometheusagents - alertmanagers/status - prometheuses - alertmanagerconfigs - thanosrulers - prometheuses apiGroups: - prometheuses/finalizers - monitoring.coreos.com - prometheuses/status verbs: - prometheusagents - patch - prometheusagents/finalizers - resources: - prometheusagents/status - alertmanagers/status - thanosrulers - podmonitors/status - thanosrulers/finalizers - probes/status - thanosrulers/status - prometheuses/status - scrapeconfigs - prometheusagents/status - scrapeconfigs/status - prometheusrules/status - servicemonitors - scrapeconfigs/status - servicemonitors/status - servicemonitors/status - podmonitors - thanosrulers/status - podmonitors/status apiGroups: - probes - monitoring.coreos.com - probes/status verbs: - prometheusrules - create - prometheusrules/status - update apiGroups: - patch - monitoring.coreos.com - delete verbs: - resources: - "*" - alertmanagers/finalizers - resources: - prometheusagents/finalizers - statefulsets - prometheuses/finalizers apiGroups: - thanosrulers/finalizers - apps apiGroups: verbs: - monitoring.coreos.com - "*" verbs: - resources: - update - configmaps - resources: - secrets - alertmanagers apiGroups: - alertmanagerconfigs - - podmonitors verbs: - probes - "*" - prometheusagents - prometheuses - prometheusrules - servicemonitors - scrapeconfigs - thanosrulers apiGroups: - monitoring.coreos.com verbs: - get - list - watch - resources: - statefulsets apiGroups: - apps verbs: - get - list - watch - create - update - patch - delete - resources: - configmaps - secrets apiGroups: - verbs: - get - list - watch - create - update - delete metadata (v1/Service/kube-privileged/node-exporter) - one map entry removed: annotations: prometheus.io/scrape: "true" spec.template.spec.containers.kube-state-metrics.image (apps/v1/Deployment/monitoring/kube-state-metrics) ± value change - registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1 + registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 spec.template.spec (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.kube-prometheus-stack.args (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) - three list entries removed: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.92.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.2" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1" + three list entries added: - "--prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.94.1" - "--thanos-default-base-image=quay.io/thanos/thanos:v0.42.4" - "--secret-field-selector=type!=kubernetes.io/dockercfg,type!=helm.sh/release.v1" spec.template.spec.containers.kube-prometheus-stack.image (apps/v1/Deployment/monitoring/kube-prometheus-stack-operator) ± value change - quay.io/prometheus-operator/prometheus-operator:v0.92.1 + quay.io/prometheus-operator/prometheus-operator:v0.94.1 spec (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) - one map entry removed: + one map entry added: hostNetwork: false nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/alertmanager:v0.33.1 + quay.io/prometheus/alertmanager:v0.34.1 spec.version (monitoring.coreos.com/v1/Alertmanager/monitoring/kube-prometheus-stack) ± value change - v0.33.1 + v0.34.1 spec (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.image (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - quay.io/prometheus/prometheus:v3.13.1-distroless + quay.io/prometheus/prometheus:v3.15.0-distroless spec.version (monitoring.coreos.com/v1/Prometheus/monitoring/kube-prometheus-stack) ± value change - v3.13.1-distroless + v3.15.0-distroless spec.groups.kube-apiserver-burnrate.rules.rules (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kube-apiserver-burnrate.rules) - ten list entries removed: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[1d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[1d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[1d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[1d])) labels: verb: read record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[3d])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[3d])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[3d])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) labels: verb: read record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[5m])) labels: verb: read record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1d])) labels: verb: write record: "apiserver_request:burnrate1d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[3d])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[3d])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[3d])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) labels: verb: write record: "apiserver_request:burnrate3d" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[5m])) labels: verb: write record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" + eleven list entries added: - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - ( ( sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope=~"resource|",le=~"1(\\.0)?"}[5m])) or vector(0) ) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="namespace",le=~"5(\\.0)?"}[5m])) + sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"LIST|GET",subresource!~"proxy|attach|log|exec|portforward",scope="cluster",le=~"30(\\.0)?"}[5m])) ) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}[5m])) ) labels: verb: read record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[5m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[5m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[5m])) ) labels: verb: write record: "cluster_verb:apiserver_request_sli_bad_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))" labels: verb: read record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: "sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))" labels: verb: write record: "cluster_verb:apiserver_request_sli_events:rate5m" - expr: | cluster_verb:apiserver_request_sli_bad_events:rate5m / cluster_verb:apiserver_request_sli_events:rate5m record: "apiserver_request:burnrate5m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[1h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[1h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[1h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[1h])) labels: verb: write record: "apiserver_request:burnrate1h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[2h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[2h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[2h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[2h])) labels: verb: write record: "apiserver_request:burnrate2h" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[30m])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[30m])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[30m])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[30m])) labels: verb: write record: "apiserver_request:burnrate30m" - expr: | ( ( # too slow sum by (cluster) (rate(apiserver_request_sli_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward"}[6h])) - sum by (cluster) (rate(apiserver_request_sli_duration_seconds_bucket{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",subresource!~"proxy|attach|log|exec|portforward",le=~"1(\\.0)?"}[6h])) ) + # errors sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}[6h])) ) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[6h])) labels: verb: write record: "apiserver_request:burnrate6h" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[1d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[1d]) record: "apiserver_request:burnrate1d" - expr: | avg_over_time(cluster_verb:apiserver_request_sli_bad_events:rate5m[3d]) / avg_over_time(cluster_verb:apiserver_request_sli_events:rate5m[3d]) record: "apiserver_request:burnrate3d" spec.groups.kubernetes-apps.rules.KubeJobNotCompleted.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, one deletion) time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job="kube-state-metrics", namespace=~".*"} and - kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + kube_job_status_active{job="kube-state-metrics", namespace=~".*"} > 0) > 43200 + unless on (namespace, job_name, cluster) + max by (namespace, job_name, cluster) (kube_job_labels{job="kube-state-metrics", namespace=~".*",label_excluded_from_alerts="true"} == 1) spec.groups.kubernetes-apps.rules.KubePdbNotEnoughHealthyPods.expr (monitoring.coreos.com/v1/PrometheusRule/monitoring/kube-prometheus-stack-kubernetes-apps) ± value change in multiline text (one insert, no deletions) ( kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"} - kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"} ) + > 0 + and + kube_poddisruptionbudget_status_expected_pods{job="kube-state-metrics", namespace=~".*"} > 0 spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-coredns) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-apiserver) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-controller-manager) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-etcd) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0 (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + one map entry added: authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token spec.endpoints.0.tlsConfig (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kube-scheduler) - one map entry removed: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + one map entry added: ca: configMap: name: kube-root-ca.crt key: ca.crt spec.endpoints (monitoring.coreos.com/v1/ServiceMonitor/monitoring/kube-prometheus-stack-kubelet) - three list entries removed: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true + three list entries added: - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true metricRelabelings: - action: drop regex: (csi_operations|storage_operation_duration)_seconds_bucket;(0.25|2.5|15|25|120|600)(\.0)? sourceLabels: - __name__ - le port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true interval: 10s metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_memory_failures_total;hierarchy sourceLabels: - __name__ - scope - action: drop regex: "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*" sourceLabels: - __name__ - interface - action: drop regex: "container_spec.*" sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true trackTimestampsStaleness: true - authorization: type: Bearer credentials: name: kube-prometheus-stack-prometheus-token key: token honorLabels: true honorTimestamps: true path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: ca: configMap: name: kube-root-ca.crt key: ca.crt insecureSkipVerify: true spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.create.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-create) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 spec.template.spec (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) + one map entry added: nodeSelector: kubernetes.io/os: linux spec.template.spec.containers.patch.image (batch/v1/Job/monitoring/kube-prometheus-stack-admission-patch) ± value change - ghcr.io/jkroepke/kube-webhook-certgen:1.8.4 + ghcr.io/jkroepke/kube-webhook-certgen:1.11.0 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/658)
Some checks failed
ci/crow/push/dns-test Pipeline was successful
Details
ci/crow/pr/dns-test Pipeline was successful
Details
ci/crow/pr/tanka-test Pipeline failed
Details
ci/crow/push/tanka-test Pipeline was successful
Details
ci/crow/pr/tanka-diff Pipeline was successful
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin renovate/kube-prometheus-stack-92.x:renovate/kube-prometheus-stack-92.x
git switch renovate/kube-prometheus-stack-92.x

Merge

Merge the changes and update on Forgejo.
git switch main
git merge --no-ff renovate/kube-prometheus-stack-92.x
git switch renovate/kube-prometheus-stack-92.x
git rebase main
git switch main
git merge --ff-only renovate/kube-prometheus-stack-92.x
git switch renovate/kube-prometheus-stack-92.x
git rebase main
git switch main
git merge --no-ff renovate/kube-prometheus-stack-92.x
git switch main
git merge --squash renovate/kube-prometheus-stack-92.x
git switch main
git merge --ff-only renovate/kube-prometheus-stack-92.x
git switch main
git merge renovate/kube-prometheus-stack-92.x
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ops/homelab!154
No description provided.