chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.37

renovate-bot commented

2026-10-12 07:10:12 +00:00

Member

This PR contains the following updates:

Package	Update	Change
registry.erwanleboucher.dev/eleboucher/charts/memini	patch	`v0.7.10` → `v0.7.37`

Configuration

📅 Schedule: (UTC)

Branch creation
- At any time (no schedule defined)
Automerge
- At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate.

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [registry.erwanleboucher.dev/eleboucher/charts/memini](https://git.erwanleboucher.dev/eleboucher/memini) | patch | `v0.7.10` → `v0.7.37` | --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).

renovate-bot added 1 commit

2026-10-12 07:10:13 +00:00

chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.32

ci/crow/push/tanka-test Pipeline was successful

Details

ci/crow/pr/dns-test Pipeline was successful

Details

ci/crow/push/dns-test Pipeline was successful

Details

ci/crow/pr/tanka-test Pipeline was successful

Details

ci/crow/pr/tanka-diff Pipeline was successful

Details

2455f07486

ci-bot commented

2026-10-12 07:14:32 +00:00

Member

flate diff

data.memini.json  (v1/ConfigMap/local-ai/memini-memini)
  ± value change in multiline text (314 inserts, 271 deletions)
      
      {
        "annotations": {
          "list": []
        },
    -   "description": "memini \u2014 what the service is actually doing for your agents. Live memory counts, write/recall traffic by tier, consolidation outcomes, store health, embedder & rerank latency & saturation, HTTP surface.",
    +   "description": "memini operator dashboard \u2014 top-to-bottom narrative: live headline health (including the deferred-repair backlog), the write path, recall & retrieval quality, aging & maintenance jobs, embedder & rerank, LLM-backed consolidation/answer, the HTTP surface, and client injection telemetry.",
        "editable": true,
        "graphTooltip": 1,
        "id": null,
        "links": [],
      
      [14 lines unchanged)]
      
          {
            "id": 2,
            "type": "stat",
            "title": "Live memories",
    -       "description": "Live (non-superseded, non-expired) memories by tier.",
    +       "description": "Total live (non-superseded, non-expired) memories across all tiers; a sudden drop means a sweep or dedup pass removed more than expected.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    +         "h": 5,
    +         "w": 4,
              "x": 0,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none"
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
      
      [five lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_memories_active",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum(memini_memories_active)",
    +           "refId": "A",
    +           "legendFormat": "live"
              }
            ]
          },
          {
            "id": 3,
            "type": "stat",
    -       "title": "Writes (5m rate)",
    -       "description": "Successful Remember calls per second, by tier.",
    +       "title": "Writes /s (5m)",
    +       "description": "Successful Remember writes per second over the last 5m; zero during active agent sessions means writes are failing upstream.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    -         "x": 6,
    +         "h": 5,
    +         "w": 3,
    +         "x": 4,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "decimals": 2
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "unit": "ops"
    -         }
    -       },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum(rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    +           "refId": "A",
    +           "legendFormat": "writes/s"
              }
            ]
          },
          {
            "id": 4,
            "type": "stat",
    -       "title": "Recalls (5m rate)",
    -       "description": "Successful Recall calls per second.",
    +       "title": "Recalls /s (5m)",
    +       "description": "Successful Recall searches per second over the last 5m; the primary read-traffic pulse.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    -         "x": 12,
    +         "h": 5,
    +         "w": 3,
    +         "x": 7,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "decimals": 2
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    +           "refId": "A",
    +           "legendFormat": "recalls/s"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 5,
    +       "type": "stat",
    +       "title": "HTTP 5xx ratio (15m)",
    +       "description": "Share of HTTP requests answered 5xx over 15m; anything above zero deserves a look, above 5% is an outage.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 5,
    +         "w": 3,
    +         "x": 10,
    +         "y": 1
    +       },
            "fieldConfig": {
              "defaults": {
    -           "unit": "ops"
    -         }
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 0.01
    +               },
    +               {
    +                 "color": "red",
    +                 "value": 0.05
    +               }
    +             ]
    +           },
    +           "unit": "percentunit"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "recalls/s",
    -           "refId": "A"
    +           "expr": "sum(rate(memini_http_requests_total{status=~\"5..\"}[15m])) / clamp_min(sum(rate(memini_http_requests_total[15m])), 1e-9)",
    +           "refId": "A",
    +           "legendFormat": "5xx ratio"
              }
            ]
          },
          {
    -       "id": 18,
    +       "id": 6,
            "type": "stat",
    -       "title": "Consolidation queue",
    -       "description": "Backlog of memories awaiting async consolidation. Sustained > 0 means the worker can't keep up with Remember writes.",
    +       "title": "Repair backlog depth",
    +       "description": "Memories currently owing deferred repair (pending + enrich + failed); green is empty, any sustained backlog means writes are storing degraded faster than repairs drain.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    -         "x": 18,
    +         "h": 5,
    +         "w": 4,
    +         "x": 13,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 1
    +               }
    +             ]
    +           },
    +           "unit": "none"
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(memini_repair_depth)",
    +           "refId": "A",
    +           "legendFormat": "backlog"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 7,
    +       "type": "stat",
    +       "title": "Repair oldest age",
    +       "description": "Age of the oldest memory still awaiting repair; amber the moment anything waits, red past 1h means the repair loop is stuck or starved.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 5,
    +         "w": 4,
    +         "x": 17,
    +         "y": 1
    +       },
            "fieldConfig": {
              "defaults": {
    -           "unit": "short",
    +           "color": {
    +             "mode": "thresholds"
    +           },
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
    -                 "color": "yellow",
    -                 "value": 50
    +                 "color": "orange",
    +                 "value": 1
                    },
                    {
                      "color": "red",
    -                 "value": 500
    +                 "value": 3600
                    }
                  ]
    -           }
    +           },
    +           "unit": "s"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "max(memini_repair_oldest_age_seconds)",
    +           "refId": "A",
    +           "legendFormat": "oldest"
              }
    +       ]
    +     },
    +     {
    +       "id": 8,
    +       "type": "stat",
    +       "title": "Consolidation queue",
    +       "description": "Depth of the async consolidation queue; a queue that grows without draining delays dedup/supersede decisions.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 5,
    +         "w": 3,
    +         "x": 21,
    +         "y": 1
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 100
    +               }
    +             ]
    +           },
    +           "unit": "none"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
                "expr": "memini_consolidate_queue_depth",
    -           "legendFormat": "queued",
    -           "refId": "A"
    +           "refId": "A",
    +           "legendFormat": "queued"
              }
            ]
          },
          {
    -       "id": 5,
    +       "id": 9,
            "type": "row",
    -       "title": "Writes & Recalls",
    +       "title": "Write path",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 7
    +         "y": 6
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 6,
    +       "id": 10,
            "type": "timeseries",
    -       "title": "Remembers per second (by tier)",
    +       "title": "Remember rate by tier",
    +       "description": "Successful Remember writes per second split by tier, plus error/dropped outcomes; red or orange series above zero mean writes are being lost.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 8
    +         "y": 7
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
                },
    -           "unit": "ops"
    -         }
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "dropped"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{tier}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_remember_results_total{result=\"error\"}[$__rate_interval]))",
    +           "refId": "B",
    +           "legendFormat": "error"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_remember_results_total{result=\"dropped\"}[$__rate_interval]))",
    +           "refId": "C",
    +           "legendFormat": "dropped"
              }
            ]
          },
          {
    -       "id": 7,
    +       "id": 11,
            "type": "timeseries",
    -       "title": "Recalls per second (by hit-count bucket)",
    -       "description": "How many memories a typical recall returns. '0' means no matches \u2014 investigate if persistent.",
    +       "title": "Degraded (vectorless) writes by reason",
    +       "description": "Writes stored without a vector (keyword-searchable only, queued for deferred repair) because the embedder timed out or errored; any sustained rate means the embedder is unhealthy.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 8
    +         "y": 7
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_timeout"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "{{hits_bucket}}",
    -           "refId": "A"
    +           "expr": "sum by (reason) (rate(memini_remember_degraded_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{reason}}"
              }
            ]
          },
          {
    -       "id": 19,
    -       "type": "row",
    -       "title": "Retrieval quality",
    -       "gridPos": {
    -         "h": 1,
    -         "w": 24,
    -         "x": 0,
    -         "y": 16
    -       },
    -       "collapsed": false,
    -       "panels": []
    -     },
    -     {
    -       "id": 20,
    +       "id": 12,
            "type": "timeseries",
    -       "title": "Rerank outcomes by backend",
    -       "description": "Per-recall reranker traffic: ok = reranker reordered the fused candidates; fallback = reranker timed out / errored and the fused order was kept.",
    +       "title": "Store mutations",
    +       "description": "Raw store write traffic: inserts vs updates (by upsert op), hard deletes from Forget, and tombstones written by consolidation; updates dominating inserts means most writes reinforce existing memories.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 17
    +         "y": 14
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend, result) (rate(memini_rerank_results_total[5m]))",
    -           "legendFormat": "{{backend}}/{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (op) (rate(memini_store_upserts_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "upsert: {{op}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "rate(memini_store_deletes_total[$__rate_interval])",
    +           "refId": "B",
    +           "legendFormat": "hard delete (forget)"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "rate(memini_store_soft_deletes_total[$__rate_interval])",
    +           "refId": "C",
    +           "legendFormat": "tombstone (supersede)"
              }
            ]
          },
          {
    -       "id": 21,
    +       "id": 13,
            "type": "timeseries",
    -       "title": "Recall degradation reasons",
    -       "description": "Recalls that fell back to keyword-only search because the embedder was slow or errored. embed_timeout = the recall embed budget elapsed; embed_error = the embed call failed.",
    +       "title": "Forget & supersede outcomes",
    +       "description": "Forget and Supersede API outcomes per second; red error series should stay at zero, orange not_found spikes usually mean a client is retrying stale ids.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 17
    +         "y": 14
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*error.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
                },
    -           "unit": "ops"
    -         }
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*not_found.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (reason) (rate(memini_recall_degraded_total[5m]))",
    -           "legendFormat": "{{reason}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_forget_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "forget: {{result}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (rate(memini_supersede_results_total[$__rate_interval]))",
    +           "refId": "B",
    +           "legendFormat": "supersede: {{result}}"
              }
            ]
          },
          {
    -       "id": 22,
    +       "id": 14,
            "type": "timeseries",
    -       "title": "Recall reinforcement (best-effort)",
    -       "description": "Background access-count bump on recalled memories. Errors are swallowed \u2014 a sustained nonzero error rate means the store write path is unhealthy but recall still served.",
    +       "title": "Content hygiene (sanitizer)",
    +       "description": "Ingestion sanitizer actions: cleaned content was rewritten in place, quarantined (orange) content was judged unsafe to store as-is and needs review if it becomes frequent.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 12,
    +         "h": 7,
    +         "w": 8,
              "x": 0,
    -         "y": 25
    +         "y": 21
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "ops"
    -         }
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "quarantined"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_reinforce_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (action) (rate(memini_write_sanitized_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{action}}"
              }
            ]
          },
          {
    -       "id": 23,
    -       "type": "stat",
    -       "title": "Recall success ratio",
    -       "description": "Fraction of recall calls that succeeded. A drop means the recall path is erroring (embed + keyword + rerank + RRF + everything).",
    +       "id": 15,
    +       "type": "timeseries",
    +       "title": "Corroboration routing",
    +       "description": "Corroboration attempts on fresh short-term writes (did this confirm something we already know); a rising red error rate means the corroboration path is failing writes silently.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 12,
    -         "x": 12,
    -         "y": 25
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 21
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    -         "textMode": "auto"
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (rate(memini_corroborate_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 16,
    +       "type": "timeseries",
    +       "title": "Contradiction routing",
    +       "description": "Contradiction attempts on fresh durable writes (did this contradict an existing fact); contradicted events drive supersession, red errors mean conflicts are going undetected.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 16,
    +         "y": 21
            },
            "fieldConfig": {
              "defaults": {
    -           "unit": "percentunit",
    -           "min": 0,
    -           "max": 1,
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
    -                 "color": "red",
    +                 "color": "green",
                      "value": null
    -               },
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
                    {
    -                 "color": "yellow",
    -                 "value": 0.95
    -               },
    -               {
    -                 "color": "green",
    -                 "value": 0.99
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
                    }
                  ]
                }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)",
    -           "legendFormat": "ok ratio",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_contradict_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 8,
    +       "id": 17,
            "type": "row",
    -       "title": "Consolidation (LLM dedup)",
    +       "title": "Recall & retrieval quality",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 33
    +         "y": 28
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 9,
    +       "id": 18,
            "type": "timeseries",
    -       "title": "Consolidation outcomes per minute",
    -       "description": "gated = similarity gate skipped the LLM; new = nothing to merge; update = merged into an existing memory; supersede = contradiction-tombstoned an old one; noop = nothing to do; error/dropped = failure modes.",
    +       "title": "Recall rate by hit-count bucket",
    +       "description": "Successful recalls per second split by how many hits they returned, plus errors; a shift toward the 0-hits bucket means retrieval quality is degrading.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 16,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 34
    +         "y": 29
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{hits_bucket}} hits"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_recall_results_total{result=\"error\"}[$__rate_interval]))",
    +           "refId": "B",
    +           "legendFormat": "error"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 19,
    +       "type": "timeseries",
    +       "title": "Recall degradation (keyword-only fallback)",
    +       "description": "Recalls that fell back to keyword-only search because embedding the query timed out or errored; these return without semantic ranking, so any sustained rate hurts result quality.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 29
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 20,
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
                },
    -           "unit": "ops"
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_timeout"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (reason) (rate(memini_recall_degraded_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{reason}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 20,
    +       "type": "timeseries",
    +       "title": "Recall candidates floored (by tier filter)",
    +       "description": "Candidates dropped from recall responses by the min_rank_score composite floor; a jump means the floor (or a scoring regression) is hiding results users would previously have seen.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 0,
    +         "y": 36
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["sum"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_consolidate_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (tier_filter) (rate(memini_recall_floored_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{tier_filter}}"
              }
            ]
          },
          {
    -       "id": 10,
    +       "id": 21,
    +       "type": "timeseries",
    +       "title": "Recall reinforcement",
    +       "description": "Best-effort reinforcement writes triggered by recall hits (strengthening recalled memories); red errors mean reinforcement is silently failing and ranking will slowly stale.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 36
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (rate(memini_reinforce_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 22,
            "type": "stat",
    -       "title": "Dedup effectiveness",
    -       "description": "Fraction of consolidations that actually used the LLM (update+supersede+new+noop) vs the similarity gate skipping it. Higher = busier corpus or noisier dedup signal.",
    +       "title": "Recall success ratio (15m)",
    +       "description": "Share of Recall calls that completed without error over 15m; below 99% something in the search path is failing.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 8,
              "x": 16,
    -         "y": 34
    +         "y": 36
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "red",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 0.9
    +               },
    +               {
    +                 "color": "green",
    +                 "value": 0.99
    +               }
    +             ]
    +           },
    +           "unit": "percentunit"
    +         },
    +         "overrides": []
            },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "unit": "percentunit",
    -           "min": 0,
    -           "max": 1
    -         }
    -       },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|new|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result!~\"error|dropped\"}[15m])), 1e-9)",
    -           "legendFormat": "LLM use ratio",
    -           "refId": "A"
    +           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)",
    +           "refId": "A",
    +           "legendFormat": "ok ratio"
              }
            ]
          },
          {
    -       "id": 11,
    +       "id": 23,
            "type": "row",
    -       "title": "Storage & decay",
    +       "title": "Aging & maintenance",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 42
    +         "y": 43
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 12,
    +       "id": 24,
            "type": "timeseries",
            "title": "Live memories by tier",
    +       "description": "Live memory count per tier over time; working/episodic should churn while semantic/procedural grow slowly \u2014 a cliff in any tier means a sweep or dedup removed too much.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 43
    +         "y": 44
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "stepAfter",
    -             "fillOpacity": 15,
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
                },
    -           "unit": "short"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_memories_active",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum by (tier) (memini_memories_active)",
    +           "refId": "A",
    +           "legendFormat": "{{tier}}"
              }
            ]
          },
          {
    -       "id": 13,
    +       "id": 25,
            "type": "timeseries",
    -       "title": "Decay sweep + tombstones (per minute)",
    -       "description": "Swept = memories purged by TTL; soft deletes = contradictions tombstoned by consolidation; dedup tombstones = tombstoned by the periodic dedup pass.",
    +       "title": "Promotion, classification & demotion (per hour)",
    +       "description": "Hourly outcomes of the episodic-to-semantic promotion pass, durable facts it wrote, and omitted-tier writes the marker classifier routed; red promote errors mean durable knowledge is not being distilled. Demotions are the mirror image: stale, never-recalled durable memories retiered down to episodic \u2014 a steady demotion stream usually means bulk-imported debris aging out.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 43
    +         "y": 44
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "stepAfter",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "promote: error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["sum"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_store_swept_total[5m]))",
    -           "legendFormat": "swept/{{tier}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (increase(memini_promote_results_total[1h]))",
    +           "refId": "A",
    +           "legendFormat": "promote: {{result}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(increase(memini_promote_facts_total[1h]))",
    +           "refId": "B",
    +           "legendFormat": "facts promoted"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "rate(memini_store_soft_deletes_total[5m])",
    -           "legendFormat": "consolidate tombstones",
    -           "refId": "B"
    +           "expr": "sum by (tier) (increase(memini_tier_classified_total[1h]))",
    +           "refId": "C",
    +           "legendFormat": "classified \u2192 {{tier}}"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "rate(memini_dedup_tombstoned_total[5m])",
    -           "legendFormat": "dedup tombstones",
    -           "refId": "C"
    +           "expr": "sum by (tier) (increase(memini_demoted_total[1h]))",
    +           "legendFormat": "demoted from {{tier}}",
    +           "refId": "D"
              }
            ]
          },
          {
    -       "id": 14,
    -       "type": "row",
    -       "title": "Embedder & rerank",
    +       "id": 26,
    +       "type": "timeseries",
    +       "title": "Decay sweeps & dedup tombstones",
    +       "description": "Memories purged by the decay sweeper (by tier) and tombstoned by the dedup pass, per second; sustained zero forever means aging jobs are not running.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
            "gridPos": {
    -         "h": 1,
    -         "w": 24,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
              "y": 51
            },
    -       "collapsed": false,
    -       "panels": []
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (tier) (rate(memini_store_swept_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "swept: {{tier}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "rate(memini_dedup_tombstoned_total[$__rate_interval])",
    +           "refId": "B",
    +           "legendFormat": "dedup tombstoned"
    +         }
    +       ]
          },
          {
    -       "id": 15,
    +       "id": 27,
            "type": "timeseries",
    -       "title": "Embedder p95 latency (by backend)",
    -       "description": "openai = real network calls; cached/diskcache/batched = local cache or fan-out; disabled = no endpoint configured.",
    +       "title": "Fsck outcomes (per hour)",
    +       "description": "Hourly consistency-sweep outcomes; red errors mean the store's self-check cannot complete and silent corruption may go unnoticed.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
    +         "x": 12,
    +         "y": 51
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (increase(memini_fsck_results_total[1h]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 28,
    +       "type": "timeseries",
    +       "title": "Repair backlog depth by state",
    +       "description": "Memories owing deferred repair, split by state; red failed entries are permanently vectorless without operator action, and a growing pending line means the repair loop cannot keep up.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
              "x": 0,
    -         "y": 52
    +         "y": 58
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "s"
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "failed"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (state) (memini_repair_depth)",
    +           "refId": "A",
    +           "legendFormat": "{{state}}"
              }
    +       ]
    +     },
    +     {
    +       "id": 29,
    +       "type": "timeseries",
    +       "title": "Repair oldest age by state",
    +       "description": "Age of the oldest memory in each repair state; growth without bound means repairs are queued faster than they drain (the red line marks the 1h alarm threshold).",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 58
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "line"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "red",
    +                 "value": 3600
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[5m])))",
    -           "legendFormat": "p95 {{backend}}",
    -           "refId": "A"
    +           "expr": "max by (state) (memini_repair_oldest_age_seconds)",
    +           "refId": "A",
    +           "legendFormat": "{{state}}"
              }
            ]
          },
          {
    -       "id": 16,
    +       "id": 30,
            "type": "timeseries",
    -       "title": "Embedder tokens + error rate",
    +       "title": "Backfill pending",
    +       "description": "Memories still awaiting embedding backfill and long memories still without chunk vectors; a number that never falls means those memories are unreachable by semantic (or chunked) recall.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
    +         "w": 8,
    +         "x": 16,
    +         "y": 58
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "memini_embed_backfill_pending",
    +           "refId": "A",
    +           "legendFormat": "pending embed"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "memini_chunk_backfill_pending",
    +           "refId": "B",
    +           "legendFormat": "pending chunk vectors"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 31,
    +       "type": "timeseries",
    +       "title": "Repair outcomes by stage",
    +       "description": "Deferred-repair attempt outcomes per second by stage; red parked means a memory gave up retrying and stays vectorless, orange retry churn means the embedder keeps failing under the repair loop.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
              "w": 12,
    -         "x": 12,
    -         "y": 52
    +         "x": 0,
    +         "y": 65
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*parked.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*retry.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
                }
    -         }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend) (rate(memini_embed_tokens_total[5m]))",
    -           "legendFormat": "tokens/s {{backend}}",
    -           "refId": "A"
    +           "expr": "sum by (stage, result) (rate(memini_repair_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{stage}}: {{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 32,
    +       "type": "timeseries",
    +       "title": "Repair duration p95 by stage",
    +       "description": "95th-percentile latency of a single deferred repair by stage; climbing durations shrink repair throughput and grow the backlog.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 65
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend) (rate(memini_embed_errors_total[5m]))",
    -           "legendFormat": "err/s {{backend}}",
    -           "refId": "B"
    +           "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(memini_repair_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{stage}}"
              }
            ]
          },
          {
    -       "id": 24,
    -       "type": "stat",
    -       "title": "Embedder in-flight",
    -       "description": "Concurrent embed calls against the backend (post-cap, excludes cache hits). Spikes during ingest storms are normal; sustained values near the cap mean the embedder is the bottleneck.",
    +       "id": 33,
    +       "type": "row",
    +       "title": "Embedder & rerank",
    +       "gridPos": {
    +         "h": 1,
    +         "w": 24,
    +         "x": 0,
    +         "y": 72
    +       },
    +       "collapsed": false,
    +       "panels": []
    +     },
    +     {
    +       "id": 34,
    +       "type": "timeseries",
    +       "title": "Embedder p95 latency by backend",
    +       "description": "95th-percentile embedder call latency per backend layer; the openai layer reflects the real provider, cached/diskcache should stay near zero.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 4,
    -         "w": 8,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 60
    +         "y": 73
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "horizontal",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    -         "textMode": "auto"
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 35,
    +       "type": "timeseries",
    +       "title": "Rerank outcomes by backend",
    +       "description": "Recall rerank attempts per second by backend and result; orange fallback means the reranker failed or timed out and results shipped with pre-rerank ordering.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 73
            },
            "fieldConfig": {
              "defaults": {
    -           "unit": "short"
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*fallback.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_embed_in_flight",
    -           "legendFormat": "embed in-flight",
    -           "refId": "A"
    +           "expr": "sum by (backend, result) (rate(memini_rerank_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}: {{result}}"
              }
            ]
          },
          {
    -       "id": 25,
    -       "type": "stat",
    -       "title": "Rerank in-flight",
    -       "description": "Concurrent rerank calls. A nonzero value indicates rerank is configured; if this stays pinned at the cap, the reranker is the bottleneck.",
    +       "id": 36,
    +       "type": "timeseries",
    +       "title": "Embedder tokens/s by backend",
    +       "description": "Embedding tokens consumed per second as reported by the API (openai backend only); this is the embed cost driver.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 4,
    +         "h": 7,
              "w": 8,
    -         "x": 8,
    -         "y": 60
    +         "x": 0,
    +         "y": 80
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "horizontal",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    -         "textMode": "auto"
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (backend) (rate(memini_embed_tokens_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 37,
    +       "type": "timeseries",
    +       "title": "Embedder errors by backend",
    +       "description": "Embedder failures per second by backend; any sustained nonzero rate here explains degraded writes and keyword-only recalls elsewhere on this board.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 80
            },
            "fieldConfig": {
              "defaults": {
    -           "unit": "short"
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_rerank_in_flight",
    -           "legendFormat": "rerank in-flight",
    -           "refId": "A"
    +           "expr": "sum by (backend) (rate(memini_embed_errors_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}"
              }
            ]
          },
          {
    -       "id": 26,
    +       "id": 38,
            "type": "timeseries",
    -       "title": "Embedder batch size (p50 vs p95)",
    -       "description": "Items per embed call, by backend. batched shows a single high-p95 bar; cached should be ~1.",
    +       "title": "Embedder batch size (p50 / p95)",
    +       "description": "Texts per Embed call at the median and 95th percentile; batches stuck at 1 mean the batcher is not coalescing and each write pays a full round trip.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 4,
    +         "h": 7,
              "w": 8,
              "x": 16,
    -         "y": 60
    +         "y": 80
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 5,
    -             "showPoints": "never"
    -           }
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.50, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "p50 {{backend}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))",
    +           "refId": "B",
    +           "legendFormat": "p95 {{backend}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 39,
    +       "type": "timeseries",
    +       "title": "In-flight embed & rerank calls",
    +       "description": "Concurrent calls currently held against the embedding and rerank backends; pinned at the concurrency cap means requests are queueing behind the limiter.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 6,
    +         "w": 24,
    +         "x": 0,
    +         "y": 87
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
            "options": {
              "legend": {
    +           "calcs": [],
                "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.5, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))",
    -           "legendFormat": "p50 {{backend}}",
    -           "refId": "A"
    +           "expr": "memini_embed_in_flight",
    +           "refId": "A",
    +           "legendFormat": "embed"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))",
    -           "legendFormat": "p95 {{backend}}",
    -           "refId": "B"
    +           "expr": "memini_rerank_in_flight",
    +           "refId": "B",
    +           "legendFormat": "rerank"
              }
            ]
          },
          {
    -       "id": 27,
    +       "id": 40,
            "type": "row",
    -       "title": "HTTP surface",
    +       "title": "LLM \u2014 consolidation & answer (empty without an LLM configured)",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 64
    +         "y": 93
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 28,
    +       "id": 41,
            "type": "timeseries",
    -       "title": "HTTP requests by route (status class)",
    -       "description": "Per-route request rate broken down by 2xx / 4xx / 5xx. 5xx spikes point at a broken handler; 4xx spikes usually mean client misuse or a misconfigured agent.",
    +       "title": "Consolidation outcomes",
    +       "description": "LLM consolidation decisions per second (gated, new, update, supersede, noop); red errors or orange drops mean writes are skipping dedup and duplicates will accumulate.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 16,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 65
    +         "y": 94
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "dropped"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (route, status) (label_replace(rate(memini_http_requests_total[5m]), \"status_class\", \"$1\", \"status\", \"(...).\"))",
    -           "legendFormat": "{{route}} {{status_class}}xx",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_consolidate_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 29,
    +       "id": 42,
            "type": "timeseries",
    -       "title": "HTTP p95 latency by route",
    +       "title": "Consolidation queue depth",
    +       "description": "Async consolidation queue depth over time; a sawtooth is normal, monotonic growth means the LLM cannot keep pace with write volume.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 16,
    -         "y": 65
    +         "h": 7,
    +         "w": 6,
    +         "x": 12,
    +         "y": 94
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "s"
    -         }
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": false
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[5m])))",
    -           "legendFormat": "p95 {{route}}",
    -           "refId": "A"
    +           "expr": "memini_consolidate_queue_depth",
    +           "refId": "A",
    +           "legendFormat": "queue depth"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 43,
    +       "type": "stat",
    +       "title": "Dedup effectiveness (15m)",
    +       "description": "Share of consolidated writes that matched an existing memory (update/supersede/noop) rather than creating a new one; a sudden drop to zero suggests the matcher is broken.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 6,
    +         "x": 18,
    +         "y": 94
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "percentunit"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result=~\"new|update|supersede|noop\"}[15m])), 1e-9)",
    +           "refId": "A",
    +           "legendFormat": "matched"
              }
            ]
          },
          {
    -       "id": 17,
    +       "id": 44,
            "type": "timeseries",
    -       "title": "End-to-end op latency p95 (by op)",
    +       "title": "Answer outcomes",
    +       "description": "Answer API (LLM-synthesized answers over recall) outcomes per second; red errors mean answer requests are burning LLM budget without returning.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 16,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 73
    +         "y": 101
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 5,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "s"
    -         }
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[5m])))",
    -           "legendFormat": "{{op}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_answer_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 34,
    +       "id": 45,
            "type": "timeseries",
    -       "title": "HTTP response size by route (p50/p90)",
    -       "description": "Body bytes shipped per response. The recall/briefing routes are what hooks inject from — watch these when tuning context-injection volume.",
    +       "title": "LLM op latency p95",
    +       "description": "95th-percentile end-to-end latency of the LLM-backed service ops (answer, promote, dedup, semantic distillation wait); these dominate tail latency when the LLM slows down.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 16,
    -         "y": 73
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 101
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "bytes"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))",
    -           "legendFormat": "p90 {{route}}",
    -           "refId": "A"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))",
    -           "legendFormat": "p50 {{route}}",
    -           "refId": "B"
    +           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket{op=~\"answer|promote|dedup|distill_sem_wait\"}[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{op}}"
              }
            ]
          },
          {
    -       "id": 30,
    +       "id": 46,
            "type": "row",
    -       "title": "Write-path signals",
    -       "collapsed": false,
    +       "title": "HTTP surface",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 82
    +         "y": 108
            },
    +       "collapsed": false,
            "panels": []
          },
          {
    -       "id": 31,
    +       "id": 47,
            "type": "timeseries",
    -       "title": "Corroboration routing",
    -       "description": "A fresh short-term write restating a durable fact: corroborated (confidence grown), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error.",
    +       "title": "HTTP request rate by route & status class",
    +       "description": "Requests per second per route, grouped into status classes; red 5xx traffic is server failure, orange 4xx spikes usually mean a misconfigured client or expired key.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 83
    +         "y": 109
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "short"
    +           "unit": "reqps",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".* 5xx"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".* 4xx"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (route, status_class) (label_replace(rate(memini_http_requests_total[$__rate_interval]), \"status_class\", \"$1\", \"status\", \"(.)..\"))",
    +           "refId": "A",
    +           "legendFormat": "{{route}} {{status_class}}xx"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 48,
    +       "type": "timeseries",
    +       "title": "HTTP p95 latency by route",
    +       "description": "95th-percentile HTTP latency per route as clients experience it; compare with the service-op view below to tell transport overhead from service work.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 109
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_corroborate_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{route}}"
              }
            ]
          },
          {
    -       "id": 33,
    +       "id": 49,
            "type": "timeseries",
    -       "title": "Contradiction routing",
    -       "description": "A fresh durable write that changes a fact's value or flips its polarity: contradicted (stale fact invalidated via valid_to + confidence shrink), no_signal (durable neighbour found but not a real change), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error. Default-on; a contradicted rate far above expectation is the misfire smoke alarm (set MEMINI_CONTRADICT_DOWNRANK=false to disable).",
    +       "title": "Service op latency p95 by op",
    +       "description": "95th-percentile end-to-end latency of every public service operation (remember, recall, briefing, forget, ...); this is the service-internal view behind the HTTP numbers.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 91
    +         "y": 116
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "short"
    -         }
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_contradict_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{op}}"
              }
            ]
          },
          {
    -       "id": 32,
    +       "id": 50,
            "type": "timeseries",
    -       "title": "Tier classification & promotion",
    -       "description": "Omitted-tier writes the marker classifier routed to a durable tier, and facts produced by the promotion pass (LLM or marker extractor).",
    +       "title": "HTTP response size (search & briefing, p50 / p90)",
    +       "description": "Response body sizes for the two injection-critical routes; growth here directly inflates the tokens injected into agent context.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 83
    +         "y": 116
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "short"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "bytes",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_tier_classified_total[5m]))",
    -           "legendFormat": "classified -> {{tier}}",
    -           "refId": "A"
    +           "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "p50 {{route}}"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_promote_facts_total[5m]))",
    -           "legendFormat": "promoted facts",
    -           "refId": "B"
    +           "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))",
    +           "refId": "B",
    +           "legendFormat": "p90 {{route}}"
              }
            ]
          },
          {
    -       "id": 40,
    +       "id": 51,
            "type": "row",
    -       "title": "Injection efficiency (client telemetry)",
    -       "collapsed": false,
    +       "title": "Client injection telemetry",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 99
    +         "y": 123
            },
    +       "collapsed": false,
            "panels": []
          },
          {
    -       "id": 41,
    +       "id": 52,
            "type": "timeseries",
    -       "title": "Injection outcomes (by result)",
    -       "description": "What hooks reported actually reaching model context vs held back by their local gates (seen/cooldown/budget/unchanged/score). Absent entirely means no integration is reporting.",
    +       "title": "Injection outcomes by result",
    +       "description": "Client-reported memory injections per second by outcome; suppressed_* series show why memories were withheld (already seen, cooldown, budget, unchanged, score) \u2014 a spike in suppressions with flat injections means the hooks are firing but delivering nothing.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 100
    +         "y": 124
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
    -           }
    -         }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_injected_memories_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_injected_memories_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 42,
    +       "id": 53,
            "type": "timeseries",
    -       "title": "Injected memories (by surface)",
    -       "description": "Memories that reached context, split by which hook surface put them there.",
    +       "title": "Injected memories by surface",
    +       "description": "Memories actually injected into model context per second, by hook surface (briefing, prompt, pretool); a silent surface means that hook is broken client-side.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 8,
    -         "y": 100
    +         "h": 7,
    +         "w": 6,
    +         "x": 12,
    +         "y": 124
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    -           }
    -         }
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[5m]))",
    -           "legendFormat": "{{surface}}",
    -           "refId": "A"
    +           "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{surface}}"
              }
            ]
          },
          {
    -       "id": 43,
    +       "id": 54,
            "type": "timeseries",
    -       "title": "Injected tokens per hour (by surface)",
    -       "description": "Client-estimated tokens the hooks added to context. The number injection tuning should push down.",
    +       "title": "Injected tokens per hour by surface",
    +       "description": "Client-side estimate of context tokens injected per hour by surface; this is the direct context-budget cost of memini to the agent.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 16,
    -         "y": 100
    +         "h": 7,
    +         "w": 6,
    +         "x": 18,
    +         "y": 124
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    -           }
    -         }
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [five lines unchanged)]
      
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
                "expr": "sum by (surface) (increase(memini_injected_tokens_total[1h]))",
    -           "legendFormat": "{{surface}}",
    -           "refId": "A"
    +           "refId": "A",
    +           "legendFormat": "{{surface}}"
              }
            ]
          }
        ],
      
      [30 lines unchanged)]
      
        "timepicker": {},
        "timezone": "",
        "title": "memini",
        "uid": "memini",
    -   "version": 2,
    +   "version": 3,
        "weekStart": ""
      }
  
  

spec.template.spec.containers.main.image  (apps/v1/StatefulSet/local-ai/memini)
  ± value change
    - registry.erwanleboucher.dev/eleboucher/memini@sha256:2d388222bca3d65673a4072e99e0970fe0d5cdab0030ce0effdfdd78307ae7e2
    + registry.erwanleboucher.dev/eleboucher/memini@sha256:1e060967d6fd11dfb79eeaa67d8745e0dde5d946ca1b7dd7469b4fad2fac0c1d
  

(root level)  (networking.k8s.io/v1/NetworkPolicy/gitlab/gitlab-redis)
+ one document added:
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: gitlab-redis
    namespace: gitlab
    labels:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  spec:
    egress:
    -   ingress:
    - ports:
      - port: 6379
    - ports:
      - port: 9121
    podSelector:
      matchLabels:
        app.kubernetes.io/instance: gitlab
        app.kubernetes.io/name: redis
    policyTypes:
    - Ingress
    - Egress

(root level)  (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-gitaly)
+ one document added:
  apiVersion: policy/v1beta1
  kind: PodDisruptionBudget
  metadata:
    name: gitlab-gitaly
    namespace: gitlab
    labels:
      app: gitaly
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxUnavailable: 1
    selector:
      matchLabels:
        app: gitaly
        release: gitlab

(root level)  (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-gitlab-shell)
+ one document added:
  apiVersion: policy/v1beta1
  kind: PodDisruptionBudget
  metadata:
    name: gitlab-gitlab-shell
    namespace: gitlab
    labels:
      app: gitlab-shell
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxUnavailable: 1
    selector:
      matchLabels:
        app: gitlab-shell
        release: gitlab

(root level)  (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-kas)
+ one document added:
  apiVersion: policy/v1beta1
  kind: PodDisruptionBudget
  metadata:
    name: gitlab-kas
    namespace: gitlab
    labels:
      app: kas
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxUnavailable: 1
    selector:
      matchLabels:
        app: kas
        release: gitlab

(root level)  (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-sidekiq-all-in-1-v1)
+ one document added:
  apiVersion: policy/v1beta1
  kind: PodDisruptionBudget
  metadata:
    name: gitlab-sidekiq-all-in-1-v1
    namespace: gitlab
    labels:
      app: sidekiq
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxUnavailable: 1
    selector:
      matchLabels:
        app: sidekiq
        queue-pod-name: all-in-1
        release: gitlab

(root level)  (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-webservice-default)
+ one document added:
  apiVersion: policy/v1beta1
  kind: PodDisruptionBudget
  metadata:
    name: gitlab-webservice-default
    namespace: gitlab
    labels:
      app: webservice
      gitlab.com/webservice-name: default
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxUnavailable: 1
    selector:
      matchLabels:
        app: webservice
        gitlab.com/webservice-name: default
        release: gitlab

(root level)  (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-registry-v1)
+ one document added:
  apiVersion: policy/v1beta1
  kind: PodDisruptionBudget
  metadata:
    name: gitlab-registry-v1
    namespace: gitlab
    labels:
      app: registry
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxUnavailable: 1
    selector:
      matchLabels:
        app: registry
        release: gitlab

(root level)  (v1/ServiceAccount/gitlab/gitlab-runner-access)
+ one document added:
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: gitlab-runner-access
    namespace: gitlab
    labels:
      app: gitlab-gitlab-runner
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab

(root level)  (v1/ServiceAccount/gitlab/gitlab-redis-master)
+ one document added:
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: gitlab-redis-master
    namespace: gitlab
    labels:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  automountServiceAccountToken: false

(root level)  (v1/ConfigMap/gitlab/gitlab-gitlab-runner)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitlab-runner
    namespace: gitlab
    labels:
      app: gitlab-gitlab-runner
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    check-live: |
      #!/bin/bash
      set -eou pipefail
      
      # default timeout is 3 seconds, can be overriden
      VERIFY_TIMEOUT=${1:-${VERIFY_TIMEOUT:-3}}
      
      if ! /usr/bin/pgrep -f ".*register-the-runner"  > /dev/null && ! /usr/bin/pgrep -f "gitlab.*runner"  > /dev/null ; then
        exit 1
      fi
      
      status=0
      # empty --url= helps `gitlab-runner verify` select all configured runners (otherwise filters for $CI_SERVER_URL)
      verify_output=$(timeout "${VERIFY_TIMEOUT}" gitlab-runner verify --url= 2>&1) || status=$?
      
      # timeout exit code is 143 with busybox, and 124 with coreutils
      if (( status == 143 )) || (( status == 124 )) ; then
        echo "'gitlab-runner verify' terminated by timeout, not a conclusive failure" >&2
        exit 0
      elif (( status > 0 )) ; then
        exit ${status}
      fi
      
      grep -qE "is (alive|valid)" <<<"${verify_output}"
      
    config.template.toml: |
      [[runners]]
        [runners.kubernetes]
        image = "archlinux/archlinux:latest"
        namespace_per_job = true
        node_selector_overwrite_allowed = "kubernetes.io/arch=.*"
        pull_policy = ["always", "always"]
      
        [runners.kubernetes.node_selector]
        "kubernetes.io/arch" = "amd64"
      
        [runners.cache]
          Type = "s3"
          Path = "gitlab-runner"
          Shared = true
          [runners.cache.s3]
            ServerAddress = "brickyard.home.arpa:3900"
            BucketName = "gitlab-runner-cache"
            BucketLocation = "apartment"
            Insecure = true
            AuthenticationType = "access-key"
            PathStyle = true
      
    config.toml: |
      shutdown_timeout = 0
      concurrent = 10
      check_interval = 3
      log_level = "info"
      
    entrypoint: |
      #!/bin/bash
      set -e
      
      export CONFIG_PATH_FOR_INIT="/home/gitlab-runner/.gitlab-runner/"
      mkdir -p ${CONFIG_PATH_FOR_INIT}
      cp /configmaps/config.toml ${CONFIG_PATH_FOR_INIT}
      
      # Set up environment variables for cache
      if [[ -f /secrets/accesskey && -f /secrets/secretkey ]]; then
        export CACHE_S3_ACCESS_KEY=$(cat /secrets/accesskey)
        export CACHE_S3_SECRET_KEY=$(cat /secrets/secretkey)
      fi
      
      if [[ -f /secrets/gcs-applicaton-credentials-file ]]; then
        export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-applicaton-credentials-file"
      elif [[ -f /secrets/gcs-application-credentials-file ]]; then
        export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-application-credentials-file"
      else
        if [[ -f /secrets/gcs-access-id && -f /secrets/gcs-private-key ]]; then
          export CACHE_GCS_ACCESS_ID=$(cat /secrets/gcs-access-id)
          # echo -e used to make private key multiline (in google json auth key private key is oneline with \n)
          export CACHE_GCS_PRIVATE_KEY=$(echo -e $(cat /secrets/gcs-private-key))
        fi
      fi
      
      if [[ -f /secrets/azure-account-name && -f /secrets/azure-account-key ]]; then
        export CACHE_AZURE_ACCOUNT_NAME=$(cat /secrets/azure-account-name)
        export CACHE_AZURE_ACCOUNT_KEY=$(cat /secrets/azure-account-key)
      fi
      
      if [[ -f /secrets/runner-registration-token ]]; then
        export REGISTRATION_TOKEN=$(cat /secrets/runner-registration-token)
      fi
      
      if [[ -f /secrets/runner-token ]]; then
        export CI_SERVER_TOKEN=$(cat /secrets/runner-token)
      fi
      
      # Register the runner
      if ! sh /configmaps/register-the-runner; then
        exit 1
      fi
      
      # Run pre-entrypoint-script
      if ! bash /configmaps/pre-entrypoint-script; then
        exit 1
      fi
      
      # Start the runner
      exec /entrypoint run \
        --working-directory=/home/gitlab-runner
      
    pre-entrypoint-script: 
    register-the-runner: |
      #!/bin/sh
      signal_handler() {
        if [ ! -d "/proc/$register_pid" ]; then
          wait $register_pid
        fi
        exit
      }
      trap 'signal_handler' QUIT INT
      
      MAX_REGISTER_ATTEMPTS=30
      
      # Reset/unset the not needed flags when an authentication token
      RUN_UNTAGGED=""
      ACCESS_LEVEL=""
      
      if [ -n "$CI_SERVER_TOKEN" ] && [ "${CI_SERVER_TOKEN#glrt-}" != "$CI_SERVER_TOKEN" ]; then
        RUN_UNTAGGED=""
        ACCESS_LEVEL=""
        unset REGISTER_LOCKED
        unset RUNNER_TAG_LIST
      fi
      
      for i in $(seq 1 "${MAX_REGISTER_ATTEMPTS}"); do
        echo "Registration attempt ${i} of ${MAX_REGISTER_ATTEMPTS}"
        /entrypoint register \
          ${RUN_UNTAGGED} \
          ${ACCESS_LEVEL} \
          --template-config /configmaps/config.template.toml \
          --non-interactive &
      
        register_pid=$!
        wait $register_pid
        retval=$?
      
        if [ ${retval} = 0 ]; then
          break
        elif [ ${i} = ${MAX_REGISTER_ATTEMPTS} ]; then
          exit 1
        fi
      
        sleep 5
      done
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-gitaly)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitaly
    namespace: gitlab
    labels:
      app: gitaly
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    config.toml.tpl: |
      # The directory where Gitaly's executables are stored
      bin_dir = "/usr/local/bin"
      
      # listen on a TCP socket. This is insecure (no authentication)
      listen_addr = ":8075"
      
      # Directory where internal sockets reside
      # note: no value will result in a `/tmp/gitlab-internal-*` path
      # internal_socket_dir = "/home/git"
      
      # If metrics collection is enabled, inform gitaly about that
      prometheus_listen_addr = ":9236"
      
      # Graceful shutdown timeout, how long to wait for in-flight requests to complete
      graceful_restart_timeout = "25s"
      
      # Storage configuration
      {% $storages := coll.Slice  "default" %}
      {% $hostid := index (.Env.HOSTNAME | strings.Split "-" | coll.Reverse) 0 %}
      {% $index := 0 %}
      {% if regexp.Match "^[0-9]+$" $hostid %}
        {% $index = conv.ToInt64 $hostid %}
      {% end %}
      {% if len $storages | lt $index %}
      [[storage]]
      name = "{% index $storages $index %}"
      path = "/home/git/repositories"
      {% else %}
      {% printf "Storage for node %d is not present in the storageNames array. Did you use kubectl to scale up? You need to solely use helm for this purpose." $index | fail %}
      {% end %}
      
      [logging]
      format = "json"
      dir = "/var/log/gitaly"
      
      [auth]
      token = {% file.Read "/etc/gitlab-secrets/gitaly/gitaly_token" | strings.TrimSpace | data.ToJSON %}
      
      [git]
      use_bundled_binaries = true
      ignore_gitconfig = true
      
      [gitlab-shell]
      # The directory where gitlab-shell is installed
      dir = "/srv/gitlab-shell"
      
      [gitlab]
      # location of shared secret for GitLab Shell / API interaction
      secret_file = "/etc/gitlab-secrets/shell/.gitlab_shell_secret"
      # URL of API
      url = "http://gitlab-webservice-default.gitlab.svc:8181/"
      
      [gitlab.http-settings]
      # read_timeout = 300
      # user = someone
      # password = somepass
      # ca_file = /etc/ssl/cert.pem
      # ca_path = /etc/pki/tls/certs
      
      [hooks]
      # directory containing custom hooks
      custom_hooks_dir = "/home/git/custom_hooks"
      
    configure: |
      set -e
      mkdir -p /init-secrets/gitaly /init-secrets/shell
      cp -v -r -L /init-config/.gitlab_shell_secret  /init-secrets/shell/.gitlab_shell_secret
      cp -v -r -L /init-config/gitaly_token  /init-secrets/gitaly/gitaly_token
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-gitlab-exporter)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitlab-exporter
    namespace: gitlab
    labels:
      app: gitlab-exporter
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    configure: |
      # BEGIN gitlab.scripts.configure.secrets
      set -e
      config_dir="/init-config"
      secret_dir="/init-secrets"
      # optional
      for secret in redis redis-sentinel postgres gitlab-exporter ; do
        if [ -e "${config_dir}/${secret}" ]; then
          mkdir -p "${secret_dir}/${secret}"
          cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
        fi
      done
      # END gitlab.scripts.configure.secrets
      
    gitlab-exporter.yml.erb: |
      server:
        name: webrick
        listen_address: "*"
        listen_port: 9168
        
      
      probes:
        db_common: &db_common
          methods:
            - probe_db
          opts:
            connection_string: dbname=gitlab-prod user=gitlab host=db-gitlab-rw.gitlab.svc port=5432 password='<%= File.read('/etc/gitlab/postgres/psql-password-main').strip.gsub(/[\'\\]/) { |esc| '\\' + esc } %>'
        database:
          multiple: true
          ci_builds:
            class_name: Database::CiBuildsProber
            <<: *db_common
          tuple_stats:
            class_name: Database::TuplesProber
            <<: *db_common
          rows_count:
            class_name: Database::RowCountProber
            <<: *db_common
        database_bloat:
          class_name: Database::BloatProber
          <<: *db_common
      
        sidekiq: &sidekiq
          methods:
            - probe_queues
            - probe_workers
            - probe_retries
            - probe_stats
          opts:
            redis_url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
            redis_enable_client: false
            probe_non_namespaced: true
      
        ruby: &ruby
          methods:
            - probe_gc
          opts:
            quantiles: false
      
        metrics:
          multiple: true
          ruby:
            <<: *ruby
          sidekiq:
            <<: *sidekiq
          ci_builds:
            class_name: Database::CiBuildsProber
            <<: *db_common
          tuple_stats:
            class_name: Database::TuplesProber
            <<: *db_common
          rows_count:
            class_name: Database::RowCountProber
            <<: *db_common
          pg_sequences:
            class_name: Database::PgSequencesProber
            <<: *db_common
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-gitlab-shell-sshd)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitlab-shell-sshd
    namespace: gitlab
    labels:
      app: gitlab-shell
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    sshd_config: |
      #	$OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $
      
      # This is the sshd server system-wide configuration file.  See
      # sshd_config(5) for more information.
      
      # This sshd was compiled with PATH=/bin:/usr/bin:/sbin:/usr/sbin
      
      # The strategy used for options in the default sshd_config shipped with
      # OpenSSH is to specify options with their default value where
      # possible, but leave them commented.  Uncommented options override the
      # default value.
      
      Port 2222
      #AddressFamily any
      #ListenAddress 0.0.0.0
      #ListenAddress ::
      
      #HostKey /etc/ssh/ssh_host_rsa_key
      #HostKey /etc/ssh/ssh_host_dsa_key
      #HostKey /etc/ssh/ssh_host_ecdsa_key
      #HostKey /etc/ssh/ssh_host_ed25519_key
      
      # Ciphers and keying
      #RekeyLimit default none
      
      # Logging
      #SyslogFacility AUTH
      LogLevel ERROR
      
      # Authentication:
      
      PermitRootLogin no
      #StrictModes yes
      #MaxAuthTries 6
      #MaxSessions 10
      
      #PubkeyAuthentication yes
      
      #AuthorizedPrincipalsFile none
      
      # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
      #HostbasedAuthentication no
      # Change to yes if you don't trust ~/.ssh/known_hosts for
      # HostbasedAuthentication
      #IgnoreUserKnownHosts no
      # Don't read the user's ~/.rhosts and ~/.shosts files
      #IgnoreRhosts yes
      
      #PermitEmptyPasswords no
      
      # Change to no to disable s/key passwords
      #ChallengeResponseAuthentication yes
      
      # Kerberos options
      #KerberosAuthentication no
      #KerberosOrLocalPasswd yes
      #KerberosTicketCleanup yes
      #KerberosGetAFSToken no
      
      # GSSAPI options
      #GSSAPIAuthentication no
      #GSSAPICleanupCredentials yes
      
      # Set this to 'yes' to enable PAM authentication, account processing,
      # and session processing. If this is enabled, PAM authentication will
      # be allowed through the ChallengeResponseAuthentication and
      # PasswordAuthentication.  Depending on your PAM configuration,
      # PAM authentication via ChallengeResponseAuthentication may bypass
      # the setting of "PermitRootLogin without-password".
      # If you just want the PAM account and session checks to run without
      # PAM authentication, then enable this but set PasswordAuthentication
      # and ChallengeResponseAuthentication to 'no'.
      #UsePAM no
      
      #AllowAgentForwarding yes
      #AllowTcpForwarding yes
      #GatewayPorts no
      #X11Forwarding no
      #X11DisplayOffset 10
      #X11UseLocalhost yes
      #PermitTTY yes
      #PrintMotd yes
      #PrintLastLog yes
      #TCPKeepAlive yes
      #UseLogin no
      #PermitUserEnvironment no
      #Compression delayed
      ClientAliveInterval 0
      #ClientAliveCountMax 3
      #UseDNS no
      PidFile /srv/sshd/sshd.pid
      #PermitTunnel no
      #ChrootDirectory none
      #VersionAddendum none
      
      # no default banner path
      #Banner none
      
      # override default of no subsystems
      # Subsystem	sftp	/usr/lib/ssh/sftp-server
      
      # the following are HPN related configuration options
      # tcp receive buffer polling. disable in non autotuning kernels
      #TcpRcvBufPoll yes
      
      # disable hpn performance boosts
      #HPNDisabled no
      
      # buffer size for hpn to non-hpn connections
      #HPNBufferSize 2048
      
      
      # Example of overriding settings on a per-user basis
      #Match User anoncvs
      #	X11Forwarding no
      #	AllowTcpForwarding no
      #	PermitTTY no
      #	ForceCommand cvs server
      
      # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
      # but this is overridden so installations will only check .ssh/authorized_keys
      AuthorizedKeysFile	.ssh/authorized_keys
      
      AuthorizedKeysCommand /authorized_keys %u %k
      AuthorizedKeysCommandUser git
      
      PasswordAuthentication no
      
      AllowUsers git
      
      # Enable the use of Git protcol v2
      AcceptEnv GIT_PROTOCOL
      
      # Hard disable all forwarding
      DisableForwarding yes
      
      # Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon.
      # See `man sshd_config(5)`
      
      MaxStartups 10:30:100
      
      
      # Specifies amount of time athat the server will disconnect after if the user has not successfully logged in
      LoginGraceTime 60
      
      # Includes
      Include /etc/ssh/sshd_config.d/*.conf
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-gitlab-shell)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitlab-shell
    namespace: gitlab
    labels:
      app: gitlab-shell
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    config.yml.tpl: |
      # GitLab user. git by default
      user: git
      
      # Url to gitlab instance. Used for api calls. Should end with a slash.
      gitlab_url: "http://gitlab-webservice-default.gitlab.svc:8181/"
      
      secret_file: /etc/gitlab-secrets/shell/.gitlab_shell_secret
      
      # File used as authorized_keys for gitlab user
      auth_file: "/home/git/.ssh/authorized_keys"
      
      # Log file.
      # Default is gitlab-shell.log in the root directory.
      log_file: "/var/log/gitlab-shell/gitlab-shell.log"
      
      log_format: json
      
      # Audit usernames.
      # Set to true to see real usernames in the logs instead of key ids, which is easier to follow, but
      # incurs an extra API call on every gitlab-shell command.
      audit_usernames: false
      
      
      lfs:
        pure_ssh_protocol: false
      pat:
        enabled: true
        allowed_scopes:
          []
      
    configure: |
      # BEGIN gitlab.scripts.configure.secrets
      set -e
      config_dir="/init-config"
      secret_dir="/init-secrets"
      # required
      for secret in shell ; do
        mkdir -p "${secret_dir}/${secret}"
        cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
      done
      # END gitlab.scripts.configure.secrets
      mkdir -p /${secret_dir}/ssh
      cp -v -r -L /${config_dir}/ssh_host_* /${secret_dir}/ssh/
      chmod 0400 /${secret_dir}/ssh/ssh_host_*
      
    krb5.conf: 
  


(root level)  (v1/ConfigMap/gitlab/gitlab-nginx-ingress-tcp)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-nginx-ingress-tcp
    namespace: gitlab
    labels:
      app: gitlab-shell
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    22222: "gitlab/gitlab-gitlab-shell:22222:PROXY:"

(root level)  (v1/ConfigMap/gitlab/gitlab-kas)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-kas
    namespace: gitlab
    labels:
      app: kas
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    config.yaml: |
      agent:
        kubernetes_api:
          listen:
            address: :8154
          url_path_prefix: /k8s-proxy
          websocket_token_secret_file: /etc/kas/.gitlab_kas_websocket_token_secret
        listen:
          address: :8150
          websocket: true
      api:
        listen:
          address: :8153
          authentication_secret_file: /etc/kas/.gitlab_kas_secret
      gitlab:
        address: http://gitlab-webservice-default.gitlab.svc:8181
        authentication_secret_file: /etc/kas/.gitlab_kas_secret
        external_url: https://code.nogweii.net
      observability:
        listen:
          address: :8151
        liveness_probe:
          url_path: /liveness
        readiness_probe:
          url_path: /readiness
      private_api:
        listen:
          address: :8155
          authentication_secret_file: /etc/kas/.gitlab_kas_private_api_secret
      redis:
        database_index: 0
        password_file: /etc/kas/redis/redis-password
        server:
          address: gitlab-redis-master.gitlab.svc:6379
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-migrations)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-migrations
    namespace: gitlab
    labels:
      app: migrations
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
        adapter: redis
      
    configure: |
      # BEGIN gitlab.scripts.configure.secrets
      set -e
      config_dir="/init-config"
      secret_dir="/init-secrets"
      # required
      for secret in rails-secrets migrations gitaly ; do
        mkdir -p "${secret_dir}/${secret}"
        cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
      done
      # optional
      for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do
        if [ -e "${config_dir}/${secret}" ]; then
          mkdir -p "${secret_dir}/${secret}"
          cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
        fi
      done
      # END gitlab.scripts.configure.secrets
      
    database.yml.erb: |
      
      production:
        main:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: true
          
          
        ci:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: false
          
          
      
    gitlab.yml.erb: |
      production: &base
        gitlab:
          host: code.nogweii.net
          https: true
          signup_enabled: false
        # Consolidated object storage configuration
        ## property local configuration will override object_store
        object_store:
          enabled: true
          proxy_download: true
          connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %>
          objects:
            artifacts:
              bucket: gitlab-artifacts
              proxy_download: true
            lfs:
              bucket: gitlab-lfs
              proxy_download: true
            uploads:
              bucket: gitlab-uploads
              proxy_download: true
            packages:
              bucket: gitlab-packages
              proxy_download: true
            external_diffs:
              bucket: gitlab-external-diffs
              proxy_download: true
            terraform_state:
              bucket: gitlab-terraform-state
            dependency_proxy:
              bucket: gitlab-dependency-proxy
              proxy_download: true
            ci_secure_files:
              bucket: gitlab-ci-secure-files
            pages:
              bucket: gitlab-pages
        artifacts:
          enabled: true
        lfs:
          enabled: true
        uploads:
          enabled: true
        packages:
          enabled: true
        external_diffs:
          enabled: true
          when: 
        terraform_state:
          enabled: true
        ci_secure_files:
          enabled: true
        gitaly:
          client_path: /home/git/gitaly/bin
          token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %>
        repositories:
          storages: # You must have at least a `default` storage path.
            default:
              path: /var/opt/gitlab/repo
              gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075
            
        
        sidekiq:
          log_format: json
        oidc_provider:
          openid_id_token_expire_in_seconds: 120
        
        
      
    installation_type: |
      gitlab-helm-chart
      
    pages_redirect_uri: "https://projects.aether.page/auth"
    redis.action_cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    resque.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-sidekiq)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-sidekiq
    namespace: gitlab
    labels:
      app: sidekiq
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
        adapter: redis
      
    configure: |
      # BEGIN gitlab.scripts.configure.secrets
      set -e
      config_dir="/init-config"
      secret_dir="/init-secrets"
      # required
      for secret in gitaly registry rails-secrets ; do
        mkdir -p "${secret_dir}/${secret}"
        cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
      done
      # optional
      for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do
        if [ -e "${config_dir}/${secret}" ]; then
          mkdir -p "${secret_dir}/${secret}"
          cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
        fi
      done
      # END gitlab.scripts.configure.secrets
      
    database.yml.erb: |
      
      production:
        main:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: true
          
          
        ci:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: false
          
          
      
    gitlab.yml.erb: |
      production: &base
        gitlab:
          host: code.nogweii.net
          https: true
          ssh_host: "code.nogweii.net"
          action_cable_allowed_origins: []
          max_request_duration_seconds: 57
          impersonation_enabled: 
          application_settings_cache_seconds: 60
          usage_ping_enabled: true
          seat_link_enabled: true
          username_changing_enabled: true
          issue_closing_pattern: 
          default_theme: 
          default_projects_features:
            issues: true
            merge_requests: true
            wiki: true
            snippets: true
            builds: true
            container_registry: true
          signup_enabled: false
          graphql_timeout: 
          webhook_timeout: 
          trusted_proxies:
          time_zone: "UTC"        
          email_from: "gitlab@aether.earth"
          email_display_name: "GitLab"
          email_reply_to: "gitlab+noreply@aether.earth"
          email_subject_suffix: ""
        
        gravatar:
          plain_url: 
          ssl_url: 
        extra:
          
          
          
          
          
          
        # Consolidated object storage configuration
        ## property local configuration will override object_store
        object_store:
          enabled: true
          proxy_download: true
          connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %>
          objects:
            artifacts:
              bucket: gitlab-artifacts
              proxy_download: true
            lfs:
              bucket: gitlab-lfs
              proxy_download: true
            uploads:
              bucket: gitlab-uploads
              proxy_download: true
            packages:
              bucket: gitlab-packages
              proxy_download: true
            external_diffs:
              bucket: gitlab-external-diffs
              proxy_download: true
            terraform_state:
              bucket: gitlab-terraform-state
            dependency_proxy:
              bucket: gitlab-dependency-proxy
              proxy_download: true
            ci_secure_files:
              bucket: gitlab-ci-secure-files
            pages:
              bucket: gitlab-pages
        artifacts:
          enabled: true
        lfs:
          enabled: true
        uploads:
          enabled: true
        packages:
          enabled: true
        external_diffs:
          enabled: true
          when: 
        terraform_state:
          enabled: true
        ci_secure_files:
          enabled: true
        dependency_proxy:
          enabled: true
        
        sentry:
          enabled: false
          dsn: 
          clientside_dsn: 
          environment: 
        
        sidekiq:
          log_format: json
        oidc_provider:
          openid_id_token_expire_in_seconds: 120
        
        pages:
          enabled: false
          access_control: false
          artifacts_server: true
          path: /srv/gitlab/shared/pages
          host: aether.page
          port: 443
          https: true
          secret_file: /etc/gitlab/pages/secret
          external_http: false
          external_https: false
          local_store:
            enabled: false
            path: 
          namespace_in_path: false
        
        mattermost:
          enabled: false
        ## Registry Integration
        registry:
          enabled: true
          host: c.code.nogweii.net
          api_url: http://gitlab-registry.gitlab.svc:5000
          key: /etc/gitlab/registry/gitlab-registry.key
          issuer: gitlab-issuer
          notification_secret: <%= YAML.load_file("/etc/gitlab/registry/notificationSecret").flatten.first %>
        gitlab_ci:
        ldap:
          enabled: false
        
        omniauth:
          enabled: true
          sync_profile_from_provider: ["openid_connect"]
          sync_profile_attributes: ["email","name"]
          allow_single_sign_on: ["openid_connect"]
          block_auto_created_users: false
          auto_link_ldap_user: false
          auto_link_saml_user: false
          auto_link_user: ["openid_connect"]
          external_providers: []
          providers:
            - <%= YAML.load_file("/etc/gitlab/omniauth/authentik-oidc/config.yaml").to_json() %>
        kerberos:
          enabled: false
          keytab: /etc/krb5.keytab
          use_dedicated_port: false
          port: 8443
          https: true
          simple_ldap_linking_allowed_realms: []
        
        duo_auth:
          enabled: false
          hostname: 
          integration_key: 
          secret_key: 
        
        duo_auth:
          enabled: false
          hostname: 
          integration_key: 
          secret_key: 
        shared:
        gitaly:
          client_path: /home/git/gitaly/bin
          token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %>
        repositories:
          storages: # You must have at least a `default` storage path.
            default:
              path: /var/opt/gitlab/repo
              gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075
            
        backup:
          path: "tmp/backups"   # Relative paths are relative to Rails.root (default: tmp/backups/)
        gitlab_kas:
          enabled: true
          secret_file: /etc/gitlab/kas/.gitlab_kas_secret
          external_url: "wss://k.code.nogweii.net"
          internal_url: "grpc://gitlab-kas.gitlab.svc:8153"
          
        
        
        
        # Zoekt secrets are present, load them.
        zoekt:
          username_file: /etc/gitlab/zoekt/.gitlab_zoekt_username
          password_file: /etc/gitlab/zoekt/.gitlab_zoekt_password
        gitlab_shell:
          path: /home/git/gitlab-shell/
          hooks_path: /home/git/gitlab-shell/hooks/
          upload_pack: true
          receive_pack: true
          ssh_port: 22222
        workhorse:
        git:
          bin_path: /usr/bin/git
        webpack:
        monitoring:
          ip_whitelist:
            - "127.0.0.0/8"
            - "::1/128"
          sidekiq_exporter:
            enabled: true
            address: "*"
            port: 3807
            log_enabled: false
          sidekiq_health_checks:
            enabled: true
            address: "*"
            port: 3808
        
      
    installation_type: |
      gitlab-helm-chart
      
    krb5.conf: 
    redis.action_cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    resque.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    session_store.yml.erb: |
      production:
          session_cookie_token_prefix: ""
      
    smtp_settings.rb: |
      smtp_settings = {
        address: "mail-relay.email.svc.cluster.local",
        port: 587,
        ca_file: "/etc/ssl/certs/ca-certificates.crt",
        
        enable_starttls_auto: false,
        tls: false,
        openssl_verify_mode: "peer",
        open_timeout: 30,
        read_timeout: 60
      }
      
      Rails.application.config.action_mailer.delivery_method = :smtp
      ActionMailer::Base.delivery_method = :smtp
      
      ActionMailer::Base.smtp_settings = smtp_settings
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-toolbox)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-toolbox
    namespace: gitlab
    labels:
      app: toolbox
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
        adapter: redis
      
    configure: |
      # BEGIN gitlab.scripts.configure.secrets
      set -e
      config_dir="/init-config"
      secret_dir="/init-secrets"
      # required
      for secret in shell gitaly registry rails-secrets ; do
        mkdir -p "${secret_dir}/${secret}"
        cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
      done
      # optional
      for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do
        if [ -e "${config_dir}/${secret}" ]; then
          mkdir -p "${secret_dir}/${secret}"
          cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
        fi
      done
      # END gitlab.scripts.configure.secrets
      
      
      if [ ! -f "/${secret_dir}/objectstorage/.s3cfg" ]; then
      cat <<EOF > "/${secret_dir}/.s3cfg"
      [default]
      access_key = $(cat /init-secrets/minio/accesskey)
      secret_key = $(cat /init-secrets/minio/secretkey)
      bucket_location = us-east-1
      host_base = minio.code.nogweii.net
      host_bucket = minio.code.nogweii.net/%(bucket)
      default_mime_type = binary/octet-stream
      enable_multipart = True
      multipart_max_chunks = 10000
      multipart_chunk_size_mb = 128
      recursive = True
      recv_chunk = 65536
      send_chunk = 65536
      server_side_encryption = False
      signature_v2 = True
      socket_timeout = 300
      use_mime_magic = False
      verbosity = WARNING
      website_endpoint = https://minio.code.nogweii.net
      EOF
      else
        mv "/${secret_dir}/objectstorage/.s3cfg" "/${secret_dir}/.s3cfg"
      fi
      
    configure-gsutil: |
      if [ -e "$GOOGLE_APPLICATION_CREDENTIALS" ]; then
        # The following script is used to configure gsutil when creating backups
        # It provides inputs to the `gsutil config -e` prompt as follows:
        # 1) Path to service account JSON key file
        # 2) Do not set permissions for key file
        # 3) GCP Project ID
        # 4) Decline anonymous usage statistics
        printf "$GOOGLE_APPLICATION_CREDENTIALS\nN\n\nN\n" | gsutil config -e
      else
        # If the backup config is not configured, assume that
        # Application Default Credentials are used with a service account.
        # This enables GKE Workload Identity Federation to work.
        echo '[GoogleCompute]\nservice_account = default' > ~/.boto
      fi
      
    database.yml.erb: |
      
      production:
        main:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: true
          
          
        ci:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: false
          
          
      
    gitlab.yml.erb: |
      production: &base
        gitlab:
          host: code.nogweii.net
          https: true
          ssh_host: "code.nogweii.net"
          max_request_duration_seconds: 57
          impersonation_enabled: 
          application_settings_cache_seconds: 60
          usage_ping_enabled: true
          seat_link_enabled: true
          username_changing_enabled: true
          issue_closing_pattern: 
          default_theme: 
          default_projects_features:
            issues: true
            merge_requests: true
            wiki: true
            snippets: true
            builds: true
            container_registry: true
          signup_enabled: false
          graphql_timeout: 
          webhook_timeout: 
          trusted_proxies:
          time_zone: "UTC"        
          email_from: "gitlab@aether.earth"
          email_display_name: "GitLab"
          email_reply_to: "gitlab+noreply@aether.earth"
          email_subject_suffix: ""
        
        # Consolidated object storage configuration
        ## property local configuration will override object_store
        object_store:
          enabled: true
          proxy_download: true
          connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %>
          objects:
            artifacts:
              bucket: gitlab-artifacts
              proxy_download: true
            lfs:
              bucket: gitlab-lfs
              proxy_download: true
            uploads:
              bucket: gitlab-uploads
              proxy_download: true
            packages:
              bucket: gitlab-packages
              proxy_download: true
            external_diffs:
              bucket: gitlab-external-diffs
              proxy_download: true
            terraform_state:
              bucket: gitlab-terraform-state
            dependency_proxy:
              bucket: gitlab-dependency-proxy
              proxy_download: true
            ci_secure_files:
              bucket: gitlab-ci-secure-files
            pages:
              bucket: gitlab-pages
        artifacts:
          enabled: true
        lfs:
          enabled: true
        uploads:
          enabled: true
        packages:
          enabled: true
        external_diffs:
          enabled: true
          when: 
        terraform_state:
          enabled: true
        ci_secure_files:
          enabled: true
        dependency_proxy:
          enabled: true
        
        sidekiq:
          log_format: json
        oidc_provider:
          openid_id_token_expire_in_seconds: 120
        
        
        pages:
          enabled: false
          access_control: false
          artifacts_server: true
          path: /srv/gitlab/shared/pages
          host: aether.page
          port: 443
          https: true
          secret_file: /etc/gitlab/pages/secret
          external_http: false
          external_https: false
          local_store:
            enabled: false
            path: 
          namespace_in_path: false
        
        mattermost:
          enabled: false
        ## Registry Integration
        registry:
          enabled: true
          host: c.code.nogweii.net
          api_url: http://gitlab-registry.gitlab.svc:5000
          key: /etc/gitlab/registry/gitlab-registry.key
          issuer: gitlab-issuer
          notification_secret: <%= YAML.load_file("/etc/gitlab/registry/notificationSecret").flatten.first %>
        gitlab_ci:
        ldap:
          enabled: false
        
        omniauth:
          enabled: true
          sync_profile_from_provider: ["openid_connect"]
          sync_profile_attributes: ["email","name"]
          allow_single_sign_on: ["openid_connect"]
          block_auto_created_users: false
          auto_link_ldap_user: false
          auto_link_saml_user: false
          auto_link_user: ["openid_connect"]
          external_providers: []
          providers:
            - <%= YAML.load_file("/etc/gitlab/omniauth/authentik-oidc/config.yaml").to_json() %>
        kerberos:
          enabled: false
          keytab: /etc/krb5.keytab
          use_dedicated_port: false
          port: 8443
          https: true
          simple_ldap_linking_allowed_realms: []
        
        duo_auth:
          enabled: false
          hostname: 
          integration_key: 
          secret_key: 
        shared:
        gitaly:
          client_path: /home/git/gitaly/bin
          token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %>
        repositories:
          storages: # You must have at least a `default` storage path.
            default:
              path: /var/opt/gitlab/repo
              gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075
            
        backup:
          path: "tmp/backups"   # Relative paths are relative to Rails.root (default: tmp/backups/)
        gitlab_kas:
          enabled: true
          secret_file: /etc/gitlab/kas/.gitlab_kas_secret
          external_url: "wss://k.code.nogweii.net"
          internal_url: "grpc://gitlab-kas.gitlab.svc:8153"
          
        
        
        
        # Zoekt secrets are present, load them.
        zoekt:
          username_file: /etc/gitlab/zoekt/.gitlab_zoekt_username
          password_file: /etc/gitlab/zoekt/.gitlab_zoekt_password
        gitlab_shell:
          path: /home/git/gitlab-shell/
          hooks_path: /home/git/gitlab-shell/hooks/
          upload_pack: true
          receive_pack: true
          ssh_port: 22222
          secret_file: /etc/gitlab/shell/.gitlab_shell_secret
        workhorse:
        git:
          bin_path: /usr/bin/git
        webpack:
        monitoring:
          ip_whitelist:
            - 127.0.0.0/8
          sidekiq_exporter:
        extra:
        
      
    krb5.conf: 
    redis.action_cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    resque.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    session_store.yml.erb: |
      production:
          session_cookie_token_prefix: ""
      
    smtp_settings.rb: |
      smtp_settings = {
        address: "mail-relay.email.svc.cluster.local",
        port: 587,
        ca_file: "/etc/ssl/certs/ca-certificates.crt",
        
        enable_starttls_auto: false,
        tls: false,
        openssl_verify_mode: "peer",
        open_timeout: 30,
        read_timeout: 60
      }
      
      Rails.application.config.action_mailer.delivery_method = :smtp
      ActionMailer::Base.delivery_method = :smtp
      
      ActionMailer::Base.smtp_settings = smtp_settings
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-webservice)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-webservice
    namespace: gitlab
    labels:
      app: webservice
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
        adapter: redis
      
    configure: |
      # BEGIN gitlab.scripts.configure.secrets
      set -e
      config_dir="/init-config"
      secret_dir="/init-secrets"
      # required
      for secret in shell gitaly registry rails-secrets gitlab-workhorse ; do
        mkdir -p "${secret_dir}/${secret}"
        cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
      done
      # optional
      for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do
        if [ -e "${config_dir}/${secret}" ]; then
          mkdir -p "${secret_dir}/${secret}"
          cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
        fi
      done
      # END gitlab.scripts.configure.secrets
    database.yml.erb: |
      
      production:
        main:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: true
          
          
        ci:
          adapter: postgresql
          encoding: unicode
          database: gitlab-prod
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %>
          host: "db-gitlab-rw.gitlab.svc"
          port: 5432
          connect_timeout: 
          keepalives: 
          keepalives_idle: 
          keepalives_interval: 
          keepalives_count: 
          tcp_user_timeout: 
          application_name: 
          prepared_statements: false
          database_tasks: false
          
          
      
    gitlab.yml.erb: |
      production: &base
        gitlab:
          host: code.nogweii.net
          https: true
          ssh_host: "code.nogweii.net"
          action_cable_allowed_origins: []
          max_request_duration_seconds: 57
          impersonation_enabled: 
          application_settings_cache_seconds: 60
          usage_ping_enabled: true
          seat_link_enabled: true
          username_changing_enabled: true
          issue_closing_pattern: 
          default_theme: 
          default_color_mode: 
          default_syntax_highlighting_theme: 
          default_projects_features:
            issues: true
            merge_requests: true
            wiki: true
            snippets: true
            builds: true
            container_registry: true
          signup_enabled: false
          graphql_timeout: 
          webhook_timeout: 
          trusted_proxies:
          time_zone: "UTC"        
          email_from: "gitlab@aether.earth"
          email_display_name: "GitLab"
          email_reply_to: "gitlab+noreply@aether.earth"
          email_subject_suffix: ""
        
        gravatar:
          plain_url: 
          ssl_url: 
        extra:
          
          
          
          
          
          
        # Consolidated object storage configuration
        ## property local configuration will override object_store
        object_store:
          enabled: true
          proxy_download: true
          connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %>
          objects:
            artifacts:
              bucket: gitlab-artifacts
              proxy_download: true
            lfs:
              bucket: gitlab-lfs
              proxy_download: true
            uploads:
              bucket: gitlab-uploads
              proxy_download: true
            packages:
              bucket: gitlab-packages
              proxy_download: true
            external_diffs:
              bucket: gitlab-external-diffs
              proxy_download: true
            terraform_state:
              bucket: gitlab-terraform-state
            dependency_proxy:
              bucket: gitlab-dependency-proxy
              proxy_download: true
            ci_secure_files:
              bucket: gitlab-ci-secure-files
            pages:
              bucket: gitlab-pages
        artifacts:
          enabled: true
        lfs:
          enabled: true
        uploads:
          enabled: true
        packages:
          enabled: true
        external_diffs:
          enabled: true
          when: 
        terraform_state:
          enabled: true
        ci_secure_files:
          enabled: true
        dependency_proxy:
          enabled: true
        
        sentry:
          enabled: false
          dsn: 
          clientside_dsn: 
          environment: 
        gitlab_docs:
          enabled: false
          host: ""
        
        sidekiq:
          log_format: json
        oidc_provider:
          openid_id_token_expire_in_seconds: 120
        
        pages:
          enabled: false
          access_control: false
          artifacts_server: true
          path: /srv/gitlab/shared/pages
          host: aether.page
          port: 443
          https: true
          secret_file: /etc/gitlab/pages/secret
          external_http: false
          external_https: false
          local_store:
            enabled: false
            path: 
          namespace_in_path: false
        
        mattermost:
          enabled: false
        gitlab_ci:
        ldap:
          enabled: false
        
        omniauth:
          enabled: true
          sync_profile_from_provider: ["openid_connect"]
          sync_profile_attributes: ["email","name"]
          allow_single_sign_on: ["openid_connect"]
          block_auto_created_users: false
          auto_link_ldap_user: false
          auto_link_saml_user: false
          auto_link_user: ["openid_connect"]
          external_providers: []
          providers:
            - <%= YAML.load_file("/etc/gitlab/omniauth/authentik-oidc/config.yaml").to_json() %>
        kerberos:
          enabled: false
          keytab: /etc/krb5.keytab
          use_dedicated_port: false
          port: 8443
          https: true
          simple_ldap_linking_allowed_realms: []
        
        duo_auth:
          enabled: false
          hostname: 
          integration_key: 
          secret_key: 
        shared:
        gitaly:
          client_path: /home/git/gitaly/bin
          token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %>
        repositories:
          storages: # You must have at least a `default` storage path.
            default:
              path: /var/opt/gitlab/repo
              gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075
            
        backup:
          path: "tmp/backups"   # Relative paths are relative to Rails.root (default: tmp/backups/)
        gitlab_kas:
          enabled: true
          secret_file: /etc/gitlab/kas/.gitlab_kas_secret
          external_url: "wss://k.code.nogweii.net"
          internal_url: "grpc://gitlab-kas.gitlab.svc:8153"
          
        
        
        
        # Zoekt secrets are present, load them.
        zoekt:
          username_file: /etc/gitlab/zoekt/.gitlab_zoekt_username
          password_file: /etc/gitlab/zoekt/.gitlab_zoekt_password
        gitlab_shell:
          path: /home/git/gitlab-shell/
          hooks_path: /home/git/gitlab-shell/hooks/
          upload_pack: true
          receive_pack: true
          ssh_port: 22222
          secret_file: /etc/gitlab/shell/.gitlab_shell_secret
        workhorse:
          secret_file: /etc/gitlab/gitlab-workhorse/secret
        git:
          bin_path: /usr/bin/git
        webpack:
        monitoring:
          ip_whitelist:
            - "0.0.0.0/0"
            - "::/0"
          web_exporter:
            enabled: true
            address: "0.0.0.0"
            port: 8083
          sidekiq_exporter:
        shutdown:
          blackout_seconds: <%= ENV["SHUTDOWN_BLACKOUT_SECONDS"] %>
        rack_attack:
          git_basic_auth:
        ## Registry Integration
        registry:
          enabled: true
          host: c.code.nogweii.net
          api_url: http://gitlab-registry.gitlab.svc:5000
          key: /etc/gitlab/registry/gitlab-registry.key
          issuer: gitlab-issuer
          notification_secret: <%= YAML.load_file("/etc/gitlab/registry/notificationSecret").flatten.first %>
        smartcard:
          enabled: false
          ca_file: '/etc/gitlab/rails-secrets/smartcard-ca.crt'
          client_certificate_required_host: smartcard.code.nogweii.net
          client_certificate_required_port: 443
          san_extensions: false
          required_for_git_access: false
        
      
    installation_type: |
      gitlab-helm-chart
      
    krb5.conf: 
    redis.action_cable.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    resque.yml.erb: |
      production:
        url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0
        
        id:
      
    session_store.yml.erb: |
      production:
          session_cookie_token_prefix: ""
      
    smtp_settings.rb: |
      smtp_settings = {
        address: "mail-relay.email.svc.cluster.local",
        port: 587,
        ca_file: "/etc/ssl/certs/ca-certificates.crt",
        
        enable_starttls_auto: false,
        tls: false,
        openssl_verify_mode: "peer",
        open_timeout: 30,
        read_timeout: 60
      }
      
      Rails.application.config.action_mailer.delivery_method = :smtp
      ActionMailer::Base.delivery_method = :smtp
      
      ActionMailer::Base.smtp_settings = smtp_settings
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-workhorse-default)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-workhorse-default
    namespace: gitlab
    labels:
      app: webservice
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    configure: |
      set -e
      mkdir -p /init-secrets-workhorse/gitlab-workhorse
      cp -v -r -L /init-config/gitlab-workhorse/secret /init-secrets-workhorse/gitlab-workhorse/secret
      mkdir -p /init-secrets-workhorse/redis
      cp -v -r -L /init-config/redis/redis-password /init-secrets-workhorse/redis/
      if [ -f /init-config/objectstorage/object_store ]; then
        mkdir -p /init-secrets-workhorse/objectstorage
        cp -v -r -L /init-config/objectstorage/object_store /init-secrets-workhorse/objectstorage/
      fi
      
    installation_type: |
      gitlab-helm-chart
      
    workhorse-config.toml.tpl: |
      shutdown_timeout = "61s"
      [redis]
      DB = 0
      URL = "redis://gitlab-redis-master.gitlab.svc:6379"
      Password = {% file.Read "/etc/gitlab/redis/redis-password" | strings.TrimSpace | data.ToJSON %}
      {%- $supported_providers := slice "AWS" "AzureRM" "Google" -%}
      {%- $connection := coll.Dict "provider" "" -%}
      {%- if file.Exists "/etc/gitlab/minio/accesskey" %}
        {%- $aws_access_key_id := file.Read "/etc/gitlab/minio/accesskey" | strings.TrimSpace -%}
        {%- $aws_secret_access_key := file.Read "/etc/gitlab/minio/secretkey" | strings.TrimSpace -%}
        {%- $connection = coll.Dict "provider" "AWS" "aws_access_key_id" $aws_access_key_id "aws_secret_access_key" $aws_secret_access_key -%}
      {%- end %}
      {%- if file.Exists "/etc/gitlab/objectstorage/object_store" %}
        {%- $connection = file.Read "/etc/gitlab/objectstorage/object_store" | strings.TrimSpace | data.YAML -%}
      {%- end %}
      {%- if has $supported_providers $connection.provider %}
      [object_storage]
      provider = "{% $connection.provider %}"
      {%-   if eq $connection.provider "AWS" %}
      {%-     $connection = coll.Merge $connection (coll.Dict "aws_access_key_id" "" "aws_secret_access_key" "" ) %}
      # AWS / S3 object storage configuration.
      [object_storage.s3]
      # access/secret can be blank!
      aws_access_key_id = {% $connection.aws_access_key_id | strings.TrimSpace | data.ToJSON %}
      aws_secret_access_key = {% $connection.aws_secret_access_key | strings.TrimSpace | data.ToJSON %}
      {%-   else if eq $connection.provider "AzureRM" %}
      {%-     $connection = coll.Merge $connection (coll.Dict "azure_storage_account_name" "" "azure_storage_access_key" "") %}
      # Azure Blob storage configuration.
      [object_storage.azurerm]
      azure_storage_account_name = {% $connection.azure_storage_account_name | strings.TrimSpace | data.ToJSON %}
      azure_storage_access_key = {% $connection.azure_storage_access_key | strings.TrimSpace | data.ToJSON %}
      {%-   else if eq $connection.provider "Google" %}
      # Google storage configuration.
      [object_storage.google]
      {% $connection | coll.Omit "provider" | data.ToTOML %}
      {%-   end %}
      {%- end %}
      [image_resizer]
      max_scaler_procs = 2
      max_filesize = 250000
      [[listeners]]
      network = "tcp"
      addr = ":8181"
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-webservice-tests)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-webservice-tests
    namespace: gitlab
    labels:
      app: webservice
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    test_login: |
      set -e
      echo 'Start Test'
      endpoint="https://code.nogweii.net"
      cookie_read="-c /tmp/test_login.cookie"
      cookie_readwrite="$cookie_read -b /tmp/test_login.cookie"
      
      signin_url="$endpoint/users/sign_in"
      echo "Login to create a session: $signin_url"
      csrf=$(curl -L $signin_url --fail -s $cookie_read | grep -Po '<meta.*name="csrf-token".*content="\K[a-zA-Z0-9\+=\-\/]*')
      curl -X POST $signin_url -s $cookie_readwrite -F "authenticity_token=$csrf" -F 'user[login]=root' -F "user[password]=$(cat /initial_root_password)"
      
      profile_url="$endpoint/-/profile"
      echo "Confirm session valid: $profile_url"
      profile_status=$(curl -L -s -o /tmp/profile_output -w "%{http_code}" $cookie_readwrite $profile_url)
      
      if [ "$profile_status" != "200" ]; then
        echo "Error: Session Invalid"
        cat /tmp/profile_output
        exit 1
      fi
      
      echo 'Test Passed'
      exit 0
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-redis-configuration)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-redis-configuration
    namespace: gitlab
    labels:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  data:
    master.conf: |
      dir /data
      # User-supplied master configuration:
      rename-command FLUSHDB ""
      rename-command FLUSHALL ""
      # End of master configuration
    redis.conf: |
      # User-supplied common configuration:
      # Enable AOF https://redis.io/topics/persistence#append-only-file
      appendonly yes
      # Disable RDB persistence, AOF persistence already enabled.
      save ""
      # End of common configuration
    replica.conf: |
      dir /data
      # User-supplied replica configuration:
      rename-command FLUSHDB ""
      rename-command FLUSHALL ""
      # End of replica configuration

(root level)  (v1/ConfigMap/gitlab/gitlab-redis-health)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-redis-health
    namespace: gitlab
    labels:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  data:
    ping_liveness_local.sh: |
      #!/bin/bash
      
      [[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")"
      [[ -n "$REDIS_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_PASSWORD"
      response=$(
        timeout -s 15 $1 \
        redis-cli \
          -h localhost \
          -p $REDIS_PORT \
          ping
      )
      if [ "$?" -eq "124" ]; then
        echo "Timed out"
        exit 1
      fi
      responseFirstWord=$(echo $response | head -n1 | awk '{print $1;}')
      if [ "$response" != "PONG" ] && [ "$responseFirstWord" != "LOADING" ] && [ "$responseFirstWord" != "MASTERDOWN" ]; then
        echo "$response"
        exit 1
      fi
    ping_liveness_local_and_master.sh: |
      script_dir="$(dirname "$0")"
      exit_status=0
      "$script_dir/ping_liveness_local.sh" $1 || exit_status=$?
      "$script_dir/ping_liveness_master.sh" $1 || exit_status=$?
      exit $exit_status
    ping_liveness_master.sh: |
      #!/bin/bash
      
      [[ -f $REDIS_MASTER_PASSWORD_FILE ]] && export REDIS_MASTER_PASSWORD="$(< "${REDIS_MASTER_PASSWORD_FILE}")"
      [[ -n "$REDIS_MASTER_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_MASTER_PASSWORD"
      response=$(
        timeout -s 15 $1 \
        redis-cli \
          -h $REDIS_MASTER_HOST \
          -p $REDIS_MASTER_PORT_NUMBER \
          ping
      )
      if [ "$?" -eq "124" ]; then
        echo "Timed out"
        exit 1
      fi
      responseFirstWord=$(echo $response | head -n1 | awk '{print $1;}')
      if [ "$response" != "PONG" ] && [ "$responseFirstWord" != "LOADING" ]; then
        echo "$response"
        exit 1
      fi
    ping_readiness_local.sh: |
      #!/bin/bash
      
      [[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")"
      [[ -n "$REDIS_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_PASSWORD"
      response=$(
        timeout -s 15 $1 \
        redis-cli \
          -h localhost \
          -p $REDIS_PORT \
          ping
      )
      if [ "$?" -eq "124" ]; then
        echo "Timed out"
        exit 1
      fi
      if [ "$response" != "PONG" ]; then
        echo "$response"
        exit 1
      fi
    ping_readiness_local_and_master.sh: |
      script_dir="$(dirname "$0")"
      exit_status=0
      "$script_dir/ping_readiness_local.sh" $1 || exit_status=$?
      "$script_dir/ping_readiness_master.sh" $1 || exit_status=$?
      exit $exit_status
    ping_readiness_master.sh: |
      #!/bin/bash
      
      [[ -f $REDIS_MASTER_PASSWORD_FILE ]] && export REDIS_MASTER_PASSWORD="$(< "${REDIS_MASTER_PASSWORD_FILE}")"
      [[ -n "$REDIS_MASTER_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_MASTER_PASSWORD"
      response=$(
        timeout -s 15 $1 \
        redis-cli \
          -h $REDIS_MASTER_HOST \
          -p $REDIS_MASTER_PORT_NUMBER \
          ping
      )
      if [ "$?" -eq "124" ]; then
        echo "Timed out"
        exit 1
      fi
      if [ "$response" != "PONG" ]; then
        echo "$response"
        exit 1
      fi
  


(root level)  (v1/ConfigMap/gitlab/gitlab-redis-scripts)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-redis-scripts
    namespace: gitlab
    labels:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  data:
    start-master.sh: |
      #!/bin/bash
      
      [[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")"
      if [[ -f /opt/bitnami/redis/mounted-etc/master.conf ]];then
          cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
      fi
      if [[ -f /opt/bitnami/redis/mounted-etc/redis.conf ]];then
          cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
      fi
      ARGS=("--port" "${REDIS_PORT}")
      ARGS+=("--requirepass" "${REDIS_PASSWORD}")
      ARGS+=("--masterauth" "${REDIS_PASSWORD}")
      ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
      ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
      exec redis-server "${ARGS[@]}"
  


(root level)  (v1/ConfigMap/gitlab/gitlab-registry)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-registry
    namespace: gitlab
    labels:
      app: registry
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    config.yml.tpl: |
      version: 0.1
      log:
        fields:
          service: registry
        level: info
      http:
        addr: :5000
        # `host` is not configurable
        # `prefix` is not configurable
        debug:
          addr: :5001
          prometheus:
            enabled: false
            path: /metrics
        draintimeout: 0
        headers:
          X-Content-Type-Options: [nosniff]
        secret: "HTTP_SECRET"
        relativeurls: false
      health:
        storagedriver:
          enabled: false
          interval: 10s
          threshold: 3
      auth:
        token:
          realm: https://code.nogweii.net/jwt/auth
          service: container_registry
          issuer: gitlab-issuer
          # This is provided from the initContainer execution, at a known path.
          rootcertbundle: /etc/docker/registry/certificate.crt
          autoredirect: false
      validation:
        disabled: true
        manifests:
          referencelimit: 0
          payloadsizelimit: 0
          urls:
            allow:
            deny:
      
      notifications:
      reporting:
      profiling:
      storage:
        maintenance:
          readonly:
            enabled: false
          uploadpurging:
            enabled: true
            age: 168h
            interval: 24h
            dryrun: false
        redirect:
          disable: true
          exceptions:
      middleware:
      
      database:
        enabled: true
        host: "db-gitlab-rw.gitlab.svc"
        port: 5432
        user: registry
        password: "DB_PASSWORD_FILE"
        dbname: gl-registry
        sslmode: disable
      gc:
        disabled: false
      redis:
      rate_limiter:
        enabled: false
      
    configure: |
      if [ -e /config/accesskey ] ; then
        sed -i -e 's@ACCESS_KEY@'"$(cat /config/accesskey)"'@' -e 's@SECRET_KEY@'"$(cat /config/secretkey)"'@' /registry/config.yml
      fi
      # Place the `http.secret` value from the kubernetes secret
      sed -i -e 's@HTTP_SECRET@'"$(cat /config/httpSecret)"'@' /registry/config.yml
      # Populate sensitive registry notification secrets in the config file
      if [ -d /config/notifications ]; then
        for i in /config/notifications/*; do
          filename=$(basename $i);
          sed -i -e 's@'"${filename}"'@'"$(cat $i)"'@' /registry/config.yml;
        done
      fi
      # Insert any provided `storage` block from kubernetes secret
      if [ -d /config/storage ]; then
        # Copy contents of storage secret(s)
        mkdir -p /registry/storage
        cp -v -r -L /config/storage/* /registry/storage/
        # Ensure there is a new line in the end
        echo '' >> /registry/storage/config
        # Default `delete.enabled: true` if not present.
        ## Note: busybox grep doesn't support multiline, so we chain `egrep`.
        if ! $(egrep -A1 '^delete:\s*$' /registry/storage/config | egrep -q '\s{2,4}enabled:') ; then
          echo 'delete:' >> /registry/storage/config
          echo '  enabled: true' >> /registry/storage/config
        fi
        # Indent /registry/storage/config 2 spaces before inserting into config.yml
        sed -i 's/^/  /' /registry/storage/config
        # Insert into /registry/config.yml after `storage:`
        sed -i '/^storage:/ r /registry/storage/config' /registry/config.yml
        # Remove the now extraneous `config` file
        rm /registry/storage/config
      fi
      # Copy any middleware.storage if present
      if [ -d /config/middleware.storage ]; then
        cp -v -r -L /config/middleware.storage  /registry/middleware.storage
      fi
      # Set to known path, to used ConfigMap
      cat /config/certificate.crt > /registry/certificate.crt
      # Copy the optional profiling keyfile to the expected location
      if [ -f /config/profiling-key.json ]; then
        cp /config/profiling-key.json /registry/profiling-key.json
      fi
      # Insert Database password, if enabled
      if [ -f /config/database_password ] ; then
        sed -i -e 's@DB_PASSWORD_FILE@'"$(cat /config/database_password)"'@' /registry/config.yml
      fi
      # Insert Redis(cache) password, if enabled
      if [ -f /config/registry/redis-password ] ; then
        sed -i -e 's@REDIS_CACHE_PASSWORD@'"$(cat /config/registry/redis-password)"'@' /registry/config.yml
      fi
      # Insert Redis(rate-limiter) password, if enabled
      if [ -f /config/registry/redis-rateLimiting-password ] ; then
        sed -i -e 's@REDIS_RATE_LIMITING_PASSWORD@'"$(cat /config/registry/redis-rateLimiting-password)"'@' /registry/config.yml
      fi
      # Insert Redis(load balancing) password, if enabled
      if [ -f /config/registry/redis-loadBalancing-password ] ; then
        sed -i -e 's@REDIS_LOAD_BALANCING_PASSWORD@'"$(cat /config/registry/redis-loadBalancing-password)"'@' /registry/config.yml
      fi
      # Copy the database TLS connection files to the expected location and set permissions
      if [ -d /config/ssl ]; then
        cp -r /config/ssl/ /registry/ssl
        chmod 700 /registry/ssl
        chmod 600 /registry/ssl/*.pem
      fi
      # Copy TLS certificates if present
      if [ -d /config/tls ]; then
        cp -r /config/tls/ /registry/tls
        chmod 700 /registry/tls
        chmod 600 /registry/tls/*
      fi
    migrations-config.yml.tpl: |
      version: 0.1
      log:
        fields:
          service: registry
        level: info
      
      database:
        enabled: true
        host: "db-gitlab-rw.gitlab.svc"
        port: 5432
        user: registry
        password: "DB_PASSWORD_FILE"
        dbname: gl-registry
        sslmode: disable
  


(root level)  (v1/ConfigMap/gitlab/gitlab-gitlab-chart-info)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitlab-chart-info
    namespace: gitlab
    labels:
      app: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  data:
    gitlabChartVersion: "9.8.2"
    gitlabVersion: "18.8.2"

(root level)  (rbac.authorization.k8s.io/v1/ClusterRole/gitlab-gitlab-runner)
+ one document added:
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: gitlab-gitlab-runner
    labels:
      app: gitlab-gitlab-runner
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  rules:
  - resources:
    - events
    apiGroups:
    -
    verbs:
    - list
    - watch
  - resources:
    - namespaces
    apiGroups:
    -
    verbs:
    - create
    - delete
  - resources:
    - pods
    apiGroups:
    -
    verbs:
    - create
    - delete
    - get
    - list
    - watch
  - resources:
    - pods/attach
    - pods/exec
    apiGroups:
    -
    verbs:
    - get
    - create
    - patch
    - delete
  - resources:
    - pods/log
    apiGroups:
    -
    verbs:
    - get
    - list
  - resources:
    - secrets
    apiGroups:
    -
    verbs:
    - create
    - delete
    - get
    - update
  - resources:
    - serviceaccounts
    apiGroups:
    -
    verbs:
    - get
  - resources:
    - services
    apiGroups:
    -
    verbs:
    - create
    - get

(root level)  (rbac.authorization.k8s.io/v1/ClusterRoleBinding/gitlab-gitlab-runner)
+ one document added:
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: gitlab-gitlab-runner
    labels:
      app: gitlab-gitlab-runner
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  roleRef:
    name: gitlab-gitlab-runner
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
  subjects:
  - name: gitlab-runner-access
    kind: ServiceAccount
    namespace: gitlab

(root level)  (v1/Service/gitlab/gitlab-gitaly)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-gitaly
    namespace: gitlab
    labels:
      app: gitaly
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    type: ClusterIP
    clusterIP: None
    selector:
      app: gitaly
      release: gitlab
    ports:
    - name: tcp-gitaly
      port: 8075
      targetPort: grpc-gitaly
    - name: http-metrics
      port: 9236
      targetPort: http-metrics

(root level)  (v1/Service/gitlab/gitlab-gitlab-exporter)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-gitlab-exporter
    namespace: gitlab
    labels:
      app: gitlab-exporter
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    type: ClusterIP
    selector:
      app: gitlab-exporter
      release: gitlab
    ports:
    - name: http-metrics
      port: 9168
      protocol: TCP
      targetPort: http-metrics

(root level)  (v1/Service/gitlab/gitlab-gitlab-shell)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-gitlab-shell
    namespace: gitlab
    annotations:
      tailscale.com/expose: "true"
      tailscale.com/hostname: gitlab-shell
    labels:
      app: gitlab-shell
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    type: ClusterIP
    selector:
      app: gitlab-shell
      release: gitlab
    ports:
    - name: ssh
      port: 22222
      protocol: TCP
      targetPort: 2222

(root level)  (v1/Service/gitlab/gitlab-kas)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-kas
    namespace: gitlab
    labels:
      app: kas
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    type: ClusterIP
    selector:
      app: kas
      release: gitlab
    ports:
    - name: tcp-kas-external-api
      port: 8150
      protocol: TCP
      targetPort: 8150
    - name: tcp-kas-internal-api
      port: 8153
      protocol: TCP
      targetPort: 8153
    - name: tcp-kas-k8s-api
      port: 8154
      protocol: TCP
      targetPort: 8154
    - name: http-metrics
      port: 8151
      protocol: TCP
      targetPort: http-metrics

(root level)  (v1/Service/gitlab/gitlab-webservice-default)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-webservice-default
    namespace: gitlab
    labels:
      app: webservice
      gitlab.com/webservice-name: default
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    type: ClusterIP
    selector:
      app: webservice
      gitlab.com/webservice-name: default
      release: gitlab
    ports:
    - name: http-webservice
      port: 8080
      protocol: TCP
      targetPort: http-webservice
    - name: http-workhorse
      port: 8181
      protocol: TCP
      targetPort: http-workhorse
    - name: http-metrics-ws
      port: 8083
      protocol: TCP
      targetPort: http-metrics-ws

(root level)  (v1/Service/gitlab/gitlab-redis-headless)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-redis-headless
    namespace: gitlab
    labels:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  spec:
    type: ClusterIP
    clusterIP: None
    selector:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/name: redis
    ports:
    - name: tcp-redis
      port: 6379
      targetPort: redis

(root level)  (v1/Service/gitlab/gitlab-redis-master)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-redis-master
    namespace: gitlab
    labels:
      app.kubernetes.io/component: master
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  spec:
    type: ClusterIP
    internalTrafficPolicy: Cluster
    sessionAffinity: None
    selector:
      app.kubernetes.io/component: master
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/name: redis
    ports:
    - name: tcp-redis
      nodePort: null
      port: 6379
      targetPort: redis

(root level)  (v1/Service/gitlab/gitlab-redis-metrics)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-redis-metrics
    namespace: gitlab
    labels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  spec:
    type: ClusterIP
    selector:
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/name: redis
    ports:
    - name: http-metrics
      port: 9121
      protocol: TCP
      targetPort: metrics

(root level)  (v1/Service/gitlab/gitlab-registry)
+ one document added:
  apiVersion: v1
  kind: Service
  metadata:
    name: gitlab-registry
    namespace: gitlab
    labels:
      app: registry
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    type: ClusterIP
    selector:
      app: registry
      release: gitlab
    ports:
    - name: http-registry
      port: 5000
      protocol: TCP
      targetPort: http

(root level)  (apps/v1/Deployment/gitlab/gitlab-gitlab-runner)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-gitlab-runner
    namespace: gitlab
    labels:
      app: gitlab-gitlab-runner
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: gitlab-gitlab-runner
    template:
      metadata:
        annotations:
          gitlab.com/prometheus_port: "9252"
          gitlab.com/prometheus_scrape: "true"
        labels:
          app: gitlab-gitlab-runner
          heritage: Helm
          release: gitlab
      spec:
        containers:
        - name: gitlab-gitlab-runner
          image: "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v18.7.2"
          imagePullPolicy: IfNotPresent
          resources: {}
          command:
          - /usr/bin/dumb-init
          - "--"
          - /bin/bash
          - /configmaps/entrypoint
          ports:
          - name: metrics
            containerPort: 9252
          volumeMounts:
          - name: projected-secrets
            mountPath: /secrets
          - name: etc-gitlab-runner
            mountPath: /home/gitlab-runner/.gitlab-runner
          - name: configmaps
            mountPath: /configmaps
          livenessProbe:
            exec:
              command:
              - /bin/bash
              - /configmaps/check-live
              - "3"
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 4
          readinessProbe:
            exec:
              command:
              - /usr/bin/pgrep
              - "gitlab.*runner"
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 4
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
          env:
          - name: CI_SERVER_URL
            value: "https://code.nogweii.net"
          - name: RUNNER_EXECUTOR
            value: kubernetes
          - name: REGISTER_LOCKED
            value: "false"
          - name: RUNNER_TAG_LIST
            value:
          - name: KUBERNETES_NAMESPACE
            value: gitlab
          - name: CACHE_S3_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: gitlab-object-store-configs
                key: runner_access_key_id
          - name: CACHE_S3_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: gitlab-object-store-configs
                key: runner_secret_access_key
          - name: SESSION_SERVER_ADDRESS
            value: null
          lifecycle:
            preStop:
              exec:
                command:
                - /entrypoint
                - unregister
                - "--all-runners"
        securityContext:
          fsGroup: 65533
          runAsUser: 100
          seccompProfile:
            type: RuntimeDefault
        serviceAccountName: gitlab-runner-access
        terminationGracePeriodSeconds: 3600
        volumes:
        - name: runner-secrets
          emptyDir:
            medium: Memory
        - name: etc-gitlab-runner
          emptyDir:
            medium: Memory
        - name: projected-secrets
          projected:
            sources:
            - secret:
                name: gitlab-gitlab-runner-secret
                items:
                - key: runner-registration-token
                  path: runner-registration-token
                - key: runner-token
                  path: runner-token
        - name: configmaps
          configMap:
            name: gitlab-gitlab-runner

(root level)  (apps/v1/Deployment/gitlab/gitlab-gitlab-exporter)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-gitlab-exporter
    namespace: gitlab
    labels:
      app: gitlab-exporter
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: gitlab-exporter
        release: gitlab
    template:
      metadata:
        annotations:
          gitlab.com/prometheus_path: /metrics
          gitlab.com/prometheus_port: "9168"
          gitlab.com/prometheus_scrape: "true"
          prometheus.io/path: /metrics
          prometheus.io/port: "9168"
          prometheus.io/scrape: "true"
        labels:
          app: gitlab-exporter
          app.kubernetes.io/name: gitlab
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: gitlab-exporter
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: gitlab-exporter
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter:16.2.0"
          env:
          - name: TZ
            value: UTC
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab-exporter/templates
          - name: CONFIG_DIRECTORY
            value: /etc/gitlab-exporter
          - name: MALLOC_CONF
            value: "dirty_decay_ms:0,muzzy_decay_ms:0"
          - name: RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO
            value: "0.111"
          - name: RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO
            value: "0.055"
          - name: RUBY_GC_HEAP_INIT_SLOTS
            value: "80000"
          ports:
          - name: http-metrics
            containerPort: 9168
          resources:
            requests:
              cpu: 75m
              memory: 100M
          volumeMounts:
          - name: gitlab-exporter-config
            mountPath: /var/opt/gitlab-exporter/templates/gitlab-exporter.yml.erb
            subPath: gitlab-exporter.yml.erb
          - name: gitlab-exporter-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          livenessProbe:
            exec:
              command:
              - pgrep
              - "-f"
              - gitlab-exporter
          readinessProbe:
            exec:
              command:
              - pgrep
              - "-f"
              - gitlab-exporter
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/bash
                - "-c"
                - "pkill -f 'gitlab-exporter'"
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          - name: MALLOC_CONF
            value: "dirty_decay_ms:0,muzzy_decay_ms:0"
          - name: RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO
            value: "0.111"
          - name: RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO
            value: "0.055"
          - name: RUBY_GC_HEAP_INIT_SLOTS
            value: "80000"
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - sh
          - /config/configure
          env:
          - name: TZ
            value: UTC
          - name: MALLOC_CONF
            value: "dirty_decay_ms:0,muzzy_decay_ms:0"
          - name: RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO
            value: "0.111"
          - name: RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO
            value: "0.055"
          - name: RUBY_GC_HEAP_INIT_SLOTS
            value: "80000"
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: gitlab-exporter-config
            mountPath: /config
            readOnly: true
          - name: init-gitlab-exporter-secrets
            mountPath: /init-config
            readOnly: true
          - name: gitlab-exporter-secrets
            mountPath: /init-secrets
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        volumes:
        - name: gitlab-exporter-config
          configMap:
            name: gitlab-gitlab-exporter
        - name: init-gitlab-exporter-secrets
          projected:
            defaultMode: 256
            sources:
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-ci
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-main
            - secret:
                name: gitlab-redis-secret
                items:
                - key: secret
                  path: redis/redis-password
        - name: gitlab-exporter-secrets
          emptyDir:
            medium: Memory
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (apps/v1/Deployment/gitlab/gitlab-gitlab-shell)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-gitlab-shell
    namespace: gitlab
    labels:
      app: gitlab-shell
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    selector:
      matchLabels:
        app: gitlab-shell
        release: gitlab
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
        labels:
          app: gitlab-shell
          app.kubernetes.io/name: gitlab
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: gitlab-shell
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: gitlab-shell
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v14.45.5"
          env:
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /etc/gitlab-shell
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab-shell
          - name: KEYS_DIRECTORY
            value: /etc/gitlab-secrets/ssh
          - name: SSH_DAEMON
            value: openssh
          - name: GODEBUG
            value: "tlsmlkem=0,tlskyber=0"
          - name: TZ
            value: UTC
          ports:
          - name: ssh
            containerPort: 2222
          readinessProbe:
            failureThreshold: 2
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            tcpSocket:
              port: 2222
            timeoutSeconds: 3
          resources:
            requests:
              cpu: 0
              memory: 6M
          volumeMounts:
          - name: shell-config
            mountPath: /etc/gitlab-shell
          - name: shell-secrets
            mountPath: /etc/gitlab-secrets
            readOnly: true
          - name: shell-config
            mountPath: /etc/krb5.conf
            readOnly: true
            subPath: krb5.conf
          - name: sshd-config
            mountPath: /etc/ssh/sshd_config
            readOnly: true
            subPath: sshd_config
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          livenessProbe:
            exec:
              command:
              - /scripts/healthcheck
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - sh
          - /config/configure
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: shell-config
            mountPath: /config
            readOnly: true
          - name: shell-init-secrets
            mountPath: /init-config
            readOnly: true
          - name: shell-secrets
            mountPath: /init-secrets
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        terminationGracePeriodSeconds: 30
        volumes:
        - name: shell-config
          configMap:
            name: gitlab-gitlab-shell
        - name: sshd-config
          configMap:
            name: gitlab-gitlab-shell-sshd
        - name: shell-init-secrets
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: gitlab-gitlab-shell-host-keys
            - secret:
                name: gitlab-gitlab-shell-secret
                items:
                - key: secret
                  path: shell/.gitlab_shell_secret
        - name: shell-secrets
          emptyDir:
            medium: Memory
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (apps/v1/Deployment/gitlab/gitlab-kas)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-kas
    namespace: gitlab
    labels:
      app: kas
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    selector:
      matchLabels:
        app: kas
        release: gitlab
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
          gitlab.com/prometheus_path: /metrics
          gitlab.com/prometheus_port: "8151"
          gitlab.com/prometheus_scrape: "true"
          prometheus.io/path: /metrics
          prometheus.io/port: "8151"
          prometheus.io/scrape: "true"
        labels:
          app: kas
          app.kubernetes.io/name: gitlab
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: kas
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: kas
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-kas:v18.8.2"
          args:
          - "--configuration-file=/etc/kas/config.yaml"
          env:
          - name: OWN_PRIVATE_API_HOST
            value: gitlab-kas.gitlab.svc
          - name: GODEBUG
            value: "tlsmlkem=0,tlskyber=0"
          - name: TZ
            value: UTC
          livenessProbe:
            httpGet:
              path: /liveness
              port: 8151
            initialDelaySeconds: 15
            periodSeconds: 20
          ports:
          - name: kas
            containerPort: 8150
          - name: kas-k8s-api
            containerPort: 8154
          - name: kas-private-api
            containerPort: 8155
          - name: http-metrics
            containerPort: 8151
          readinessProbe:
            httpGet:
              path: /readiness
              port: 8151
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
          volumeMounts:
          - name: init-etc-kas
            mountPath: /etc/kas
            readOnly: true
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 65532
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        securityContext:
          fsGroup: 65532
          runAsGroup: 65532
          runAsUser: 65532
          seccompProfile:
            type: RuntimeDefault
        terminationGracePeriodSeconds: 300
        volumes:
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
        - name: init-etc-kas
          projected:
            defaultMode: 288
            sources:
            - configMap:
                name: gitlab-kas
            - secret:
                name: gitlab-gitlab-kas-secret
                items:
                - key: kas_shared_secret
                  path: .gitlab_kas_secret
            - secret:
                name: gitlab-kas-private-api
                items:
                - key: kas_private_api_secret
                  path: .gitlab_kas_private_api_secret
            - secret:
                name: gitlab-kas-websocket-token
                items:
                - key: kas_websocket_token_secret
                  path: .gitlab_kas_websocket_token_secret
            - secret:
                name: gitlab-redis-secret
                items:
                - key: secret
                  path: redis/redis-password

(root level)  (apps/v1/Deployment/gitlab/gitlab-sidekiq-all-in-1-v2)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-sidekiq-all-in-1-v2
    namespace: gitlab
    labels:
      app: sidekiq
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      queue-pod-name: all-in-1
      release: gitlab
  spec:
    selector:
      matchLabels:
        app: sidekiq
        queue-pod-name: all-in-1
        release: gitlab
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
          gitlab.com/prometheus_path: /metrics
          gitlab.com/prometheus_port: "3807"
          gitlab.com/prometheus_scrape: "true"
          prometheus.io/path: /metrics
          prometheus.io/port: "3807"
          prometheus.io/scrape: "true"
        labels:
          app: sidekiq
          app.kubernetes.io/name: gitlab
          heritage: Helm
          queue-pod-name: all-in-1
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: sidekiq
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: sidekiq
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v18.8.2"
          env:
          - name: prometheus_multiproc_dir
            value: /metrics
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: SIDEKIQ_CONCURRENCY
            value: "20"
          - name: SIDEKIQ_TIMEOUT
            value: "25"
          - name: SIDEKIQ_QUEUES
            value: null
          - name: SIDEKIQ_DAEMON_MEMORY_KILLER
            value: "1"
          - name: SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL
            value: "3"
          - name: SIDEKIQ_MEMORY_KILLER_MAX_RSS
            value: "2000000"
          - name: SIDEKIQ_MEMORY_KILLER_GRACE_TIME
            value: "900"
          - name: SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT
            value: "30"
          - name: ENABLE_BOOTSNAP
            value: "1"
          - name: TZ
            value: UTC
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /liveness
              port: 3808
            initialDelaySeconds: 20
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 30
          ports:
          - name: http-metrics
            containerPort: 3807
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /readiness
              port: 3808
            initialDelaySeconds: 0
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 2
          resources:
            requests:
              cpu: 900m
              memory: 2G
          volumeMounts:
          - name: sidekiq-metrics
            mountPath: /metrics
          - name: sidekiq-config
            mountPath: /var/opt/gitlab/templates
            readOnly: true
          - name: sidekiq-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: sidekiq-secrets
            mountPath: /srv/gitlab/config/secrets.yml
            subPath: rails-secrets/secrets.yml
          - name: sidekiq-config
            mountPath: /etc/krb5.conf
            subPath: krb5.conf
          - name: sidekiq-config
            mountPath: /srv/gitlab/config/initializers/smtp_settings.rb
            subPath: smtp_settings.rb
          - name: sidekiq-config
            mountPath: /srv/gitlab/INSTALLATION_TYPE
            subPath: installation_type
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/bash
                - "-c"
                - "pkill -f 'sidekiq'"
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - sh
          - /config/configure
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: sidekiq-config
            mountPath: /config
            readOnly: true
          - name: init-sidekiq-secrets
            mountPath: /init-config
            readOnly: true
          - name: sidekiq-secrets
            mountPath: /init-secrets
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: dependencies
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v18.8.2"
          args:
          - /scripts/wait-for-deps
          env:
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: SIDEKIQ_CONCURRENCY
            value: "20"
          - name: SIDEKIQ_TIMEOUT
            value: "25"
          - name: ENABLE_BOOTSNAP
            value: "1"
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          - name: sidekiq-config
            mountPath: /var/opt/gitlab/templates
            readOnly: true
          - name: sidekiq-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: sidekiq-secrets
            mountPath: /srv/gitlab/config/secrets.yml
            readOnly: true
            subPath: rails-secrets/secrets.yml
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        terminationGracePeriodSeconds: 30
        volumes:
        - name: sidekiq-metrics
          emptyDir:
            medium: Memory
        - name: sidekiq-config
          projected:
            sources:
            - configMap:
                name: gitlab-sidekiq
        - name: init-sidekiq-secrets
          projected:
            defaultMode: 256
            sources:
            - secret:
                name: gitlab-rails-secret
                items:
                - key: secrets.yml
                  path: rails-secrets/secrets.yml
            - secret:
                name: gitlab-gitaly-secret
                items:
                - key: token
                  path: gitaly/gitaly_token
            - secret:
                name: gitlab-redis-secret
                items:
                - key: secret
                  path: redis/redis-password
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-ci
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-main
            - secret:
                name: gitlab-registry-secret
                items:
                - key: registry-auth.key
                  path: registry/gitlab-registry.key
            - secret:
                name: gitlab-registry-notification
                items:
                - key: secret
                  path: registry/notificationSecret
            - secret:
                name: gitlab-gitlab-kas-secret
                items:
                - key: kas_shared_secret
                  path: kas/.gitlab_kas_secret
            - secret:
                name: gitlab-zoekt-basicauth
                optional: true
                items:
                - key: gitlab_username
                  path: zoekt/.gitlab_zoekt_username
                - key: gitlab_password
                  path: zoekt/.gitlab_zoekt_password
            - secret:
                name: gitlab-object-store-configs
                items:
                - key: connection.yaml
                  path: objectstorage/object_store
            - secret:
                name: authentik-oidc
                items:
                - key: config.yaml
                  path: omniauth/authentik-oidc/config.yaml
        - name: sidekiq-secrets
          emptyDir:
            medium: Memory
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (apps/v1/Deployment/gitlab/gitlab-toolbox)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-toolbox
    namespace: gitlab
    labels:
      app: toolbox
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: toolbox
        release: gitlab
    strategy:
      type: Recreate
      rollingUpdate: null
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
        labels:
          app: toolbox
          app.kubernetes.io/name: gitlab
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: gitaly
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: toolbox
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce:v18.8.2"
          args:
          - /bin/bash
          - "-c"
          - "cp -v -r -L /etc/gitlab/.s3cfg $HOME/.s3cfg && while sleep 3600; do :; done"
          env:
          - name: ARTIFACTS_BUCKET_NAME
            value: gitlab-artifacts
          - name: REGISTRY_BUCKET_NAME
            value: gitlab-container-registry
          - name: LFS_BUCKET_NAME
            value: gitlab-lfs
          - name: UPLOADS_BUCKET_NAME
            value: gitlab-uploads
          - name: PACKAGES_BUCKET_NAME
            value: gitlab-packages
          - name: EXTERNAL_DIFFS_BUCKET_NAME
            value: gitlab-external-diffs
          - name: TERRAFORM_STATE_BUCKET_NAME
            value: gitlab-terraform-state
          - name: CI_SECURE_FILES_BUCKET_NAME
            value: gitlab-ci-secure-files
          - name: BACKUP_BUCKET_NAME
            value: gitlab-backup
          - name: BACKUP_BACKEND
            value: s3
          - name: TMP_BUCKET_NAME
            value: gitlab-tmp-storage
          - name: PAGES_BUCKET_NAME
            value: gitlab-pages
          - name: ENABLE_BOOTSNAP
            value: "1"
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
              memory: 350M
          volumeMounts:
          - name: toolbox-config
            mountPath: /var/opt/gitlab/templates
          - name: toolbox-config
            mountPath: /etc/krb5.conf
            subPath: krb5.conf
          - name: toolbox-config
            mountPath: /srv/gitlab/config/initializers/smtp_settings.rb
            subPath: smtp_settings.rb
          - name: toolbox-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: toolbox-secrets
            mountPath: /srv/gitlab/config/secrets.yml
            subPath: rails-secrets/secrets.yml
          - name: toolbox-tmp
            mountPath: /srv/gitlab/tmp
            readOnly: false
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - sh
          - /config/configure
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: toolbox-config
            mountPath: /config
            readOnly: true
          - name: init-toolbox-secrets
            mountPath: /init-config
            readOnly: true
          - name: toolbox-secrets
            mountPath: /init-secrets
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        securityContext:
          fsGroup: 1000
          runAsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        volumes:
        - name: toolbox-config
          projected:
            sources:
            - configMap:
                name: gitlab-toolbox
        - name: toolbox-tmp
          emptyDir: {}
        - name: init-toolbox-secrets
          projected:
            defaultMode: 256
            sources:
            - secret:
                name: gitlab-rails-secret
                items:
                - key: secrets.yml
                  path: rails-secrets/secrets.yml
            - secret:
                name: gitlab-gitlab-shell-secret
                items:
                - key: secret
                  path: shell/.gitlab_shell_secret
            - secret:
                name: gitlab-gitaly-secret
                items:
                - key: token
                  path: gitaly/gitaly_token
            - secret:
                name: gitlab-redis-secret
                items:
                - key: secret
                  path: redis/redis-password
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-ci
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-main
            - secret:
                name: gitlab-registry-secret
                items:
                - key: registry-auth.key
                  path: registry/gitlab-registry.key
            - secret:
                name: gitlab-registry-notification
                items:
                - key: secret
                  path: registry/notificationSecret
            - secret:
                name: gitlab-object-store-configs
                items:
                - key: s3cmd.cfg
                  path: objectstorage/.s3cfg
            - secret:
                name: gitlab-gitlab-kas-secret
                items:
                - key: kas_shared_secret
                  path: kas/.gitlab_kas_secret
            - secret:
                name: gitlab-zoekt-basicauth
                optional: true
                items:
                - key: gitlab_username
                  path: zoekt/.gitlab_zoekt_username
                - key: gitlab_password
                  path: zoekt/.gitlab_zoekt_password
            - secret:
                name: gitlab-object-store-configs
                items:
                - key: connection.yaml
                  path: objectstorage/object_store
            - secret:
                name: authentik-oidc
                items:
                - key: config.yaml
                  path: omniauth/authentik-oidc/config.yaml
        - name: toolbox-secrets
          emptyDir:
            medium: Memory
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (apps/v1/Deployment/gitlab/gitlab-webservice-default)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-webservice-default
    namespace: gitlab
    labels:
      app: webservice
      app.kubernetes.io/name: gitlab
      gitlab.com/webservice-name: default
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    selector:
      matchLabels:
        app: webservice
        gitlab.com/webservice-name: default
        release: gitlab
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
          gitlab.com/prometheus_path: /metrics
          gitlab.com/prometheus_port: "8083"
          gitlab.com/prometheus_scrape: "true"
          prometheus.io/path: /metrics
          prometheus.io/port: "8083"
          prometheus.io/scrape: "true"
        labels:
          app: webservice
          app.kubernetes.io/name: gitlab
          gitlab.com/webservice-name: default
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: webservice
                    gitlab.com/webservice-name: default
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: webservice
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce:v18.8.2"
          env:
          - name: TZ
            value: UTC
          - name: GITLAB_WEBSERVER
            value: puma
          - name: TMPDIR
            value: /tmp/gitlab
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: prometheus_multiproc_dir
            value: /metrics
          - name: ENABLE_BOOTSNAP
            value: "1"
          - name: WORKER_PROCESSES
            value: "2"
          - name: WORKER_TIMEOUT
            value: "60"
          - name: INTERNAL_PORT
            value: "8080"
          - name: PUMA_THREADS_MIN
            value: "4"
          - name: PUMA_THREADS_MAX
            value: "4"
          - name: PUMA_WORKER_MAX_MEMORY
            value: 
          - name: DISABLE_PUMA_WORKER_KILLER
            value: "true"
          - name: BIND_IP6
            value: "false"
          - name: PUMA_CONTROL_PORT
            value: "9293"
          - name: SHUTDOWN_BLACKOUT_SECONDS
            value: "10"
          - name: WORKHORSE_ARCHIVE_CACHE_DISABLED
            value: "true"
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /-/liveness
              port: 8080
            initialDelaySeconds: 20
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 30
          ports:
          - name: http-webservice
            containerPort: 8080
          - name: http-metrics-ws
            containerPort: 8083
          readinessProbe:
            failureThreshold: 2
            httpGet:
              path: /-/readiness
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 2
          resources:
            requests:
              cpu: 300m
              memory: 2.5G
          volumeMounts:
          - name: webservice-metrics
            mountPath: /metrics
          - name: webservice-config
            mountPath: /var/opt/gitlab/templates
          - name: webservice-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: webservice-secrets
            mountPath: /srv/gitlab/config/secrets.yml
            subPath: rails-secrets/secrets.yml
          - name: webservice-config
            mountPath: /etc/krb5.conf
            subPath: krb5.conf
          - name: webservice-config
            mountPath: /srv/gitlab/config/initializers/smtp_settings.rb
            subPath: smtp_settings.rb
          - name: webservice-config
            mountPath: /srv/gitlab/INSTALLATION_TYPE
            subPath: installation_type
          - name: shared-upload-directory
            mountPath: /srv/gitlab/public/uploads/tmp
            readOnly: false
          - name: shared-tmp
            mountPath: /tmp
            readOnly: false
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/bash
                - "-c"
                - "pkill -SIGINT -o ruby"
        - name: gitlab-workhorse
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce:v18.8.2"
          env:
          - name: TZ
            value: UTC
          - name: GODEBUG
            value: "tlsmlkem=0,tlskyber=0"
          - name: TMPDIR
            value: /tmp/gitlab
          - name: GITLAB_WORKHORSE_AUTH_BACKEND
            value: "http://localhost:8080"
          - name: GITLAB_WORKHORSE_EXTRA_ARGS
            value: 
          - name: GITLAB_WORKHORSE_LISTEN_PORT
            value: "8181"
          - name: GITLAB_WORKHORSE_LOG_FORMAT
            value: json
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: SHUTDOWN_BLACKOUT_SECONDS
            value: "10"
          - name: PUMA_CONTROL_PORT
            value: "9293"
          ports:
          - name: http-workhorse
            containerPort: 8181
          resources:
            requests:
              cpu: 100m
              memory: 100M
          volumeMounts:
          - name: workhorse-config
            mountPath: /var/opt/gitlab/templates
          - name: workhorse-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: shared-upload-directory
            mountPath: /srv/gitlab/public/uploads/tmp
            readOnly: false
          - name: shared-tmp
            mountPath: /tmp
            readOnly: false
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          livenessProbe:
            exec:
              command:
              - /scripts/healthcheck
            failureThreshold: 3
            initialDelaySeconds: 20
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 30
          readinessProbe:
            exec:
              command:
              - /scripts/healthcheck
            failureThreshold: 3
            initialDelaySeconds: 0
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 2
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/bash
                - "-c"
                - "sleep $SHUTDOWN_BLACKOUT_SECONDS"
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          args:
          - "-c"
          - "sh -x /config-webservice/configure ; sh -x /config-workhorse/configure ; mkdir -p -m 3770 /tmp/gitlab"
          command:
          - sh
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: webservice-config
            mountPath: /config-webservice
            readOnly: true
          - name: workhorse-config
            mountPath: /config-workhorse
            readOnly: true
          - name: init-webservice-secrets
            mountPath: /init-config
            readOnly: true
          - name: webservice-secrets
            mountPath: /init-secrets
            readOnly: false
          - name: workhorse-secrets
            mountPath: /init-secrets-workhorse
            readOnly: false
          - name: shared-tmp
            mountPath: /tmp
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: dependencies
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce:v18.8.2"
          args:
          - /scripts/wait-for-deps
          env:
          - name: TZ
            value: UTC
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: WORKHORSE_ARCHIVE_CACHE_DISABLED
            value: "1"
          - name: ENABLE_BOOTSNAP
            value: "1"
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          - name: webservice-config
            mountPath: /var/opt/gitlab/templates
          - name: webservice-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: webservice-secrets
            mountPath: /srv/gitlab/config/secrets.yml
            readOnly: true
            subPath: rails-secrets/secrets.yml
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        volumes:
        - name: shared-tmp
          emptyDir: {}
        - name: webservice-metrics
          emptyDir:
            medium: Memory
        - name: webservice-config
          configMap:
            name: gitlab-webservice
        - name: workhorse-config
          configMap:
            name: gitlab-workhorse-default
        - name: init-webservice-secrets
          projected:
            defaultMode: 256
            sources:
            - secret:
                name: gitlab-rails-secret
                items:
                - key: secrets.yml
                  path: rails-secrets/secrets.yml
            - secret:
                name: gitlab-gitlab-shell-secret
                items:
                - key: secret
                  path: shell/.gitlab_shell_secret
            - secret:
                name: gitlab-gitaly-secret
                items:
                - key: token
                  path: gitaly/gitaly_token
            - secret:
                name: gitlab-redis-secret
                items:
                - key: secret
                  path: redis/redis-password
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-ci
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-main
            - secret:
                name: gitlab-registry-secret
                items:
                - key: registry-auth.key
                  path: registry/gitlab-registry.key
            - secret:
                name: gitlab-registry-notification
                items:
                - key: secret
                  path: registry/notificationSecret
            - secret:
                name: gitlab-gitlab-workhorse-secret
                items:
                - key: shared_secret
                  path: gitlab-workhorse/secret
            - secret:
                name: gitlab-gitlab-kas-secret
                items:
                - key: kas_shared_secret
                  path: kas/.gitlab_kas_secret
            - secret:
                name: gitlab-zoekt-basicauth
                optional: true
                items:
                - key: gitlab_username
                  path: zoekt/.gitlab_zoekt_username
                - key: gitlab_password
                  path: zoekt/.gitlab_zoekt_password
            - secret:
                name: gitlab-object-store-configs
                items:
                - key: connection.yaml
                  path: objectstorage/object_store
            - secret:
                name: authentik-oidc
                items:
                - key: config.yaml
                  path: omniauth/authentik-oidc/config.yaml
        - name: webservice-secrets
          emptyDir:
            medium: Memory
        - name: workhorse-secrets
          emptyDir:
            medium: Memory
        - name: shared-upload-directory
          emptyDir: {}
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (apps/v1/Deployment/gitlab/gitlab-registry)
+ one document added:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: gitlab-registry
    namespace: gitlab
    labels:
      app: registry
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    selector:
      matchLabels:
        app: registry
        release: gitlab
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
        labels:
          app: registry
          app.kubernetes.io/name: gitlab
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: registry
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: registry
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab"
          env:
          - name: GODEBUG
            value: "tlsmlkem=0,tlskyber=0"
          - name: TZ
            value: UTC
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /debug/health
              port: debug
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
          - name: http
            containerPort: 5000
          - name: debug
            containerPort: 5001
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /debug/health
              port: debug
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
          volumeMounts:
          - name: registry-server-config
            mountPath: /etc/docker/registry/
            readOnly: true
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/bash
                - "-c"
                - "sleep 5"
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          args:
          - sh
          - /config/configure
          env:
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /templates
          - name: CONFIG_DIRECTORY
            value: /registry
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: registry-secrets
            mountPath: /config
          - name: registry-server-config-templates
            mountPath: /templates
            readOnly: true
          - name: registry-server-config
            mountPath: /registry
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: dependencies
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab"
          command:
          - /scripts/wait-for-deps
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: registry-server-config
            mountPath: /etc/docker/registry/
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        terminationGracePeriodSeconds: 30
        volumes:
        - name: registry-server-config-templates
          configMap:
            name: gitlab-registry
            items:
            - key: config.yml.tpl
              path: config.yml.tpl
        - name: registry-server-config
          emptyDir:
            medium: Memory
        - name: registry-secrets
          projected:
            sources:
            - configMap:
                name: gitlab-registry
            - secret:
                name: gitlab-registry-secret
                items:
                - key: registry-auth.crt
                  path: certificate.crt
            - secret:
                name: gitlab-registry-httpsecret
                items:
                - key: secret
                  path: httpSecret
            - secret:
                name: gitlab-object-store-configs
                items:
                - key: registry.yaml
                  path: storage/config
            - secret:
                name: gitlab-registry-db-pass
                items:
                - key: password
                  path: database_password
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-gitlab-shell)
+ one document added:
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: gitlab-gitlab-shell
    namespace: gitlab
    labels:
      app: gitlab-shell
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxReplicas: 10
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageValue: 100m
    minReplicas: 2
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: gitlab-gitlab-shell

(root level)  (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-kas)
+ one document added:
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: gitlab-kas
    namespace: gitlab
    labels:
      app: kas
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxReplicas: 10
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageValue: 100m
    minReplicas: 2
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: gitlab-kas

(root level)  (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-sidekiq-all-in-1-v2)
+ one document added:
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: gitlab-sidekiq-all-in-1-v2
    namespace: gitlab
    labels:
      app: sidekiq
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxReplicas: 10
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageValue: 350m
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: gitlab-sidekiq-all-in-1-v2

(root level)  (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-webservice-default)
+ one document added:
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: gitlab-webservice-default
    namespace: gitlab
    labels:
      app: webservice
      gitlab.com/webservice-name: default
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxReplicas: 10
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageValue: 1
    minReplicas: 2
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: gitlab-webservice-default

(root level)  (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-registry)
+ one document added:
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: gitlab-registry
    namespace: gitlab
    labels:
      app: registry
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    maxReplicas: 10
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 75
    minReplicas: 2
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: gitlab-registry

(root level)  (apps/v1/StatefulSet/gitlab/gitlab-gitaly)
+ one document added:
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: gitlab-gitaly
    namespace: gitlab
    labels:
      app: gitaly
      app.kubernetes.io/name: gitlab
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    podManagementPolicy: Parallel
    replicas: 1
    selector:
      matchLabels:
        app: gitaly
        release: gitlab
    serviceName: gitlab-gitaly
    template:
      metadata:
        annotations:
          gitlab.com/prometheus_path: /metrics
          gitlab.com/prometheus_port: "9236"
          gitlab.com/prometheus_scrape: "true"
          prometheus.io/path: /metrics
          prometheus.io/port: "9236"
          prometheus.io/scrape: "true"
        labels:
          app: gitaly
          app.kubernetes.io/name: gitlab
          heritage: Helm
          release: gitlab
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: gitaly
                    release: gitlab
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: gitaly
          image: "registry.gitlab.com/gitlab-org/build/cng/gitaly:v18.8.2"
          env:
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /etc/gitaly/templates
          - name: CONFIG_DIRECTORY
            value: /etc/gitaly
          - name: GITALY_CONFIG_FILE
            value: /etc/gitaly/config.toml
          - name: SSL_CERT_DIR
            value: /etc/ssl/certs
          - name: GODEBUG
            value: "tlsmlkem=0,tlskyber=0"
          - name: TZ
            value: UTC
          livenessProbe:
            failureThreshold: 3
            grpc:
              port: 8075
            initialDelaySeconds: 0
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          ports:
          - name: grpc-gitaly
            containerPort: 8075
          - name: http-metrics
            containerPort: 9236
          readinessProbe:
            failureThreshold: 3
            grpc:
              port: 8075
            initialDelaySeconds: 0
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            requests:
              cpu: 100m
              memory: 200Mi
          startupProbe:
            failureThreshold: 60
            grpc:
              port: 8075
            initialDelaySeconds: 1
            periodSeconds: 1
            successThreshold: 1
            timeoutSeconds: 1
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          - name: gitaly-config
            mountPath: /etc/gitaly/templates
          - name: gitaly-secrets
            mountPath: /etc/gitlab-secrets
            readOnly: true
          - name: repo-data
            mountPath: /home/git/repositories
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - sh
          - /config/configure
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: gitaly-config
            mountPath: /config
            readOnly: true
          - name: init-gitaly-secrets
            mountPath: /init-config
            readOnly: true
          - name: gitaly-secrets
            mountPath: /init-secrets
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
        terminationGracePeriodSeconds: 30
        volumes:
        - name: gitaly-config
          configMap:
            name: gitlab-gitaly
        - name: gitaly-secrets
          emptyDir:
            medium: Memory
        - name: init-gitaly-secrets
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: gitlab-gitaly-secret
                items:
                - key: token
                  path: gitaly_token
            - secret:
                name: gitlab-gitlab-shell-secret
                items:
                - key: secret
                  path: .gitlab_shell_secret
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
    volumeClaimTemplates:
    - metadata:
        name: repo-data
        annotations: null
        labels:
          app: gitaly
          release: gitlab
      spec:
        resources:
          requests:
            storage: 50Gi
        selector: null
        storageClassName: nfs-subdir
        accessModes:
        - ReadWriteOnce

(root level)  (apps/v1/StatefulSet/gitlab/gitlab-redis-master)
+ one document added:
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: gitlab-redis-master
    namespace: gitlab
    labels:
      app.kubernetes.io/component: master
      app.kubernetes.io/instance: gitlab
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: redis
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/component: master
        app.kubernetes.io/instance: gitlab
        app.kubernetes.io/name: redis
    serviceName: gitlab-redis-headless
    template:
      metadata:
        annotations:
          prometheus.io/port: "9121"
          prometheus.io/scrape: "true"
        labels:
          app.kubernetes.io/component: master
          app.kubernetes.io/instance: gitlab
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/name: redis
      spec:
        affinity:
          nodeAffinity: null
          podAffinity: null
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: master
                    app.kubernetes.io/instance: gitlab
                    app.kubernetes.io/name: redis
                topologyKey: kubernetes.io/hostname
              weight: 1
        automountServiceAccountToken: false
        containers:
        - name: redis
          image: "docker.io/bitnamilegacy/redis:7.2.4-debian-12-r9"
          imagePullPolicy: IfNotPresent
          args:
          - "-c"
          - /opt/bitnami/scripts/start-scripts/start-master.sh
          command:
          - /bin/bash
          env:
          - name: BITNAMI_DEBUG
            value: "false"
          - name: REDIS_REPLICATION_MODE
            value: master
          - name: ALLOW_EMPTY_PASSWORD
            value: no
          - name: REDIS_PASSWORD_FILE
            value: /opt/bitnami/redis/secrets/redis-password
          - name: REDIS_TLS_ENABLED
            value: no
          - name: REDIS_PORT
            value: "6379"
          ports:
          - name: redis
            containerPort: 6379
          volumeMounts:
          - name: start-scripts
            mountPath: /opt/bitnami/scripts/start-scripts
          - name: health
            mountPath: /health
          - name: redis-password
            mountPath: /opt/bitnami/redis/secrets/
          - name: redis-data
            mountPath: /data
          - name: config
            mountPath: /opt/bitnami/redis/mounted-etc
          - name: empty-dir
            mountPath: /opt/bitnami/redis/etc/
            subPath: app-conf-dir
          - name: empty-dir
            mountPath: /tmp
            subPath: tmp-dir
          livenessProbe:
            exec:
              command:
              - sh
              - "-c"
              - "/health/ping_liveness_local.sh 5"
            failureThreshold: 5
            initialDelaySeconds: 20
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 6
          readinessProbe:
            exec:
              command:
              - sh
              - "-c"
              - "/health/ping_readiness_local.sh 1"
            failureThreshold: 5
            initialDelaySeconds: 20
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 2
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: false
            runAsGroup: 0
            runAsNonRoot: true
            runAsUser: 1001
            seLinuxOptions: null
            seccompProfile:
              type: RuntimeDefault
        - name: metrics
          image: "docker.io/bitnamilegacy/redis-exporter:1.58.0-debian-12-r4"
          imagePullPolicy: IfNotPresent
          command:
          - /bin/bash
          - "-c"
          - |
            if [[ -f '/secrets/redis-password' ]]; then
            export REDIS_PASSWORD=$(cat /secrets/redis-password)
            fi
            redis_exporter

          env:
          - name: REDIS_ALIAS
            value: gitlab-redis
          - name: REDIS_EXPORTER_WEB_LISTEN_ADDRESS
            value: ":9121"
          - name: REDIS_USER
            value: default
          livenessProbe:
            failureThreshold: 5
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: metrics
            timeoutSeconds: 5
          ports:
          - name: metrics
            containerPort: 9121
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: metrics
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          volumeMounts:
          - name: empty-dir
            mountPath: /tmp
            subPath: app-tmp-dir
          - name: redis-password
            mountPath: /secrets/
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: false
            runAsGroup: 0
            runAsNonRoot: true
            runAsUser: 1001
            seLinuxOptions: null
            seccompProfile:
              type: RuntimeDefault
        enableServiceLinks: true
        securityContext:
          fsGroup: 1001
          fsGroupChangePolicy: Always
          supplementalGroups: []
          sysctls: []
        serviceAccountName: gitlab-redis-master
        terminationGracePeriodSeconds: 30
        volumes:
        - name: start-scripts
          configMap:
            name: gitlab-redis-scripts
            defaultMode: 493
        - name: health
          configMap:
            name: gitlab-redis-health
            defaultMode: 493
        - name: redis-password
          secret:
            items:
            - key: secret
              path: redis-password
            secretName: gitlab-redis-secret
        - name: config
          configMap:
            name: gitlab-redis-configuration
        - name: empty-dir
          emptyDir: {}
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: redis-data
        labels:
          app.kubernetes.io/component: master
          app.kubernetes.io/instance: gitlab
          app.kubernetes.io/name: redis
      spec:
        resources:
          requests:
            storage: 8Gi
        storageClassName: longhorn
        accessModes:
        - ReadWriteOnce

(root level)  (batch/v1/Job/gitlab/gitlab-migrations-c0164a8)
+ one document added:
  apiVersion: batch/v1
  kind: Job
  metadata:
    name: gitlab-migrations-c0164a8
    namespace: gitlab
    labels:
      app: migrations
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    activeDeadlineSeconds: 3600
    backoffLimit: 6
    template:
      metadata:
        labels:
          app: migrations
          heritage: Helm
          release: gitlab
      spec:
        automountServiceAccountToken: false
        containers:
        - name: migrations
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce:v18.8.2"
          args:
          - /bin/bash
          - "-c"
          - |
            set -e;
            /scripts/wait-for-deps;
            /scripts/db-migrate;
                                
            
          env:
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /var/opt/gitlab/templates
          - name: CONFIG_DIRECTORY
            value: /srv/gitlab/config
          - name: BYPASS_SCHEMA_VERSION
            value: "true"
          - name: ENABLE_BOOTSNAP
            value: "1"
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 250m
              memory: 200Mi
          volumeMounts:
          - name: migrations-config
            mountPath: /var/opt/gitlab/templates
          - name: migrations-secrets
            mountPath: /etc/gitlab
            readOnly: true
          - name: migrations-secrets
            mountPath: /srv/gitlab/config/secrets.yml
            subPath: rails-secrets/secrets.yml
          - name: migrations-secrets
            mountPath: /srv/gitlab/config/initial_root_password
            subPath: migrations/initial_root_password
          - name: migrations-secrets
            mountPath: /srv/gitlab/config/gitlab_shared_runners_registration_token
            subPath: migrations/gitlab_shared_runners_registration_token
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - sh
          - /config/configure
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: migrations-config
            mountPath: /config
            readOnly: true
          - name: init-migrations-secrets
            mountPath: /init-config
            readOnly: true
          - name: migrations-secrets
            mountPath: /init-secrets
            readOnly: false
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        restartPolicy: OnFailure
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        volumes:
        - name: migrations-config
          configMap:
            name: gitlab-migrations
        - name: init-migrations-secrets
          projected:
            defaultMode: 256
            sources:
            - secret:
                name: gitlab-rails-secret
                items:
                - key: secrets.yml
                  path: rails-secrets/secrets.yml
            - secret:
                name: gitlab-gitaly-secret
                items:
                - key: token
                  path: gitaly/gitaly_token
            - secret:
                name: gitlab-redis-secret
                items:
                - key: secret
                  path: redis/redis-password
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-ci
            - secret:
                name: db-gitlab-app
                items:
                - key: password
                  path: postgres/psql-password-main
            - secret:
                name: gitlab-object-store-configs
                items:
                - key: connection.yaml
                  path: objectstorage/object_store
            - secret:
                name: gitlab-gitlab-initial-root-password
                items:
                - key: password
                  path: migrations/initial_root_password
            - secret:
                name: gitlab-gitlab-runner-secret
                optional: true
                items:
                - key: runner-registration-token
                  path: migrations/gitlab_shared_runners_registration_token
        - name: migrations-secrets
          emptyDir:
            medium: Memory
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (batch/v1/Job/gitlab/gitlab-registry-migrations-8e2a9a9)
+ one document added:
  apiVersion: batch/v1
  kind: Job
  metadata:
    name: gitlab-registry-migrations-8e2a9a9
    namespace: gitlab
    labels:
      app: registry-migrations
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    activeDeadlineSeconds: 3600
    backoffLimit: 6
    template:
      metadata:
        labels:
          app: registry-migrations
          heritage: Helm
          release: gitlab
      spec:
        containers:
        - name: migrations
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab"
          args:
          - /scripts/wait-for-deps
          - /scripts/db-migrate
          env:
          - name: BYPASS_SCHEMA_VERSION
            value: "true"
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
          volumeMounts:
          - name: registry-server-config
            mountPath: /etc/docker/registry/
            readOnly: true
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs/
            readOnly: true
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        initContainers:
        - name: certificates
          image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
          env:
          - name: TZ
            value: UTC
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: etc-ssl-certs
            mountPath: /etc/ssl/certs
            readOnly: false
          - name: etc-pki-ca-trust-extracted-pem
            mountPath: /etc/pki/ca-trust/extracted/pem
            readOnly: false
          - name: custom-ca-certificates
            mountPath: /usr/local/share/ca-certificates
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        - name: configure
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          args:
          - sh
          - /config/configure
          env:
          - name: CONFIG_TEMPLATE_DIRECTORY
            value: /templates
          - name: CONFIG_DIRECTORY
            value: /registry
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: registry-secrets
            mountPath: /config
          - name: registry-server-config-templates
            mountPath: /templates
            readOnly: true
          - name: registry-server-config
            mountPath: /registry
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
        restartPolicy: OnFailure
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        volumes:
        - name: registry-server-config-templates
          configMap:
            name: gitlab-registry
            items:
            - key: migrations-config.yml.tpl
              path: config.yml.tpl
        - name: registry-server-config
          emptyDir:
            medium: Memory
        - name: registry-secrets
          projected:
            sources:
            - configMap:
                name: gitlab-registry
                items:
                - key: configure
                  path: configure
            - secret:
                name: gitlab-registry-db-pass
                items:
                - key: password
                  path: database_password
        - name: etc-ssl-certs
          emptyDir:
            medium: Memory
        - name: etc-pki-ca-trust-extracted-pem
          emptyDir:
            medium: Memory
        - name: custom-ca-certificates
          projected:
            defaultMode: 288
            sources:
            - secret:
                name: aethernet.crt
            - secret:
                name: gitlab-wildcard-tls-ca
  


(root level)  (batch/v1beta1/CronJob/gitlab/gitlab-toolbox-backup)
+ one document added:
  apiVersion: batch/v1beta1
  kind: CronJob
  metadata:
    name: gitlab-toolbox-backup
    namespace: gitlab
    labels:
      app: toolbox
      helm.toolkit.fluxcd.io/name: gitlab
      helm.toolkit.fluxcd.io/namespace: flux-system
      heritage: Helm
      release: gitlab
  spec:
    concurrencyPolicy: Replace
    failedJobsHistoryLimit: 1
    jobTemplate:
      spec:
        backoffLimit: 6
        template:
          metadata:
            annotations:
              cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
            labels:
              app: toolbox
              heritage: Helm
              release: gitlab
          spec:
            containers:
            - name: toolbox-backup
              image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce:v18.8.2"
              args:
              - /bin/bash
              - "-c"
              - "cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility --skip registry --skip packages --skip pages --skip artifacts"
              env:
              - name: ARTIFACTS_BUCKET_NAME
                value: gitlab-artifacts
              - name: REGISTRY_BUCKET_NAME
                value: gitlab-container-registry
              - name: LFS_BUCKET_NAME
                value: gitlab-lfs
              - name: UPLOADS_BUCKET_NAME
                value: gitlab-uploads
              - name: PACKAGES_BUCKET_NAME
                value: gitlab-packages
              - name: EXTERNAL_DIFFS_BUCKET_NAME
                value: gitlab-external-diffs
              - name: TERRAFORM_STATE_BUCKET_NAME
                value: gitlab-terraform-state
              - name: CI_SECURE_FILES_BUCKET_NAME
                value: gitlab-ci-secure-files
              - name: BACKUP_BUCKET_NAME
                value: gitlab-backup
              - name: BACKUP_BACKEND
                value: s3
              - name: TMP_BUCKET_NAME
                value: gitlab-tmp-storage
              - name: PAGES_BUCKET_NAME
                value: gitlab-pages
              - name: CONFIG_TEMPLATE_DIRECTORY
                value: /var/opt/gitlab/templates
              - name: CONFIG_DIRECTORY
                value: /srv/gitlab/config
              - name: TZ
                value: UTC
              resources:
                requests:
                  cpu: 50m
                  memory: 350M
              volumeMounts:
              - name: toolbox-config
                mountPath: /var/opt/gitlab/templates
              - name: toolbox-secrets
                mountPath: /etc/gitlab
                readOnly: true
              - name: toolbox-secrets
                mountPath: /srv/gitlab/config/secrets.yml
                subPath: rails-secrets/secrets.yml
              - name: toolbox-tmp
                mountPath: /srv/gitlab/tmp
                readOnly: false
              - name: etc-ssl-certs
                mountPath: /etc/ssl/certs/
                readOnly: true
              - name: etc-pki-ca-trust-extracted-pem
                mountPath: /etc/pki/ca-trust/extracted/pem
                readOnly: true
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                  - ALL
                runAsNonRoot: true
                runAsUser: 1000
            initContainers:
            - name: certificates
              image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2"
              env:
              - name: TZ
                value: UTC
              resources:
                requests:
                  cpu: 50m
              volumeMounts:
              - name: etc-ssl-certs
                mountPath: /etc/ssl/certs
                readOnly: false
              - name: etc-pki-ca-trust-extracted-pem
                mountPath: /etc/pki/ca-trust/extracted/pem
                readOnly: false
              - name: custom-ca-certificates
                mountPath: /usr/local/share/ca-certificates
                readOnly: true
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                  - ALL
                runAsNonRoot: true
                runAsUser: 1000
            - name: configure
              image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
              command:
              - sh
              - /config/configure
              env:
              - name: TZ
                value: UTC
              resources:
                requests:
                  cpu: 50m
              volumeMounts:
              - name: toolbox-config
                mountPath: /config
                readOnly: true
              - name: init-toolbox-secrets
                mountPath: /init-config
                readOnly: true
              - name: toolbox-secrets
                mountPath: /init-secrets
                readOnly: false
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                  - ALL
                runAsNonRoot: true
                runAsUser: 1000
            restartPolicy: OnFailure
            securityContext:
              fsGroup: 1000
              runAsGroup: 1000
              runAsUser: 1000
              seccompProfile:
                type: RuntimeDefault
            volumes:
            - name: toolbox-config
              projected:
                sources:
                - configMap:
                    name: gitlab-toolbox
            - name: toolbox-tmp
              emptyDir: {}
            - name: init-toolbox-secrets
              projected:
                defaultMode: 256
                sources:
                - secret:
                    name: gitlab-rails-secret
                    items:
                    - key: secrets.yml
                      path: rails-secrets/secrets.yml
                - secret:
                    name: gitlab-gitlab-shell-secret
                    items:
                    - key: secret
                      path: shell/.gitlab_shell_secret
                - secret:
                    name: gitlab-gitaly-secret
                    items:
                    - key: token
                      path: gitaly/gitaly_token
                - secret:
                    name: gitlab-redis-secret
                    items:
                    - key: secret
                      path: redis/redis-password
                - secret:
                    name: db-gitlab-app
                    items:
                    - key: password
                      path: postgres/psql-password-ci
                - secret:
                    name: db-gitlab-app
                    items:
                    - key: password
                      path: postgres/psql-password-main
                - secret:
                    name: gitlab-registry-secret
                    items:
                    - key: registry-auth.key
                      path: registry/gitlab-registry.key
                - secret:
                    name: gitlab-registry-notification
                    items:
                    - key: secret
                      path: registry/notificationSecret
                - secret:
                    name: gitlab-object-store-configs
                    items:
                    - key: s3cmd.cfg
                      path: objectstorage/.s3cfg
                - secret:
                    name: gitlab-gitlab-kas-secret
                    items:
                    - key: kas_shared_secret
                      path: kas/.gitlab_kas_secret
                - secret:
                    name: gitlab-object-store-configs
                    items:
                    - key: connection.yaml
                      path: objectstorage/object_store
                - secret:
                    name: authentik-oidc
                    items:
                    - key: config.yaml
                      path: omniauth/authentik-oidc/config.yaml
            - name: toolbox-secrets
              emptyDir:
                medium: Memory
            - name: etc-ssl-certs
              emptyDir:
                medium: Memory
            - name: etc-pki-ca-trust-extracted-pem
              emptyDir:
                medium: Memory
            - name: custom-ca-certificates
              projected:
                defaultMode: 288
                sources:
                - secret:
                    name: aethernet.crt
                - secret:
                    name: gitlab-wildcard-tls-ca
    schedule: @daily
    startingDeadlineSeconds: null
    successfulJobsHistoryLimit: 3
    suspend: false
  


(root level)  (v1/ServiceAccount/gitlab/gitlab-shared-secrets)
+ one document added:
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: gitlab-shared-secrets
    namespace: gitlab
    annotations:
      helm.sh/hook: "pre-install,pre-upgrade"
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
      helm.sh/hook-weight: "-5"
    labels:
      app: gitlab
      heritage: Helm
      release: gitlab

(root level)  (v1/ConfigMap/gitlab/gitlab-shared-secrets)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-shared-secrets
    namespace: gitlab
    annotations:
      helm.sh/hook: "pre-install,pre-upgrade"
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
      helm.sh/hook-weight: "-3"
    labels:
      app: gitlab
      heritage: Helm
      release: gitlab
  data:
    generate-secrets: |
      # vim: set filetype=sh:
      
      namespace=gitlab
      release=gitlab
      env=production
      
      pushd $(mktemp -d)
      
      # Args pattern, length
      function gen_random(){
        head -c 4096 /dev/urandom | LC_CTYPE=C tr -cd $1 | head -c $2
      }
      
      # Args: length
      function gen_random_base64(){
        local len="$1"
        gen_random_bytes "$len" | base64 -w0
      }
      
      # Args: length
      function gen_random_bytes(){
        local len="$1"
        head -c "$len" /dev/urandom
      }
      
      # Args: yaml file, search path
      function fetch_rails_value(){
        local value=$(yq --prettyPrint --no-colors ".${2}" $1)
      
        # Don't return null values
        if [ "${value}" != "null" ]; then echo "${value}"; fi
      }
      
      # Args: secretname
      function label_secret(){
        local secret_name=$1
      # Remove application labels if they exist
        kubectl --namespace=$namespace label \
          secret $secret_name $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g')
      
        kubectl --namespace=$namespace label \
          --overwrite \
          secret $secret_name app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm 
      }
      
      # Args: secretname, args
      function generate_secret_if_needed(){
        local secret_args=( "${@:2}")
        local secret_name=$1
      
        if ! $(kubectl --namespace=$namespace get secret $secret_name > /dev/null 2>&1); then
          kubectl --namespace=$namespace create secret generic $secret_name ${secret_args[@]}
        else
          echo "secret \"$secret_name\" already exists."
      
          for arg in "${secret_args[@]}"; do
            local from=$(echo -n ${arg} | cut -d '=' -f1)
      
            if [ -z "${from##*literal*}" ]; then
              local key=$(echo -n ${arg} | cut -d '=' -f2)
              local desiredValue=$(echo -n ${arg} | cut -d '=' -f3-)
              local flags="--namespace=$namespace --allow-missing-template-keys=false"
      
              if ! $(kubectl $flags get secret $secret_name -ojsonpath="{.data.${key}}" > /dev/null 2>&1); then
                echo "key \"${key}\" does not exist. patching it in."
      
                if [ "${desiredValue}" != "" ]; then
                  desiredValue=$(echo -n "${desiredValue}" | base64 -w 0)
                fi
      
                kubectl --namespace=$namespace patch secret ${secret_name} -p "{\"data\":{\"$key\":\"${desiredValue}\"}}"
              fi
            fi
          done
        fi
      
        label_secret $secret_name
      }
      
      # Initial root password
      generate_secret_if_needed "gitlab-gitlab-initial-root-password" --from-literal="password"=$(gen_random 'a-zA-Z0-9' 64)
      
      
      # Redis password
      generate_secret_if_needed "gitlab-redis-secret" --from-literal="secret"=$(gen_random 'a-zA-Z0-9' 64)
      
      
      
      
      # Gitlab shell
      generate_secret_if_needed "gitlab-gitlab-shell-secret" --from-literal="secret"=$(gen_random 'a-zA-Z0-9' 64)
      
      # Gitaly secret
      generate_secret_if_needed "gitlab-gitaly-secret" --from-literal="token"=$(gen_random 'a-zA-Z0-9' 64)
      
      
      
      # Gitlab runner secret
      generate_secret_if_needed "gitlab-gitlab-runner-secret" --from-literal=runner-registration-token=$(gen_random 'a-zA-Z0-9' 64) --from-literal=runner-token=""
      
      # GitLab Pages API secret
      
      
      # GitLab Pages auth secret for hashing cookie store when using access control
      
      
      # GitLab Pages OAuth secret
      
      
      # Gitlab-kas secret
      generate_secret_if_needed "gitlab-gitlab-kas-secret" --from-literal="kas_shared_secret"=$(gen_random 'a-zA-Z0-9' 32 | base64)
      
      # Gitlab-kas private API secret
      generate_secret_if_needed "gitlab-kas-private-api" --from-literal="kas_private_api_secret"=$(gen_random 'a-zA-Z0-9' 32 | base64)
      
      # Gitlab-kas WebSocket Token secret
      generate_secret_if_needed "gitlab-kas-websocket-token" --from-literal="kas_websocket_token_secret"=$(gen_random_base64 72)
      
      
      
      
      
      
      
      
      # Registry certificates
      mkdir -p certs
      openssl req -new -newkey rsa:4096 -subj "/CN=gitlab-issuer" -nodes -x509 -keyout certs/registry-example-com.key -out certs/registry-example-com.crt -days 3650
      generate_secret_if_needed "gitlab-registry-secret" --from-file=registry-auth.key=certs/registry-example-com.key --from-file=registry-auth.crt=certs/registry-example-com.crt
      
      # config/secrets.yaml
      if [ -n "$env" ]; then
        rails_secret="gitlab-rails-secret"
      
        # Fetch the values from the existing secret if it exists
        if $(kubectl --namespace=$namespace get secret $rails_secret > /dev/null 2>&1); then
          kubectl --namespace=$namespace get secret $rails_secret -o jsonpath="{.data.secrets\.yml}" | base64 --decode > secrets.yml
          secret_key_base=$(fetch_rails_value secrets.yml "${env}.secret_key_base")
          otp_key_base=$(fetch_rails_value secrets.yml "${env}.otp_key_base")
          db_key_base=$(fetch_rails_value secrets.yml "${env}.db_key_base")
          openid_connect_signing_key=$(fetch_rails_value secrets.yml "${env}.openid_connect_signing_key")
          encrypted_settings_key_base=$(fetch_rails_value secrets.yml "${env}.encrypted_settings_key_base")
      
          active_record_encryption_primary_keys=$(fetch_rails_value secrets.yml "${env}.active_record_encryption_primary_key")
          active_record_encryption_deterministic_keys=$(fetch_rails_value secrets.yml "${env}.active_record_encryption_deterministic_key")
          active_record_encryption_key_derivation_salt=$(fetch_rails_value secrets.yml "${env}.active_record_encryption_key_derivation_salt")
        fi;
      
        # Generate defaults for any unset secrets
        secret_key_base="${secret_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64)
        otp_key_base="${otp_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64)
        db_key_base="${db_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64)
        openid_connect_signing_key="${openid_connect_signing_key:-$(openssl genrsa 2048)}"
        encrypted_settings_key_base="${encrypted_settings_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64)
      
        # 1. We set the following two keys as an array to support keys rotation.
        #    The last key in the array is always used to encrypt data:
        #    https://github.com/rails/rails/blob/v7.0.8.4/activerecord/lib/active_record/encryption/key_provider.rb#L21
        #    while all the keys are used (in the order they're defined) to decrypt data:
        #    https://github.com/rails/rails/blob/v7.0.8.4/activerecord/lib/active_record/encryption/cipher.rb#L26.
        #    This allows to rotate keys by adding a new key as the last key, and start a re-encryption process that
        #    runs in the background: https://gitlab.com/gitlab-org/gitlab/-/issues/494976
        # 2. We use the same method and length as Rails' defaults:
        #    https://github.com/rails/rails/blob/v7.0.8.4/activerecord/lib/active_record/railties/databases.rake#L537-L540
        active_record_encryption_primary_keys=${active_record_encryption_primary_keys:-"- $(gen_random 'a-zA-Z0-9' 32)"}
        active_record_encryption_deterministic_keys=${active_record_encryption_deterministic_keys:-"- $(gen_random 'a-zA-Z0-9' 32)"}
        active_record_encryption_key_derivation_salt=${active_record_encryption_key_derivation_salt:-$(gen_random 'a-zA-Z0-9' 32)}
      
        # Update the existing secret
        cat << EOF > rails-secrets.yml
      apiVersion: v1
      kind: Secret
      metadata:
        name: $rails_secret
      type: Opaque
      stringData:
        secrets.yml: |-
          $env:
            secret_key_base: $secret_key_base
            otp_key_base: $otp_key_base
            db_key_base: $db_key_base
            encrypted_settings_key_base: $encrypted_settings_key_base
            openid_connect_signing_key: |
      $(echo "${openid_connect_signing_key}" | awk '{print "        " $0}')
            active_record_encryption_primary_key:
              $active_record_encryption_primary_keys
            active_record_encryption_deterministic_key:
              $active_record_encryption_deterministic_keys
            active_record_encryption_key_derivation_salt: $active_record_encryption_key_derivation_salt
      EOF
        kubectl --validate=false --namespace=$namespace apply -f rails-secrets.yml
        label_secret $rails_secret
      fi
      
      # Shell ssh host keys
      ssh-keygen -A
      mkdir -p host_keys
      cp /etc/ssh/ssh_host_* host_keys/
      generate_secret_if_needed "gitlab-gitlab-shell-host-keys" --from-file host_keys
      
      # Gitlab-workhorse secret
      generate_secret_if_needed "gitlab-gitlab-workhorse-secret" --from-literal="shared_secret"=$(gen_random 'a-zA-Z0-9' 32 | base64)
      
      # Registry http.secret secret
      generate_secret_if_needed "gitlab-registry-httpsecret" --from-literal="secret"=$(gen_random 'a-z0-9' 128 | base64 -w 0)
      
      # Container Registry notification_secret
      generate_secret_if_needed "gitlab-registry-notification" --from-literal="secret"=[\"$(gen_random 'a-zA-Z0-9' 32)\"]
      
  


(root level)  (v1/ConfigMap/gitlab/gitlab-gitlab-upgrade-check)
+ one document added:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: gitlab-gitlab-upgrade-check
    namespace: gitlab
    annotations:
      helm.sh/hook: pre-upgrade
      helm.sh/hook-delete-policy: before-hook-creation
      helm.sh/hook-weight: "-11"
    labels:
      app: gitlab
      heritage: Helm
      release: gitlab
  data:
    runcheck: |
      #!/bin/sh
      
      set -e
      
      notify() {
        echo "$1"
        echo -n "$1 " >> /dev/termination-log
      }
      
      greater_version()
      {
        test "$(printf '%s\n' "$@" | sort -V | tail -n 1)" = "$1";
      }
      
      is_semver()
      {
        printf '%s' "$1" | grep -Eq "^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"
        return "$?"
      }
      
      # For the PostgreSQL upgrade, you either need both secrets, or no secrets.
      # If there are no secrets, we will create them for you.
      # If the secrets aren't in either of these states, we assume you are upgrading from an older version
      # This is running ahead of version checks to ensure this always runs. This is to account for
      # installations outside of the official helm repo.
      secrets_dir="/etc/secrets/postgresql"
      if [ -d "${secrets_dir}" ]; then
        if [ ! "$(ls -A ${secrets_dir}/..data/)" = "" ]; then
          if [ ! -f "${secrets_dir}/postgresql-postgres-password" ] || [ ! -f "${secrets_dir}/password" ]; then
            notify "You seem to be upgrading from a previous version of GitLab using the bundled PostgreSQL chart"
            notify "There are some manual steps which need to be performed in order to upgrade the database"
            notify "Please see the upgrade documentation for instructions on how to proceed:"
            notify "https://docs.gitlab.com/charts/installation/upgrade.html"
            exit 1
          fi
        fi
      fi
      MIN_VERSION=18.5
      CHART_MIN_VERSION=9.5
      
      # Remove 'v' prefix from GitLab version if present (set in Chart.yaml appVersions)
      GITLAB_VERSION=${GITLAB_VERSION#v}
      
      # Only run check for semver releases
      if ! is_semver "$GITLAB_VERSION"; then
        exit 0
      fi
      
      NEW_MAJOR_VERSION=$(echo $GITLAB_VERSION | awk -F "." '{print $1}')
      NEW_MINOR_VERSION=$(echo $GITLAB_VERSION | awk -F "." '{print $1"."$2}')
      
      NEW_CHART_MAJOR_VERSION=$(echo $CHART_VERSION | awk -F "." '{print $1}')
      NEW_CHART_MINOR_VERSION=$(echo $CHART_VERSION | awk -F "." '{print $1"."$2}')
      
      if [ ! -f /chart-info/gitlabVersion ]; then
        notify "It seems you are attempting an unsupported upgrade path."
        notify "Please follow the upgrade documentation at https://docs.gitlab.com/update/#upgrade-paths"
        exit 1
      fi
      
      OLD_VERSION_STRING=$(cat /chart-info/gitlabVersion)
      OLD_VERSION_STRING=${OLD_VERSION_STRING#v}
      OLD_CHART_VERSION_STRING=$(cat /chart-info/gitlabChartVersion)
      
      # Skip check if old version wasn't semver
      if ! is_semver "$OLD_VERSION_STRING"; then
        exit 0
      fi
      
      OLD_MAJOR_VERSION=$(echo $OLD_VERSION_STRING | awk -F "." '{print $1}')
      OLD_MINOR_VERSION=$(echo $OLD_VERSION_STRING | awk -F "." '{print $1"."$2}')
      OLD_CHART_MAJOR_VERSION=$(echo $OLD_CHART_VERSION_STRING | awk -F "." '{print $1}')
      OLD_CHART_MINOR_VERSION=$(echo $OLD_CHART_VERSION_STRING | awk -F "." '{print $1"."$2}')
      
      # Checking Version
      if ( ! greater_version "$OLD_MINOR_VERSION" "$MIN_VERSION" ) || ( ! greater_version "$OLD_CHART_MINOR_VERSION" "$CHART_MIN_VERSION" ); then
        notify "It seems you are upgrading the GitLab Helm Chart from ${OLD_CHART_VERSION_STRING} (GitLab ${OLD_VERSION_STRING}) to ${CHART_VERSION} (GitLab ${GITLAB_VERSION})."
        notify "It is required to upgrade to the latest ${CHART_MIN_VERSION}.x version first before proceeding."
        # Link to upgrade docs if it is a major version jump
        if [ "${OLD_MAJOR_VERSION}" -lt "${NEW_MAJOR_VERSION}" ] || [ "${OLD_CHART_MAJOR_VERSION}" -lt "${NEW_CHART_MAJOR_VERSION}" ]; then
          notify "Please follow the upgrade documentation at https://docs.gitlab.com/charts/releases/8_0.html"
          notify "and upgrade to GitLab Helm Chart version ${CHART_MIN_VERSION}.x before upgrading to ${CHART_VERSION}."
        fi
        exit 1
      fi
  


(root level)  (rbac.authorization.k8s.io/v1/Role/gitlab/gitlab-shared-secrets)
+ one document added:
  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: gitlab-shared-secrets
    namespace: gitlab
    annotations:
      helm.sh/hook: "pre-install,pre-upgrade"
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
      helm.sh/hook-weight: "-5"
    labels:
      app: shared-secrets
      heritage: Helm
      release: gitlab
  rules:
  - resources:
    - secrets
    apiGroups:
    -
    verbs:
    - get
    - list
    - create
    - patch

(root level)  (rbac.authorization.k8s.io/v1/RoleBinding/gitlab/gitlab-shared-secrets)
+ one document added:
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: gitlab-shared-secrets
    namespace: gitlab
    annotations:
      helm.sh/hook: "pre-install,pre-upgrade"
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
      helm.sh/hook-weight: "-5"
    labels:
      app: shared-secrets
      heritage: Helm
      release: gitlab
  roleRef:
    name: gitlab-shared-secrets
    apiGroup: rbac.authorization.k8s.io
    kind: Role
  subjects:
  - name: gitlab-shared-secrets
    kind: ServiceAccount
    namespace: gitlab

(root level)  (batch/v1/Job/gitlab/gitlab-shared-secrets-6fff130)
+ one document added:
  apiVersion: batch/v1
  kind: Job
  metadata:
    name: gitlab-shared-secrets-6fff130
    namespace: gitlab
    annotations:
      helm.sh/hook: "pre-install,pre-upgrade"
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
    labels:
      app: gitlab
      heritage: Helm
      release: gitlab
  spec:
    template:
      metadata:
        annotations: null
        labels:
          app: gitlab
          heritage: Helm
          release: gitlab
      spec:
        containers:
        - name: gitlab
          image: "registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.8.2"
          command:
          - /bin/bash
          - /scripts/generate-secrets
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: scripts
            mountPath: /scripts
          - name: ssh
            mountPath: /etc/ssh
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
        restartPolicy: Never
        securityContext:
          fsGroup: 65534
          runAsUser: 65534
          seccompProfile:
            type: RuntimeDefault
        serviceAccountName: gitlab-shared-secrets
        volumes:
        - name: scripts
          configMap:
            name: gitlab-shared-secrets
        - name: ssh
          emptyDir: {}

(root level)  (batch/v1/Job/gitlab/gitlab-shared-secrets-6fff130-selfsign)
+ one document added:
  apiVersion: batch/v1
  kind: Job
  metadata:
    name: gitlab-shared-secrets-6fff130-selfsign
    namespace: gitlab
    annotations:
      helm.sh/hook: "pre-install,pre-upgrade"
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
    labels:
      app: gitlab
      heritage: Helm
      release: gitlab
  spec:
    template:
      metadata:
        annotations: null
        labels:
          app: gitlab
          heritage: Helm
          release: gitlab
      spec:
        containers:
        - name: kubectl
          image: "registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.8.2"
          command:
          - /bin/bash
          - "-exc"
          - |
            certname=gitlab-wildcard-tls
            # create wildcard certificate secret
            kubectl create secret tls $certname \
              --cert=/output/wildcard.pem --key=/output/wildcard-key.pem || true
            kubectl --namespace=$namespace label \
              secret $certname  $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g')
            kubectl --namespace=$namespace label --overwrite \
              secret $certname app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm 
            # create CA certificate secret
            kubectl create secret generic ${certname}-ca \
              --from-file=cfssl_ca=/output/ca.pem || true
            kubectl --namespace=$namespace label \
              secret ${certname}-ca  $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g')
            kubectl --namespace=$namespace label --overwrite \
              secret ${certname}-ca app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm 
            # create certificate chain for GitLab Runner
            cat /output/ca.pem /output/wildcard.pem > /tmp/code.nogweii.net.crt
            kubectl create secret generic ${certname}-chain \
              --from-file=/tmp/code.nogweii.net.crt || true
            kubectl --namespace=$namespace label \
              secret ${certname}-chain  $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g')
            kubectl --namespace=$namespace label --overwrite \
              secret ${certname}-chain app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm 
            
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: certs-path
            mountPath: /output
        initContainers:
        - name: cfssl-self-sign
          image: "registry.gitlab.com/gitlab-org/build/cng/cfssl-self-sign:v18.8.2"
          env:
          - name: CA_SUBJECT
            value: "GitLab Helm Chart"
          - name: CA_ORG
            value: gitlab
          - name: CA_ORG_UNIT
            value: gitlab
          - name: CERT_SUBJECT
            value: code.nogweii.net
          - name: CERT_DOMAIN
            value: code.nogweii.net
          - name: EXPIRY
            value: 3650d
          - name: ALGORITHM
            value: rsa
          - name: KEY_SIZE
            value: "4096"
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: certs-path
            mountPath: /output
        restartPolicy: Never
        securityContext:
          fsGroup: 65534
          runAsUser: 65534
        serviceAccountName: gitlab-shared-secrets
        volumes:
        - name: certs-path
          emptyDir: {}
  


(root level)  (batch/v1/Job/gitlab/gitlab-gitlab-upgrade-check)
+ one document added:
  apiVersion: batch/v1
  kind: Job
  metadata:
    name: gitlab-gitlab-upgrade-check
    namespace: gitlab
    annotations:
      helm.sh/hook: pre-upgrade
      helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation"
      helm.sh/hook-weight: "-10"
    labels:
      app: gitlab
      heritage: Helm
      release: gitlab
  spec:
    backoffLimit: 2
    template:
      metadata:
        labels:
          app: gitlab
          heritage: Helm
          release: gitlab
      spec:
        containers:
        - name: run-check
          image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2"
          command:
          - /bin/sh
          - /scripts/runcheck
          env:
          - name: GITLAB_VERSION
            value: "18.8.2"
          - name: CHART_VERSION
            value: "9.8.2"
          resources:
            requests:
              cpu: 50m
          volumeMounts:
          - name: chart-info
            mountPath: /chart-info
          - name: scripts
            mountPath: /scripts
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 65534
        restartPolicy: Never
        securityContext:
          fsGroup: 65534
          runAsUser: 65534
          seccompProfile:
            type: RuntimeDefault
        volumes:
        - name: chart-info
          configMap:
            name: gitlab-gitlab-chart-info
            optional: true
        - name: scripts
          configMap:
            name: gitlab-gitlab-upgrade-check

flate error: orig snapshot: reconcile completed with 1 failure(s):
  HelmRelease/flux-system/gitlab: helm template flux-system/gitlab: execution error at (gitlab/templates/NOTES.txt:222:3): 
CONFIGURATION CHECKS:

redis:
  You must configure an Redis connetion. Please set `global.redis.host`. Since chart v10.0.0,
  external Redis became required. If you were using the bundled Redis chart refer to
  https://docs.gitlab.com/charts/installation/migration/bundled_chart_migration/.

Pipeline

## flate diff ``` data.memini.json (v1/ConfigMap/local-ai/memini-memini) ± value change in multiline text (314 inserts, 271 deletions) { "annotations": { "list": [] }, - "description": "memini \u2014 what the service is actually doing for your agents. Live memory counts, write/recall traffic by tier, consolidation outcomes, store health, embedder & rerank latency & saturation, HTTP surface.", + "description": "memini operator dashboard \u2014 top-to-bottom narrative: live headline health (including the deferred-repair backlog), the write path, recall & retrieval quality, aging & maintenance jobs, embedder & rerank, LLM-backed consolidation/answer, the HTTP surface, and client injection telemetry.", "editable": true, "graphTooltip": 1, "id": null, "links": [], [14 lines unchanged)] { "id": 2, "type": "stat", "title": "Live memories", - "description": "Live (non-superseded, non-expired) memories by tier.", + "description": "Total live (non-superseded, non-expired) memories across all tiers; a sudden drop means a sweep or dedup pass removed more than expected.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, + "h": 5, + "w": 4, "x": 0, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false [five lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_memories_active", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum(memini_memories_active)", + "refId": "A", + "legendFormat": "live" } ] }, { "id": 3, "type": "stat", - "title": "Writes (5m rate)", - "description": "Successful Remember calls per second, by tier.", + "title": "Writes /s (5m)", + "description": "Successful Remember writes per second over the last 5m; zero during active agent sessions means writes are failing upstream.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, - "x": 6, + "h": 5, + "w": 3, + "x": 4, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "decimals": 2 + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, - "fieldConfig": { - "defaults": { - "unit": "ops" - } - }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum(rate(memini_remember_results_total{result=\"ok\"}[5m]))", + "refId": "A", + "legendFormat": "writes/s" } ] }, { "id": 4, "type": "stat", - "title": "Recalls (5m rate)", - "description": "Successful Recall calls per second.", + "title": "Recalls /s (5m)", + "description": "Successful Recall searches per second over the last 5m; the primary read-traffic pulse.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, - "x": 12, + "h": 5, + "w": 3, + "x": 7, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "decimals": 2 + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))", + "refId": "A", + "legendFormat": "recalls/s" + } + ] + }, + { + "id": 5, + "type": "stat", + "title": "HTTP 5xx ratio (15m)", + "description": "Share of HTTP requests answered 5xx over 15m; anything above zero deserves a look, above 5% is an outage.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 10, + "y": 1 + }, "fieldConfig": { "defaults": { - "unit": "ops" - } + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 0.01 + }, + { + "color": "red", + "value": 0.05 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))", - "legendFormat": "recalls/s", - "refId": "A" + "expr": "sum(rate(memini_http_requests_total{status=~\"5..\"}[15m])) / clamp_min(sum(rate(memini_http_requests_total[15m])), 1e-9)", + "refId": "A", + "legendFormat": "5xx ratio" } ] }, { - "id": 18, + "id": 6, "type": "stat", - "title": "Consolidation queue", - "description": "Backlog of memories awaiting async consolidation. Sustained > 0 means the worker can't keep up with Remember writes.", + "title": "Repair backlog depth", + "description": "Memories currently owing deferred repair (pending + enrich + failed); green is empty, any sustained backlog means writes are storing degraded faster than repairs drain.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, - "x": 18, + "h": 5, + "w": 4, + "x": 13, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(memini_repair_depth)", + "refId": "A", + "legendFormat": "backlog" + } + ] + }, + { + "id": 7, + "type": "stat", + "title": "Repair oldest age", + "description": "Age of the oldest memory still awaiting repair; amber the moment anything waits, red past 1h means the repair loop is stuck or starved.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 17, + "y": 1 + }, "fieldConfig": { "defaults": { - "unit": "short", + "color": { + "mode": "thresholds" + }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { - "color": "yellow", - "value": 50 + "color": "orange", + "value": 1 }, { "color": "red", - "value": 500 + "value": 3600 } ] - } + }, + "unit": "s" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "max(memini_repair_oldest_age_seconds)", + "refId": "A", + "legendFormat": "oldest" } + ] + }, + { + "id": 8, + "type": "stat", + "title": "Consolidation queue", + "description": "Depth of the async consolidation queue; a queue that grows without draining delays dedup/supersede decisions.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 21, + "y": 1 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 100 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "memini_consolidate_queue_depth", - "legendFormat": "queued", - "refId": "A" + "refId": "A", + "legendFormat": "queued" } ] }, { - "id": 5, + "id": 9, "type": "row", - "title": "Writes & Recalls", + "title": "Write path", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 7 + "y": 6 }, "collapsed": false, "panels": [] }, { - "id": 6, + "id": 10, "type": "timeseries", - "title": "Remembers per second (by tier)", + "title": "Remember rate by tier", + "description": "Successful Remember writes per second split by tier, plus error/dropped outcomes; red or orange series above zero mean writes are being lost.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 8 + "y": 7 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] }, - "unit": "ops" - } + { + "matcher": { + "id": "byName", + "options": "dropped" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{tier}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_remember_results_total{result=\"error\"}[$__rate_interval]))", + "refId": "B", + "legendFormat": "error" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_remember_results_total{result=\"dropped\"}[$__rate_interval]))", + "refId": "C", + "legendFormat": "dropped" } ] }, { - "id": 7, + "id": 11, "type": "timeseries", - "title": "Recalls per second (by hit-count bucket)", - "description": "How many memories a typical recall returns. '0' means no matches \u2014 investigate if persistent.", + "title": "Degraded (vectorless) writes by reason", + "description": "Writes stored without a vector (keyword-searchable only, queued for deferred repair) because the embedder timed out or errored; any sustained rate means the embedder is unhealthy.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 8 + "y": 7 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "embed_error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "embed_timeout" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[5m]))", - "legendFormat": "{{hits_bucket}}", - "refId": "A" + "expr": "sum by (reason) (rate(memini_remember_degraded_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{reason}}" } ] }, { - "id": 19, - "type": "row", - "title": "Retrieval quality", - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 16 - }, - "collapsed": false, - "panels": [] - }, - { - "id": 20, + "id": 12, "type": "timeseries", - "title": "Rerank outcomes by backend", - "description": "Per-recall reranker traffic: ok = reranker reordered the fused candidates; fallback = reranker timed out / errored and the fused order was kept.", + "title": "Store mutations", + "description": "Raw store write traffic: inserts vs updates (by upsert op), hard deletes from Forget, and tombstones written by consolidation; updates dominating inserts means most writes reinforce existing memories.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 17 + "y": 14 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend, result) (rate(memini_rerank_results_total[5m]))", - "legendFormat": "{{backend}}/{{result}}", - "refId": "A" + "expr": "sum by (op) (rate(memini_store_upserts_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "upsert: {{op}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(memini_store_deletes_total[$__rate_interval])", + "refId": "B", + "legendFormat": "hard delete (forget)" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(memini_store_soft_deletes_total[$__rate_interval])", + "refId": "C", + "legendFormat": "tombstone (supersede)" } ] }, { - "id": 21, + "id": 13, "type": "timeseries", - "title": "Recall degradation reasons", - "description": "Recalls that fell back to keyword-only search because the embedder was slow or errored. embed_timeout = the recall embed budget elapsed; embed_error = the embed call failed.", + "title": "Forget & supersede outcomes", + "description": "Forget and Supersede API outcomes per second; red error series should stay at zero, orange not_found spikes usually mean a client is retrying stale ids.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 17 + "y": 14 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*error.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] }, - "unit": "ops" - } + { + "matcher": { + "id": "byRegexp", + "options": ".*not_found.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (reason) (rate(memini_recall_degraded_total[5m]))", - "legendFormat": "{{reason}}", - "refId": "A" + "expr": "sum by (result) (rate(memini_forget_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "forget: {{result}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (rate(memini_supersede_results_total[$__rate_interval]))", + "refId": "B", + "legendFormat": "supersede: {{result}}" } ] }, { - "id": 22, + "id": 14, "type": "timeseries", - "title": "Recall reinforcement (best-effort)", - "description": "Background access-count bump on recalled memories. Errors are swallowed \u2014 a sustained nonzero error rate means the store write path is unhealthy but recall still served.", + "title": "Content hygiene (sanitizer)", + "description": "Ingestion sanitizer actions: cleaned content was rewritten in place, quarantined (orange) content was judged unsafe to store as-is and needs review if it becomes frequent.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 12, + "h": 7, + "w": 8, "x": 0, - "y": 25 + "y": 21 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "ops" - } + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "quarantined" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_reinforce_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "sum by (action) (rate(memini_write_sanitized_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{action}}" } ] }, { - "id": 23, - "type": "stat", - "title": "Recall success ratio", - "description": "Fraction of recall calls that succeeded. A drop means the recall path is erroring (embed + keyword + rerank + RRF + everything).", + "id": 15, + "type": "timeseries", + "title": "Corroboration routing", + "description": "Corroboration attempts on fresh short-term writes (did this confirm something we already know); a rising red error rate means the corroboration path is failing writes silently.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 25 + "h": 7, + "w": 8, + "x": 8, + "y": 21 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] }, "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, - "textMode": "auto" + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (rate(memini_corroborate_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" + } + ] + }, + { + "id": 16, + "type": "timeseries", + "title": "Contradiction routing", + "description": "Contradiction attempts on fresh durable writes (did this contradict an existing fact); contradicted events drive supersession, red errors mean conflicts are going undetected.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 21 }, "fieldConfig": { "defaults": { - "unit": "percentunit", - "min": 0, - "max": 1, + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, "thresholds": { "mode": "absolute", "steps": [ { - "color": "red", + "color": "green", "value": null - }, + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ { - "color": "yellow", - "value": 0.95 - }, - { - "color": "green", - "value": 0.99 + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } } ] } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)", - "legendFormat": "ok ratio", - "refId": "A" + "expr": "sum by (result) (rate(memini_contradict_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 8, + "id": 17, "type": "row", - "title": "Consolidation (LLM dedup)", + "title": "Recall & retrieval quality", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 33 + "y": 28 }, "collapsed": false, "panels": [] }, { - "id": 9, + "id": 18, "type": "timeseries", - "title": "Consolidation outcomes per minute", - "description": "gated = similarity gate skipped the LLM; new = nothing to merge; update = merged into an existing memory; supersede = contradiction-tombstoned an old one; noop = nothing to do; error/dropped = failure modes.", + "title": "Recall rate by hit-count bucket", + "description": "Successful recalls per second split by how many hits they returned, plus errors; a shift toward the 0-hits bucket means retrieval quality is degrading.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 16, + "h": 7, + "w": 12, "x": 0, - "y": 34 + "y": 29 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{hits_bucket}} hits" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_recall_results_total{result=\"error\"}[$__rate_interval]))", + "refId": "B", + "legendFormat": "error" + } + ] + }, + { + "id": 19, + "type": "timeseries", + "title": "Recall degradation (keyword-only fallback)", + "description": "Recalls that fell back to keyword-only search because embedding the query timed out or errored; these return without semantic ranking, so any sustained rate hurts result quality.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 29 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 20, + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } }, - "unit": "ops" + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "embed_error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "embed_timeout" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (reason) (rate(memini_recall_degraded_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{reason}}" + } + ] + }, + { + "id": 20, + "type": "timeseries", + "title": "Recall candidates floored (by tier filter)", + "description": "Candidates dropped from recall responses by the min_rank_score composite floor; a jump means the floor (or a scoring regression) is hiding results users would previously have seen.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 36 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] + }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["sum"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_consolidate_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "sum by (tier_filter) (rate(memini_recall_floored_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{tier_filter}}" } ] }, { - "id": 10, + "id": 21, + "type": "timeseries", + "title": "Recall reinforcement", + "description": "Best-effort reinforcement writes triggered by recall hits (strengthening recalled memories); red errors mean reinforcement is silently failing and ranking will slowly stale.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 36 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (rate(memini_reinforce_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" + } + ] + }, + { + "id": 22, "type": "stat", - "title": "Dedup effectiveness", - "description": "Fraction of consolidations that actually used the LLM (update+supersede+new+noop) vs the similarity gate skipping it. Higher = busier corpus or noisier dedup signal.", + "title": "Recall success ratio (15m)", + "description": "Share of Recall calls that completed without error over 15m; below 99% something in the search path is failing.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 8, "x": 16, - "y": 34 + "y": 36 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "orange", + "value": 0.9 + }, + { + "color": "green", + "value": 0.99 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] }, "options": { "colorMode": "value", "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "min": 0, - "max": 1 - } - }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|new|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result!~\"error|dropped\"}[15m])), 1e-9)", - "legendFormat": "LLM use ratio", - "refId": "A" + "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)", + "refId": "A", + "legendFormat": "ok ratio" } ] }, { - "id": 11, + "id": 23, "type": "row", - "title": "Storage & decay", + "title": "Aging & maintenance", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 42 + "y": 43 }, "collapsed": false, "panels": [] }, { - "id": 12, + "id": 24, "type": "timeseries", "title": "Live memories by tier", + "description": "Live memory count per tier over time; working/episodic should churn while semantic/procedural grow slowly \u2014 a cliff in any tier means a sweep or dedup removed too much.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 43 + "y": 44 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "stepAfter", - "fillOpacity": 15, + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } }, - "unit": "short" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["mean", "max"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_memories_active", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum by (tier) (memini_memories_active)", + "refId": "A", + "legendFormat": "{{tier}}" } ] }, { - "id": 13, + "id": 25, "type": "timeseries", - "title": "Decay sweep + tombstones (per minute)", - "description": "Swept = memories purged by TTL; soft deletes = contradictions tombstoned by consolidation; dedup tombstones = tombstoned by the periodic dedup pass.", + "title": "Promotion, classification & demotion (per hour)", + "description": "Hourly outcomes of the episodic-to-semantic promotion pass, durable facts it wrote, and omitted-tier writes the marker classifier routed; red promote errors mean durable knowledge is not being distilled. Demotions are the mirror image: stale, never-recalled durable memories retiered down to episodic \u2014 a steady demotion stream usually means bulk-imported debris aging out.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 43 + "y": 44 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "stepAfter", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "promote: error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["sum"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_store_swept_total[5m]))", - "legendFormat": "swept/{{tier}}", - "refId": "A" + "expr": "sum by (result) (increase(memini_promote_results_total[1h]))", + "refId": "A", + "legendFormat": "promote: {{result}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(increase(memini_promote_facts_total[1h]))", + "refId": "B", + "legendFormat": "facts promoted" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "rate(memini_store_soft_deletes_total[5m])", - "legendFormat": "consolidate tombstones", - "refId": "B" + "expr": "sum by (tier) (increase(memini_tier_classified_total[1h]))", + "refId": "C", + "legendFormat": "classified \u2192 {{tier}}" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "rate(memini_dedup_tombstoned_total[5m])", - "legendFormat": "dedup tombstones", - "refId": "C" + "expr": "sum by (tier) (increase(memini_demoted_total[1h]))", + "legendFormat": "demoted from {{tier}}", + "refId": "D" } ] }, { - "id": 14, - "type": "row", - "title": "Embedder & rerank", + "id": 26, + "type": "timeseries", + "title": "Decay sweeps & dedup tombstones", + "description": "Memories purged by the decay sweeper (by tier) and tombstoned by the dedup pass, per second; sustained zero forever means aging jobs are not running.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "gridPos": { - "h": 1, - "w": 24, + "h": 7, + "w": 12, "x": 0, "y": 51 }, - "collapsed": false, - "panels": [] + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (tier) (rate(memini_store_swept_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "swept: {{tier}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(memini_dedup_tombstoned_total[$__rate_interval])", + "refId": "B", + "legendFormat": "dedup tombstoned" + } + ] }, { - "id": 15, + "id": 27, "type": "timeseries", - "title": "Embedder p95 latency (by backend)", - "description": "openai = real network calls; cached/diskcache/batched = local cache or fan-out; disabled = no endpoint configured.", + "title": "Fsck outcomes (per hour)", + "description": "Hourly consistency-sweep outcomes; red errors mean the store's self-check cannot complete and silent corruption may go unnoticed.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, + "x": 12, + "y": 51 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (increase(memini_fsck_results_total[1h]))", + "refId": "A", + "legendFormat": "{{result}}" + } + ] + }, + { + "id": 28, + "type": "timeseries", + "title": "Repair backlog depth by state", + "description": "Memories owing deferred repair, split by state; red failed entries are permanently vectorless without operator action, and a growing pending line means the repair loop cannot keep up.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, "x": 0, - "y": 52 + "y": 58 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "s" + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "failed" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (state) (memini_repair_depth)", + "refId": "A", + "legendFormat": "{{state}}" } + ] + }, + { + "id": 29, + "type": "timeseries", + "title": "Repair oldest age by state", + "description": "Age of the oldest memory in each repair state; growth without bound means repairs are queued faster than they drain (the red line marks the 1h alarm threshold).", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 58 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 3600 + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[5m])))", - "legendFormat": "p95 {{backend}}", - "refId": "A" + "expr": "max by (state) (memini_repair_oldest_age_seconds)", + "refId": "A", + "legendFormat": "{{state}}" } ] }, { - "id": 16, + "id": 30, "type": "timeseries", - "title": "Embedder tokens + error rate", + "title": "Backfill pending", + "description": "Memories still awaiting embedding backfill and long memories still without chunk vectors; a number that never falls means those memories are unreachable by semantic (or chunked) recall.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, + "w": 8, + "x": 16, + "y": 58 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "memini_embed_backfill_pending", + "refId": "A", + "legendFormat": "pending embed" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "memini_chunk_backfill_pending", + "refId": "B", + "legendFormat": "pending chunk vectors" + } + ] + }, + { + "id": 31, + "type": "timeseries", + "title": "Repair outcomes by stage", + "description": "Deferred-repair attempt outcomes per second by stage; red parked means a memory gave up retrying and stays vectorless, orange retry churn means the embedder keeps failing under the repair loop.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, "w": 12, - "x": 12, - "y": 52 + "x": 0, + "y": 65 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*parked.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*retry.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] } - } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend) (rate(memini_embed_tokens_total[5m]))", - "legendFormat": "tokens/s {{backend}}", - "refId": "A" + "expr": "sum by (stage, result) (rate(memini_repair_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{stage}}: {{result}}" + } + ] + }, + { + "id": 32, + "type": "timeseries", + "title": "Repair duration p95 by stage", + "description": "95th-percentile latency of a single deferred repair by stage; climbing durations shrink repair throughput and grow the backlog.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 65 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend) (rate(memini_embed_errors_total[5m]))", - "legendFormat": "err/s {{backend}}", - "refId": "B" + "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(memini_repair_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{stage}}" } ] }, { - "id": 24, - "type": "stat", - "title": "Embedder in-flight", - "description": "Concurrent embed calls against the backend (post-cap, excludes cache hits). Spikes during ingest storms are normal; sustained values near the cap mean the embedder is the bottleneck.", + "id": 33, + "type": "row", + "title": "Embedder & rerank", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 72 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 34, + "type": "timeseries", + "title": "Embedder p95 latency by backend", + "description": "95th-percentile embedder call latency per backend layer; the openai layer reflects the real provider, cached/diskcache should stay near zero.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 4, - "w": 8, + "h": 7, + "w": 12, "x": 0, - "y": 60 + "y": 73 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, - "textMode": "auto" + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{backend}}" + } + ] + }, + { + "id": 35, + "type": "timeseries", + "title": "Rerank outcomes by backend", + "description": "Recall rerank attempts per second by backend and result; orange fallback means the reranker failed or timed out and results shipped with pre-rerank ordering.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 73 }, "fieldConfig": { "defaults": { - "unit": "short" + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*fallback.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_embed_in_flight", - "legendFormat": "embed in-flight", - "refId": "A" + "expr": "sum by (backend, result) (rate(memini_rerank_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{backend}}: {{result}}" } ] }, { - "id": 25, - "type": "stat", - "title": "Rerank in-flight", - "description": "Concurrent rerank calls. A nonzero value indicates rerank is configured; if this stays pinned at the cap, the reranker is the bottleneck.", + "id": 36, + "type": "timeseries", + "title": "Embedder tokens/s by backend", + "description": "Embedding tokens consumed per second as reported by the API (openai backend only); this is the embed cost driver.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 4, + "h": 7, "w": 8, - "x": 8, - "y": 60 + "x": 0, + "y": 80 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, - "textMode": "auto" + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (backend) (rate(memini_embed_tokens_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{backend}}" + } + ] + }, + { + "id": 37, + "type": "timeseries", + "title": "Embedder errors by backend", + "description": "Embedder failures per second by backend; any sustained nonzero rate here explains degraded writes and keyword-only recalls elsewhere on this board.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 80 }, "fieldConfig": { "defaults": { - "unit": "short" + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_rerank_in_flight", - "legendFormat": "rerank in-flight", - "refId": "A" + "expr": "sum by (backend) (rate(memini_embed_errors_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{backend}}" } ] }, { - "id": 26, + "id": 38, "type": "timeseries", - "title": "Embedder batch size (p50 vs p95)", - "description": "Items per embed call, by backend. batched shows a single high-p95 bar; cached should be ~1.", + "title": "Embedder batch size (p50 / p95)", + "description": "Texts per Embed call at the median and 95th percentile; batches stuck at 1 mean the batcher is not coalescing and each write pays a full round trip.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 4, + "h": 7, "w": 8, "x": 16, - "y": 60 + "y": 80 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 5, - "showPoints": "never" - } + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.50, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "p50 {{backend}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))", + "refId": "B", + "legendFormat": "p95 {{backend}}" + } + ] + }, + { + "id": 39, + "type": "timeseries", + "title": "In-flight embed & rerank calls", + "description": "Concurrent calls currently held against the embedding and rerank backends; pinned at the concurrency cap means requests are queueing behind the limiter.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 87 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] + }, "options": { "legend": { + "calcs": [], "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.5, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))", - "legendFormat": "p50 {{backend}}", - "refId": "A" + "expr": "memini_embed_in_flight", + "refId": "A", + "legendFormat": "embed" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))", - "legendFormat": "p95 {{backend}}", - "refId": "B" + "expr": "memini_rerank_in_flight", + "refId": "B", + "legendFormat": "rerank" } ] }, { - "id": 27, + "id": 40, "type": "row", - "title": "HTTP surface", + "title": "LLM \u2014 consolidation & answer (empty without an LLM configured)", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 64 + "y": 93 }, "collapsed": false, "panels": [] }, { - "id": 28, + "id": 41, "type": "timeseries", - "title": "HTTP requests by route (status class)", - "description": "Per-route request rate broken down by 2xx / 4xx / 5xx. 5xx spikes point at a broken handler; 4xx spikes usually mean client misuse or a misconfigured agent.", + "title": "Consolidation outcomes", + "description": "LLM consolidation decisions per second (gated, new, update, supersede, noop); red errors or orange drops mean writes are skipping dedup and duplicates will accumulate.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 16, + "h": 7, + "w": 12, "x": 0, - "y": 65 + "y": 94 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "dropped" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["mean", "max"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (route, status) (label_replace(rate(memini_http_requests_total[5m]), \"status_class\", \"$1\", \"status\", \"(...).\"))", - "legendFormat": "{{route}} {{status_class}}xx", - "refId": "A" + "expr": "sum by (result) (rate(memini_consolidate_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 29, + "id": 42, "type": "timeseries", - "title": "HTTP p95 latency by route", + "title": "Consolidation queue depth", + "description": "Async consolidation queue depth over time; a sawtooth is normal, monotonic growth means the LLM cannot keep pace with write volume.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 65 + "h": 7, + "w": 6, + "x": 12, + "y": 94 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "s" - } + "unit": "none", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": false }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[5m])))", - "legendFormat": "p95 {{route}}", - "refId": "A" + "expr": "memini_consolidate_queue_depth", + "refId": "A", + "legendFormat": "queue depth" + } + ] + }, + { + "id": 43, + "type": "stat", + "title": "Dedup effectiveness (15m)", + "description": "Share of consolidated writes that matched an existing memory (update/supersede/noop) rather than creating a new one; a sudden drop to zero suggests the matcher is broken.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 18, + "y": 94 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result=~\"new|update|supersede|noop\"}[15m])), 1e-9)", + "refId": "A", + "legendFormat": "matched" } ] }, { - "id": 17, + "id": 44, "type": "timeseries", - "title": "End-to-end op latency p95 (by op)", + "title": "Answer outcomes", + "description": "Answer API (LLM-synthesized answers over recall) outcomes per second; red errors mean answer requests are burning LLM budget without returning.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 16, + "h": 7, + "w": 12, "x": 0, - "y": 73 + "y": 101 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 5, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "s" - } + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["mean", "max"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[5m])))", - "legendFormat": "{{op}}", - "refId": "A" + "expr": "sum by (result) (rate(memini_answer_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 34, + "id": 45, "type": "timeseries", - "title": "HTTP response size by route (p50/p90)", - "description": "Body bytes shipped per response. The recall/briefing routes are what hooks inject from — watch these when tuning context-injection volume.", + "title": "LLM op latency p95", + "description": "95th-percentile end-to-end latency of the LLM-backed service ops (answer, promote, dedup, semantic distillation wait); these dominate tail latency when the LLM slows down.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 73 + "h": 7, + "w": 12, + "x": 12, + "y": 101 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "bytes" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))", - "legendFormat": "p90 {{route}}", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))", - "legendFormat": "p50 {{route}}", - "refId": "B" + "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket{op=~\"answer|promote|dedup|distill_sem_wait\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{op}}" } ] }, { - "id": 30, + "id": 46, "type": "row", - "title": "Write-path signals", - "collapsed": false, + "title": "HTTP surface", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 82 + "y": 108 }, + "collapsed": false, "panels": [] }, { - "id": 31, + "id": 47, "type": "timeseries", - "title": "Corroboration routing", - "description": "A fresh short-term write restating a durable fact: corroborated (confidence grown), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error.", + "title": "HTTP request rate by route & status class", + "description": "Requests per second per route, grouped into status classes; red 5xx traffic is server failure, orange 4xx spikes usually mean a misconfigured client or expired key.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 83 + "y": 109 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "short" + "unit": "reqps", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".* 5xx" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".* 4xx" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (route, status_class) (label_replace(rate(memini_http_requests_total[$__rate_interval]), \"status_class\", \"$1\", \"status\", \"(.)..\"))", + "refId": "A", + "legendFormat": "{{route}} {{status_class}}xx" + } + ] + }, + { + "id": 48, + "type": "timeseries", + "title": "HTTP p95 latency by route", + "description": "95th-percentile HTTP latency per route as clients experience it; compare with the service-op view below to tell transport overhead from service work.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 109 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] + }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_corroborate_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{route}}" } ] }, { - "id": 33, + "id": 49, "type": "timeseries", - "title": "Contradiction routing", - "description": "A fresh durable write that changes a fact's value or flips its polarity: contradicted (stale fact invalidated via valid_to + confidence shrink), no_signal (durable neighbour found but not a real change), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error. Default-on; a contradicted rate far above expectation is the misfire smoke alarm (set MEMINI_CONTRADICT_DOWNRANK=false to disable).", + "title": "Service op latency p95 by op", + "description": "95th-percentile end-to-end latency of every public service operation (remember, recall, briefing, forget, ...); this is the service-internal view behind the HTTP numbers.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 91 + "y": 116 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "short" - } + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_contradict_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{op}}" } ] }, { - "id": 32, + "id": 50, "type": "timeseries", - "title": "Tier classification & promotion", - "description": "Omitted-tier writes the marker classifier routed to a durable tier, and facts produced by the promotion pass (LLM or marker extractor).", + "title": "HTTP response size (search & briefing, p50 / p90)", + "description": "Response body sizes for the two injection-critical routes; growth here directly inflates the tokens injected into agent context.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 83 + "y": 116 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "short" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "bytes", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_tier_classified_total[5m]))", - "legendFormat": "classified -> {{tier}}", - "refId": "A" + "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "p50 {{route}}" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_promote_facts_total[5m]))", - "legendFormat": "promoted facts", - "refId": "B" + "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))", + "refId": "B", + "legendFormat": "p90 {{route}}" } ] }, { - "id": 40, + "id": 51, "type": "row", - "title": "Injection efficiency (client telemetry)", - "collapsed": false, + "title": "Client injection telemetry", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 99 + "y": 123 }, + "collapsed": false, "panels": [] }, { - "id": 41, + "id": 52, "type": "timeseries", - "title": "Injection outcomes (by result)", - "description": "What hooks reported actually reaching model context vs held back by their local gates (seen/cooldown/budget/unchanged/score). Absent entirely means no integration is reporting.", + "title": "Injection outcomes by result", + "description": "Client-reported memory injections per second by outcome; suppressed_* series show why memories were withheld (already seen, cooldown, budget, unchanged, score) \u2014 a spike in suppressions with flat injections means the hooks are firing but delivering nothing.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, + "h": 7, + "w": 12, "x": 0, - "y": 100 + "y": 124 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } - } - } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_injected_memories_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "sum by (result) (rate(memini_injected_memories_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 42, + "id": 53, "type": "timeseries", - "title": "Injected memories (by surface)", - "description": "Memories that reached context, split by which hook surface put them there.", + "title": "Injected memories by surface", + "description": "Memories actually injected into model context per second, by hook surface (briefing, prompt, pretool); a silent surface means that hook is broken client-side.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 100 + "h": 7, + "w": 6, + "x": 12, + "y": 124 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" - } - } + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[5m]))", - "legendFormat": "{{surface}}", - "refId": "A" + "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{surface}}" } ] }, { - "id": 43, + "id": 54, "type": "timeseries", - "title": "Injected tokens per hour (by surface)", - "description": "Client-estimated tokens the hooks added to context. The number injection tuning should push down.", + "title": "Injected tokens per hour by surface", + "description": "Client-side estimate of context tokens injected per hour by surface; this is the direct context-budget cost of memini to the agent.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 100 + "h": 7, + "w": 6, + "x": 18, + "y": 124 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" - } - } + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [five lines unchanged)] "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "sum by (surface) (increase(memini_injected_tokens_total[1h]))", - "legendFormat": "{{surface}}", - "refId": "A" + "refId": "A", + "legendFormat": "{{surface}}" } ] } ], [30 lines unchanged)] "timepicker": {}, "timezone": "", "title": "memini", "uid": "memini", - "version": 2, + "version": 3, "weekStart": "" } spec.template.spec.containers.main.image (apps/v1/StatefulSet/local-ai/memini) ± value change - registry.erwanleboucher.dev/eleboucher/memini@sha256:2d388222bca3d65673a4072e99e0970fe0d5cdab0030ce0effdfdd78307ae7e2 + registry.erwanleboucher.dev/eleboucher/memini@sha256:1e060967d6fd11dfb79eeaa67d8745e0dde5d946ca1b7dd7469b4fad2fac0c1d (root level) (networking.k8s.io/v1/NetworkPolicy/gitlab/gitlab-redis) + one document added: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: gitlab-redis namespace: gitlab labels: app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system spec: egress: - ingress: - ports: - port: 6379 - ports: - port: 9121 podSelector: matchLabels: app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis policyTypes: - Ingress - Egress (root level) (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-gitaly) + one document added: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-gitaly namespace: gitlab labels: app: gitaly helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxUnavailable: 1 selector: matchLabels: app: gitaly release: gitlab (root level) (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-gitlab-shell) + one document added: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-gitlab-shell namespace: gitlab labels: app: gitlab-shell helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxUnavailable: 1 selector: matchLabels: app: gitlab-shell release: gitlab (root level) (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-kas) + one document added: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-kas namespace: gitlab labels: app: kas helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxUnavailable: 1 selector: matchLabels: app: kas release: gitlab (root level) (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-sidekiq-all-in-1-v1) + one document added: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-sidekiq-all-in-1-v1 namespace: gitlab labels: app: sidekiq helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxUnavailable: 1 selector: matchLabels: app: sidekiq queue-pod-name: all-in-1 release: gitlab (root level) (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-webservice-default) + one document added: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-webservice-default namespace: gitlab labels: app: webservice gitlab.com/webservice-name: default helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxUnavailable: 1 selector: matchLabels: app: webservice gitlab.com/webservice-name: default release: gitlab (root level) (policy/v1beta1/PodDisruptionBudget/gitlab/gitlab-registry-v1) + one document added: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-registry-v1 namespace: gitlab labels: app: registry helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxUnavailable: 1 selector: matchLabels: app: registry release: gitlab (root level) (v1/ServiceAccount/gitlab/gitlab-runner-access) + one document added: apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-runner-access namespace: gitlab labels: app: gitlab-gitlab-runner helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab (root level) (v1/ServiceAccount/gitlab/gitlab-redis-master) + one document added: apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-redis-master namespace: gitlab labels: app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system automountServiceAccountToken: false (root level) (v1/ConfigMap/gitlab/gitlab-gitlab-runner) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitlab-runner namespace: gitlab labels: app: gitlab-gitlab-runner helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: check-live: | #!/bin/bash set -eou pipefail # default timeout is 3 seconds, can be overriden VERIFY_TIMEOUT=${1:-${VERIFY_TIMEOUT:-3}} if ! /usr/bin/pgrep -f ".*register-the-runner" > /dev/null && ! /usr/bin/pgrep -f "gitlab.*runner" > /dev/null ; then exit 1 fi status=0 # empty --url= helps `gitlab-runner verify` select all configured runners (otherwise filters for $CI_SERVER_URL) verify_output=$(timeout "${VERIFY_TIMEOUT}" gitlab-runner verify --url= 2>&1) || status=$? # timeout exit code is 143 with busybox, and 124 with coreutils if (( status == 143 )) || (( status == 124 )) ; then echo "'gitlab-runner verify' terminated by timeout, not a conclusive failure" >&2 exit 0 elif (( status > 0 )) ; then exit ${status} fi grep -qE "is (alive|valid)" <<<"${verify_output}" config.template.toml: | [[runners]] [runners.kubernetes] image = "archlinux/archlinux:latest" namespace_per_job = true node_selector_overwrite_allowed = "kubernetes.io/arch=.*" pull_policy = ["always", "always"] [runners.kubernetes.node_selector] "kubernetes.io/arch" = "amd64" [runners.cache] Type = "s3" Path = "gitlab-runner" Shared = true [runners.cache.s3] ServerAddress = "brickyard.home.arpa:3900" BucketName = "gitlab-runner-cache" BucketLocation = "apartment" Insecure = true AuthenticationType = "access-key" PathStyle = true config.toml: | shutdown_timeout = 0 concurrent = 10 check_interval = 3 log_level = "info" entrypoint: | #!/bin/bash set -e export CONFIG_PATH_FOR_INIT="/home/gitlab-runner/.gitlab-runner/" mkdir -p ${CONFIG_PATH_FOR_INIT} cp /configmaps/config.toml ${CONFIG_PATH_FOR_INIT} # Set up environment variables for cache if [[ -f /secrets/accesskey && -f /secrets/secretkey ]]; then export CACHE_S3_ACCESS_KEY=$(cat /secrets/accesskey) export CACHE_S3_SECRET_KEY=$(cat /secrets/secretkey) fi if [[ -f /secrets/gcs-applicaton-credentials-file ]]; then export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-applicaton-credentials-file" elif [[ -f /secrets/gcs-application-credentials-file ]]; then export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-application-credentials-file" else if [[ -f /secrets/gcs-access-id && -f /secrets/gcs-private-key ]]; then export CACHE_GCS_ACCESS_ID=$(cat /secrets/gcs-access-id) # echo -e used to make private key multiline (in google json auth key private key is oneline with \n) export CACHE_GCS_PRIVATE_KEY=$(echo -e $(cat /secrets/gcs-private-key)) fi fi if [[ -f /secrets/azure-account-name && -f /secrets/azure-account-key ]]; then export CACHE_AZURE_ACCOUNT_NAME=$(cat /secrets/azure-account-name) export CACHE_AZURE_ACCOUNT_KEY=$(cat /secrets/azure-account-key) fi if [[ -f /secrets/runner-registration-token ]]; then export REGISTRATION_TOKEN=$(cat /secrets/runner-registration-token) fi if [[ -f /secrets/runner-token ]]; then export CI_SERVER_TOKEN=$(cat /secrets/runner-token) fi # Register the runner if ! sh /configmaps/register-the-runner; then exit 1 fi # Run pre-entrypoint-script if ! bash /configmaps/pre-entrypoint-script; then exit 1 fi # Start the runner exec /entrypoint run \ --working-directory=/home/gitlab-runner pre-entrypoint-script: register-the-runner: | #!/bin/sh signal_handler() { if [ ! -d "/proc/$register_pid" ]; then wait $register_pid fi exit } trap 'signal_handler' QUIT INT MAX_REGISTER_ATTEMPTS=30 # Reset/unset the not needed flags when an authentication token RUN_UNTAGGED="" ACCESS_LEVEL="" if [ -n "$CI_SERVER_TOKEN" ] && [ "${CI_SERVER_TOKEN#glrt-}" != "$CI_SERVER_TOKEN" ]; then RUN_UNTAGGED="" ACCESS_LEVEL="" unset REGISTER_LOCKED unset RUNNER_TAG_LIST fi for i in $(seq 1 "${MAX_REGISTER_ATTEMPTS}"); do echo "Registration attempt ${i} of ${MAX_REGISTER_ATTEMPTS}" /entrypoint register \ ${RUN_UNTAGGED} \ ${ACCESS_LEVEL} \ --template-config /configmaps/config.template.toml \ --non-interactive & register_pid=$! wait $register_pid retval=$? if [ ${retval} = 0 ]; then break elif [ ${i} = ${MAX_REGISTER_ATTEMPTS} ]; then exit 1 fi sleep 5 done (root level) (v1/ConfigMap/gitlab/gitlab-gitaly) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitaly namespace: gitlab labels: app: gitaly helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: config.toml.tpl: | # The directory where Gitaly's executables are stored bin_dir = "/usr/local/bin" # listen on a TCP socket. This is insecure (no authentication) listen_addr = ":8075" # Directory where internal sockets reside # note: no value will result in a `/tmp/gitlab-internal-*` path # internal_socket_dir = "/home/git" # If metrics collection is enabled, inform gitaly about that prometheus_listen_addr = ":9236" # Graceful shutdown timeout, how long to wait for in-flight requests to complete graceful_restart_timeout = "25s" # Storage configuration {% $storages := coll.Slice "default" %} {% $hostid := index (.Env.HOSTNAME | strings.Split "-" | coll.Reverse) 0 %} {% $index := 0 %} {% if regexp.Match "^[0-9]+$" $hostid %} {% $index = conv.ToInt64 $hostid %} {% end %} {% if len $storages | lt $index %} [[storage]] name = "{% index $storages $index %}" path = "/home/git/repositories" {% else %} {% printf "Storage for node %d is not present in the storageNames array. Did you use kubectl to scale up? You need to solely use helm for this purpose." $index | fail %} {% end %} [logging] format = "json" dir = "/var/log/gitaly" [auth] token = {% file.Read "/etc/gitlab-secrets/gitaly/gitaly_token" | strings.TrimSpace | data.ToJSON %} [git] use_bundled_binaries = true ignore_gitconfig = true [gitlab-shell] # The directory where gitlab-shell is installed dir = "/srv/gitlab-shell" [gitlab] # location of shared secret for GitLab Shell / API interaction secret_file = "/etc/gitlab-secrets/shell/.gitlab_shell_secret" # URL of API url = "http://gitlab-webservice-default.gitlab.svc:8181/" [gitlab.http-settings] # read_timeout = 300 # user = someone # password = somepass # ca_file = /etc/ssl/cert.pem # ca_path = /etc/pki/tls/certs [hooks] # directory containing custom hooks custom_hooks_dir = "/home/git/custom_hooks" configure: | set -e mkdir -p /init-secrets/gitaly /init-secrets/shell cp -v -r -L /init-config/.gitlab_shell_secret /init-secrets/shell/.gitlab_shell_secret cp -v -r -L /init-config/gitaly_token /init-secrets/gitaly/gitaly_token (root level) (v1/ConfigMap/gitlab/gitlab-gitlab-exporter) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitlab-exporter namespace: gitlab labels: app: gitlab-exporter helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: configure: | # BEGIN gitlab.scripts.configure.secrets set -e config_dir="/init-config" secret_dir="/init-secrets" # optional for secret in redis redis-sentinel postgres gitlab-exporter ; do if [ -e "${config_dir}/${secret}" ]; then mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" fi done # END gitlab.scripts.configure.secrets gitlab-exporter.yml.erb: | server: name: webrick listen_address: "*" listen_port: 9168 probes: db_common: &db_common methods: - probe_db opts: connection_string: dbname=gitlab-prod user=gitlab host=db-gitlab-rw.gitlab.svc port=5432 password='<%= File.read('/etc/gitlab/postgres/psql-password-main').strip.gsub(/[\'\\]/) { |esc| '\\' + esc } %>' database: multiple: true ci_builds: class_name: Database::CiBuildsProber <<: *db_common tuple_stats: class_name: Database::TuplesProber <<: *db_common rows_count: class_name: Database::RowCountProber <<: *db_common database_bloat: class_name: Database::BloatProber <<: *db_common sidekiq: &sidekiq methods: - probe_queues - probe_workers - probe_retries - probe_stats opts: redis_url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 redis_enable_client: false probe_non_namespaced: true ruby: &ruby methods: - probe_gc opts: quantiles: false metrics: multiple: true ruby: <<: *ruby sidekiq: <<: *sidekiq ci_builds: class_name: Database::CiBuildsProber <<: *db_common tuple_stats: class_name: Database::TuplesProber <<: *db_common rows_count: class_name: Database::RowCountProber <<: *db_common pg_sequences: class_name: Database::PgSequencesProber <<: *db_common (root level) (v1/ConfigMap/gitlab/gitlab-gitlab-shell-sshd) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitlab-shell-sshd namespace: gitlab labels: app: gitlab-shell helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: sshd_config: | # $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/bin:/usr/bin:/sbin:/usr/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. Port 2222 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key #HostKey /etc/ssh/ssh_host_ed25519_key # Ciphers and keying #RekeyLimit default none # Logging #SyslogFacility AUTH LogLevel ERROR # Authentication: PermitRootLogin no #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #PubkeyAuthentication yes #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes #PermitEmptyPasswords no # Change to no to disable s/key passwords #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'yes' to enable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM no #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding no #X11DisplayOffset 10 #X11UseLocalhost yes #PermitTTY yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #PermitUserEnvironment no #Compression delayed ClientAliveInterval 0 #ClientAliveCountMax 3 #UseDNS no PidFile /srv/sshd/sshd.pid #PermitTunnel no #ChrootDirectory none #VersionAddendum none # no default banner path #Banner none # override default of no subsystems # Subsystem sftp /usr/lib/ssh/sftp-server # the following are HPN related configuration options # tcp receive buffer polling. disable in non autotuning kernels #TcpRcvBufPoll yes # disable hpn performance boosts #HPNDisabled no # buffer size for hpn to non-hpn connections #HPNBufferSize 2048 # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # PermitTTY no # ForceCommand cvs server # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys AuthorizedKeysCommand /authorized_keys %u %k AuthorizedKeysCommandUser git PasswordAuthentication no AllowUsers git # Enable the use of Git protcol v2 AcceptEnv GIT_PROTOCOL # Hard disable all forwarding DisableForwarding yes # Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. # See `man sshd_config(5)` MaxStartups 10:30:100 # Specifies amount of time athat the server will disconnect after if the user has not successfully logged in LoginGraceTime 60 # Includes Include /etc/ssh/sshd_config.d/*.conf (root level) (v1/ConfigMap/gitlab/gitlab-gitlab-shell) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitlab-shell namespace: gitlab labels: app: gitlab-shell helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: config.yml.tpl: | # GitLab user. git by default user: git # Url to gitlab instance. Used for api calls. Should end with a slash. gitlab_url: "http://gitlab-webservice-default.gitlab.svc:8181/" secret_file: /etc/gitlab-secrets/shell/.gitlab_shell_secret # File used as authorized_keys for gitlab user auth_file: "/home/git/.ssh/authorized_keys" # Log file. # Default is gitlab-shell.log in the root directory. log_file: "/var/log/gitlab-shell/gitlab-shell.log" log_format: json # Audit usernames. # Set to true to see real usernames in the logs instead of key ids, which is easier to follow, but # incurs an extra API call on every gitlab-shell command. audit_usernames: false lfs: pure_ssh_protocol: false pat: enabled: true allowed_scopes: [] configure: | # BEGIN gitlab.scripts.configure.secrets set -e config_dir="/init-config" secret_dir="/init-secrets" # required for secret in shell ; do mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" done # END gitlab.scripts.configure.secrets mkdir -p /${secret_dir}/ssh cp -v -r -L /${config_dir}/ssh_host_* /${secret_dir}/ssh/ chmod 0400 /${secret_dir}/ssh/ssh_host_* krb5.conf: (root level) (v1/ConfigMap/gitlab/gitlab-nginx-ingress-tcp) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-nginx-ingress-tcp namespace: gitlab labels: app: gitlab-shell helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: 22222: "gitlab/gitlab-gitlab-shell:22222:PROXY:" (root level) (v1/ConfigMap/gitlab/gitlab-kas) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-kas namespace: gitlab labels: app: kas helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: config.yaml: | agent: kubernetes_api: listen: address: :8154 url_path_prefix: /k8s-proxy websocket_token_secret_file: /etc/kas/.gitlab_kas_websocket_token_secret listen: address: :8150 websocket: true api: listen: address: :8153 authentication_secret_file: /etc/kas/.gitlab_kas_secret gitlab: address: http://gitlab-webservice-default.gitlab.svc:8181 authentication_secret_file: /etc/kas/.gitlab_kas_secret external_url: https://code.nogweii.net observability: listen: address: :8151 liveness_probe: url_path: /liveness readiness_probe: url_path: /readiness private_api: listen: address: :8155 authentication_secret_file: /etc/kas/.gitlab_kas_private_api_secret redis: database_index: 0 password_file: /etc/kas/redis/redis-password server: address: gitlab-redis-master.gitlab.svc:6379 (root level) (v1/ConfigMap/gitlab/gitlab-migrations) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-migrations namespace: gitlab labels: app: migrations helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: adapter: redis configure: | # BEGIN gitlab.scripts.configure.secrets set -e config_dir="/init-config" secret_dir="/init-secrets" # required for secret in rails-secrets migrations gitaly ; do mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" done # optional for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do if [ -e "${config_dir}/${secret}" ]; then mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" fi done # END gitlab.scripts.configure.secrets database.yml.erb: | production: main: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: true ci: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: false gitlab.yml.erb: | production: &base gitlab: host: code.nogweii.net https: true signup_enabled: false # Consolidated object storage configuration ## property local configuration will override object_store object_store: enabled: true proxy_download: true connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %> objects: artifacts: bucket: gitlab-artifacts proxy_download: true lfs: bucket: gitlab-lfs proxy_download: true uploads: bucket: gitlab-uploads proxy_download: true packages: bucket: gitlab-packages proxy_download: true external_diffs: bucket: gitlab-external-diffs proxy_download: true terraform_state: bucket: gitlab-terraform-state dependency_proxy: bucket: gitlab-dependency-proxy proxy_download: true ci_secure_files: bucket: gitlab-ci-secure-files pages: bucket: gitlab-pages artifacts: enabled: true lfs: enabled: true uploads: enabled: true packages: enabled: true external_diffs: enabled: true when: terraform_state: enabled: true ci_secure_files: enabled: true gitaly: client_path: /home/git/gitaly/bin token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %> repositories: storages: # You must have at least a `default` storage path. default: path: /var/opt/gitlab/repo gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075 sidekiq: log_format: json oidc_provider: openid_id_token_expire_in_seconds: 120 installation_type: | gitlab-helm-chart pages_redirect_uri: "https://projects.aether.page/auth" redis.action_cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: resque.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: (root level) (v1/ConfigMap/gitlab/gitlab-sidekiq) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-sidekiq namespace: gitlab labels: app: sidekiq helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: adapter: redis configure: | # BEGIN gitlab.scripts.configure.secrets set -e config_dir="/init-config" secret_dir="/init-secrets" # required for secret in gitaly registry rails-secrets ; do mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" done # optional for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do if [ -e "${config_dir}/${secret}" ]; then mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" fi done # END gitlab.scripts.configure.secrets database.yml.erb: | production: main: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: true ci: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: false gitlab.yml.erb: | production: &base gitlab: host: code.nogweii.net https: true ssh_host: "code.nogweii.net" action_cable_allowed_origins: [] max_request_duration_seconds: 57 impersonation_enabled: application_settings_cache_seconds: 60 usage_ping_enabled: true seat_link_enabled: true username_changing_enabled: true issue_closing_pattern: default_theme: default_projects_features: issues: true merge_requests: true wiki: true snippets: true builds: true container_registry: true signup_enabled: false graphql_timeout: webhook_timeout: trusted_proxies: time_zone: "UTC" email_from: "gitlab@aether.earth" email_display_name: "GitLab" email_reply_to: "gitlab+noreply@aether.earth" email_subject_suffix: "" gravatar: plain_url: ssl_url: extra: # Consolidated object storage configuration ## property local configuration will override object_store object_store: enabled: true proxy_download: true connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %> objects: artifacts: bucket: gitlab-artifacts proxy_download: true lfs: bucket: gitlab-lfs proxy_download: true uploads: bucket: gitlab-uploads proxy_download: true packages: bucket: gitlab-packages proxy_download: true external_diffs: bucket: gitlab-external-diffs proxy_download: true terraform_state: bucket: gitlab-terraform-state dependency_proxy: bucket: gitlab-dependency-proxy proxy_download: true ci_secure_files: bucket: gitlab-ci-secure-files pages: bucket: gitlab-pages artifacts: enabled: true lfs: enabled: true uploads: enabled: true packages: enabled: true external_diffs: enabled: true when: terraform_state: enabled: true ci_secure_files: enabled: true dependency_proxy: enabled: true sentry: enabled: false dsn: clientside_dsn: environment: sidekiq: log_format: json oidc_provider: openid_id_token_expire_in_seconds: 120 pages: enabled: false access_control: false artifacts_server: true path: /srv/gitlab/shared/pages host: aether.page port: 443 https: true secret_file: /etc/gitlab/pages/secret external_http: false external_https: false local_store: enabled: false path: namespace_in_path: false mattermost: enabled: false ## Registry Integration registry: enabled: true host: c.code.nogweii.net api_url: http://gitlab-registry.gitlab.svc:5000 key: /etc/gitlab/registry/gitlab-registry.key issuer: gitlab-issuer notification_secret: <%= YAML.load_file("/etc/gitlab/registry/notificationSecret").flatten.first %> gitlab_ci: ldap: enabled: false omniauth: enabled: true sync_profile_from_provider: ["openid_connect"] sync_profile_attributes: ["email","name"] allow_single_sign_on: ["openid_connect"] block_auto_created_users: false auto_link_ldap_user: false auto_link_saml_user: false auto_link_user: ["openid_connect"] external_providers: [] providers: - <%= YAML.load_file("/etc/gitlab/omniauth/authentik-oidc/config.yaml").to_json() %> kerberos: enabled: false keytab: /etc/krb5.keytab use_dedicated_port: false port: 8443 https: true simple_ldap_linking_allowed_realms: [] duo_auth: enabled: false hostname: integration_key: secret_key: duo_auth: enabled: false hostname: integration_key: secret_key: shared: gitaly: client_path: /home/git/gitaly/bin token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %> repositories: storages: # You must have at least a `default` storage path. default: path: /var/opt/gitlab/repo gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075 backup: path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/) gitlab_kas: enabled: true secret_file: /etc/gitlab/kas/.gitlab_kas_secret external_url: "wss://k.code.nogweii.net" internal_url: "grpc://gitlab-kas.gitlab.svc:8153" # Zoekt secrets are present, load them. zoekt: username_file: /etc/gitlab/zoekt/.gitlab_zoekt_username password_file: /etc/gitlab/zoekt/.gitlab_zoekt_password gitlab_shell: path: /home/git/gitlab-shell/ hooks_path: /home/git/gitlab-shell/hooks/ upload_pack: true receive_pack: true ssh_port: 22222 workhorse: git: bin_path: /usr/bin/git webpack: monitoring: ip_whitelist: - "127.0.0.0/8" - "::1/128" sidekiq_exporter: enabled: true address: "*" port: 3807 log_enabled: false sidekiq_health_checks: enabled: true address: "*" port: 3808 installation_type: | gitlab-helm-chart krb5.conf: redis.action_cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: resque.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: session_store.yml.erb: | production: session_cookie_token_prefix: "" smtp_settings.rb: | smtp_settings = { address: "mail-relay.email.svc.cluster.local", port: 587, ca_file: "/etc/ssl/certs/ca-certificates.crt", enable_starttls_auto: false, tls: false, openssl_verify_mode: "peer", open_timeout: 30, read_timeout: 60 } Rails.application.config.action_mailer.delivery_method = :smtp ActionMailer::Base.delivery_method = :smtp ActionMailer::Base.smtp_settings = smtp_settings (root level) (v1/ConfigMap/gitlab/gitlab-toolbox) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-toolbox namespace: gitlab labels: app: toolbox helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: adapter: redis configure: | # BEGIN gitlab.scripts.configure.secrets set -e config_dir="/init-config" secret_dir="/init-secrets" # required for secret in shell gitaly registry rails-secrets ; do mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" done # optional for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do if [ -e "${config_dir}/${secret}" ]; then mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" fi done # END gitlab.scripts.configure.secrets if [ ! -f "/${secret_dir}/objectstorage/.s3cfg" ]; then cat <<EOF > "/${secret_dir}/.s3cfg" [default] access_key = $(cat /init-secrets/minio/accesskey) secret_key = $(cat /init-secrets/minio/secretkey) bucket_location = us-east-1 host_base = minio.code.nogweii.net host_bucket = minio.code.nogweii.net/%(bucket) default_mime_type = binary/octet-stream enable_multipart = True multipart_max_chunks = 10000 multipart_chunk_size_mb = 128 recursive = True recv_chunk = 65536 send_chunk = 65536 server_side_encryption = False signature_v2 = True socket_timeout = 300 use_mime_magic = False verbosity = WARNING website_endpoint = https://minio.code.nogweii.net EOF else mv "/${secret_dir}/objectstorage/.s3cfg" "/${secret_dir}/.s3cfg" fi configure-gsutil: | if [ -e "$GOOGLE_APPLICATION_CREDENTIALS" ]; then # The following script is used to configure gsutil when creating backups # It provides inputs to the `gsutil config -e` prompt as follows: # 1) Path to service account JSON key file # 2) Do not set permissions for key file # 3) GCP Project ID # 4) Decline anonymous usage statistics printf "$GOOGLE_APPLICATION_CREDENTIALS\nN\n\nN\n" | gsutil config -e else # If the backup config is not configured, assume that # Application Default Credentials are used with a service account. # This enables GKE Workload Identity Federation to work. echo '[GoogleCompute]\nservice_account = default' > ~/.boto fi database.yml.erb: | production: main: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: true ci: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: false gitlab.yml.erb: | production: &base gitlab: host: code.nogweii.net https: true ssh_host: "code.nogweii.net" max_request_duration_seconds: 57 impersonation_enabled: application_settings_cache_seconds: 60 usage_ping_enabled: true seat_link_enabled: true username_changing_enabled: true issue_closing_pattern: default_theme: default_projects_features: issues: true merge_requests: true wiki: true snippets: true builds: true container_registry: true signup_enabled: false graphql_timeout: webhook_timeout: trusted_proxies: time_zone: "UTC" email_from: "gitlab@aether.earth" email_display_name: "GitLab" email_reply_to: "gitlab+noreply@aether.earth" email_subject_suffix: "" # Consolidated object storage configuration ## property local configuration will override object_store object_store: enabled: true proxy_download: true connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %> objects: artifacts: bucket: gitlab-artifacts proxy_download: true lfs: bucket: gitlab-lfs proxy_download: true uploads: bucket: gitlab-uploads proxy_download: true packages: bucket: gitlab-packages proxy_download: true external_diffs: bucket: gitlab-external-diffs proxy_download: true terraform_state: bucket: gitlab-terraform-state dependency_proxy: bucket: gitlab-dependency-proxy proxy_download: true ci_secure_files: bucket: gitlab-ci-secure-files pages: bucket: gitlab-pages artifacts: enabled: true lfs: enabled: true uploads: enabled: true packages: enabled: true external_diffs: enabled: true when: terraform_state: enabled: true ci_secure_files: enabled: true dependency_proxy: enabled: true sidekiq: log_format: json oidc_provider: openid_id_token_expire_in_seconds: 120 pages: enabled: false access_control: false artifacts_server: true path: /srv/gitlab/shared/pages host: aether.page port: 443 https: true secret_file: /etc/gitlab/pages/secret external_http: false external_https: false local_store: enabled: false path: namespace_in_path: false mattermost: enabled: false ## Registry Integration registry: enabled: true host: c.code.nogweii.net api_url: http://gitlab-registry.gitlab.svc:5000 key: /etc/gitlab/registry/gitlab-registry.key issuer: gitlab-issuer notification_secret: <%= YAML.load_file("/etc/gitlab/registry/notificationSecret").flatten.first %> gitlab_ci: ldap: enabled: false omniauth: enabled: true sync_profile_from_provider: ["openid_connect"] sync_profile_attributes: ["email","name"] allow_single_sign_on: ["openid_connect"] block_auto_created_users: false auto_link_ldap_user: false auto_link_saml_user: false auto_link_user: ["openid_connect"] external_providers: [] providers: - <%= YAML.load_file("/etc/gitlab/omniauth/authentik-oidc/config.yaml").to_json() %> kerberos: enabled: false keytab: /etc/krb5.keytab use_dedicated_port: false port: 8443 https: true simple_ldap_linking_allowed_realms: [] duo_auth: enabled: false hostname: integration_key: secret_key: shared: gitaly: client_path: /home/git/gitaly/bin token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %> repositories: storages: # You must have at least a `default` storage path. default: path: /var/opt/gitlab/repo gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075 backup: path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/) gitlab_kas: enabled: true secret_file: /etc/gitlab/kas/.gitlab_kas_secret external_url: "wss://k.code.nogweii.net" internal_url: "grpc://gitlab-kas.gitlab.svc:8153" # Zoekt secrets are present, load them. zoekt: username_file: /etc/gitlab/zoekt/.gitlab_zoekt_username password_file: /etc/gitlab/zoekt/.gitlab_zoekt_password gitlab_shell: path: /home/git/gitlab-shell/ hooks_path: /home/git/gitlab-shell/hooks/ upload_pack: true receive_pack: true ssh_port: 22222 secret_file: /etc/gitlab/shell/.gitlab_shell_secret workhorse: git: bin_path: /usr/bin/git webpack: monitoring: ip_whitelist: - 127.0.0.0/8 sidekiq_exporter: extra: krb5.conf: redis.action_cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: resque.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: session_store.yml.erb: | production: session_cookie_token_prefix: "" smtp_settings.rb: | smtp_settings = { address: "mail-relay.email.svc.cluster.local", port: 587, ca_file: "/etc/ssl/certs/ca-certificates.crt", enable_starttls_auto: false, tls: false, openssl_verify_mode: "peer", open_timeout: 30, read_timeout: 60 } Rails.application.config.action_mailer.delivery_method = :smtp ActionMailer::Base.delivery_method = :smtp ActionMailer::Base.smtp_settings = smtp_settings (root level) (v1/ConfigMap/gitlab/gitlab-webservice) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-webservice namespace: gitlab labels: app: webservice helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: adapter: redis configure: | # BEGIN gitlab.scripts.configure.secrets set -e config_dir="/init-config" secret_dir="/init-secrets" # required for secret in shell gitaly registry rails-secrets gitlab-workhorse ; do mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" done # optional for secret in redis redis-sentinel minio objectstorage postgres ldap duo omniauth smtp kas pages oauth-secrets mailroom gitlab-exporter microsoft_graph_mailer zoekt clickhouse openbao ; do if [ -e "${config_dir}/${secret}" ]; then mkdir -p "${secret_dir}/${secret}" cp -f -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/" fi done # END gitlab.scripts.configure.secrets database.yml.erb: | production: main: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: true ci: adapter: postgresql encoding: unicode database: gitlab-prod username: gitlab password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %> host: "db-gitlab-rw.gitlab.svc" port: 5432 connect_timeout: keepalives: keepalives_idle: keepalives_interval: keepalives_count: tcp_user_timeout: application_name: prepared_statements: false database_tasks: false gitlab.yml.erb: | production: &base gitlab: host: code.nogweii.net https: true ssh_host: "code.nogweii.net" action_cable_allowed_origins: [] max_request_duration_seconds: 57 impersonation_enabled: application_settings_cache_seconds: 60 usage_ping_enabled: true seat_link_enabled: true username_changing_enabled: true issue_closing_pattern: default_theme: default_color_mode: default_syntax_highlighting_theme: default_projects_features: issues: true merge_requests: true wiki: true snippets: true builds: true container_registry: true signup_enabled: false graphql_timeout: webhook_timeout: trusted_proxies: time_zone: "UTC" email_from: "gitlab@aether.earth" email_display_name: "GitLab" email_reply_to: "gitlab+noreply@aether.earth" email_subject_suffix: "" gravatar: plain_url: ssl_url: extra: # Consolidated object storage configuration ## property local configuration will override object_store object_store: enabled: true proxy_download: true connection: <%= YAML.load_file("/etc/gitlab/objectstorage/object_store").to_json() %> objects: artifacts: bucket: gitlab-artifacts proxy_download: true lfs: bucket: gitlab-lfs proxy_download: true uploads: bucket: gitlab-uploads proxy_download: true packages: bucket: gitlab-packages proxy_download: true external_diffs: bucket: gitlab-external-diffs proxy_download: true terraform_state: bucket: gitlab-terraform-state dependency_proxy: bucket: gitlab-dependency-proxy proxy_download: true ci_secure_files: bucket: gitlab-ci-secure-files pages: bucket: gitlab-pages artifacts: enabled: true lfs: enabled: true uploads: enabled: true packages: enabled: true external_diffs: enabled: true when: terraform_state: enabled: true ci_secure_files: enabled: true dependency_proxy: enabled: true sentry: enabled: false dsn: clientside_dsn: environment: gitlab_docs: enabled: false host: "" sidekiq: log_format: json oidc_provider: openid_id_token_expire_in_seconds: 120 pages: enabled: false access_control: false artifacts_server: true path: /srv/gitlab/shared/pages host: aether.page port: 443 https: true secret_file: /etc/gitlab/pages/secret external_http: false external_https: false local_store: enabled: false path: namespace_in_path: false mattermost: enabled: false gitlab_ci: ldap: enabled: false omniauth: enabled: true sync_profile_from_provider: ["openid_connect"] sync_profile_attributes: ["email","name"] allow_single_sign_on: ["openid_connect"] block_auto_created_users: false auto_link_ldap_user: false auto_link_saml_user: false auto_link_user: ["openid_connect"] external_providers: [] providers: - <%= YAML.load_file("/etc/gitlab/omniauth/authentik-oidc/config.yaml").to_json() %> kerberos: enabled: false keytab: /etc/krb5.keytab use_dedicated_port: false port: 8443 https: true simple_ldap_linking_allowed_realms: [] duo_auth: enabled: false hostname: integration_key: secret_key: shared: gitaly: client_path: /home/git/gitaly/bin token: <%= File.read('/etc/gitlab/gitaly/gitaly_token').strip.to_json %> repositories: storages: # You must have at least a `default` storage path. default: path: /var/opt/gitlab/repo gitaly_address: tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075 backup: path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/) gitlab_kas: enabled: true secret_file: /etc/gitlab/kas/.gitlab_kas_secret external_url: "wss://k.code.nogweii.net" internal_url: "grpc://gitlab-kas.gitlab.svc:8153" # Zoekt secrets are present, load them. zoekt: username_file: /etc/gitlab/zoekt/.gitlab_zoekt_username password_file: /etc/gitlab/zoekt/.gitlab_zoekt_password gitlab_shell: path: /home/git/gitlab-shell/ hooks_path: /home/git/gitlab-shell/hooks/ upload_pack: true receive_pack: true ssh_port: 22222 secret_file: /etc/gitlab/shell/.gitlab_shell_secret workhorse: secret_file: /etc/gitlab/gitlab-workhorse/secret git: bin_path: /usr/bin/git webpack: monitoring: ip_whitelist: - "0.0.0.0/0" - "::/0" web_exporter: enabled: true address: "0.0.0.0" port: 8083 sidekiq_exporter: shutdown: blackout_seconds: <%= ENV["SHUTDOWN_BLACKOUT_SECONDS"] %> rack_attack: git_basic_auth: ## Registry Integration registry: enabled: true host: c.code.nogweii.net api_url: http://gitlab-registry.gitlab.svc:5000 key: /etc/gitlab/registry/gitlab-registry.key issuer: gitlab-issuer notification_secret: <%= YAML.load_file("/etc/gitlab/registry/notificationSecret").flatten.first %> smartcard: enabled: false ca_file: '/etc/gitlab/rails-secrets/smartcard-ca.crt' client_certificate_required_host: smartcard.code.nogweii.net client_certificate_required_port: 443 san_extensions: false required_for_git_access: false installation_type: | gitlab-helm-chart krb5.conf: redis.action_cable.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: resque.yml.erb: | production: url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379/0 id: session_store.yml.erb: | production: session_cookie_token_prefix: "" smtp_settings.rb: | smtp_settings = { address: "mail-relay.email.svc.cluster.local", port: 587, ca_file: "/etc/ssl/certs/ca-certificates.crt", enable_starttls_auto: false, tls: false, openssl_verify_mode: "peer", open_timeout: 30, read_timeout: 60 } Rails.application.config.action_mailer.delivery_method = :smtp ActionMailer::Base.delivery_method = :smtp ActionMailer::Base.smtp_settings = smtp_settings (root level) (v1/ConfigMap/gitlab/gitlab-workhorse-default) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-workhorse-default namespace: gitlab labels: app: webservice helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: configure: | set -e mkdir -p /init-secrets-workhorse/gitlab-workhorse cp -v -r -L /init-config/gitlab-workhorse/secret /init-secrets-workhorse/gitlab-workhorse/secret mkdir -p /init-secrets-workhorse/redis cp -v -r -L /init-config/redis/redis-password /init-secrets-workhorse/redis/ if [ -f /init-config/objectstorage/object_store ]; then mkdir -p /init-secrets-workhorse/objectstorage cp -v -r -L /init-config/objectstorage/object_store /init-secrets-workhorse/objectstorage/ fi installation_type: | gitlab-helm-chart workhorse-config.toml.tpl: | shutdown_timeout = "61s" [redis] DB = 0 URL = "redis://gitlab-redis-master.gitlab.svc:6379" Password = {% file.Read "/etc/gitlab/redis/redis-password" | strings.TrimSpace | data.ToJSON %} {%- $supported_providers := slice "AWS" "AzureRM" "Google" -%} {%- $connection := coll.Dict "provider" "" -%} {%- if file.Exists "/etc/gitlab/minio/accesskey" %} {%- $aws_access_key_id := file.Read "/etc/gitlab/minio/accesskey" | strings.TrimSpace -%} {%- $aws_secret_access_key := file.Read "/etc/gitlab/minio/secretkey" | strings.TrimSpace -%} {%- $connection = coll.Dict "provider" "AWS" "aws_access_key_id" $aws_access_key_id "aws_secret_access_key" $aws_secret_access_key -%} {%- end %} {%- if file.Exists "/etc/gitlab/objectstorage/object_store" %} {%- $connection = file.Read "/etc/gitlab/objectstorage/object_store" | strings.TrimSpace | data.YAML -%} {%- end %} {%- if has $supported_providers $connection.provider %} [object_storage] provider = "{% $connection.provider %}" {%- if eq $connection.provider "AWS" %} {%- $connection = coll.Merge $connection (coll.Dict "aws_access_key_id" "" "aws_secret_access_key" "" ) %} # AWS / S3 object storage configuration. [object_storage.s3] # access/secret can be blank! aws_access_key_id = {% $connection.aws_access_key_id | strings.TrimSpace | data.ToJSON %} aws_secret_access_key = {% $connection.aws_secret_access_key | strings.TrimSpace | data.ToJSON %} {%- else if eq $connection.provider "AzureRM" %} {%- $connection = coll.Merge $connection (coll.Dict "azure_storage_account_name" "" "azure_storage_access_key" "") %} # Azure Blob storage configuration. [object_storage.azurerm] azure_storage_account_name = {% $connection.azure_storage_account_name | strings.TrimSpace | data.ToJSON %} azure_storage_access_key = {% $connection.azure_storage_access_key | strings.TrimSpace | data.ToJSON %} {%- else if eq $connection.provider "Google" %} # Google storage configuration. [object_storage.google] {% $connection | coll.Omit "provider" | data.ToTOML %} {%- end %} {%- end %} [image_resizer] max_scaler_procs = 2 max_filesize = 250000 [[listeners]] network = "tcp" addr = ":8181" (root level) (v1/ConfigMap/gitlab/gitlab-webservice-tests) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-webservice-tests namespace: gitlab labels: app: webservice helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: test_login: | set -e echo 'Start Test' endpoint="https://code.nogweii.net" cookie_read="-c /tmp/test_login.cookie" cookie_readwrite="$cookie_read -b /tmp/test_login.cookie" signin_url="$endpoint/users/sign_in" echo "Login to create a session: $signin_url" csrf=$(curl -L $signin_url --fail -s $cookie_read | grep -Po '<meta.*name="csrf-token".*content="\K[a-zA-Z0-9\+=\-\/]*') curl -X POST $signin_url -s $cookie_readwrite -F "authenticity_token=$csrf" -F 'user[login]=root' -F "user[password]=$(cat /initial_root_password)" profile_url="$endpoint/-/profile" echo "Confirm session valid: $profile_url" profile_status=$(curl -L -s -o /tmp/profile_output -w "%{http_code}" $cookie_readwrite $profile_url) if [ "$profile_status" != "200" ]; then echo "Error: Session Invalid" cat /tmp/profile_output exit 1 fi echo 'Test Passed' exit 0 (root level) (v1/ConfigMap/gitlab/gitlab-redis-configuration) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-redis-configuration namespace: gitlab labels: app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system data: master.conf: | dir /data # User-supplied master configuration: rename-command FLUSHDB "" rename-command FLUSHALL "" # End of master configuration redis.conf: | # User-supplied common configuration: # Enable AOF https://redis.io/topics/persistence#append-only-file appendonly yes # Disable RDB persistence, AOF persistence already enabled. save "" # End of common configuration replica.conf: | dir /data # User-supplied replica configuration: rename-command FLUSHDB "" rename-command FLUSHALL "" # End of replica configuration (root level) (v1/ConfigMap/gitlab/gitlab-redis-health) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-redis-health namespace: gitlab labels: app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system data: ping_liveness_local.sh: | #!/bin/bash [[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")" [[ -n "$REDIS_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_PASSWORD" response=$( timeout -s 15 $1 \ redis-cli \ -h localhost \ -p $REDIS_PORT \ ping ) if [ "$?" -eq "124" ]; then echo "Timed out" exit 1 fi responseFirstWord=$(echo $response | head -n1 | awk '{print $1;}') if [ "$response" != "PONG" ] && [ "$responseFirstWord" != "LOADING" ] && [ "$responseFirstWord" != "MASTERDOWN" ]; then echo "$response" exit 1 fi ping_liveness_local_and_master.sh: | script_dir="$(dirname "$0")" exit_status=0 "$script_dir/ping_liveness_local.sh" $1 || exit_status=$? "$script_dir/ping_liveness_master.sh" $1 || exit_status=$? exit $exit_status ping_liveness_master.sh: | #!/bin/bash [[ -f $REDIS_MASTER_PASSWORD_FILE ]] && export REDIS_MASTER_PASSWORD="$(< "${REDIS_MASTER_PASSWORD_FILE}")" [[ -n "$REDIS_MASTER_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_MASTER_PASSWORD" response=$( timeout -s 15 $1 \ redis-cli \ -h $REDIS_MASTER_HOST \ -p $REDIS_MASTER_PORT_NUMBER \ ping ) if [ "$?" -eq "124" ]; then echo "Timed out" exit 1 fi responseFirstWord=$(echo $response | head -n1 | awk '{print $1;}') if [ "$response" != "PONG" ] && [ "$responseFirstWord" != "LOADING" ]; then echo "$response" exit 1 fi ping_readiness_local.sh: | #!/bin/bash [[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")" [[ -n "$REDIS_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_PASSWORD" response=$( timeout -s 15 $1 \ redis-cli \ -h localhost \ -p $REDIS_PORT \ ping ) if [ "$?" -eq "124" ]; then echo "Timed out" exit 1 fi if [ "$response" != "PONG" ]; then echo "$response" exit 1 fi ping_readiness_local_and_master.sh: | script_dir="$(dirname "$0")" exit_status=0 "$script_dir/ping_readiness_local.sh" $1 || exit_status=$? "$script_dir/ping_readiness_master.sh" $1 || exit_status=$? exit $exit_status ping_readiness_master.sh: | #!/bin/bash [[ -f $REDIS_MASTER_PASSWORD_FILE ]] && export REDIS_MASTER_PASSWORD="$(< "${REDIS_MASTER_PASSWORD_FILE}")" [[ -n "$REDIS_MASTER_PASSWORD" ]] && export REDISCLI_AUTH="$REDIS_MASTER_PASSWORD" response=$( timeout -s 15 $1 \ redis-cli \ -h $REDIS_MASTER_HOST \ -p $REDIS_MASTER_PORT_NUMBER \ ping ) if [ "$?" -eq "124" ]; then echo "Timed out" exit 1 fi if [ "$response" != "PONG" ]; then echo "$response" exit 1 fi (root level) (v1/ConfigMap/gitlab/gitlab-redis-scripts) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-redis-scripts namespace: gitlab labels: app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system data: start-master.sh: | #!/bin/bash [[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")" if [[ -f /opt/bitnami/redis/mounted-etc/master.conf ]];then cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf fi if [[ -f /opt/bitnami/redis/mounted-etc/redis.conf ]];then cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf fi ARGS=("--port" "${REDIS_PORT}") ARGS+=("--requirepass" "${REDIS_PASSWORD}") ARGS+=("--masterauth" "${REDIS_PASSWORD}") ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf") ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf") exec redis-server "${ARGS[@]}" (root level) (v1/ConfigMap/gitlab/gitlab-registry) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-registry namespace: gitlab labels: app: registry helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: config.yml.tpl: | version: 0.1 log: fields: service: registry level: info http: addr: :5000 # `host` is not configurable # `prefix` is not configurable debug: addr: :5001 prometheus: enabled: false path: /metrics draintimeout: 0 headers: X-Content-Type-Options: [nosniff] secret: "HTTP_SECRET" relativeurls: false health: storagedriver: enabled: false interval: 10s threshold: 3 auth: token: realm: https://code.nogweii.net/jwt/auth service: container_registry issuer: gitlab-issuer # This is provided from the initContainer execution, at a known path. rootcertbundle: /etc/docker/registry/certificate.crt autoredirect: false validation: disabled: true manifests: referencelimit: 0 payloadsizelimit: 0 urls: allow: deny: notifications: reporting: profiling: storage: maintenance: readonly: enabled: false uploadpurging: enabled: true age: 168h interval: 24h dryrun: false redirect: disable: true exceptions: middleware: database: enabled: true host: "db-gitlab-rw.gitlab.svc" port: 5432 user: registry password: "DB_PASSWORD_FILE" dbname: gl-registry sslmode: disable gc: disabled: false redis: rate_limiter: enabled: false configure: | if [ -e /config/accesskey ] ; then sed -i -e 's@ACCESS_KEY@'"$(cat /config/accesskey)"'@' -e 's@SECRET_KEY@'"$(cat /config/secretkey)"'@' /registry/config.yml fi # Place the `http.secret` value from the kubernetes secret sed -i -e 's@HTTP_SECRET@'"$(cat /config/httpSecret)"'@' /registry/config.yml # Populate sensitive registry notification secrets in the config file if [ -d /config/notifications ]; then for i in /config/notifications/*; do filename=$(basename $i); sed -i -e 's@'"${filename}"'@'"$(cat $i)"'@' /registry/config.yml; done fi # Insert any provided `storage` block from kubernetes secret if [ -d /config/storage ]; then # Copy contents of storage secret(s) mkdir -p /registry/storage cp -v -r -L /config/storage/* /registry/storage/ # Ensure there is a new line in the end echo '' >> /registry/storage/config # Default `delete.enabled: true` if not present. ## Note: busybox grep doesn't support multiline, so we chain `egrep`. if ! $(egrep -A1 '^delete:\s*$' /registry/storage/config | egrep -q '\s{2,4}enabled:') ; then echo 'delete:' >> /registry/storage/config echo ' enabled: true' >> /registry/storage/config fi # Indent /registry/storage/config 2 spaces before inserting into config.yml sed -i 's/^/ /' /registry/storage/config # Insert into /registry/config.yml after `storage:` sed -i '/^storage:/ r /registry/storage/config' /registry/config.yml # Remove the now extraneous `config` file rm /registry/storage/config fi # Copy any middleware.storage if present if [ -d /config/middleware.storage ]; then cp -v -r -L /config/middleware.storage /registry/middleware.storage fi # Set to known path, to used ConfigMap cat /config/certificate.crt > /registry/certificate.crt # Copy the optional profiling keyfile to the expected location if [ -f /config/profiling-key.json ]; then cp /config/profiling-key.json /registry/profiling-key.json fi # Insert Database password, if enabled if [ -f /config/database_password ] ; then sed -i -e 's@DB_PASSWORD_FILE@'"$(cat /config/database_password)"'@' /registry/config.yml fi # Insert Redis(cache) password, if enabled if [ -f /config/registry/redis-password ] ; then sed -i -e 's@REDIS_CACHE_PASSWORD@'"$(cat /config/registry/redis-password)"'@' /registry/config.yml fi # Insert Redis(rate-limiter) password, if enabled if [ -f /config/registry/redis-rateLimiting-password ] ; then sed -i -e 's@REDIS_RATE_LIMITING_PASSWORD@'"$(cat /config/registry/redis-rateLimiting-password)"'@' /registry/config.yml fi # Insert Redis(load balancing) password, if enabled if [ -f /config/registry/redis-loadBalancing-password ] ; then sed -i -e 's@REDIS_LOAD_BALANCING_PASSWORD@'"$(cat /config/registry/redis-loadBalancing-password)"'@' /registry/config.yml fi # Copy the database TLS connection files to the expected location and set permissions if [ -d /config/ssl ]; then cp -r /config/ssl/ /registry/ssl chmod 700 /registry/ssl chmod 600 /registry/ssl/*.pem fi # Copy TLS certificates if present if [ -d /config/tls ]; then cp -r /config/tls/ /registry/tls chmod 700 /registry/tls chmod 600 /registry/tls/* fi migrations-config.yml.tpl: | version: 0.1 log: fields: service: registry level: info database: enabled: true host: "db-gitlab-rw.gitlab.svc" port: 5432 user: registry password: "DB_PASSWORD_FILE" dbname: gl-registry sslmode: disable (root level) (v1/ConfigMap/gitlab/gitlab-gitlab-chart-info) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitlab-chart-info namespace: gitlab labels: app: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab data: gitlabChartVersion: "9.8.2" gitlabVersion: "18.8.2" (root level) (rbac.authorization.k8s.io/v1/ClusterRole/gitlab-gitlab-runner) + one document added: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: gitlab-gitlab-runner labels: app: gitlab-gitlab-runner helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab rules: - resources: - events apiGroups: - verbs: - list - watch - resources: - namespaces apiGroups: - verbs: - create - delete - resources: - pods apiGroups: - verbs: - create - delete - get - list - watch - resources: - pods/attach - pods/exec apiGroups: - verbs: - get - create - patch - delete - resources: - pods/log apiGroups: - verbs: - get - list - resources: - secrets apiGroups: - verbs: - create - delete - get - update - resources: - serviceaccounts apiGroups: - verbs: - get - resources: - services apiGroups: - verbs: - create - get (root level) (rbac.authorization.k8s.io/v1/ClusterRoleBinding/gitlab-gitlab-runner) + one document added: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gitlab-gitlab-runner labels: app: gitlab-gitlab-runner helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab roleRef: name: gitlab-gitlab-runner apiGroup: rbac.authorization.k8s.io kind: ClusterRole subjects: - name: gitlab-runner-access kind: ServiceAccount namespace: gitlab (root level) (v1/Service/gitlab/gitlab-gitaly) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-gitaly namespace: gitlab labels: app: gitaly helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: type: ClusterIP clusterIP: None selector: app: gitaly release: gitlab ports: - name: tcp-gitaly port: 8075 targetPort: grpc-gitaly - name: http-metrics port: 9236 targetPort: http-metrics (root level) (v1/Service/gitlab/gitlab-gitlab-exporter) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-gitlab-exporter namespace: gitlab labels: app: gitlab-exporter helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: type: ClusterIP selector: app: gitlab-exporter release: gitlab ports: - name: http-metrics port: 9168 protocol: TCP targetPort: http-metrics (root level) (v1/Service/gitlab/gitlab-gitlab-shell) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-gitlab-shell namespace: gitlab annotations: tailscale.com/expose: "true" tailscale.com/hostname: gitlab-shell labels: app: gitlab-shell helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: type: ClusterIP selector: app: gitlab-shell release: gitlab ports: - name: ssh port: 22222 protocol: TCP targetPort: 2222 (root level) (v1/Service/gitlab/gitlab-kas) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-kas namespace: gitlab labels: app: kas helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: type: ClusterIP selector: app: kas release: gitlab ports: - name: tcp-kas-external-api port: 8150 protocol: TCP targetPort: 8150 - name: tcp-kas-internal-api port: 8153 protocol: TCP targetPort: 8153 - name: tcp-kas-k8s-api port: 8154 protocol: TCP targetPort: 8154 - name: http-metrics port: 8151 protocol: TCP targetPort: http-metrics (root level) (v1/Service/gitlab/gitlab-webservice-default) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-webservice-default namespace: gitlab labels: app: webservice gitlab.com/webservice-name: default helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: type: ClusterIP selector: app: webservice gitlab.com/webservice-name: default release: gitlab ports: - name: http-webservice port: 8080 protocol: TCP targetPort: http-webservice - name: http-workhorse port: 8181 protocol: TCP targetPort: http-workhorse - name: http-metrics-ws port: 8083 protocol: TCP targetPort: http-metrics-ws (root level) (v1/Service/gitlab/gitlab-redis-headless) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-redis-headless namespace: gitlab labels: app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system spec: type: ClusterIP clusterIP: None selector: app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis ports: - name: tcp-redis port: 6379 targetPort: redis (root level) (v1/Service/gitlab/gitlab-redis-master) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-redis-master namespace: gitlab labels: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system spec: type: ClusterIP internalTrafficPolicy: Cluster sessionAffinity: None selector: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis ports: - name: tcp-redis nodePort: null port: 6379 targetPort: redis (root level) (v1/Service/gitlab/gitlab-redis-metrics) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-redis-metrics namespace: gitlab labels: app.kubernetes.io/component: metrics app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system spec: type: ClusterIP selector: app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis ports: - name: http-metrics port: 9121 protocol: TCP targetPort: metrics (root level) (v1/Service/gitlab/gitlab-registry) + one document added: apiVersion: v1 kind: Service metadata: name: gitlab-registry namespace: gitlab labels: app: registry helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: type: ClusterIP selector: app: registry release: gitlab ports: - name: http-registry port: 5000 protocol: TCP targetPort: http (root level) (apps/v1/Deployment/gitlab/gitlab-gitlab-runner) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-gitlab-runner namespace: gitlab labels: app: gitlab-gitlab-runner helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: revisionHistoryLimit: 10 selector: matchLabels: app: gitlab-gitlab-runner template: metadata: annotations: gitlab.com/prometheus_port: "9252" gitlab.com/prometheus_scrape: "true" labels: app: gitlab-gitlab-runner heritage: Helm release: gitlab spec: containers: - name: gitlab-gitlab-runner image: "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v18.7.2" imagePullPolicy: IfNotPresent resources: {} command: - /usr/bin/dumb-init - "--" - /bin/bash - /configmaps/entrypoint ports: - name: metrics containerPort: 9252 volumeMounts: - name: projected-secrets mountPath: /secrets - name: etc-gitlab-runner mountPath: /home/gitlab-runner/.gitlab-runner - name: configmaps mountPath: /configmaps livenessProbe: exec: command: - /bin/bash - /configmaps/check-live - "3" failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 4 readinessProbe: exec: command: - /usr/bin/pgrep - "gitlab.*runner" failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 4 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL privileged: false readOnlyRootFilesystem: false runAsNonRoot: true env: - name: CI_SERVER_URL value: "https://code.nogweii.net" - name: RUNNER_EXECUTOR value: kubernetes - name: REGISTER_LOCKED value: "false" - name: RUNNER_TAG_LIST value: - name: KUBERNETES_NAMESPACE value: gitlab - name: CACHE_S3_ACCESS_KEY valueFrom: secretKeyRef: name: gitlab-object-store-configs key: runner_access_key_id - name: CACHE_S3_SECRET_KEY valueFrom: secretKeyRef: name: gitlab-object-store-configs key: runner_secret_access_key - name: SESSION_SERVER_ADDRESS value: null lifecycle: preStop: exec: command: - /entrypoint - unregister - "--all-runners" securityContext: fsGroup: 65533 runAsUser: 100 seccompProfile: type: RuntimeDefault serviceAccountName: gitlab-runner-access terminationGracePeriodSeconds: 3600 volumes: - name: runner-secrets emptyDir: medium: Memory - name: etc-gitlab-runner emptyDir: medium: Memory - name: projected-secrets projected: sources: - secret: name: gitlab-gitlab-runner-secret items: - key: runner-registration-token path: runner-registration-token - key: runner-token path: runner-token - name: configmaps configMap: name: gitlab-gitlab-runner (root level) (apps/v1/Deployment/gitlab/gitlab-gitlab-exporter) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-gitlab-exporter namespace: gitlab labels: app: gitlab-exporter app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: replicas: 1 selector: matchLabels: app: gitlab-exporter release: gitlab template: metadata: annotations: gitlab.com/prometheus_path: /metrics gitlab.com/prometheus_port: "9168" gitlab.com/prometheus_scrape: "true" prometheus.io/path: /metrics prometheus.io/port: "9168" prometheus.io/scrape: "true" labels: app: gitlab-exporter app.kubernetes.io/name: gitlab heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: gitlab-exporter release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: gitlab-exporter image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter:16.2.0" env: - name: TZ value: UTC - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab-exporter/templates - name: CONFIG_DIRECTORY value: /etc/gitlab-exporter - name: MALLOC_CONF value: "dirty_decay_ms:0,muzzy_decay_ms:0" - name: RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO value: "0.111" - name: RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO value: "0.055" - name: RUBY_GC_HEAP_INIT_SLOTS value: "80000" ports: - name: http-metrics containerPort: 9168 resources: requests: cpu: 75m memory: 100M volumeMounts: - name: gitlab-exporter-config mountPath: /var/opt/gitlab-exporter/templates/gitlab-exporter.yml.erb subPath: gitlab-exporter.yml.erb - name: gitlab-exporter-secrets mountPath: /etc/gitlab readOnly: true - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true livenessProbe: exec: command: - pgrep - "-f" - gitlab-exporter readinessProbe: exec: command: - pgrep - "-f" - gitlab-exporter securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 lifecycle: preStop: exec: command: - /bin/bash - "-c" - "pkill -f 'gitlab-exporter'" initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC - name: MALLOC_CONF value: "dirty_decay_ms:0,muzzy_decay_ms:0" - name: RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO value: "0.111" - name: RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO value: "0.055" - name: RUBY_GC_HEAP_INIT_SLOTS value: "80000" resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC - name: MALLOC_CONF value: "dirty_decay_ms:0,muzzy_decay_ms:0" - name: RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO value: "0.111" - name: RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO value: "0.055" - name: RUBY_GC_HEAP_INIT_SLOTS value: "80000" resources: requests: cpu: 50m volumeMounts: - name: gitlab-exporter-config mountPath: /config readOnly: true - name: init-gitlab-exporter-secrets mountPath: /init-config readOnly: true - name: gitlab-exporter-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault volumes: - name: gitlab-exporter-config configMap: name: gitlab-gitlab-exporter - name: init-gitlab-exporter-secrets projected: defaultMode: 256 sources: - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-ci - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-main - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password - name: gitlab-exporter-secrets emptyDir: medium: Memory - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (apps/v1/Deployment/gitlab/gitlab-gitlab-shell) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-gitlab-shell namespace: gitlab labels: app: gitlab-shell app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: selector: matchLabels: app: gitlab-shell release: gitlab template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" labels: app: gitlab-shell app.kubernetes.io/name: gitlab heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: gitlab-shell release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: gitlab-shell image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v14.45.5" env: - name: CONFIG_TEMPLATE_DIRECTORY value: /etc/gitlab-shell - name: CONFIG_DIRECTORY value: /srv/gitlab-shell - name: KEYS_DIRECTORY value: /etc/gitlab-secrets/ssh - name: SSH_DAEMON value: openssh - name: GODEBUG value: "tlsmlkem=0,tlskyber=0" - name: TZ value: UTC ports: - name: ssh containerPort: 2222 readinessProbe: failureThreshold: 2 initialDelaySeconds: 10 periodSeconds: 5 successThreshold: 1 tcpSocket: port: 2222 timeoutSeconds: 3 resources: requests: cpu: 0 memory: 6M volumeMounts: - name: shell-config mountPath: /etc/gitlab-shell - name: shell-secrets mountPath: /etc/gitlab-secrets readOnly: true - name: shell-config mountPath: /etc/krb5.conf readOnly: true subPath: krb5.conf - name: sshd-config mountPath: /etc/ssh/sshd_config readOnly: true subPath: sshd_config - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true livenessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: shell-config mountPath: /config readOnly: true - name: shell-init-secrets mountPath: /init-config readOnly: true - name: shell-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault terminationGracePeriodSeconds: 30 volumes: - name: shell-config configMap: name: gitlab-gitlab-shell - name: sshd-config configMap: name: gitlab-gitlab-shell-sshd - name: shell-init-secrets projected: defaultMode: 288 sources: - secret: name: gitlab-gitlab-shell-host-keys - secret: name: gitlab-gitlab-shell-secret items: - key: secret path: shell/.gitlab_shell_secret - name: shell-secrets emptyDir: medium: Memory - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (apps/v1/Deployment/gitlab/gitlab-kas) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-kas namespace: gitlab labels: app: kas app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: selector: matchLabels: app: kas release: gitlab template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" gitlab.com/prometheus_path: /metrics gitlab.com/prometheus_port: "8151" gitlab.com/prometheus_scrape: "true" prometheus.io/path: /metrics prometheus.io/port: "8151" prometheus.io/scrape: "true" labels: app: kas app.kubernetes.io/name: gitlab heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: kas release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: kas image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-kas:v18.8.2" args: - "--configuration-file=/etc/kas/config.yaml" env: - name: OWN_PRIVATE_API_HOST value: gitlab-kas.gitlab.svc - name: GODEBUG value: "tlsmlkem=0,tlskyber=0" - name: TZ value: UTC livenessProbe: httpGet: path: /liveness port: 8151 initialDelaySeconds: 15 periodSeconds: 20 ports: - name: kas containerPort: 8150 - name: kas-k8s-api containerPort: 8154 - name: kas-private-api containerPort: 8155 - name: http-metrics containerPort: 8151 readinessProbe: httpGet: path: /readiness port: 8151 initialDelaySeconds: 5 periodSeconds: 10 resources: requests: cpu: 100m memory: 256Mi volumeMounts: - name: init-etc-kas mountPath: /etc/kas readOnly: true - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 65532 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true securityContext: fsGroup: 65532 runAsGroup: 65532 runAsUser: 65532 seccompProfile: type: RuntimeDefault terminationGracePeriodSeconds: 300 volumes: - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca - name: init-etc-kas projected: defaultMode: 288 sources: - configMap: name: gitlab-kas - secret: name: gitlab-gitlab-kas-secret items: - key: kas_shared_secret path: .gitlab_kas_secret - secret: name: gitlab-kas-private-api items: - key: kas_private_api_secret path: .gitlab_kas_private_api_secret - secret: name: gitlab-kas-websocket-token items: - key: kas_websocket_token_secret path: .gitlab_kas_websocket_token_secret - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password (root level) (apps/v1/Deployment/gitlab/gitlab-sidekiq-all-in-1-v2) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-sidekiq-all-in-1-v2 namespace: gitlab labels: app: sidekiq app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm queue-pod-name: all-in-1 release: gitlab spec: selector: matchLabels: app: sidekiq queue-pod-name: all-in-1 release: gitlab template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" gitlab.com/prometheus_path: /metrics gitlab.com/prometheus_port: "3807" gitlab.com/prometheus_scrape: "true" prometheus.io/path: /metrics prometheus.io/port: "3807" prometheus.io/scrape: "true" labels: app: sidekiq app.kubernetes.io/name: gitlab heritage: Helm queue-pod-name: all-in-1 release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: sidekiq release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: sidekiq image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v18.8.2" env: - name: prometheus_multiproc_dir value: /metrics - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: SIDEKIQ_CONCURRENCY value: "20" - name: SIDEKIQ_TIMEOUT value: "25" - name: SIDEKIQ_QUEUES value: null - name: SIDEKIQ_DAEMON_MEMORY_KILLER value: "1" - name: SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL value: "3" - name: SIDEKIQ_MEMORY_KILLER_MAX_RSS value: "2000000" - name: SIDEKIQ_MEMORY_KILLER_GRACE_TIME value: "900" - name: SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT value: "30" - name: ENABLE_BOOTSNAP value: "1" - name: TZ value: UTC livenessProbe: failureThreshold: 3 httpGet: path: /liveness port: 3808 initialDelaySeconds: 20 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 30 ports: - name: http-metrics containerPort: 3807 readinessProbe: failureThreshold: 3 httpGet: path: /readiness port: 3808 initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 resources: requests: cpu: 900m memory: 2G volumeMounts: - name: sidekiq-metrics mountPath: /metrics - name: sidekiq-config mountPath: /var/opt/gitlab/templates readOnly: true - name: sidekiq-secrets mountPath: /etc/gitlab readOnly: true - name: sidekiq-secrets mountPath: /srv/gitlab/config/secrets.yml subPath: rails-secrets/secrets.yml - name: sidekiq-config mountPath: /etc/krb5.conf subPath: krb5.conf - name: sidekiq-config mountPath: /srv/gitlab/config/initializers/smtp_settings.rb subPath: smtp_settings.rb - name: sidekiq-config mountPath: /srv/gitlab/INSTALLATION_TYPE subPath: installation_type - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 lifecycle: preStop: exec: command: - /bin/bash - "-c" - "pkill -f 'sidekiq'" initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: sidekiq-config mountPath: /config readOnly: true - name: init-sidekiq-secrets mountPath: /init-config readOnly: true - name: sidekiq-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: dependencies image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v18.8.2" args: - /scripts/wait-for-deps env: - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: SIDEKIQ_CONCURRENCY value: "20" - name: SIDEKIQ_TIMEOUT value: "25" - name: ENABLE_BOOTSNAP value: "1" - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true - name: sidekiq-config mountPath: /var/opt/gitlab/templates readOnly: true - name: sidekiq-secrets mountPath: /etc/gitlab readOnly: true - name: sidekiq-secrets mountPath: /srv/gitlab/config/secrets.yml readOnly: true subPath: rails-secrets/secrets.yml securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault terminationGracePeriodSeconds: 30 volumes: - name: sidekiq-metrics emptyDir: medium: Memory - name: sidekiq-config projected: sources: - configMap: name: gitlab-sidekiq - name: init-sidekiq-secrets projected: defaultMode: 256 sources: - secret: name: gitlab-rails-secret items: - key: secrets.yml path: rails-secrets/secrets.yml - secret: name: gitlab-gitaly-secret items: - key: token path: gitaly/gitaly_token - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-ci - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-main - secret: name: gitlab-registry-secret items: - key: registry-auth.key path: registry/gitlab-registry.key - secret: name: gitlab-registry-notification items: - key: secret path: registry/notificationSecret - secret: name: gitlab-gitlab-kas-secret items: - key: kas_shared_secret path: kas/.gitlab_kas_secret - secret: name: gitlab-zoekt-basicauth optional: true items: - key: gitlab_username path: zoekt/.gitlab_zoekt_username - key: gitlab_password path: zoekt/.gitlab_zoekt_password - secret: name: gitlab-object-store-configs items: - key: connection.yaml path: objectstorage/object_store - secret: name: authentik-oidc items: - key: config.yaml path: omniauth/authentik-oidc/config.yaml - name: sidekiq-secrets emptyDir: medium: Memory - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (apps/v1/Deployment/gitlab/gitlab-toolbox) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-toolbox namespace: gitlab labels: app: toolbox app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: replicas: 1 selector: matchLabels: app: toolbox release: gitlab strategy: type: Recreate rollingUpdate: null template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" labels: app: toolbox app.kubernetes.io/name: gitlab heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: gitaly release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: toolbox image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce:v18.8.2" args: - /bin/bash - "-c" - "cp -v -r -L /etc/gitlab/.s3cfg $HOME/.s3cfg && while sleep 3600; do :; done" env: - name: ARTIFACTS_BUCKET_NAME value: gitlab-artifacts - name: REGISTRY_BUCKET_NAME value: gitlab-container-registry - name: LFS_BUCKET_NAME value: gitlab-lfs - name: UPLOADS_BUCKET_NAME value: gitlab-uploads - name: PACKAGES_BUCKET_NAME value: gitlab-packages - name: EXTERNAL_DIFFS_BUCKET_NAME value: gitlab-external-diffs - name: TERRAFORM_STATE_BUCKET_NAME value: gitlab-terraform-state - name: CI_SECURE_FILES_BUCKET_NAME value: gitlab-ci-secure-files - name: BACKUP_BUCKET_NAME value: gitlab-backup - name: BACKUP_BACKEND value: s3 - name: TMP_BUCKET_NAME value: gitlab-tmp-storage - name: PAGES_BUCKET_NAME value: gitlab-pages - name: ENABLE_BOOTSNAP value: "1" - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: TZ value: UTC resources: requests: cpu: 50m memory: 350M volumeMounts: - name: toolbox-config mountPath: /var/opt/gitlab/templates - name: toolbox-config mountPath: /etc/krb5.conf subPath: krb5.conf - name: toolbox-config mountPath: /srv/gitlab/config/initializers/smtp_settings.rb subPath: smtp_settings.rb - name: toolbox-secrets mountPath: /etc/gitlab readOnly: true - name: toolbox-secrets mountPath: /srv/gitlab/config/secrets.yml subPath: rails-secrets/secrets.yml - name: toolbox-tmp mountPath: /srv/gitlab/tmp readOnly: false - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: toolbox-config mountPath: /config readOnly: true - name: init-toolbox-secrets mountPath: /init-config readOnly: true - name: toolbox-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 securityContext: fsGroup: 1000 runAsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault volumes: - name: toolbox-config projected: sources: - configMap: name: gitlab-toolbox - name: toolbox-tmp emptyDir: {} - name: init-toolbox-secrets projected: defaultMode: 256 sources: - secret: name: gitlab-rails-secret items: - key: secrets.yml path: rails-secrets/secrets.yml - secret: name: gitlab-gitlab-shell-secret items: - key: secret path: shell/.gitlab_shell_secret - secret: name: gitlab-gitaly-secret items: - key: token path: gitaly/gitaly_token - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-ci - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-main - secret: name: gitlab-registry-secret items: - key: registry-auth.key path: registry/gitlab-registry.key - secret: name: gitlab-registry-notification items: - key: secret path: registry/notificationSecret - secret: name: gitlab-object-store-configs items: - key: s3cmd.cfg path: objectstorage/.s3cfg - secret: name: gitlab-gitlab-kas-secret items: - key: kas_shared_secret path: kas/.gitlab_kas_secret - secret: name: gitlab-zoekt-basicauth optional: true items: - key: gitlab_username path: zoekt/.gitlab_zoekt_username - key: gitlab_password path: zoekt/.gitlab_zoekt_password - secret: name: gitlab-object-store-configs items: - key: connection.yaml path: objectstorage/object_store - secret: name: authentik-oidc items: - key: config.yaml path: omniauth/authentik-oidc/config.yaml - name: toolbox-secrets emptyDir: medium: Memory - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (apps/v1/Deployment/gitlab/gitlab-webservice-default) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-webservice-default namespace: gitlab labels: app: webservice app.kubernetes.io/name: gitlab gitlab.com/webservice-name: default helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: selector: matchLabels: app: webservice gitlab.com/webservice-name: default release: gitlab template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" gitlab.com/prometheus_path: /metrics gitlab.com/prometheus_port: "8083" gitlab.com/prometheus_scrape: "true" prometheus.io/path: /metrics prometheus.io/port: "8083" prometheus.io/scrape: "true" labels: app: webservice app.kubernetes.io/name: gitlab gitlab.com/webservice-name: default heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: webservice gitlab.com/webservice-name: default release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: webservice image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce:v18.8.2" env: - name: TZ value: UTC - name: GITLAB_WEBSERVER value: puma - name: TMPDIR value: /tmp/gitlab - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: prometheus_multiproc_dir value: /metrics - name: ENABLE_BOOTSNAP value: "1" - name: WORKER_PROCESSES value: "2" - name: WORKER_TIMEOUT value: "60" - name: INTERNAL_PORT value: "8080" - name: PUMA_THREADS_MIN value: "4" - name: PUMA_THREADS_MAX value: "4" - name: PUMA_WORKER_MAX_MEMORY value: - name: DISABLE_PUMA_WORKER_KILLER value: "true" - name: BIND_IP6 value: "false" - name: PUMA_CONTROL_PORT value: "9293" - name: SHUTDOWN_BLACKOUT_SECONDS value: "10" - name: WORKHORSE_ARCHIVE_CACHE_DISABLED value: "true" livenessProbe: failureThreshold: 3 httpGet: path: /-/liveness port: 8080 initialDelaySeconds: 20 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 30 ports: - name: http-webservice containerPort: 8080 - name: http-metrics-ws containerPort: 8083 readinessProbe: failureThreshold: 2 httpGet: path: /-/readiness port: 8080 initialDelaySeconds: 0 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 2 resources: requests: cpu: 300m memory: 2.5G volumeMounts: - name: webservice-metrics mountPath: /metrics - name: webservice-config mountPath: /var/opt/gitlab/templates - name: webservice-secrets mountPath: /etc/gitlab readOnly: true - name: webservice-secrets mountPath: /srv/gitlab/config/secrets.yml subPath: rails-secrets/secrets.yml - name: webservice-config mountPath: /etc/krb5.conf subPath: krb5.conf - name: webservice-config mountPath: /srv/gitlab/config/initializers/smtp_settings.rb subPath: smtp_settings.rb - name: webservice-config mountPath: /srv/gitlab/INSTALLATION_TYPE subPath: installation_type - name: shared-upload-directory mountPath: /srv/gitlab/public/uploads/tmp readOnly: false - name: shared-tmp mountPath: /tmp readOnly: false - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 lifecycle: preStop: exec: command: - /bin/bash - "-c" - "pkill -SIGINT -o ruby" - name: gitlab-workhorse image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce:v18.8.2" env: - name: TZ value: UTC - name: GODEBUG value: "tlsmlkem=0,tlskyber=0" - name: TMPDIR value: /tmp/gitlab - name: GITLAB_WORKHORSE_AUTH_BACKEND value: "http://localhost:8080" - name: GITLAB_WORKHORSE_EXTRA_ARGS value: - name: GITLAB_WORKHORSE_LISTEN_PORT value: "8181" - name: GITLAB_WORKHORSE_LOG_FORMAT value: json - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: SHUTDOWN_BLACKOUT_SECONDS value: "10" - name: PUMA_CONTROL_PORT value: "9293" ports: - name: http-workhorse containerPort: 8181 resources: requests: cpu: 100m memory: 100M volumeMounts: - name: workhorse-config mountPath: /var/opt/gitlab/templates - name: workhorse-secrets mountPath: /etc/gitlab readOnly: true - name: shared-upload-directory mountPath: /srv/gitlab/public/uploads/tmp readOnly: false - name: shared-tmp mountPath: /tmp readOnly: false - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true livenessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 initialDelaySeconds: 20 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 30 readinessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 lifecycle: preStop: exec: command: - /bin/bash - "-c" - "sleep $SHUTDOWN_BLACKOUT_SECONDS" initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" args: - "-c" - "sh -x /config-webservice/configure ; sh -x /config-workhorse/configure ; mkdir -p -m 3770 /tmp/gitlab" command: - sh env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: webservice-config mountPath: /config-webservice readOnly: true - name: workhorse-config mountPath: /config-workhorse readOnly: true - name: init-webservice-secrets mountPath: /init-config readOnly: true - name: webservice-secrets mountPath: /init-secrets readOnly: false - name: workhorse-secrets mountPath: /init-secrets-workhorse readOnly: false - name: shared-tmp mountPath: /tmp readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: dependencies image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce:v18.8.2" args: - /scripts/wait-for-deps env: - name: TZ value: UTC - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: WORKHORSE_ARCHIVE_CACHE_DISABLED value: "1" - name: ENABLE_BOOTSNAP value: "1" resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true - name: webservice-config mountPath: /var/opt/gitlab/templates - name: webservice-secrets mountPath: /etc/gitlab readOnly: true - name: webservice-secrets mountPath: /srv/gitlab/config/secrets.yml readOnly: true subPath: rails-secrets/secrets.yml securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault volumes: - name: shared-tmp emptyDir: {} - name: webservice-metrics emptyDir: medium: Memory - name: webservice-config configMap: name: gitlab-webservice - name: workhorse-config configMap: name: gitlab-workhorse-default - name: init-webservice-secrets projected: defaultMode: 256 sources: - secret: name: gitlab-rails-secret items: - key: secrets.yml path: rails-secrets/secrets.yml - secret: name: gitlab-gitlab-shell-secret items: - key: secret path: shell/.gitlab_shell_secret - secret: name: gitlab-gitaly-secret items: - key: token path: gitaly/gitaly_token - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-ci - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-main - secret: name: gitlab-registry-secret items: - key: registry-auth.key path: registry/gitlab-registry.key - secret: name: gitlab-registry-notification items: - key: secret path: registry/notificationSecret - secret: name: gitlab-gitlab-workhorse-secret items: - key: shared_secret path: gitlab-workhorse/secret - secret: name: gitlab-gitlab-kas-secret items: - key: kas_shared_secret path: kas/.gitlab_kas_secret - secret: name: gitlab-zoekt-basicauth optional: true items: - key: gitlab_username path: zoekt/.gitlab_zoekt_username - key: gitlab_password path: zoekt/.gitlab_zoekt_password - secret: name: gitlab-object-store-configs items: - key: connection.yaml path: objectstorage/object_store - secret: name: authentik-oidc items: - key: config.yaml path: omniauth/authentik-oidc/config.yaml - name: webservice-secrets emptyDir: medium: Memory - name: workhorse-secrets emptyDir: medium: Memory - name: shared-upload-directory emptyDir: {} - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (apps/v1/Deployment/gitlab/gitlab-registry) + one document added: apiVersion: apps/v1 kind: Deployment metadata: name: gitlab-registry namespace: gitlab labels: app: registry app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: selector: matchLabels: app: registry release: gitlab template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" labels: app: registry app.kubernetes.io/name: gitlab heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: registry release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: registry image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab" env: - name: GODEBUG value: "tlsmlkem=0,tlskyber=0" - name: TZ value: UTC livenessProbe: failureThreshold: 3 httpGet: path: /debug/health port: debug scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 ports: - name: http containerPort: 5000 - name: debug containerPort: 5001 readinessProbe: failureThreshold: 3 httpGet: path: /debug/health port: debug scheme: HTTP initialDelaySeconds: 5 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 50m memory: 32Mi volumeMounts: - name: registry-server-config mountPath: /etc/docker/registry/ readOnly: true - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 lifecycle: preStop: exec: command: - /bin/bash - "-c" - "sleep 5" initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" args: - sh - /config/configure env: - name: CONFIG_TEMPLATE_DIRECTORY value: /templates - name: CONFIG_DIRECTORY value: /registry - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: registry-secrets mountPath: /config - name: registry-server-config-templates mountPath: /templates readOnly: true - name: registry-server-config mountPath: /registry securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: dependencies image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab" command: - /scripts/wait-for-deps env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: registry-server-config mountPath: /etc/docker/registry/ readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault terminationGracePeriodSeconds: 30 volumes: - name: registry-server-config-templates configMap: name: gitlab-registry items: - key: config.yml.tpl path: config.yml.tpl - name: registry-server-config emptyDir: medium: Memory - name: registry-secrets projected: sources: - configMap: name: gitlab-registry - secret: name: gitlab-registry-secret items: - key: registry-auth.crt path: certificate.crt - secret: name: gitlab-registry-httpsecret items: - key: secret path: httpSecret - secret: name: gitlab-object-store-configs items: - key: registry.yaml path: storage/config - secret: name: gitlab-registry-db-pass items: - key: password path: database_password - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-gitlab-shell) + one document added: apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: gitlab-gitlab-shell namespace: gitlab labels: app: gitlab-shell helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageValue: 100m minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gitlab-gitlab-shell (root level) (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-kas) + one document added: apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: gitlab-kas namespace: gitlab labels: app: kas helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageValue: 100m minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gitlab-kas (root level) (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-sidekiq-all-in-1-v2) + one document added: apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: gitlab-sidekiq-all-in-1-v2 namespace: gitlab labels: app: sidekiq helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageValue: 350m minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gitlab-sidekiq-all-in-1-v2 (root level) (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-webservice-default) + one document added: apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: gitlab-webservice-default namespace: gitlab labels: app: webservice gitlab.com/webservice-name: default helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageValue: 1 minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gitlab-webservice-default (root level) (autoscaling/v2beta1/HorizontalPodAutoscaler/gitlab/gitlab-registry) + one document added: apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: gitlab-registry namespace: gitlab labels: app: registry helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 75 minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gitlab-registry (root level) (apps/v1/StatefulSet/gitlab/gitlab-gitaly) + one document added: apiVersion: apps/v1 kind: StatefulSet metadata: name: gitlab-gitaly namespace: gitlab labels: app: gitaly app.kubernetes.io/name: gitlab helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: podManagementPolicy: Parallel replicas: 1 selector: matchLabels: app: gitaly release: gitlab serviceName: gitlab-gitaly template: metadata: annotations: gitlab.com/prometheus_path: /metrics gitlab.com/prometheus_port: "9236" gitlab.com/prometheus_scrape: "true" prometheus.io/path: /metrics prometheus.io/port: "9236" prometheus.io/scrape: "true" labels: app: gitaly app.kubernetes.io/name: gitlab heritage: Helm release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: gitaly release: gitlab topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: gitaly image: "registry.gitlab.com/gitlab-org/build/cng/gitaly:v18.8.2" env: - name: CONFIG_TEMPLATE_DIRECTORY value: /etc/gitaly/templates - name: CONFIG_DIRECTORY value: /etc/gitaly - name: GITALY_CONFIG_FILE value: /etc/gitaly/config.toml - name: SSL_CERT_DIR value: /etc/ssl/certs - name: GODEBUG value: "tlsmlkem=0,tlskyber=0" - name: TZ value: UTC livenessProbe: failureThreshold: 3 grpc: port: 8075 initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 ports: - name: grpc-gitaly containerPort: 8075 - name: http-metrics containerPort: 9236 readinessProbe: failureThreshold: 3 grpc: port: 8075 initialDelaySeconds: 0 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 3 resources: requests: cpu: 100m memory: 200Mi startupProbe: failureThreshold: 60 grpc: port: 8075 initialDelaySeconds: 1 periodSeconds: 1 successThreshold: 1 timeoutSeconds: 1 volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true - name: gitaly-config mountPath: /etc/gitaly/templates - name: gitaly-secrets mountPath: /etc/gitlab-secrets readOnly: true - name: repo-data mountPath: /home/git/repositories securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: gitaly-config mountPath: /config readOnly: true - name: init-gitaly-secrets mountPath: /init-config readOnly: true - name: gitaly-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true securityContext: fsGroup: 1000 runAsUser: 1000 terminationGracePeriodSeconds: 30 volumes: - name: gitaly-config configMap: name: gitlab-gitaly - name: gitaly-secrets emptyDir: medium: Memory - name: init-gitaly-secrets projected: defaultMode: 288 sources: - secret: name: gitlab-gitaly-secret items: - key: token path: gitaly_token - secret: name: gitlab-gitlab-shell-secret items: - key: secret path: .gitlab_shell_secret - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca volumeClaimTemplates: - metadata: name: repo-data annotations: null labels: app: gitaly release: gitlab spec: resources: requests: storage: 50Gi selector: null storageClassName: nfs-subdir accessModes: - ReadWriteOnce (root level) (apps/v1/StatefulSet/gitlab/gitlab-redis-master) + one document added: apiVersion: apps/v1 kind: StatefulSet metadata: name: gitlab-redis-master namespace: gitlab labels: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system spec: replicas: 1 selector: matchLabels: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis serviceName: gitlab-redis-headless template: metadata: annotations: prometheus.io/port: "9121" prometheus.io/scrape: "true" labels: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: redis spec: affinity: nodeAffinity: null podAffinity: null podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - name: redis image: "docker.io/bitnamilegacy/redis:7.2.4-debian-12-r9" imagePullPolicy: IfNotPresent args: - "-c" - /opt/bitnami/scripts/start-scripts/start-master.sh command: - /bin/bash env: - name: BITNAMI_DEBUG value: "false" - name: REDIS_REPLICATION_MODE value: master - name: ALLOW_EMPTY_PASSWORD value: no - name: REDIS_PASSWORD_FILE value: /opt/bitnami/redis/secrets/redis-password - name: REDIS_TLS_ENABLED value: no - name: REDIS_PORT value: "6379" ports: - name: redis containerPort: 6379 volumeMounts: - name: start-scripts mountPath: /opt/bitnami/scripts/start-scripts - name: health mountPath: /health - name: redis-password mountPath: /opt/bitnami/redis/secrets/ - name: redis-data mountPath: /data - name: config mountPath: /opt/bitnami/redis/mounted-etc - name: empty-dir mountPath: /opt/bitnami/redis/etc/ subPath: app-conf-dir - name: empty-dir mountPath: /tmp subPath: tmp-dir livenessProbe: exec: command: - sh - "-c" - "/health/ping_liveness_local.sh 5" failureThreshold: 5 initialDelaySeconds: 20 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 6 readinessProbe: exec: command: - sh - "-c" - "/health/ping_readiness_local.sh 1" failureThreshold: 5 initialDelaySeconds: 20 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 2 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsGroup: 0 runAsNonRoot: true runAsUser: 1001 seLinuxOptions: null seccompProfile: type: RuntimeDefault - name: metrics image: "docker.io/bitnamilegacy/redis-exporter:1.58.0-debian-12-r4" imagePullPolicy: IfNotPresent command: - /bin/bash - "-c" - | if [[ -f '/secrets/redis-password' ]]; then export REDIS_PASSWORD=$(cat /secrets/redis-password) fi redis_exporter env: - name: REDIS_ALIAS value: gitlab-redis - name: REDIS_EXPORTER_WEB_LISTEN_ADDRESS value: ":9121" - name: REDIS_USER value: default livenessProbe: failureThreshold: 5 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: metrics timeoutSeconds: 5 ports: - name: metrics containerPort: 9121 readinessProbe: failureThreshold: 3 httpGet: path: / port: metrics initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 volumeMounts: - name: empty-dir mountPath: /tmp subPath: app-tmp-dir - name: redis-password mountPath: /secrets/ securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsGroup: 0 runAsNonRoot: true runAsUser: 1001 seLinuxOptions: null seccompProfile: type: RuntimeDefault enableServiceLinks: true securityContext: fsGroup: 1001 fsGroupChangePolicy: Always supplementalGroups: [] sysctls: [] serviceAccountName: gitlab-redis-master terminationGracePeriodSeconds: 30 volumes: - name: start-scripts configMap: name: gitlab-redis-scripts defaultMode: 493 - name: health configMap: name: gitlab-redis-health defaultMode: 493 - name: redis-password secret: items: - key: secret path: redis-password secretName: gitlab-redis-secret - name: config configMap: name: gitlab-redis-configuration - name: empty-dir emptyDir: {} updateStrategy: type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data labels: app.kubernetes.io/component: master app.kubernetes.io/instance: gitlab app.kubernetes.io/name: redis spec: resources: requests: storage: 8Gi storageClassName: longhorn accessModes: - ReadWriteOnce (root level) (batch/v1/Job/gitlab/gitlab-migrations-c0164a8) + one document added: apiVersion: batch/v1 kind: Job metadata: name: gitlab-migrations-c0164a8 namespace: gitlab labels: app: migrations helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: activeDeadlineSeconds: 3600 backoffLimit: 6 template: metadata: labels: app: migrations heritage: Helm release: gitlab spec: automountServiceAccountToken: false containers: - name: migrations image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce:v18.8.2" args: - /bin/bash - "-c" - | set -e; /scripts/wait-for-deps; /scripts/db-migrate; env: - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: BYPASS_SCHEMA_VERSION value: "true" - name: ENABLE_BOOTSNAP value: "1" - name: TZ value: UTC resources: requests: cpu: 250m memory: 200Mi volumeMounts: - name: migrations-config mountPath: /var/opt/gitlab/templates - name: migrations-secrets mountPath: /etc/gitlab readOnly: true - name: migrations-secrets mountPath: /srv/gitlab/config/secrets.yml subPath: rails-secrets/secrets.yml - name: migrations-secrets mountPath: /srv/gitlab/config/initial_root_password subPath: migrations/initial_root_password - name: migrations-secrets mountPath: /srv/gitlab/config/gitlab_shared_runners_registration_token subPath: migrations/gitlab_shared_runners_registration_token - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: migrations-config mountPath: /config readOnly: true - name: init-migrations-secrets mountPath: /init-config readOnly: true - name: migrations-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true restartPolicy: OnFailure securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault volumes: - name: migrations-config configMap: name: gitlab-migrations - name: init-migrations-secrets projected: defaultMode: 256 sources: - secret: name: gitlab-rails-secret items: - key: secrets.yml path: rails-secrets/secrets.yml - secret: name: gitlab-gitaly-secret items: - key: token path: gitaly/gitaly_token - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-ci - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-main - secret: name: gitlab-object-store-configs items: - key: connection.yaml path: objectstorage/object_store - secret: name: gitlab-gitlab-initial-root-password items: - key: password path: migrations/initial_root_password - secret: name: gitlab-gitlab-runner-secret optional: true items: - key: runner-registration-token path: migrations/gitlab_shared_runners_registration_token - name: migrations-secrets emptyDir: medium: Memory - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (batch/v1/Job/gitlab/gitlab-registry-migrations-8e2a9a9) + one document added: apiVersion: batch/v1 kind: Job metadata: name: gitlab-registry-migrations-8e2a9a9 namespace: gitlab labels: app: registry-migrations helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: activeDeadlineSeconds: 3600 backoffLimit: 6 template: metadata: labels: app: registry-migrations heritage: Helm release: gitlab spec: containers: - name: migrations image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab" args: - /scripts/wait-for-deps - /scripts/db-migrate env: - name: BYPASS_SCHEMA_VERSION value: "true" resources: requests: cpu: 50m memory: 32Mi volumeMounts: - name: registry-server-config mountPath: /etc/docker/registry/ readOnly: true - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" args: - sh - /config/configure env: - name: CONFIG_TEMPLATE_DIRECTORY value: /templates - name: CONFIG_DIRECTORY value: /registry resources: requests: cpu: 50m volumeMounts: - name: registry-secrets mountPath: /config - name: registry-server-config-templates mountPath: /templates readOnly: true - name: registry-server-config mountPath: /registry securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 restartPolicy: OnFailure securityContext: fsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault volumes: - name: registry-server-config-templates configMap: name: gitlab-registry items: - key: migrations-config.yml.tpl path: config.yml.tpl - name: registry-server-config emptyDir: medium: Memory - name: registry-secrets projected: sources: - configMap: name: gitlab-registry items: - key: configure path: configure - secret: name: gitlab-registry-db-pass items: - key: password path: database_password - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca (root level) (batch/v1beta1/CronJob/gitlab/gitlab-toolbox-backup) + one document added: apiVersion: batch/v1beta1 kind: CronJob metadata: name: gitlab-toolbox-backup namespace: gitlab labels: app: toolbox helm.toolkit.fluxcd.io/name: gitlab helm.toolkit.fluxcd.io/namespace: flux-system heritage: Helm release: gitlab spec: concurrencyPolicy: Replace failedJobsHistoryLimit: 1 jobTemplate: spec: backoffLimit: 6 template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false" labels: app: toolbox heritage: Helm release: gitlab spec: containers: - name: toolbox-backup image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce:v18.8.2" args: - /bin/bash - "-c" - "cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility --skip registry --skip packages --skip pages --skip artifacts" env: - name: ARTIFACTS_BUCKET_NAME value: gitlab-artifacts - name: REGISTRY_BUCKET_NAME value: gitlab-container-registry - name: LFS_BUCKET_NAME value: gitlab-lfs - name: UPLOADS_BUCKET_NAME value: gitlab-uploads - name: PACKAGES_BUCKET_NAME value: gitlab-packages - name: EXTERNAL_DIFFS_BUCKET_NAME value: gitlab-external-diffs - name: TERRAFORM_STATE_BUCKET_NAME value: gitlab-terraform-state - name: CI_SECURE_FILES_BUCKET_NAME value: gitlab-ci-secure-files - name: BACKUP_BUCKET_NAME value: gitlab-backup - name: BACKUP_BACKEND value: s3 - name: TMP_BUCKET_NAME value: gitlab-tmp-storage - name: PAGES_BUCKET_NAME value: gitlab-pages - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: TZ value: UTC resources: requests: cpu: 50m memory: 350M volumeMounts: - name: toolbox-config mountPath: /var/opt/gitlab/templates - name: toolbox-secrets mountPath: /etc/gitlab readOnly: true - name: toolbox-secrets mountPath: /srv/gitlab/config/secrets.yml subPath: rails-secrets/secrets.yml - name: toolbox-tmp mountPath: /srv/gitlab/tmp readOnly: false - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 initContainers: - name: certificates image: "registry.gitlab.com/gitlab-org/build/cng/certificates:v18.8.2" env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false - name: custom-ca-certificates mountPath: /usr/local/share/ca-certificates readOnly: true securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 - name: configure image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - sh - /config/configure env: - name: TZ value: UTC resources: requests: cpu: 50m volumeMounts: - name: toolbox-config mountPath: /config readOnly: true - name: init-toolbox-secrets mountPath: /init-config readOnly: true - name: toolbox-secrets mountPath: /init-secrets readOnly: false securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 restartPolicy: OnFailure securityContext: fsGroup: 1000 runAsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault volumes: - name: toolbox-config projected: sources: - configMap: name: gitlab-toolbox - name: toolbox-tmp emptyDir: {} - name: init-toolbox-secrets projected: defaultMode: 256 sources: - secret: name: gitlab-rails-secret items: - key: secrets.yml path: rails-secrets/secrets.yml - secret: name: gitlab-gitlab-shell-secret items: - key: secret path: shell/.gitlab_shell_secret - secret: name: gitlab-gitaly-secret items: - key: token path: gitaly/gitaly_token - secret: name: gitlab-redis-secret items: - key: secret path: redis/redis-password - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-ci - secret: name: db-gitlab-app items: - key: password path: postgres/psql-password-main - secret: name: gitlab-registry-secret items: - key: registry-auth.key path: registry/gitlab-registry.key - secret: name: gitlab-registry-notification items: - key: secret path: registry/notificationSecret - secret: name: gitlab-object-store-configs items: - key: s3cmd.cfg path: objectstorage/.s3cfg - secret: name: gitlab-gitlab-kas-secret items: - key: kas_shared_secret path: kas/.gitlab_kas_secret - secret: name: gitlab-object-store-configs items: - key: connection.yaml path: objectstorage/object_store - secret: name: authentik-oidc items: - key: config.yaml path: omniauth/authentik-oidc/config.yaml - name: toolbox-secrets emptyDir: medium: Memory - name: etc-ssl-certs emptyDir: medium: Memory - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: Memory - name: custom-ca-certificates projected: defaultMode: 288 sources: - secret: name: aethernet.crt - secret: name: gitlab-wildcard-tls-ca schedule: @daily startingDeadlineSeconds: null successfulJobsHistoryLimit: 3 suspend: false (root level) (v1/ServiceAccount/gitlab/gitlab-shared-secrets) + one document added: apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-shared-secrets namespace: gitlab annotations: helm.sh/hook: "pre-install,pre-upgrade" helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" helm.sh/hook-weight: "-5" labels: app: gitlab heritage: Helm release: gitlab (root level) (v1/ConfigMap/gitlab/gitlab-shared-secrets) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-shared-secrets namespace: gitlab annotations: helm.sh/hook: "pre-install,pre-upgrade" helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" helm.sh/hook-weight: "-3" labels: app: gitlab heritage: Helm release: gitlab data: generate-secrets: | # vim: set filetype=sh: namespace=gitlab release=gitlab env=production pushd $(mktemp -d) # Args pattern, length function gen_random(){ head -c 4096 /dev/urandom | LC_CTYPE=C tr -cd $1 | head -c $2 } # Args: length function gen_random_base64(){ local len="$1" gen_random_bytes "$len" | base64 -w0 } # Args: length function gen_random_bytes(){ local len="$1" head -c "$len" /dev/urandom } # Args: yaml file, search path function fetch_rails_value(){ local value=$(yq --prettyPrint --no-colors ".${2}" $1) # Don't return null values if [ "${value}" != "null" ]; then echo "${value}"; fi } # Args: secretname function label_secret(){ local secret_name=$1 # Remove application labels if they exist kubectl --namespace=$namespace label \ secret $secret_name $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g') kubectl --namespace=$namespace label \ --overwrite \ secret $secret_name app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm } # Args: secretname, args function generate_secret_if_needed(){ local secret_args=( "${@:2}") local secret_name=$1 if ! $(kubectl --namespace=$namespace get secret $secret_name > /dev/null 2>&1); then kubectl --namespace=$namespace create secret generic $secret_name ${secret_args[@]} else echo "secret \"$secret_name\" already exists." for arg in "${secret_args[@]}"; do local from=$(echo -n ${arg} | cut -d '=' -f1) if [ -z "${from##*literal*}" ]; then local key=$(echo -n ${arg} | cut -d '=' -f2) local desiredValue=$(echo -n ${arg} | cut -d '=' -f3-) local flags="--namespace=$namespace --allow-missing-template-keys=false" if ! $(kubectl $flags get secret $secret_name -ojsonpath="{.data.${key}}" > /dev/null 2>&1); then echo "key \"${key}\" does not exist. patching it in." if [ "${desiredValue}" != "" ]; then desiredValue=$(echo -n "${desiredValue}" | base64 -w 0) fi kubectl --namespace=$namespace patch secret ${secret_name} -p "{\"data\":{\"$key\":\"${desiredValue}\"}}" fi fi done fi label_secret $secret_name } # Initial root password generate_secret_if_needed "gitlab-gitlab-initial-root-password" --from-literal="password"=$(gen_random 'a-zA-Z0-9' 64) # Redis password generate_secret_if_needed "gitlab-redis-secret" --from-literal="secret"=$(gen_random 'a-zA-Z0-9' 64) # Gitlab shell generate_secret_if_needed "gitlab-gitlab-shell-secret" --from-literal="secret"=$(gen_random 'a-zA-Z0-9' 64) # Gitaly secret generate_secret_if_needed "gitlab-gitaly-secret" --from-literal="token"=$(gen_random 'a-zA-Z0-9' 64) # Gitlab runner secret generate_secret_if_needed "gitlab-gitlab-runner-secret" --from-literal=runner-registration-token=$(gen_random 'a-zA-Z0-9' 64) --from-literal=runner-token="" # GitLab Pages API secret # GitLab Pages auth secret for hashing cookie store when using access control # GitLab Pages OAuth secret # Gitlab-kas secret generate_secret_if_needed "gitlab-gitlab-kas-secret" --from-literal="kas_shared_secret"=$(gen_random 'a-zA-Z0-9' 32 | base64) # Gitlab-kas private API secret generate_secret_if_needed "gitlab-kas-private-api" --from-literal="kas_private_api_secret"=$(gen_random 'a-zA-Z0-9' 32 | base64) # Gitlab-kas WebSocket Token secret generate_secret_if_needed "gitlab-kas-websocket-token" --from-literal="kas_websocket_token_secret"=$(gen_random_base64 72) # Registry certificates mkdir -p certs openssl req -new -newkey rsa:4096 -subj "/CN=gitlab-issuer" -nodes -x509 -keyout certs/registry-example-com.key -out certs/registry-example-com.crt -days 3650 generate_secret_if_needed "gitlab-registry-secret" --from-file=registry-auth.key=certs/registry-example-com.key --from-file=registry-auth.crt=certs/registry-example-com.crt # config/secrets.yaml if [ -n "$env" ]; then rails_secret="gitlab-rails-secret" # Fetch the values from the existing secret if it exists if $(kubectl --namespace=$namespace get secret $rails_secret > /dev/null 2>&1); then kubectl --namespace=$namespace get secret $rails_secret -o jsonpath="{.data.secrets\.yml}" | base64 --decode > secrets.yml secret_key_base=$(fetch_rails_value secrets.yml "${env}.secret_key_base") otp_key_base=$(fetch_rails_value secrets.yml "${env}.otp_key_base") db_key_base=$(fetch_rails_value secrets.yml "${env}.db_key_base") openid_connect_signing_key=$(fetch_rails_value secrets.yml "${env}.openid_connect_signing_key") encrypted_settings_key_base=$(fetch_rails_value secrets.yml "${env}.encrypted_settings_key_base") active_record_encryption_primary_keys=$(fetch_rails_value secrets.yml "${env}.active_record_encryption_primary_key") active_record_encryption_deterministic_keys=$(fetch_rails_value secrets.yml "${env}.active_record_encryption_deterministic_key") active_record_encryption_key_derivation_salt=$(fetch_rails_value secrets.yml "${env}.active_record_encryption_key_derivation_salt") fi; # Generate defaults for any unset secrets secret_key_base="${secret_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64) otp_key_base="${otp_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64) db_key_base="${db_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64) openid_connect_signing_key="${openid_connect_signing_key:-$(openssl genrsa 2048)}" encrypted_settings_key_base="${encrypted_settings_key_base:-$(gen_random 'a-f0-9' 128)}" # equivalent to secureRandom.hex(64) # 1. We set the following two keys as an array to support keys rotation. # The last key in the array is always used to encrypt data: # https://github.com/rails/rails/blob/v7.0.8.4/activerecord/lib/active_record/encryption/key_provider.rb#L21 # while all the keys are used (in the order they're defined) to decrypt data: # https://github.com/rails/rails/blob/v7.0.8.4/activerecord/lib/active_record/encryption/cipher.rb#L26. # This allows to rotate keys by adding a new key as the last key, and start a re-encryption process that # runs in the background: https://gitlab.com/gitlab-org/gitlab/-/issues/494976 # 2. We use the same method and length as Rails' defaults: # https://github.com/rails/rails/blob/v7.0.8.4/activerecord/lib/active_record/railties/databases.rake#L537-L540 active_record_encryption_primary_keys=${active_record_encryption_primary_keys:-"- $(gen_random 'a-zA-Z0-9' 32)"} active_record_encryption_deterministic_keys=${active_record_encryption_deterministic_keys:-"- $(gen_random 'a-zA-Z0-9' 32)"} active_record_encryption_key_derivation_salt=${active_record_encryption_key_derivation_salt:-$(gen_random 'a-zA-Z0-9' 32)} # Update the existing secret cat << EOF > rails-secrets.yml apiVersion: v1 kind: Secret metadata: name: $rails_secret type: Opaque stringData: secrets.yml: |- $env: secret_key_base: $secret_key_base otp_key_base: $otp_key_base db_key_base: $db_key_base encrypted_settings_key_base: $encrypted_settings_key_base openid_connect_signing_key: | $(echo "${openid_connect_signing_key}" | awk '{print " " $0}') active_record_encryption_primary_key: $active_record_encryption_primary_keys active_record_encryption_deterministic_key: $active_record_encryption_deterministic_keys active_record_encryption_key_derivation_salt: $active_record_encryption_key_derivation_salt EOF kubectl --validate=false --namespace=$namespace apply -f rails-secrets.yml label_secret $rails_secret fi # Shell ssh host keys ssh-keygen -A mkdir -p host_keys cp /etc/ssh/ssh_host_* host_keys/ generate_secret_if_needed "gitlab-gitlab-shell-host-keys" --from-file host_keys # Gitlab-workhorse secret generate_secret_if_needed "gitlab-gitlab-workhorse-secret" --from-literal="shared_secret"=$(gen_random 'a-zA-Z0-9' 32 | base64) # Registry http.secret secret generate_secret_if_needed "gitlab-registry-httpsecret" --from-literal="secret"=$(gen_random 'a-z0-9' 128 | base64 -w 0) # Container Registry notification_secret generate_secret_if_needed "gitlab-registry-notification" --from-literal="secret"=[\"$(gen_random 'a-zA-Z0-9' 32)\"] (root level) (v1/ConfigMap/gitlab/gitlab-gitlab-upgrade-check) + one document added: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-gitlab-upgrade-check namespace: gitlab annotations: helm.sh/hook: pre-upgrade helm.sh/hook-delete-policy: before-hook-creation helm.sh/hook-weight: "-11" labels: app: gitlab heritage: Helm release: gitlab data: runcheck: | #!/bin/sh set -e notify() { echo "$1" echo -n "$1 " >> /dev/termination-log } greater_version() { test "$(printf '%s\n' "$@" | sort -V | tail -n 1)" = "$1"; } is_semver() { printf '%s' "$1" | grep -Eq "^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" return "$?" } # For the PostgreSQL upgrade, you either need both secrets, or no secrets. # If there are no secrets, we will create them for you. # If the secrets aren't in either of these states, we assume you are upgrading from an older version # This is running ahead of version checks to ensure this always runs. This is to account for # installations outside of the official helm repo. secrets_dir="/etc/secrets/postgresql" if [ -d "${secrets_dir}" ]; then if [ ! "$(ls -A ${secrets_dir}/..data/)" = "" ]; then if [ ! -f "${secrets_dir}/postgresql-postgres-password" ] || [ ! -f "${secrets_dir}/password" ]; then notify "You seem to be upgrading from a previous version of GitLab using the bundled PostgreSQL chart" notify "There are some manual steps which need to be performed in order to upgrade the database" notify "Please see the upgrade documentation for instructions on how to proceed:" notify "https://docs.gitlab.com/charts/installation/upgrade.html" exit 1 fi fi fi MIN_VERSION=18.5 CHART_MIN_VERSION=9.5 # Remove 'v' prefix from GitLab version if present (set in Chart.yaml appVersions) GITLAB_VERSION=${GITLAB_VERSION#v} # Only run check for semver releases if ! is_semver "$GITLAB_VERSION"; then exit 0 fi NEW_MAJOR_VERSION=$(echo $GITLAB_VERSION | awk -F "." '{print $1}') NEW_MINOR_VERSION=$(echo $GITLAB_VERSION | awk -F "." '{print $1"."$2}') NEW_CHART_MAJOR_VERSION=$(echo $CHART_VERSION | awk -F "." '{print $1}') NEW_CHART_MINOR_VERSION=$(echo $CHART_VERSION | awk -F "." '{print $1"."$2}') if [ ! -f /chart-info/gitlabVersion ]; then notify "It seems you are attempting an unsupported upgrade path." notify "Please follow the upgrade documentation at https://docs.gitlab.com/update/#upgrade-paths" exit 1 fi OLD_VERSION_STRING=$(cat /chart-info/gitlabVersion) OLD_VERSION_STRING=${OLD_VERSION_STRING#v} OLD_CHART_VERSION_STRING=$(cat /chart-info/gitlabChartVersion) # Skip check if old version wasn't semver if ! is_semver "$OLD_VERSION_STRING"; then exit 0 fi OLD_MAJOR_VERSION=$(echo $OLD_VERSION_STRING | awk -F "." '{print $1}') OLD_MINOR_VERSION=$(echo $OLD_VERSION_STRING | awk -F "." '{print $1"."$2}') OLD_CHART_MAJOR_VERSION=$(echo $OLD_CHART_VERSION_STRING | awk -F "." '{print $1}') OLD_CHART_MINOR_VERSION=$(echo $OLD_CHART_VERSION_STRING | awk -F "." '{print $1"."$2}') # Checking Version if ( ! greater_version "$OLD_MINOR_VERSION" "$MIN_VERSION" ) || ( ! greater_version "$OLD_CHART_MINOR_VERSION" "$CHART_MIN_VERSION" ); then notify "It seems you are upgrading the GitLab Helm Chart from ${OLD_CHART_VERSION_STRING} (GitLab ${OLD_VERSION_STRING}) to ${CHART_VERSION} (GitLab ${GITLAB_VERSION})." notify "It is required to upgrade to the latest ${CHART_MIN_VERSION}.x version first before proceeding." # Link to upgrade docs if it is a major version jump if [ "${OLD_MAJOR_VERSION}" -lt "${NEW_MAJOR_VERSION}" ] || [ "${OLD_CHART_MAJOR_VERSION}" -lt "${NEW_CHART_MAJOR_VERSION}" ]; then notify "Please follow the upgrade documentation at https://docs.gitlab.com/charts/releases/8_0.html" notify "and upgrade to GitLab Helm Chart version ${CHART_MIN_VERSION}.x before upgrading to ${CHART_VERSION}." fi exit 1 fi (root level) (rbac.authorization.k8s.io/v1/Role/gitlab/gitlab-shared-secrets) + one document added: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: gitlab-shared-secrets namespace: gitlab annotations: helm.sh/hook: "pre-install,pre-upgrade" helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" helm.sh/hook-weight: "-5" labels: app: shared-secrets heritage: Helm release: gitlab rules: - resources: - secrets apiGroups: - verbs: - get - list - create - patch (root level) (rbac.authorization.k8s.io/v1/RoleBinding/gitlab/gitlab-shared-secrets) + one document added: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: gitlab-shared-secrets namespace: gitlab annotations: helm.sh/hook: "pre-install,pre-upgrade" helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" helm.sh/hook-weight: "-5" labels: app: shared-secrets heritage: Helm release: gitlab roleRef: name: gitlab-shared-secrets apiGroup: rbac.authorization.k8s.io kind: Role subjects: - name: gitlab-shared-secrets kind: ServiceAccount namespace: gitlab (root level) (batch/v1/Job/gitlab/gitlab-shared-secrets-6fff130) + one document added: apiVersion: batch/v1 kind: Job metadata: name: gitlab-shared-secrets-6fff130 namespace: gitlab annotations: helm.sh/hook: "pre-install,pre-upgrade" helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" labels: app: gitlab heritage: Helm release: gitlab spec: template: metadata: annotations: null labels: app: gitlab heritage: Helm release: gitlab spec: containers: - name: gitlab image: "registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.8.2" command: - /bin/bash - /scripts/generate-secrets resources: requests: cpu: 50m volumeMounts: - name: scripts mountPath: /scripts - name: ssh mountPath: /etc/ssh securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true restartPolicy: Never securityContext: fsGroup: 65534 runAsUser: 65534 seccompProfile: type: RuntimeDefault serviceAccountName: gitlab-shared-secrets volumes: - name: scripts configMap: name: gitlab-shared-secrets - name: ssh emptyDir: {} (root level) (batch/v1/Job/gitlab/gitlab-shared-secrets-6fff130-selfsign) + one document added: apiVersion: batch/v1 kind: Job metadata: name: gitlab-shared-secrets-6fff130-selfsign namespace: gitlab annotations: helm.sh/hook: "pre-install,pre-upgrade" helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" labels: app: gitlab heritage: Helm release: gitlab spec: template: metadata: annotations: null labels: app: gitlab heritage: Helm release: gitlab spec: containers: - name: kubectl image: "registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.8.2" command: - /bin/bash - "-exc" - | certname=gitlab-wildcard-tls # create wildcard certificate secret kubectl create secret tls $certname \ --cert=/output/wildcard.pem --key=/output/wildcard-key.pem || true kubectl --namespace=$namespace label \ secret $certname $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g') kubectl --namespace=$namespace label --overwrite \ secret $certname app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm # create CA certificate secret kubectl create secret generic ${certname}-ca \ --from-file=cfssl_ca=/output/ca.pem || true kubectl --namespace=$namespace label \ secret ${certname}-ca $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g') kubectl --namespace=$namespace label --overwrite \ secret ${certname}-ca app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm # create certificate chain for GitLab Runner cat /output/ca.pem /output/wildcard.pem > /tmp/code.nogweii.net.crt kubectl create secret generic ${certname}-chain \ --from-file=/tmp/code.nogweii.net.crt || true kubectl --namespace=$namespace label \ secret ${certname}-chain $(echo 'app.kubernetes.io/name=gitlab' | sed -E 's/=[^ ]*/-/g') kubectl --namespace=$namespace label --overwrite \ secret ${certname}-chain app=gitlab chart=gitlab-9.8.2 release=gitlab heritage=Helm resources: requests: cpu: 50m volumeMounts: - name: certs-path mountPath: /output initContainers: - name: cfssl-self-sign image: "registry.gitlab.com/gitlab-org/build/cng/cfssl-self-sign:v18.8.2" env: - name: CA_SUBJECT value: "GitLab Helm Chart" - name: CA_ORG value: gitlab - name: CA_ORG_UNIT value: gitlab - name: CERT_SUBJECT value: code.nogweii.net - name: CERT_DOMAIN value: code.nogweii.net - name: EXPIRY value: 3650d - name: ALGORITHM value: rsa - name: KEY_SIZE value: "4096" resources: requests: cpu: 50m volumeMounts: - name: certs-path mountPath: /output restartPolicy: Never securityContext: fsGroup: 65534 runAsUser: 65534 serviceAccountName: gitlab-shared-secrets volumes: - name: certs-path emptyDir: {} (root level) (batch/v1/Job/gitlab/gitlab-gitlab-upgrade-check) + one document added: apiVersion: batch/v1 kind: Job metadata: name: gitlab-gitlab-upgrade-check namespace: gitlab annotations: helm.sh/hook: pre-upgrade helm.sh/hook-delete-policy: "hook-succeeded,before-hook-creation" helm.sh/hook-weight: "-10" labels: app: gitlab heritage: Helm release: gitlab spec: backoffLimit: 2 template: metadata: labels: app: gitlab heritage: Helm release: gitlab spec: containers: - name: run-check image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.8.2" command: - /bin/sh - /scripts/runcheck env: - name: GITLAB_VERSION value: "18.8.2" - name: CHART_VERSION value: "9.8.2" resources: requests: cpu: 50m volumeMounts: - name: chart-info mountPath: /chart-info - name: scripts mountPath: /scripts securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 65534 restartPolicy: Never securityContext: fsGroup: 65534 runAsUser: 65534 seccompProfile: type: RuntimeDefault volumes: - name: chart-info configMap: name: gitlab-gitlab-chart-info optional: true - name: scripts configMap: name: gitlab-gitlab-upgrade-check flate error: orig snapshot: reconcile completed with 1 failure(s): HelmRelease/flux-system/gitlab: helm template flux-system/gitlab: execution error at (gitlab/templates/NOTES.txt:222:3): CONFIGURATION CHECKS: redis: You must configure an Redis connetion. Please set `global.redis.host`. Since chart v10.0.0, external Redis became required. If you were using the bundled Redis chart refer to https://docs.gitlab.com/charts/installation/migration/bundled_chart_migration/. ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/524)

renovate-bot changed title from ~~chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.32~~ to chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.36

2026-10-15 03:07:38 +00:00

renovate-bot changed title from ~~chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.36~~ to chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.37

2026-10-22 16:07:01 +00:00

renovate-bot force-pushed renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x from f4db47876a to e4601ab41d

ci/crow/pr/dns-test Pipeline was successful

Details

ci/crow/push/dns-test Pipeline was successful

Details

ci/crow/push/tanka-test Pipeline was successful

Details

ci/crow/pr/tanka-test Pipeline was successful

Details

ci/crow/pr/tanka-diff Pipeline was successful

Details

2026-10-22 16:07:02 +00:00

Compare

ci-bot commented

2026-10-22 16:09:53 +00:00

Member

flate diff

data.memini.json  (v1/ConfigMap/local-ai/memini-memini)
  ± value change in multiline text (314 inserts, 271 deletions)
      
      {
        "annotations": {
          "list": []
        },
    -   "description": "memini \u2014 what the service is actually doing for your agents. Live memory counts, write/recall traffic by tier, consolidation outcomes, store health, embedder & rerank latency & saturation, HTTP surface.",
    +   "description": "memini operator dashboard \u2014 top-to-bottom narrative: live headline health (including the deferred-repair backlog), the write path, recall & retrieval quality, aging & maintenance jobs, embedder & rerank, LLM-backed consolidation/answer, the HTTP surface, and client injection telemetry.",
        "editable": true,
        "graphTooltip": 1,
        "id": null,
        "links": [],
      
      [14 lines unchanged)]
      
          {
            "id": 2,
            "type": "stat",
            "title": "Live memories",
    -       "description": "Live (non-superseded, non-expired) memories by tier.",
    +       "description": "Total live (non-superseded, non-expired) memories across all tiers; a sudden drop means a sweep or dedup pass removed more than expected.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    +         "h": 5,
    +         "w": 4,
              "x": 0,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none"
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
      
      [five lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_memories_active",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum(memini_memories_active)",
    +           "refId": "A",
    +           "legendFormat": "live"
              }
            ]
          },
          {
            "id": 3,
            "type": "stat",
    -       "title": "Writes (5m rate)",
    -       "description": "Successful Remember calls per second, by tier.",
    +       "title": "Writes /s (5m)",
    +       "description": "Successful Remember writes per second over the last 5m; zero during active agent sessions means writes are failing upstream.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    -         "x": 6,
    +         "h": 5,
    +         "w": 3,
    +         "x": 4,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "decimals": 2
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "unit": "ops"
    -         }
    -       },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum(rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    +           "refId": "A",
    +           "legendFormat": "writes/s"
              }
            ]
          },
          {
            "id": 4,
            "type": "stat",
    -       "title": "Recalls (5m rate)",
    -       "description": "Successful Recall calls per second.",
    +       "title": "Recalls /s (5m)",
    +       "description": "Successful Recall searches per second over the last 5m; the primary read-traffic pulse.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    -         "x": 12,
    +         "h": 5,
    +         "w": 3,
    +         "x": 7,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "decimals": 2
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    +           "refId": "A",
    +           "legendFormat": "recalls/s"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 5,
    +       "type": "stat",
    +       "title": "HTTP 5xx ratio (15m)",
    +       "description": "Share of HTTP requests answered 5xx over 15m; anything above zero deserves a look, above 5% is an outage.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 5,
    +         "w": 3,
    +         "x": 10,
    +         "y": 1
    +       },
            "fieldConfig": {
              "defaults": {
    -           "unit": "ops"
    -         }
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 0.01
    +               },
    +               {
    +                 "color": "red",
    +                 "value": 0.05
    +               }
    +             ]
    +           },
    +           "unit": "percentunit"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "recalls/s",
    -           "refId": "A"
    +           "expr": "sum(rate(memini_http_requests_total{status=~\"5..\"}[15m])) / clamp_min(sum(rate(memini_http_requests_total[15m])), 1e-9)",
    +           "refId": "A",
    +           "legendFormat": "5xx ratio"
              }
            ]
          },
          {
    -       "id": 18,
    +       "id": 6,
            "type": "stat",
    -       "title": "Consolidation queue",
    -       "description": "Backlog of memories awaiting async consolidation. Sustained > 0 means the worker can't keep up with Remember writes.",
    +       "title": "Repair backlog depth",
    +       "description": "Memories currently owing deferred repair (pending + enrich + failed); green is empty, any sustained backlog means writes are storing degraded faster than repairs drain.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 6,
    -         "w": 6,
    -         "x": 18,
    +         "h": 5,
    +         "w": 4,
    +         "x": 13,
              "y": 1
            },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 1
    +               }
    +             ]
    +           },
    +           "unit": "none"
    +         },
    +         "overrides": []
    +       },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
              "justifyMode": "auto",
    -         "orientation": "horizontal",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(memini_repair_depth)",
    +           "refId": "A",
    +           "legendFormat": "backlog"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 7,
    +       "type": "stat",
    +       "title": "Repair oldest age",
    +       "description": "Age of the oldest memory still awaiting repair; amber the moment anything waits, red past 1h means the repair loop is stuck or starved.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 5,
    +         "w": 4,
    +         "x": 17,
    +         "y": 1
    +       },
            "fieldConfig": {
              "defaults": {
    -           "unit": "short",
    +           "color": {
    +             "mode": "thresholds"
    +           },
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
    -                 "color": "yellow",
    -                 "value": 50
    +                 "color": "orange",
    +                 "value": 1
                    },
                    {
                      "color": "red",
    -                 "value": 500
    +                 "value": 3600
                    }
                  ]
    -           }
    +           },
    +           "unit": "s"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "max(memini_repair_oldest_age_seconds)",
    +           "refId": "A",
    +           "legendFormat": "oldest"
              }
    +       ]
    +     },
    +     {
    +       "id": 8,
    +       "type": "stat",
    +       "title": "Consolidation queue",
    +       "description": "Depth of the async consolidation queue; a queue that grows without draining delays dedup/supersede decisions.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 5,
    +         "w": 3,
    +         "x": 21,
    +         "y": 1
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 100
    +               }
    +             ]
    +           },
    +           "unit": "none"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
                "expr": "memini_consolidate_queue_depth",
    -           "legendFormat": "queued",
    -           "refId": "A"
    +           "refId": "A",
    +           "legendFormat": "queued"
              }
            ]
          },
          {
    -       "id": 5,
    +       "id": 9,
            "type": "row",
    -       "title": "Writes & Recalls",
    +       "title": "Write path",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 7
    +         "y": 6
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 6,
    +       "id": 10,
            "type": "timeseries",
    -       "title": "Remembers per second (by tier)",
    +       "title": "Remember rate by tier",
    +       "description": "Successful Remember writes per second split by tier, plus error/dropped outcomes; red or orange series above zero mean writes are being lost.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 8
    +         "y": 7
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
                },
    -           "unit": "ops"
    -         }
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "dropped"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{tier}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_remember_results_total{result=\"error\"}[$__rate_interval]))",
    +           "refId": "B",
    +           "legendFormat": "error"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_remember_results_total{result=\"dropped\"}[$__rate_interval]))",
    +           "refId": "C",
    +           "legendFormat": "dropped"
              }
            ]
          },
          {
    -       "id": 7,
    +       "id": 11,
            "type": "timeseries",
    -       "title": "Recalls per second (by hit-count bucket)",
    -       "description": "How many memories a typical recall returns. '0' means no matches \u2014 investigate if persistent.",
    +       "title": "Degraded (vectorless) writes by reason",
    +       "description": "Writes stored without a vector (keyword-searchable only, queued for deferred repair) because the embedder timed out or errored; any sustained rate means the embedder is unhealthy.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 8
    +         "y": 7
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_timeout"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[5m]))",
    -           "legendFormat": "{{hits_bucket}}",
    -           "refId": "A"
    +           "expr": "sum by (reason) (rate(memini_remember_degraded_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{reason}}"
              }
            ]
          },
          {
    -       "id": 19,
    -       "type": "row",
    -       "title": "Retrieval quality",
    -       "gridPos": {
    -         "h": 1,
    -         "w": 24,
    -         "x": 0,
    -         "y": 16
    -       },
    -       "collapsed": false,
    -       "panels": []
    -     },
    -     {
    -       "id": 20,
    +       "id": 12,
            "type": "timeseries",
    -       "title": "Rerank outcomes by backend",
    -       "description": "Per-recall reranker traffic: ok = reranker reordered the fused candidates; fallback = reranker timed out / errored and the fused order was kept.",
    +       "title": "Store mutations",
    +       "description": "Raw store write traffic: inserts vs updates (by upsert op), hard deletes from Forget, and tombstones written by consolidation; updates dominating inserts means most writes reinforce existing memories.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 17
    +         "y": 14
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend, result) (rate(memini_rerank_results_total[5m]))",
    -           "legendFormat": "{{backend}}/{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (op) (rate(memini_store_upserts_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "upsert: {{op}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "rate(memini_store_deletes_total[$__rate_interval])",
    +           "refId": "B",
    +           "legendFormat": "hard delete (forget)"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "rate(memini_store_soft_deletes_total[$__rate_interval])",
    +           "refId": "C",
    +           "legendFormat": "tombstone (supersede)"
              }
            ]
          },
          {
    -       "id": 21,
    +       "id": 13,
            "type": "timeseries",
    -       "title": "Recall degradation reasons",
    -       "description": "Recalls that fell back to keyword-only search because the embedder was slow or errored. embed_timeout = the recall embed budget elapsed; embed_error = the embed call failed.",
    +       "title": "Forget & supersede outcomes",
    +       "description": "Forget and Supersede API outcomes per second; red error series should stay at zero, orange not_found spikes usually mean a client is retrying stale ids.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 17
    +         "y": 14
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*error.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
                },
    -           "unit": "ops"
    -         }
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*not_found.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (reason) (rate(memini_recall_degraded_total[5m]))",
    -           "legendFormat": "{{reason}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_forget_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "forget: {{result}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (rate(memini_supersede_results_total[$__rate_interval]))",
    +           "refId": "B",
    +           "legendFormat": "supersede: {{result}}"
              }
            ]
          },
          {
    -       "id": 22,
    +       "id": 14,
            "type": "timeseries",
    -       "title": "Recall reinforcement (best-effort)",
    -       "description": "Background access-count bump on recalled memories. Errors are swallowed \u2014 a sustained nonzero error rate means the store write path is unhealthy but recall still served.",
    +       "title": "Content hygiene (sanitizer)",
    +       "description": "Ingestion sanitizer actions: cleaned content was rewritten in place, quarantined (orange) content was judged unsafe to store as-is and needs review if it becomes frequent.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 12,
    +         "h": 7,
    +         "w": 8,
              "x": 0,
    -         "y": 25
    +         "y": 21
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "ops"
    -         }
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "quarantined"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_reinforce_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (action) (rate(memini_write_sanitized_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{action}}"
              }
            ]
          },
          {
    -       "id": 23,
    -       "type": "stat",
    -       "title": "Recall success ratio",
    -       "description": "Fraction of recall calls that succeeded. A drop means the recall path is erroring (embed + keyword + rerank + RRF + everything).",
    +       "id": 15,
    +       "type": "timeseries",
    +       "title": "Corroboration routing",
    +       "description": "Corroboration attempts on fresh short-term writes (did this confirm something we already know); a rising red error rate means the corroboration path is failing writes silently.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 12,
    -         "x": 12,
    -         "y": 25
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 21
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    -         "textMode": "auto"
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (rate(memini_corroborate_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 16,
    +       "type": "timeseries",
    +       "title": "Contradiction routing",
    +       "description": "Contradiction attempts on fresh durable writes (did this contradict an existing fact); contradicted events drive supersession, red errors mean conflicts are going undetected.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 16,
    +         "y": 21
            },
            "fieldConfig": {
              "defaults": {
    -           "unit": "percentunit",
    -           "min": 0,
    -           "max": 1,
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
                "thresholds": {
                  "mode": "absolute",
                  "steps": [
                    {
    -                 "color": "red",
    +                 "color": "green",
                      "value": null
    -               },
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
                    {
    -                 "color": "yellow",
    -                 "value": 0.95
    -               },
    -               {
    -                 "color": "green",
    -                 "value": 0.99
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
                    }
                  ]
                }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)",
    -           "legendFormat": "ok ratio",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_contradict_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 8,
    +       "id": 17,
            "type": "row",
    -       "title": "Consolidation (LLM dedup)",
    +       "title": "Recall & retrieval quality",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 33
    +         "y": 28
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 9,
    +       "id": 18,
            "type": "timeseries",
    -       "title": "Consolidation outcomes per minute",
    -       "description": "gated = similarity gate skipped the LLM; new = nothing to merge; update = merged into an existing memory; supersede = contradiction-tombstoned an old one; noop = nothing to do; error/dropped = failure modes.",
    +       "title": "Recall rate by hit-count bucket",
    +       "description": "Successful recalls per second split by how many hits they returned, plus errors; a shift toward the 0-hits bucket means retrieval quality is degrading.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 16,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 34
    +         "y": 29
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{hits_bucket}} hits"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_recall_results_total{result=\"error\"}[$__rate_interval]))",
    +           "refId": "B",
    +           "legendFormat": "error"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 19,
    +       "type": "timeseries",
    +       "title": "Recall degradation (keyword-only fallback)",
    +       "description": "Recalls that fell back to keyword-only search because embedding the query timed out or errored; these return without semantic ranking, so any sustained rate hurts result quality.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 29
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 20,
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
                },
    -           "unit": "ops"
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "embed_timeout"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (reason) (rate(memini_recall_degraded_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{reason}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 20,
    +       "type": "timeseries",
    +       "title": "Recall candidates floored (by tier filter)",
    +       "description": "Candidates dropped from recall responses by the min_rank_score composite floor; a jump means the floor (or a scoring regression) is hiding results users would previously have seen.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 0,
    +         "y": 36
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["sum"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_consolidate_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (tier_filter) (rate(memini_recall_floored_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{tier_filter}}"
              }
            ]
          },
          {
    -       "id": 10,
    +       "id": 21,
    +       "type": "timeseries",
    +       "title": "Recall reinforcement",
    +       "description": "Best-effort reinforcement writes triggered by recall hits (strengthening recalled memories); red errors mean reinforcement is silently failing and ranking will slowly stale.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 36
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (rate(memini_reinforce_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 22,
            "type": "stat",
    -       "title": "Dedup effectiveness",
    -       "description": "Fraction of consolidations that actually used the LLM (update+supersede+new+noop) vs the similarity gate skipping it. Higher = busier corpus or noisier dedup signal.",
    +       "title": "Recall success ratio (15m)",
    +       "description": "Share of Recall calls that completed without error over 15m; below 99% something in the search path is failing.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 8,
              "x": 16,
    -         "y": 34
    +         "y": 36
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "red",
    +                 "value": null
    +               },
    +               {
    +                 "color": "orange",
    +                 "value": 0.9
    +               },
    +               {
    +                 "color": "green",
    +                 "value": 0.99
    +               }
    +             ]
    +           },
    +           "unit": "percentunit"
    +         },
    +         "overrides": []
            },
            "options": {
              "colorMode": "value",
              "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
              "reduceOptions": {
                "calcs": ["lastNotNull"],
                "fields": "",
                "values": false
              },
              "textMode": "auto"
            },
    -       "fieldConfig": {
    -         "defaults": {
    -           "unit": "percentunit",
    -           "min": 0,
    -           "max": 1
    -         }
    -       },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|new|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result!~\"error|dropped\"}[15m])), 1e-9)",
    -           "legendFormat": "LLM use ratio",
    -           "refId": "A"
    +           "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)",
    +           "refId": "A",
    +           "legendFormat": "ok ratio"
              }
            ]
          },
          {
    -       "id": 11,
    +       "id": 23,
            "type": "row",
    -       "title": "Storage & decay",
    +       "title": "Aging & maintenance",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 42
    +         "y": 43
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 12,
    +       "id": 24,
            "type": "timeseries",
            "title": "Live memories by tier",
    +       "description": "Live memory count per tier over time; working/episodic should churn while semantic/procedural grow slowly \u2014 a cliff in any tier means a sweep or dedup removed too much.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 43
    +         "y": 44
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "stepAfter",
    -             "fillOpacity": 15,
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
                },
    -           "unit": "short"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_memories_active",
    -           "legendFormat": "{{tier}}",
    -           "refId": "A"
    +           "expr": "sum by (tier) (memini_memories_active)",
    +           "refId": "A",
    +           "legendFormat": "{{tier}}"
              }
            ]
          },
          {
    -       "id": 13,
    +       "id": 25,
            "type": "timeseries",
    -       "title": "Decay sweep + tombstones (per minute)",
    -       "description": "Swept = memories purged by TTL; soft deletes = contradictions tombstoned by consolidation; dedup tombstones = tombstoned by the periodic dedup pass.",
    +       "title": "Promotion, classification & demotion (per hour)",
    +       "description": "Hourly outcomes of the episodic-to-semantic promotion pass, durable facts it wrote, and omitted-tier writes the marker classifier routed; red promote errors mean durable knowledge is not being distilled. Demotions are the mirror image: stale, never-recalled durable memories retiered down to episodic \u2014 a steady demotion stream usually means bulk-imported debris aging out.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 43
    +         "y": 44
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "stepAfter",
    -             "fillOpacity": 15,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "promote: error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["sum"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_store_swept_total[5m]))",
    -           "legendFormat": "swept/{{tier}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (increase(memini_promote_results_total[1h]))",
    +           "refId": "A",
    +           "legendFormat": "promote: {{result}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(increase(memini_promote_facts_total[1h]))",
    +           "refId": "B",
    +           "legendFormat": "facts promoted"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "rate(memini_store_soft_deletes_total[5m])",
    -           "legendFormat": "consolidate tombstones",
    -           "refId": "B"
    +           "expr": "sum by (tier) (increase(memini_tier_classified_total[1h]))",
    +           "refId": "C",
    +           "legendFormat": "classified \u2192 {{tier}}"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "rate(memini_dedup_tombstoned_total[5m])",
    -           "legendFormat": "dedup tombstones",
    -           "refId": "C"
    +           "expr": "sum by (tier) (increase(memini_demoted_total[1h]))",
    +           "legendFormat": "demoted from {{tier}}",
    +           "refId": "D"
              }
            ]
          },
          {
    -       "id": 14,
    -       "type": "row",
    -       "title": "Embedder & rerank",
    +       "id": 26,
    +       "type": "timeseries",
    +       "title": "Decay sweeps & dedup tombstones",
    +       "description": "Memories purged by the decay sweeper (by tier) and tombstoned by the dedup pass, per second; sustained zero forever means aging jobs are not running.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
            "gridPos": {
    -         "h": 1,
    -         "w": 24,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
              "y": 51
            },
    -       "collapsed": false,
    -       "panels": []
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (tier) (rate(memini_store_swept_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "swept: {{tier}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "rate(memini_dedup_tombstoned_total[$__rate_interval])",
    +           "refId": "B",
    +           "legendFormat": "dedup tombstoned"
    +         }
    +       ]
          },
          {
    -       "id": 15,
    +       "id": 27,
            "type": "timeseries",
    -       "title": "Embedder p95 latency (by backend)",
    -       "description": "openai = real network calls; cached/diskcache/batched = local cache or fan-out; disabled = no endpoint configured.",
    +       "title": "Fsck outcomes (per hour)",
    +       "description": "Hourly consistency-sweep outcomes; red errors mean the store's self-check cannot complete and silent corruption may go unnoticed.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
    +         "x": 12,
    +         "y": 51
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (result) (increase(memini_fsck_results_total[1h]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 28,
    +       "type": "timeseries",
    +       "title": "Repair backlog depth by state",
    +       "description": "Memories owing deferred repair, split by state; red failed entries are permanently vectorless without operator action, and a growing pending line means the repair loop cannot keep up.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
              "x": 0,
    -         "y": 52
    +         "y": 58
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "s"
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "failed"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (state) (memini_repair_depth)",
    +           "refId": "A",
    +           "legendFormat": "{{state}}"
              }
    +       ]
    +     },
    +     {
    +       "id": 29,
    +       "type": "timeseries",
    +       "title": "Repair oldest age by state",
    +       "description": "Age of the oldest memory in each repair state; growth without bound means repairs are queued faster than they drain (the red line marks the 1h alarm threshold).",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 58
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "line"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               },
    +               {
    +                 "color": "red",
    +                 "value": 3600
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[5m])))",
    -           "legendFormat": "p95 {{backend}}",
    -           "refId": "A"
    +           "expr": "max by (state) (memini_repair_oldest_age_seconds)",
    +           "refId": "A",
    +           "legendFormat": "{{state}}"
              }
            ]
          },
          {
    -       "id": 16,
    +       "id": 30,
            "type": "timeseries",
    -       "title": "Embedder tokens + error rate",
    +       "title": "Backfill pending",
    +       "description": "Memories still awaiting embedding backfill and long memories still without chunk vectors; a number that never falls means those memories are unreachable by semantic (or chunked) recall.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
    +         "w": 8,
    +         "x": 16,
    +         "y": 58
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "memini_embed_backfill_pending",
    +           "refId": "A",
    +           "legendFormat": "pending embed"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "memini_chunk_backfill_pending",
    +           "refId": "B",
    +           "legendFormat": "pending chunk vectors"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 31,
    +       "type": "timeseries",
    +       "title": "Repair outcomes by stage",
    +       "description": "Deferred-repair attempt outcomes per second by stage; red parked means a memory gave up retrying and stays vectorless, orange retry churn means the embedder keeps failing under the repair loop.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
              "w": 12,
    -         "x": 12,
    -         "y": 52
    +         "x": 0,
    +         "y": 65
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*parked.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*retry.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
                }
    -         }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend) (rate(memini_embed_tokens_total[5m]))",
    -           "legendFormat": "tokens/s {{backend}}",
    -           "refId": "A"
    +           "expr": "sum by (stage, result) (rate(memini_repair_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{stage}}: {{result}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 32,
    +       "type": "timeseries",
    +       "title": "Repair duration p95 by stage",
    +       "description": "95th-percentile latency of a single deferred repair by stage; climbing durations shrink repair throughput and grow the backlog.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 65
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (backend) (rate(memini_embed_errors_total[5m]))",
    -           "legendFormat": "err/s {{backend}}",
    -           "refId": "B"
    +           "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(memini_repair_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{stage}}"
              }
            ]
          },
          {
    -       "id": 24,
    -       "type": "stat",
    -       "title": "Embedder in-flight",
    -       "description": "Concurrent embed calls against the backend (post-cap, excludes cache hits). Spikes during ingest storms are normal; sustained values near the cap mean the embedder is the bottleneck.",
    +       "id": 33,
    +       "type": "row",
    +       "title": "Embedder & rerank",
    +       "gridPos": {
    +         "h": 1,
    +         "w": 24,
    +         "x": 0,
    +         "y": 72
    +       },
    +       "collapsed": false,
    +       "panels": []
    +     },
    +     {
    +       "id": 34,
    +       "type": "timeseries",
    +       "title": "Embedder p95 latency by backend",
    +       "description": "95th-percentile embedder call latency per backend layer; the openai layer reflects the real provider, cached/diskcache should stay near zero.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 4,
    -         "w": 8,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 60
    +         "y": 73
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "horizontal",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    -         "textMode": "auto"
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 35,
    +       "type": "timeseries",
    +       "title": "Rerank outcomes by backend",
    +       "description": "Recall rerank attempts per second by backend and result; orange fallback means the reranker failed or timed out and results shipped with pre-rerank ordering.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 73
            },
            "fieldConfig": {
              "defaults": {
    -           "unit": "short"
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".*fallback.*"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_embed_in_flight",
    -           "legendFormat": "embed in-flight",
    -           "refId": "A"
    +           "expr": "sum by (backend, result) (rate(memini_rerank_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}: {{result}}"
              }
            ]
          },
          {
    -       "id": 25,
    -       "type": "stat",
    -       "title": "Rerank in-flight",
    -       "description": "Concurrent rerank calls. A nonzero value indicates rerank is configured; if this stays pinned at the cap, the reranker is the bottleneck.",
    +       "id": 36,
    +       "type": "timeseries",
    +       "title": "Embedder tokens/s by backend",
    +       "description": "Embedding tokens consumed per second as reported by the API (openai backend only); this is the embed cost driver.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 4,
    +         "h": 7,
              "w": 8,
    -         "x": 8,
    -         "y": 60
    +         "x": 0,
    +         "y": 80
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
    -         "colorMode": "value",
    -         "graphMode": "area",
    -         "justifyMode": "auto",
    -         "orientation": "horizontal",
    -         "reduceOptions": {
    -           "calcs": ["lastNotNull"],
    -           "fields": "",
    -           "values": false
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
    -         "textMode": "auto"
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
    +         }
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (backend) (rate(memini_embed_tokens_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 37,
    +       "type": "timeseries",
    +       "title": "Embedder errors by backend",
    +       "description": "Embedder failures per second by backend; any sustained nonzero rate here explains degraded writes and keyword-only recalls elsewhere on this board.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 8,
    +         "x": 8,
    +         "y": 80
            },
            "fieldConfig": {
              "defaults": {
    -           "unit": "short"
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
            "targets": [
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "memini_rerank_in_flight",
    -           "legendFormat": "rerank in-flight",
    -           "refId": "A"
    +           "expr": "sum by (backend) (rate(memini_embed_errors_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{backend}}"
              }
            ]
          },
          {
    -       "id": 26,
    +       "id": 38,
            "type": "timeseries",
    -       "title": "Embedder batch size (p50 vs p95)",
    -       "description": "Items per embed call, by backend. batched shows a single high-p95 bar; cached should be ~1.",
    +       "title": "Embedder batch size (p50 / p95)",
    +       "description": "Texts per Embed call at the median and 95th percentile; batches stuck at 1 mean the batcher is not coalescing and each write pays a full round trip.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 4,
    +         "h": 7,
              "w": 8,
              "x": 16,
    -         "y": 60
    +         "y": 80
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 5,
    -             "showPoints": "never"
    -           }
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.50, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "p50 {{backend}}"
    +         },
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))",
    +           "refId": "B",
    +           "legendFormat": "p95 {{backend}}"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 39,
    +       "type": "timeseries",
    +       "title": "In-flight embed & rerank calls",
    +       "description": "Concurrent calls currently held against the embedding and rerank backends; pinned at the concurrency cap means requests are queueing behind the limiter.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 6,
    +         "w": 24,
    +         "x": 0,
    +         "y": 87
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
            "options": {
              "legend": {
    +           "calcs": [],
                "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.5, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))",
    -           "legendFormat": "p50 {{backend}}",
    -           "refId": "A"
    +           "expr": "memini_embed_in_flight",
    +           "refId": "A",
    +           "legendFormat": "embed"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))",
    -           "legendFormat": "p95 {{backend}}",
    -           "refId": "B"
    +           "expr": "memini_rerank_in_flight",
    +           "refId": "B",
    +           "legendFormat": "rerank"
              }
            ]
          },
          {
    -       "id": 27,
    +       "id": 40,
            "type": "row",
    -       "title": "HTTP surface",
    +       "title": "LLM \u2014 consolidation & answer (empty without an LLM configured)",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 64
    +         "y": 93
            },
            "collapsed": false,
            "panels": []
          },
          {
    -       "id": 28,
    +       "id": 41,
            "type": "timeseries",
    -       "title": "HTTP requests by route (status class)",
    -       "description": "Per-route request rate broken down by 2xx / 4xx / 5xx. 5xx spikes point at a broken handler; 4xx spikes usually mean client misuse or a misconfigured agent.",
    +       "title": "Consolidation outcomes",
    +       "description": "LLM consolidation decisions per second (gated, new, update, supersede, noop); red errors or orange drops mean writes are skipping dedup and duplicates will accumulate.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 16,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 65
    +         "y": 94
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 15,
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
                },
    -           "unit": "ops"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "dropped"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (route, status) (label_replace(rate(memini_http_requests_total[5m]), \"status_class\", \"$1\", \"status\", \"(...).\"))",
    -           "legendFormat": "{{route}} {{status_class}}xx",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_consolidate_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 29,
    +       "id": 42,
            "type": "timeseries",
    -       "title": "HTTP p95 latency by route",
    +       "title": "Consolidation queue depth",
    +       "description": "Async consolidation queue depth over time; a sawtooth is normal, monotonic growth means the LLM cannot keep pace with write volume.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 16,
    -         "y": 65
    +         "h": 7,
    +         "w": 6,
    +         "x": 12,
    +         "y": 94
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "s"
    -         }
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": false
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[5m])))",
    -           "legendFormat": "p95 {{route}}",
    -           "refId": "A"
    +           "expr": "memini_consolidate_queue_depth",
    +           "refId": "A",
    +           "legendFormat": "queue depth"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 43,
    +       "type": "stat",
    +       "title": "Dedup effectiveness (15m)",
    +       "description": "Share of consolidated writes that matched an existing memory (update/supersede/noop) rather than creating a new one; a sudden drop to zero suggests the matcher is broken.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 6,
    +         "x": 18,
    +         "y": 94
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "thresholds"
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "percentunit"
    +         },
    +         "overrides": []
    +       },
    +       "options": {
    +         "colorMode": "value",
    +         "graphMode": "area",
    +         "justifyMode": "auto",
    +         "orientation": "auto",
    +         "reduceOptions": {
    +           "calcs": ["lastNotNull"],
    +           "fields": "",
    +           "values": false
    +         },
    +         "textMode": "auto"
    +       },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result=~\"new|update|supersede|noop\"}[15m])), 1e-9)",
    +           "refId": "A",
    +           "legendFormat": "matched"
              }
            ]
          },
          {
    -       "id": 17,
    +       "id": 44,
            "type": "timeseries",
    -       "title": "End-to-end op latency p95 (by op)",
    +       "title": "Answer outcomes",
    +       "description": "Answer API (LLM-synthesized answers over recall) outcomes per second; red errors mean answer requests are burning LLM budget without returning.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 16,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 73
    +         "y": 101
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    -             "fillOpacity": 5,
    -             "showPoints": "never"
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "s"
    -         }
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byName",
    +               "options": "error"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    -           "placement": "right",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[5m])))",
    -           "legendFormat": "{{op}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_answer_results_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 34,
    +       "id": 45,
            "type": "timeseries",
    -       "title": "HTTP response size by route (p50/p90)",
    -       "description": "Body bytes shipped per response. The recall/briefing routes are what hooks inject from — watch these when tuning context-injection volume.",
    +       "title": "LLM op latency p95",
    +       "description": "95th-percentile end-to-end latency of the LLM-backed service ops (answer, promote, dedup, semantic distillation wait); these dominate tail latency when the LLM slows down.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 16,
    -         "y": 73
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 101
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "bytes"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))",
    -           "legendFormat": "p90 {{route}}",
    -           "refId": "A"
    -         },
    -         {
    -           "datasource": {
    -             "type": "prometheus",
    -             "uid": "${DS_PROMETHEUS}"
    -           },
    -           "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))",
    -           "legendFormat": "p50 {{route}}",
    -           "refId": "B"
    +           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket{op=~\"answer|promote|dedup|distill_sem_wait\"}[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{op}}"
              }
            ]
          },
          {
    -       "id": 30,
    +       "id": 46,
            "type": "row",
    -       "title": "Write-path signals",
    -       "collapsed": false,
    +       "title": "HTTP surface",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 82
    +         "y": 108
            },
    +       "collapsed": false,
            "panels": []
          },
          {
    -       "id": 31,
    +       "id": 47,
            "type": "timeseries",
    -       "title": "Corroboration routing",
    -       "description": "A fresh short-term write restating a durable fact: corroborated (confidence grown), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error.",
    +       "title": "HTTP request rate by route & status class",
    +       "description": "Requests per second per route, grouped into status classes; red 5xx traffic is server failure, orange 4xx spikes usually mean a misconfigured client or expired key.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 83
    +         "y": 109
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "short"
    +           "unit": "reqps",
    +           "min": 0
    +         },
    +         "overrides": [
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".* 5xx"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "red"
    +                 }
    +               }
    +             ]
    +           },
    +           {
    +             "matcher": {
    +               "id": "byRegexp",
    +               "options": ".* 4xx"
    +             },
    +             "properties": [
    +               {
    +                 "id": "color",
    +                 "value": {
    +                   "mode": "fixed",
    +                   "fixedColor": "orange"
    +                 }
    +               }
    +             ]
    +           }
    +         ]
    +       },
    +       "options": {
    +         "legend": {
    +           "calcs": [],
    +           "displayMode": "list",
    +           "placement": "bottom",
    +           "showLegend": true
    +         },
    +         "tooltip": {
    +           "mode": "multi",
    +           "sort": "desc"
              }
            },
    +       "targets": [
    +         {
    +           "datasource": {
    +             "type": "prometheus",
    +             "uid": "${DS_PROMETHEUS}"
    +           },
    +           "expr": "sum by (route, status_class) (label_replace(rate(memini_http_requests_total[$__rate_interval]), \"status_class\", \"$1\", \"status\", \"(.)..\"))",
    +           "refId": "A",
    +           "legendFormat": "{{route}} {{status_class}}xx"
    +         }
    +       ]
    +     },
    +     {
    +       "id": 48,
    +       "type": "timeseries",
    +       "title": "HTTP p95 latency by route",
    +       "description": "95th-percentile HTTP latency per route as clients experience it; compare with the service-op view below to tell transport overhead from service work.",
    +       "datasource": {
    +         "type": "prometheus",
    +         "uid": "${DS_PROMETHEUS}"
    +       },
    +       "gridPos": {
    +         "h": 7,
    +         "w": 12,
    +         "x": 12,
    +         "y": 109
    +       },
    +       "fieldConfig": {
    +         "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
    +           "custom": {
    +             "drawStyle": "line",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
    +             "fillOpacity": 10,
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
    +       },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_corroborate_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{route}}"
              }
            ]
          },
          {
    -       "id": 33,
    +       "id": 49,
            "type": "timeseries",
    -       "title": "Contradiction routing",
    -       "description": "A fresh durable write that changes a fact's value or flips its polarity: contradicted (stale fact invalidated via valid_to + confidence shrink), no_signal (durable neighbour found but not a real change), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error. Default-on; a contradicted rate far above expectation is the misfire smoke alarm (set MEMINI_CONTRADICT_DOWNRANK=false to disable).",
    +       "title": "Service op latency p95 by op",
    +       "description": "95th-percentile end-to-end latency of every public service operation (remember, recall, briefing, forget, ...); this is the service-internal view behind the HTTP numbers.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 0,
    -         "y": 91
    +         "y": 116
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
                },
    -           "unit": "short"
    -         }
    +           "unit": "s",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_contradict_results_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "{{op}}"
              }
            ]
          },
          {
    -       "id": 32,
    +       "id": 50,
            "type": "timeseries",
    -       "title": "Tier classification & promotion",
    -       "description": "Omitted-tier writes the marker classifier routed to a durable tier, and facts produced by the promotion pass (LLM or marker extractor).",
    +       "title": "HTTP response size (search & briefing, p50 / p90)",
    +       "description": "Response body sizes for the two injection-critical routes; growth here directly inflates the tokens injected into agent context.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    +         "h": 7,
              "w": 12,
              "x": 12,
    -         "y": 83
    +         "y": 116
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
                },
    -           "unit": "short"
    -         }
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "bytes",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (tier) (rate(memini_tier_classified_total[5m]))",
    -           "legendFormat": "classified -> {{tier}}",
    -           "refId": "A"
    +           "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))",
    +           "refId": "A",
    +           "legendFormat": "p50 {{route}}"
              },
              {
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum(rate(memini_promote_facts_total[5m]))",
    -           "legendFormat": "promoted facts",
    -           "refId": "B"
    +           "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))",
    +           "refId": "B",
    +           "legendFormat": "p90 {{route}}"
              }
            ]
          },
          {
    -       "id": 40,
    +       "id": 51,
            "type": "row",
    -       "title": "Injection efficiency (client telemetry)",
    -       "collapsed": false,
    +       "title": "Client injection telemetry",
            "gridPos": {
              "h": 1,
              "w": 24,
              "x": 0,
    -         "y": 99
    +         "y": 123
            },
    +       "collapsed": false,
            "panels": []
          },
          {
    -       "id": 41,
    +       "id": 52,
            "type": "timeseries",
    -       "title": "Injection outcomes (by result)",
    -       "description": "What hooks reported actually reaching model context vs held back by their local gates (seen/cooldown/budget/unchanged/score). Absent entirely means no integration is reporting.",
    +       "title": "Injection outcomes by result",
    +       "description": "Client-reported memory injections per second by outcome; suppressed_* series show why memories were withheld (already seen, cooldown, budget, unchanged, score) \u2014 a spike in suppressions with flat injections means the hooks are firing but delivering nothing.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    +         "h": 7,
    +         "w": 12,
              "x": 0,
    -         "y": 100
    +         "y": 124
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    +             "gradientMode": "none",
                  "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
                  "stacking": {
    -               "mode": "normal"
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
                  }
    -           }
    -         }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (result) (rate(memini_injected_memories_total[5m]))",
    -           "legendFormat": "{{result}}",
    -           "refId": "A"
    +           "expr": "sum by (result) (rate(memini_injected_memories_total[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{result}}"
              }
            ]
          },
          {
    -       "id": 42,
    +       "id": 53,
            "type": "timeseries",
    -       "title": "Injected memories (by surface)",
    -       "description": "Memories that reached context, split by which hook surface put them there.",
    +       "title": "Injected memories by surface",
    +       "description": "Memories actually injected into model context per second, by hook surface (briefing, prompt, pretool); a silent surface means that hook is broken client-side.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 8,
    -         "y": 100
    +         "h": 7,
    +         "w": 6,
    +         "x": 12,
    +         "y": 124
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    -           }
    -         }
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "ops",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [four lines unchanged)]
      
                "datasource": {
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
    -           "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[5m]))",
    -           "legendFormat": "{{surface}}",
    -           "refId": "A"
    +           "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[$__rate_interval]))",
    +           "refId": "A",
    +           "legendFormat": "{{surface}}"
              }
            ]
          },
          {
    -       "id": 43,
    +       "id": 54,
            "type": "timeseries",
    -       "title": "Injected tokens per hour (by surface)",
    -       "description": "Client-estimated tokens the hooks added to context. The number injection tuning should push down.",
    +       "title": "Injected tokens per hour by surface",
    +       "description": "Client-side estimate of context tokens injected per hour by surface; this is the direct context-budget cost of memini to the agent.",
            "datasource": {
              "type": "prometheus",
              "uid": "${DS_PROMETHEUS}"
            },
            "gridPos": {
    -         "h": 8,
    -         "w": 8,
    -         "x": 16,
    -         "y": 100
    +         "h": 7,
    +         "w": 6,
    +         "x": 18,
    +         "y": 124
            },
            "fieldConfig": {
              "defaults": {
    +           "color": {
    +             "mode": "palette-classic"
    +           },
                "custom": {
                  "drawStyle": "line",
    -             "lineInterpolation": "smooth",
    +             "lineInterpolation": "linear",
    +             "lineWidth": 1,
                  "fillOpacity": 10,
    -             "showPoints": "never"
    -           }
    -         }
    +             "gradientMode": "none",
    +             "showPoints": "never",
    +             "pointSize": 5,
    +             "spanNulls": false,
    +             "stacking": {
    +               "mode": "none",
    +               "group": "A"
    +             },
    +             "axisPlacement": "auto",
    +             "axisLabel": "",
    +             "axisColorMode": "text",
    +             "scaleDistribution": {
    +               "type": "linear"
    +             },
    +             "hideFrom": {
    +               "legend": false,
    +               "tooltip": false,
    +               "viz": false
    +             },
    +             "thresholdsStyle": {
    +               "mode": "off"
    +             }
    +           },
    +           "thresholds": {
    +             "mode": "absolute",
    +             "steps": [
    +               {
    +                 "color": "green",
    +                 "value": null
    +               }
    +             ]
    +           },
    +           "unit": "none",
    +           "min": 0
    +         },
    +         "overrides": []
            },
            "options": {
              "legend": {
    -           "displayMode": "table",
    +           "calcs": [],
    +           "displayMode": "list",
                "placement": "bottom",
    -           "showLegend": true,
    -           "calcs": ["mean", "max"]
    +           "showLegend": true
              },
              "tooltip": {
                "mode": "multi",
                "sort": "desc"
      
      [five lines unchanged)]
      
                  "type": "prometheus",
                  "uid": "${DS_PROMETHEUS}"
                },
                "expr": "sum by (surface) (increase(memini_injected_tokens_total[1h]))",
    -           "legendFormat": "{{surface}}",
    -           "refId": "A"
    +           "refId": "A",
    +           "legendFormat": "{{surface}}"
              }
            ]
          }
        ],
      
      [30 lines unchanged)]
      
        "timepicker": {},
        "timezone": "",
        "title": "memini",
        "uid": "memini",
    -   "version": 2,
    +   "version": 3,
        "weekStart": ""
      }
  
  

spec.template.spec.containers.main.image  (apps/v1/StatefulSet/local-ai/memini)
  ± value change
    - registry.erwanleboucher.dev/eleboucher/memini@sha256:2d388222bca3d65673a4072e99e0970fe0d5cdab0030ce0effdfdd78307ae7e2
    + registry.erwanleboucher.dev/eleboucher/memini@sha256:6d28c3bc3a68ef3c528632a9b012144d4843f56a2f3f0c4812105d4713a49340

Pipeline

## flate diff ``` data.memini.json (v1/ConfigMap/local-ai/memini-memini) ± value change in multiline text (314 inserts, 271 deletions) { "annotations": { "list": [] }, - "description": "memini \u2014 what the service is actually doing for your agents. Live memory counts, write/recall traffic by tier, consolidation outcomes, store health, embedder & rerank latency & saturation, HTTP surface.", + "description": "memini operator dashboard \u2014 top-to-bottom narrative: live headline health (including the deferred-repair backlog), the write path, recall & retrieval quality, aging & maintenance jobs, embedder & rerank, LLM-backed consolidation/answer, the HTTP surface, and client injection telemetry.", "editable": true, "graphTooltip": 1, "id": null, "links": [], [14 lines unchanged)] { "id": 2, "type": "stat", "title": "Live memories", - "description": "Live (non-superseded, non-expired) memories by tier.", + "description": "Total live (non-superseded, non-expired) memories across all tiers; a sudden drop means a sweep or dedup pass removed more than expected.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, + "h": 5, + "w": 4, "x": 0, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false [five lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_memories_active", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum(memini_memories_active)", + "refId": "A", + "legendFormat": "live" } ] }, { "id": 3, "type": "stat", - "title": "Writes (5m rate)", - "description": "Successful Remember calls per second, by tier.", + "title": "Writes /s (5m)", + "description": "Successful Remember writes per second over the last 5m; zero during active agent sessions means writes are failing upstream.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, - "x": 6, + "h": 5, + "w": 3, + "x": 4, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "decimals": 2 + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, - "fieldConfig": { - "defaults": { - "unit": "ops" - } - }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum(rate(memini_remember_results_total{result=\"ok\"}[5m]))", + "refId": "A", + "legendFormat": "writes/s" } ] }, { "id": 4, "type": "stat", - "title": "Recalls (5m rate)", - "description": "Successful Recall calls per second.", + "title": "Recalls /s (5m)", + "description": "Successful Recall searches per second over the last 5m; the primary read-traffic pulse.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, - "x": 12, + "h": 5, + "w": 3, + "x": 7, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "decimals": 2 + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))", + "refId": "A", + "legendFormat": "recalls/s" + } + ] + }, + { + "id": 5, + "type": "stat", + "title": "HTTP 5xx ratio (15m)", + "description": "Share of HTTP requests answered 5xx over 15m; anything above zero deserves a look, above 5% is an outage.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 10, + "y": 1 + }, "fieldConfig": { "defaults": { - "unit": "ops" - } + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 0.01 + }, + { + "color": "red", + "value": 0.05 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[5m]))", - "legendFormat": "recalls/s", - "refId": "A" + "expr": "sum(rate(memini_http_requests_total{status=~\"5..\"}[15m])) / clamp_min(sum(rate(memini_http_requests_total[15m])), 1e-9)", + "refId": "A", + "legendFormat": "5xx ratio" } ] }, { - "id": 18, + "id": 6, "type": "stat", - "title": "Consolidation queue", - "description": "Backlog of memories awaiting async consolidation. Sustained > 0 means the worker can't keep up with Remember writes.", + "title": "Repair backlog depth", + "description": "Memories currently owing deferred repair (pending + enrich + failed); green is empty, any sustained backlog means writes are storing degraded faster than repairs drain.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 6, - "w": 6, - "x": 18, + "h": 5, + "w": 4, + "x": 13, "y": 1 }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", - "orientation": "horizontal", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(memini_repair_depth)", + "refId": "A", + "legendFormat": "backlog" + } + ] + }, + { + "id": 7, + "type": "stat", + "title": "Repair oldest age", + "description": "Age of the oldest memory still awaiting repair; amber the moment anything waits, red past 1h means the repair loop is stuck or starved.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 17, + "y": 1 + }, "fieldConfig": { "defaults": { - "unit": "short", + "color": { + "mode": "thresholds" + }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { - "color": "yellow", - "value": 50 + "color": "orange", + "value": 1 }, { "color": "red", - "value": 500 + "value": 3600 } ] - } + }, + "unit": "s" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "max(memini_repair_oldest_age_seconds)", + "refId": "A", + "legendFormat": "oldest" } + ] + }, + { + "id": 8, + "type": "stat", + "title": "Consolidation queue", + "description": "Depth of the async consolidation queue; a queue that grows without draining delays dedup/supersede decisions.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 21, + "y": 1 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 100 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "memini_consolidate_queue_depth", - "legendFormat": "queued", - "refId": "A" + "refId": "A", + "legendFormat": "queued" } ] }, { - "id": 5, + "id": 9, "type": "row", - "title": "Writes & Recalls", + "title": "Write path", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 7 + "y": 6 }, "collapsed": false, "panels": [] }, { - "id": 6, + "id": 10, "type": "timeseries", - "title": "Remembers per second (by tier)", + "title": "Remember rate by tier", + "description": "Successful Remember writes per second split by tier, plus error/dropped outcomes; red or orange series above zero mean writes are being lost.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 8 + "y": 7 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] }, - "unit": "ops" - } + { + "matcher": { + "id": "byName", + "options": "dropped" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[5m]))", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum by (tier) (rate(memini_remember_results_total{result=\"ok\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{tier}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_remember_results_total{result=\"error\"}[$__rate_interval]))", + "refId": "B", + "legendFormat": "error" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_remember_results_total{result=\"dropped\"}[$__rate_interval]))", + "refId": "C", + "legendFormat": "dropped" } ] }, { - "id": 7, + "id": 11, "type": "timeseries", - "title": "Recalls per second (by hit-count bucket)", - "description": "How many memories a typical recall returns. '0' means no matches \u2014 investigate if persistent.", + "title": "Degraded (vectorless) writes by reason", + "description": "Writes stored without a vector (keyword-searchable only, queued for deferred repair) because the embedder timed out or errored; any sustained rate means the embedder is unhealthy.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 8 + "y": 7 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "embed_error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "embed_timeout" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[5m]))", - "legendFormat": "{{hits_bucket}}", - "refId": "A" + "expr": "sum by (reason) (rate(memini_remember_degraded_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{reason}}" } ] }, { - "id": 19, - "type": "row", - "title": "Retrieval quality", - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 16 - }, - "collapsed": false, - "panels": [] - }, - { - "id": 20, + "id": 12, "type": "timeseries", - "title": "Rerank outcomes by backend", - "description": "Per-recall reranker traffic: ok = reranker reordered the fused candidates; fallback = reranker timed out / errored and the fused order was kept.", + "title": "Store mutations", + "description": "Raw store write traffic: inserts vs updates (by upsert op), hard deletes from Forget, and tombstones written by consolidation; updates dominating inserts means most writes reinforce existing memories.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 17 + "y": 14 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend, result) (rate(memini_rerank_results_total[5m]))", - "legendFormat": "{{backend}}/{{result}}", - "refId": "A" + "expr": "sum by (op) (rate(memini_store_upserts_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "upsert: {{op}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(memini_store_deletes_total[$__rate_interval])", + "refId": "B", + "legendFormat": "hard delete (forget)" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(memini_store_soft_deletes_total[$__rate_interval])", + "refId": "C", + "legendFormat": "tombstone (supersede)" } ] }, { - "id": 21, + "id": 13, "type": "timeseries", - "title": "Recall degradation reasons", - "description": "Recalls that fell back to keyword-only search because the embedder was slow or errored. embed_timeout = the recall embed budget elapsed; embed_error = the embed call failed.", + "title": "Forget & supersede outcomes", + "description": "Forget and Supersede API outcomes per second; red error series should stay at zero, orange not_found spikes usually mean a client is retrying stale ids.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 17 + "y": 14 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*error.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] }, - "unit": "ops" - } + { + "matcher": { + "id": "byRegexp", + "options": ".*not_found.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (reason) (rate(memini_recall_degraded_total[5m]))", - "legendFormat": "{{reason}}", - "refId": "A" + "expr": "sum by (result) (rate(memini_forget_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "forget: {{result}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (rate(memini_supersede_results_total[$__rate_interval]))", + "refId": "B", + "legendFormat": "supersede: {{result}}" } ] }, { - "id": 22, + "id": 14, "type": "timeseries", - "title": "Recall reinforcement (best-effort)", - "description": "Background access-count bump on recalled memories. Errors are swallowed \u2014 a sustained nonzero error rate means the store write path is unhealthy but recall still served.", + "title": "Content hygiene (sanitizer)", + "description": "Ingestion sanitizer actions: cleaned content was rewritten in place, quarantined (orange) content was judged unsafe to store as-is and needs review if it becomes frequent.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 12, + "h": 7, + "w": 8, "x": 0, - "y": 25 + "y": 21 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "ops" - } + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "quarantined" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_reinforce_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "sum by (action) (rate(memini_write_sanitized_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{action}}" } ] }, { - "id": 23, - "type": "stat", - "title": "Recall success ratio", - "description": "Fraction of recall calls that succeeded. A drop means the recall path is erroring (embed + keyword + rerank + RRF + everything).", + "id": 15, + "type": "timeseries", + "title": "Corroboration routing", + "description": "Corroboration attempts on fresh short-term writes (did this confirm something we already know); a rising red error rate means the corroboration path is failing writes silently.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 25 + "h": 7, + "w": 8, + "x": 8, + "y": 21 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] }, "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, - "textMode": "auto" + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (rate(memini_corroborate_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" + } + ] + }, + { + "id": 16, + "type": "timeseries", + "title": "Contradiction routing", + "description": "Contradiction attempts on fresh durable writes (did this contradict an existing fact); contradicted events drive supersession, red errors mean conflicts are going undetected.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 21 }, "fieldConfig": { "defaults": { - "unit": "percentunit", - "min": 0, - "max": 1, + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, "thresholds": { "mode": "absolute", "steps": [ { - "color": "red", + "color": "green", "value": null - }, + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ { - "color": "yellow", - "value": 0.95 - }, - { - "color": "green", - "value": 0.99 + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } } ] } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)", - "legendFormat": "ok ratio", - "refId": "A" + "expr": "sum by (result) (rate(memini_contradict_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 8, + "id": 17, "type": "row", - "title": "Consolidation (LLM dedup)", + "title": "Recall & retrieval quality", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 33 + "y": 28 }, "collapsed": false, "panels": [] }, { - "id": 9, + "id": 18, "type": "timeseries", - "title": "Consolidation outcomes per minute", - "description": "gated = similarity gate skipped the LLM; new = nothing to merge; update = merged into an existing memory; supersede = contradiction-tombstoned an old one; noop = nothing to do; error/dropped = failure modes.", + "title": "Recall rate by hit-count bucket", + "description": "Successful recalls per second split by how many hits they returned, plus errors; a shift toward the 0-hits bucket means retrieval quality is degrading.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 16, + "h": 7, + "w": 12, "x": 0, - "y": 34 + "y": 29 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (hits_bucket) (rate(memini_recall_results_total{result=\"ok\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{hits_bucket}} hits" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_recall_results_total{result=\"error\"}[$__rate_interval]))", + "refId": "B", + "legendFormat": "error" + } + ] + }, + { + "id": 19, + "type": "timeseries", + "title": "Recall degradation (keyword-only fallback)", + "description": "Recalls that fell back to keyword-only search because embedding the query timed out or errored; these return without semantic ranking, so any sustained rate hurts result quality.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 29 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 20, + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } }, - "unit": "ops" + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "embed_error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "embed_timeout" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (reason) (rate(memini_recall_degraded_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{reason}}" + } + ] + }, + { + "id": 20, + "type": "timeseries", + "title": "Recall candidates floored (by tier filter)", + "description": "Candidates dropped from recall responses by the min_rank_score composite floor; a jump means the floor (or a scoring regression) is hiding results users would previously have seen.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 36 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] + }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["sum"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_consolidate_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "sum by (tier_filter) (rate(memini_recall_floored_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{tier_filter}}" } ] }, { - "id": 10, + "id": 21, + "type": "timeseries", + "title": "Recall reinforcement", + "description": "Best-effort reinforcement writes triggered by recall hits (strengthening recalled memories); red errors mean reinforcement is silently failing and ranking will slowly stale.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 36 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (rate(memini_reinforce_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" + } + ] + }, + { + "id": 22, "type": "stat", - "title": "Dedup effectiveness", - "description": "Fraction of consolidations that actually used the LLM (update+supersede+new+noop) vs the similarity gate skipping it. Higher = busier corpus or noisier dedup signal.", + "title": "Recall success ratio (15m)", + "description": "Share of Recall calls that completed without error over 15m; below 99% something in the search path is failing.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 8, "x": 16, - "y": 34 + "y": 36 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "orange", + "value": 0.9 + }, + { + "color": "green", + "value": 0.99 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] }, "options": { "colorMode": "value", "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "min": 0, - "max": 1 - } - }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|new|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result!~\"error|dropped\"}[15m])), 1e-9)", - "legendFormat": "LLM use ratio", - "refId": "A" + "expr": "sum(rate(memini_recall_results_total{result=\"ok\"}[15m])) / clamp_min(sum(rate(memini_recall_results_total[15m])), 1e-9)", + "refId": "A", + "legendFormat": "ok ratio" } ] }, { - "id": 11, + "id": 23, "type": "row", - "title": "Storage & decay", + "title": "Aging & maintenance", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 42 + "y": 43 }, "collapsed": false, "panels": [] }, { - "id": 12, + "id": 24, "type": "timeseries", "title": "Live memories by tier", + "description": "Live memory count per tier over time; working/episodic should churn while semantic/procedural grow slowly \u2014 a cliff in any tier means a sweep or dedup removed too much.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 43 + "y": 44 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "stepAfter", - "fillOpacity": 15, + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } }, - "unit": "short" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["mean", "max"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_memories_active", - "legendFormat": "{{tier}}", - "refId": "A" + "expr": "sum by (tier) (memini_memories_active)", + "refId": "A", + "legendFormat": "{{tier}}" } ] }, { - "id": 13, + "id": 25, "type": "timeseries", - "title": "Decay sweep + tombstones (per minute)", - "description": "Swept = memories purged by TTL; soft deletes = contradictions tombstoned by consolidation; dedup tombstones = tombstoned by the periodic dedup pass.", + "title": "Promotion, classification & demotion (per hour)", + "description": "Hourly outcomes of the episodic-to-semantic promotion pass, durable facts it wrote, and omitted-tier writes the marker classifier routed; red promote errors mean durable knowledge is not being distilled. Demotions are the mirror image: stale, never-recalled durable memories retiered down to episodic \u2014 a steady demotion stream usually means bulk-imported debris aging out.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 43 + "y": 44 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "stepAfter", - "fillOpacity": 15, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "promote: error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["sum"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_store_swept_total[5m]))", - "legendFormat": "swept/{{tier}}", - "refId": "A" + "expr": "sum by (result) (increase(memini_promote_results_total[1h]))", + "refId": "A", + "legendFormat": "promote: {{result}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(increase(memini_promote_facts_total[1h]))", + "refId": "B", + "legendFormat": "facts promoted" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "rate(memini_store_soft_deletes_total[5m])", - "legendFormat": "consolidate tombstones", - "refId": "B" + "expr": "sum by (tier) (increase(memini_tier_classified_total[1h]))", + "refId": "C", + "legendFormat": "classified \u2192 {{tier}}" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "rate(memini_dedup_tombstoned_total[5m])", - "legendFormat": "dedup tombstones", - "refId": "C" + "expr": "sum by (tier) (increase(memini_demoted_total[1h]))", + "legendFormat": "demoted from {{tier}}", + "refId": "D" } ] }, { - "id": 14, - "type": "row", - "title": "Embedder & rerank", + "id": 26, + "type": "timeseries", + "title": "Decay sweeps & dedup tombstones", + "description": "Memories purged by the decay sweeper (by tier) and tombstoned by the dedup pass, per second; sustained zero forever means aging jobs are not running.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "gridPos": { - "h": 1, - "w": 24, + "h": 7, + "w": 12, "x": 0, "y": 51 }, - "collapsed": false, - "panels": [] + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (tier) (rate(memini_store_swept_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "swept: {{tier}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(memini_dedup_tombstoned_total[$__rate_interval])", + "refId": "B", + "legendFormat": "dedup tombstoned" + } + ] }, { - "id": 15, + "id": 27, "type": "timeseries", - "title": "Embedder p95 latency (by backend)", - "description": "openai = real network calls; cached/diskcache/batched = local cache or fan-out; disabled = no endpoint configured.", + "title": "Fsck outcomes (per hour)", + "description": "Hourly consistency-sweep outcomes; red errors mean the store's self-check cannot complete and silent corruption may go unnoticed.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, + "x": 12, + "y": 51 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (result) (increase(memini_fsck_results_total[1h]))", + "refId": "A", + "legendFormat": "{{result}}" + } + ] + }, + { + "id": 28, + "type": "timeseries", + "title": "Repair backlog depth by state", + "description": "Memories owing deferred repair, split by state; red failed entries are permanently vectorless without operator action, and a growing pending line means the repair loop cannot keep up.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, "x": 0, - "y": 52 + "y": 58 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "s" + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "failed" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (state) (memini_repair_depth)", + "refId": "A", + "legendFormat": "{{state}}" } + ] + }, + { + "id": 29, + "type": "timeseries", + "title": "Repair oldest age by state", + "description": "Age of the oldest memory in each repair state; growth without bound means repairs are queued faster than they drain (the red line marks the 1h alarm threshold).", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 58 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 3600 + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[5m])))", - "legendFormat": "p95 {{backend}}", - "refId": "A" + "expr": "max by (state) (memini_repair_oldest_age_seconds)", + "refId": "A", + "legendFormat": "{{state}}" } ] }, { - "id": 16, + "id": 30, "type": "timeseries", - "title": "Embedder tokens + error rate", + "title": "Backfill pending", + "description": "Memories still awaiting embedding backfill and long memories still without chunk vectors; a number that never falls means those memories are unreachable by semantic (or chunked) recall.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, + "w": 8, + "x": 16, + "y": 58 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "memini_embed_backfill_pending", + "refId": "A", + "legendFormat": "pending embed" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "memini_chunk_backfill_pending", + "refId": "B", + "legendFormat": "pending chunk vectors" + } + ] + }, + { + "id": 31, + "type": "timeseries", + "title": "Repair outcomes by stage", + "description": "Deferred-repair attempt outcomes per second by stage; red parked means a memory gave up retrying and stays vectorless, orange retry churn means the embedder keeps failing under the repair loop.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, "w": 12, - "x": 12, - "y": 52 + "x": 0, + "y": 65 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*parked.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*retry.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] } - } + ] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend) (rate(memini_embed_tokens_total[5m]))", - "legendFormat": "tokens/s {{backend}}", - "refId": "A" + "expr": "sum by (stage, result) (rate(memini_repair_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{stage}}: {{result}}" + } + ] + }, + { + "id": 32, + "type": "timeseries", + "title": "Repair duration p95 by stage", + "description": "95th-percentile latency of a single deferred repair by stage; climbing durations shrink repair throughput and grow the backlog.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 65 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (backend) (rate(memini_embed_errors_total[5m]))", - "legendFormat": "err/s {{backend}}", - "refId": "B" + "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(memini_repair_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{stage}}" } ] }, { - "id": 24, - "type": "stat", - "title": "Embedder in-flight", - "description": "Concurrent embed calls against the backend (post-cap, excludes cache hits). Spikes during ingest storms are normal; sustained values near the cap mean the embedder is the bottleneck.", + "id": 33, + "type": "row", + "title": "Embedder & rerank", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 72 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 34, + "type": "timeseries", + "title": "Embedder p95 latency by backend", + "description": "95th-percentile embedder call latency per backend layer; the openai layer reflects the real provider, cached/diskcache should stay near zero.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 4, - "w": 8, + "h": 7, + "w": 12, "x": 0, - "y": 60 + "y": 73 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, - "textMode": "auto" + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{backend}}" + } + ] + }, + { + "id": 35, + "type": "timeseries", + "title": "Rerank outcomes by backend", + "description": "Recall rerank attempts per second by backend and result; orange fallback means the reranker failed or timed out and results shipped with pre-rerank ordering.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 73 }, "fieldConfig": { "defaults": { - "unit": "short" + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*fallback.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_embed_in_flight", - "legendFormat": "embed in-flight", - "refId": "A" + "expr": "sum by (backend, result) (rate(memini_rerank_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{backend}}: {{result}}" } ] }, { - "id": 25, - "type": "stat", - "title": "Rerank in-flight", - "description": "Concurrent rerank calls. A nonzero value indicates rerank is configured; if this stays pinned at the cap, the reranker is the bottleneck.", + "id": 36, + "type": "timeseries", + "title": "Embedder tokens/s by backend", + "description": "Embedding tokens consumed per second as reported by the API (openai backend only); this is the embed cost driver.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 4, + "h": 7, "w": 8, - "x": 8, - "y": 60 + "x": 0, + "y": 80 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, - "textMode": "auto" + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (backend) (rate(memini_embed_tokens_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{backend}}" + } + ] + }, + { + "id": 37, + "type": "timeseries", + "title": "Embedder errors by backend", + "description": "Embedder failures per second by backend; any sustained nonzero rate here explains degraded writes and keyword-only recalls elsewhere on this board.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 80 }, "fieldConfig": { "defaults": { - "unit": "short" + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "memini_rerank_in_flight", - "legendFormat": "rerank in-flight", - "refId": "A" + "expr": "sum by (backend) (rate(memini_embed_errors_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{backend}}" } ] }, { - "id": 26, + "id": 38, "type": "timeseries", - "title": "Embedder batch size (p50 vs p95)", - "description": "Items per embed call, by backend. batched shows a single high-p95 bar; cached should be ~1.", + "title": "Embedder batch size (p50 / p95)", + "description": "Texts per Embed call at the median and 95th percentile; batches stuck at 1 mean the batcher is not coalescing and each write pays a full round trip.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 4, + "h": 7, "w": 8, "x": 16, - "y": 60 + "y": 80 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 5, - "showPoints": "never" - } + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.50, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "p50 {{backend}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[$__rate_interval])))", + "refId": "B", + "legendFormat": "p95 {{backend}}" + } + ] + }, + { + "id": 39, + "type": "timeseries", + "title": "In-flight embed & rerank calls", + "description": "Concurrent calls currently held against the embedding and rerank backends; pinned at the concurrency cap means requests are queueing behind the limiter.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 87 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] + }, "options": { "legend": { + "calcs": [], "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.5, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))", - "legendFormat": "p50 {{backend}}", - "refId": "A" + "expr": "memini_embed_in_flight", + "refId": "A", + "legendFormat": "embed" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (backend, le) (rate(memini_embed_items_bucket[5m])))", - "legendFormat": "p95 {{backend}}", - "refId": "B" + "expr": "memini_rerank_in_flight", + "refId": "B", + "legendFormat": "rerank" } ] }, { - "id": 27, + "id": 40, "type": "row", - "title": "HTTP surface", + "title": "LLM \u2014 consolidation & answer (empty without an LLM configured)", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 64 + "y": 93 }, "collapsed": false, "panels": [] }, { - "id": 28, + "id": 41, "type": "timeseries", - "title": "HTTP requests by route (status class)", - "description": "Per-route request rate broken down by 2xx / 4xx / 5xx. 5xx spikes point at a broken handler; 4xx spikes usually mean client misuse or a misconfigured agent.", + "title": "Consolidation outcomes", + "description": "LLM consolidation decisions per second (gated, new, update, supersede, noop); red errors or orange drops mean writes are skipping dedup and duplicates will accumulate.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 16, + "h": 7, + "w": 12, "x": 0, - "y": 65 + "y": 94 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 15, + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } }, - "unit": "ops" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "dropped" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["mean", "max"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (route, status) (label_replace(rate(memini_http_requests_total[5m]), \"status_class\", \"$1\", \"status\", \"(...).\"))", - "legendFormat": "{{route}} {{status_class}}xx", - "refId": "A" + "expr": "sum by (result) (rate(memini_consolidate_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 29, + "id": 42, "type": "timeseries", - "title": "HTTP p95 latency by route", + "title": "Consolidation queue depth", + "description": "Async consolidation queue depth over time; a sawtooth is normal, monotonic growth means the LLM cannot keep pace with write volume.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 65 + "h": 7, + "w": 6, + "x": 12, + "y": 94 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "s" - } + "unit": "none", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": false }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[5m])))", - "legendFormat": "p95 {{route}}", - "refId": "A" + "expr": "memini_consolidate_queue_depth", + "refId": "A", + "legendFormat": "queue depth" + } + ] + }, + { + "id": 43, + "type": "stat", + "title": "Dedup effectiveness (15m)", + "description": "Share of consolidated writes that matched an existing memory (update/supersede/noop) rather than creating a new one; a sudden drop to zero suggests the matcher is broken.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 18, + "y": 94 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(rate(memini_consolidate_results_total{result=~\"update|supersede|noop\"}[15m])) / clamp_min(sum(rate(memini_consolidate_results_total{result=~\"new|update|supersede|noop\"}[15m])), 1e-9)", + "refId": "A", + "legendFormat": "matched" } ] }, { - "id": 17, + "id": 44, "type": "timeseries", - "title": "End-to-end op latency p95 (by op)", + "title": "Answer outcomes", + "description": "Answer API (LLM-synthesized answers over recall) outcomes per second; red errors mean answer requests are burning LLM budget without returning.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 16, + "h": 7, + "w": 12, "x": 0, - "y": 73 + "y": 101 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", - "fillOpacity": 5, - "showPoints": "never" + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "s" - } + "unit": "ops", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + } + ] }, "options": { "legend": { - "displayMode": "table", - "placement": "right", - "showLegend": true, - "calcs": ["mean", "max"] + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[5m])))", - "legendFormat": "{{op}}", - "refId": "A" + "expr": "sum by (result) (rate(memini_answer_results_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 34, + "id": 45, "type": "timeseries", - "title": "HTTP response size by route (p50/p90)", - "description": "Body bytes shipped per response. The recall/briefing routes are what hooks inject from — watch these when tuning context-injection volume.", + "title": "LLM op latency p95", + "description": "95th-percentile end-to-end latency of the LLM-backed service ops (answer, promote, dedup, semantic distillation wait); these dominate tail latency when the LLM slows down.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 73 + "h": 7, + "w": 12, + "x": 12, + "y": 101 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "bytes" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))", - "legendFormat": "p90 {{route}}", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[5m])))", - "legendFormat": "p50 {{route}}", - "refId": "B" + "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket{op=~\"answer|promote|dedup|distill_sem_wait\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{op}}" } ] }, { - "id": 30, + "id": 46, "type": "row", - "title": "Write-path signals", - "collapsed": false, + "title": "HTTP surface", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 82 + "y": 108 }, + "collapsed": false, "panels": [] }, { - "id": 31, + "id": 47, "type": "timeseries", - "title": "Corroboration routing", - "description": "A fresh short-term write restating a durable fact: corroborated (confidence grown), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error.", + "title": "HTTP request rate by route & status class", + "description": "Requests per second per route, grouped into status classes; red 5xx traffic is server failure, orange 4xx spikes usually mean a misconfigured client or expired key.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 83 + "y": 109 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "short" + "unit": "reqps", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".* 5xx" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".* 4xx" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" } }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum by (route, status_class) (label_replace(rate(memini_http_requests_total[$__rate_interval]), \"status_class\", \"$1\", \"status\", \"(.)..\"))", + "refId": "A", + "legendFormat": "{{route}} {{status_class}}xx" + } + ] + }, + { + "id": 48, + "type": "timeseries", + "title": "HTTP p95 latency by route", + "description": "95th-percentile HTTP latency per route as clients experience it; compare with the service-op view below to tell transport overhead from service work.", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 109 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s", + "min": 0 + }, + "overrides": [] + }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_corroborate_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "histogram_quantile(0.95, sum by (route, le) (rate(memini_http_request_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{route}}" } ] }, { - "id": 33, + "id": 49, "type": "timeseries", - "title": "Contradiction routing", - "description": "A fresh durable write that changes a fact's value or flips its polarity: contradicted (stale fact invalidated via valid_to + confidence shrink), no_signal (durable neighbour found but not a real change), cooldown (inside the 24h per-fact window), miss (no durable neighbour), error. Default-on; a contradicted rate far above expectation is the misfire smoke alarm (set MEMINI_CONTRADICT_DOWNRANK=false to disable).", + "title": "Service op latency p95 by op", + "description": "95th-percentile end-to-end latency of every public service operation (remember, recall, briefing, forget, ...); this is the service-internal view behind the HTTP numbers.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 0, - "y": 91 + "y": 116 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] }, - "unit": "short" - } + "unit": "s", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_contradict_results_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "histogram_quantile(0.95, sum by (op, le) (rate(memini_op_duration_seconds_bucket[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{op}}" } ] }, { - "id": 32, + "id": 50, "type": "timeseries", - "title": "Tier classification & promotion", - "description": "Omitted-tier writes the marker classifier routed to a durable tier, and facts produced by the promotion pass (LLM or marker extractor).", + "title": "HTTP response size (search & briefing, p50 / p90)", + "description": "Response body sizes for the two injection-critical routes; growth here directly inflates the tokens injected into agent context.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, + "h": 7, "w": 12, "x": 12, - "y": 83 + "y": 116 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } }, - "unit": "short" - } + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "bytes", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (tier) (rate(memini_tier_classified_total[5m]))", - "legendFormat": "classified -> {{tier}}", - "refId": "A" + "expr": "histogram_quantile(0.50, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "p50 {{route}}" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum(rate(memini_promote_facts_total[5m]))", - "legendFormat": "promoted facts", - "refId": "B" + "expr": "histogram_quantile(0.90, sum by (route, le) (rate(memini_http_response_bytes_bucket{route=~\"/v1/search|/v1/namespaces/briefing\"}[$__rate_interval])))", + "refId": "B", + "legendFormat": "p90 {{route}}" } ] }, { - "id": 40, + "id": 51, "type": "row", - "title": "Injection efficiency (client telemetry)", - "collapsed": false, + "title": "Client injection telemetry", "gridPos": { "h": 1, "w": 24, "x": 0, - "y": 99 + "y": 123 }, + "collapsed": false, "panels": [] }, { - "id": 41, + "id": 52, "type": "timeseries", - "title": "Injection outcomes (by result)", - "description": "What hooks reported actually reaching model context vs held back by their local gates (seen/cooldown/budget/unchanged/score). Absent entirely means no integration is reporting.", + "title": "Injection outcomes by result", + "description": "Client-reported memory injections per second by outcome; suppressed_* series show why memories were withheld (already seen, cooldown, budget, unchanged, score) \u2014 a spike in suppressions with flat injections means the hooks are firing but delivering nothing.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, + "h": 7, + "w": 12, "x": 0, - "y": 100 + "y": 124 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, + "gradientMode": "none", "showPoints": "never", + "pointSize": 5, + "spanNulls": false, "stacking": { - "mode": "normal" + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" } - } - } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (result) (rate(memini_injected_memories_total[5m]))", - "legendFormat": "{{result}}", - "refId": "A" + "expr": "sum by (result) (rate(memini_injected_memories_total[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{result}}" } ] }, { - "id": 42, + "id": 53, "type": "timeseries", - "title": "Injected memories (by surface)", - "description": "Memories that reached context, split by which hook surface put them there.", + "title": "Injected memories by surface", + "description": "Memories actually injected into model context per second, by hook surface (briefing, prompt, pretool); a silent surface means that hook is broken client-side.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 100 + "h": 7, + "w": 6, + "x": 12, + "y": 124 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" - } - } + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [four lines unchanged)] "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[5m]))", - "legendFormat": "{{surface}}", - "refId": "A" + "expr": "sum by (surface) (rate(memini_injected_memories_total{result=\"injected\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{surface}}" } ] }, { - "id": 43, + "id": 54, "type": "timeseries", - "title": "Injected tokens per hour (by surface)", - "description": "Client-estimated tokens the hooks added to context. The number injection tuning should push down.", + "title": "Injected tokens per hour by surface", + "description": "Client-side estimate of context tokens injected per hour by surface; this is the direct context-budget cost of memini to the agent.", "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 100 + "h": 7, + "w": 6, + "x": 18, + "y": 124 }, "fieldConfig": { "defaults": { + "color": { + "mode": "palette-classic" + }, "custom": { "drawStyle": "line", - "lineInterpolation": "smooth", + "lineInterpolation": "linear", + "lineWidth": 1, "fillOpacity": 10, - "showPoints": "never" - } - } + "gradientMode": "none", + "showPoints": "never", + "pointSize": 5, + "spanNulls": false, + "stacking": { + "mode": "none", + "group": "A" + }, + "axisPlacement": "auto", + "axisLabel": "", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "min": 0 + }, + "overrides": [] }, "options": { "legend": { - "displayMode": "table", + "calcs": [], + "displayMode": "list", "placement": "bottom", - "showLegend": true, - "calcs": ["mean", "max"] + "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" [five lines unchanged)] "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "sum by (surface) (increase(memini_injected_tokens_total[1h]))", - "legendFormat": "{{surface}}", - "refId": "A" + "refId": "A", + "legendFormat": "{{surface}}" } ] } ], [30 lines unchanged)] "timepicker": {}, "timezone": "", "title": "memini", "uid": "memini", - "version": 2, + "version": 3, "weekStart": "" } spec.template.spec.containers.main.image (apps/v1/StatefulSet/local-ai/memini) ± value change - registry.erwanleboucher.dev/eleboucher/memini@sha256:2d388222bca3d65673a4072e99e0970fe0d5cdab0030ce0effdfdd78307ae7e2 + registry.erwanleboucher.dev/eleboucher/memini@sha256:6d28c3bc3a68ef3c528632a9b012144d4843f56a2f3f0c4812105d4713a49340 ``` [Pipeline](https://ci.aether.earth/repos/4/pipeline/699)

ci/crow/pr/dns-test Pipeline was successful

Details

ci/crow/push/dns-test Pipeline was successful

Details

ci/crow/push/tanka-test Pipeline was successful

Details

ci/crow/pr/tanka-test Pipeline was successful

Details

ci/crow/pr/tanka-diff Pipeline was successful

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x:renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git switch renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

Merge

Merge the changes and update on Forgejo.

git switch main

git merge --no-ff renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git switch renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git rebase main

git switch main

git merge --ff-only renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git switch renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git rebase main

git switch main

git merge --no-ff renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git switch main

git merge --squash renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git switch main

git merge --ff-only renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git switch main

git merge renovate/registry.erwanleboucher.dev-eleboucher-charts-memini-0.x

git push origin main

Rows
Columns

chore(deps): update registry.erwanleboucher.dev/eleboucher/charts/memini docker tag to v0.7.37 #159

Configuration

flate diff

flate diff

Checkout

Merge