UpgradePath
Sample runbook

Prometheus 3.11 → 3.12

caution

Claims testedrun 2026-10-24
2 observations

prometheus suite: Downgrades and replays the write-ahead log with the st-storage flag on and off; with it on, data loss is the expected result.

  • [prometheus] st-storage=ON 3.12->3.11 WAL replay: REPLAY_DATA_LOSS
  • [prometheus] st-storage=OFF 3.12->3.11 WAL replay: REPLAY_CLEAN

One of 158 published runbooks, shown in full. Agents fetch the others with get_runbook.

Found on a real cluster. With the st-storage flag on, 3.12 writes histogram samples as a new WAL record type that 3.11 skips on replay, so a rollback loses them. The 3.12 release notes don't mention the rollback effect. A real cluster upgrade and rollback surfaced it, and a control run with the flag off replayed clean.

Summary

Prometheus 3.12.0 is primarily a security, feature and performance release. The severity here is caution rather than safe because Remote Write now rejects snappy payloads declaring a decoded length over 32MB, gzip-encoded OTLP bodies are capped at a decompressed size, and one bugfix converts a previously-silent remote_write misconfiguration into a load-time error (v3.12.0 release notes).

Two CVE-class fixes make this a recommended upgrade for anyone running Prometheus as a Remote Write receiver or using STACKIT service discovery (v3.12.0 release notes).

Notable additions you may want to adopt afterwards: /api/v1/status/self_metrics, a UI for deleting series and cleaning tombstones, DigitalOcean Managed Databases and Outscale VM SD, AWS SD external_id and EC2 IPv6, and the experimental use-start-timestamps / st-synthesis feature flags (v3.12.0 release notes).

The matching Helm chart for appVersion v3.12.0 is prometheus-29.13.1 (Chart.yaml).

Breaking changes, explained

1. Remote Write rejects snappy payloads declaring >32MB decoded length (remote-write-snappy-32mb-limit)

Prometheus now rejects snappy-compressed requests received via Remote Write whose declared decoded length exceeds 32MB — a decompression-bomb DoS fix (v3.12.0 release notes, #18642). If this Prometheus acts as a Remote Write receiver, senders emitting very large batches may now have writes rejected; review sender batch sizing.

2. STACKIT SD secret exposure fix (stackit-sd-secret-redaction)

Secrets in stackit_sd_configs were exposed in plaintext via the config endpoint; this is fixed under GHSA-39j6-789q-qxvh (v3.12.0 release notes, #18649). The release note names it /-/config, but 3.12.0 has no such route: config is served at /api/v1/status/config and shown on the /config page (web.go, api.go at v3.12.0). If you use STACKIT SD, treat any credential that was in the config as leaked and rotate it.

3. OTLP gzip decompressed body size limit (otlp-gzip-decompressed-size-limit)

gzip-encoded OTLP write requests now have their decompressed body size limited (v3.12.0 release notes, #18408). The decoder reads at most 32 MiB of decompressed body (decodeReadLimit in codec.go at v3.12.0).

4. remote_write queue_config validated at load time (remote-write-queue-config-validation)

queue_config fields are now validated when the configuration is loaded, to prevent runtime panic and silent misconfiguration (v3.12.0 release notes, #18209). Validate with a 3.12 promtool check config before rolling, so any config that 3.11 accepted but 3.12 rejects is caught off-cluster.

5. Consul SD: health_filter split from filter (consul-sd-health-filter-split)

A new health_filter field was added for Health API filtering, "fixing breakage when using Catalog-only fields like ServiceTags in filter" (v3.12.0 release notes, #18479, #18499). #18499 passes health_filter only to the Health API and keeps filter Catalog-only, so move Health-API predicates into health_filter and keep Catalog predicates in filter. v3.11.2 already ships this (#18499), so from 3.11.2 or later nothing changes here.

6. WAL histogram samples carry a Start Timestamp field (wal-histogram-start-timestamp-field)

The TSDB/Agent WAL now adds a Start Timestamp field to all WAL histogram samples in memory when the st-storage flag is enabled (v3.12.0 release notes, #18221). With the flag on, 3.12.0 writes histogram samples as the new WAL record types HistogramSamplesV2 and FloatHistogramSamplesV2 (record.go at v3.12.0). 3.11.3's WAL replay has no case for them and ends in default: // Noop. (head_wal.go at v3.11.3), so after a rollback to 3.11 those records are skipped and their histogram samples are not replayed.

7. SD metrics deleted on scrape job removal (sd-metric-deletion-on-job-removal)

prometheus_sd_refresh* and prometheus_sd_discovered_targets for a scrape job are now deleted when that job is removed (v3.12.0 release notes, #17614). Previously stale series lingered.

8. Behavioural changes worth flagging (non-breaking but visible)

Migration steps

  1. Pre-flight config validation. Run against your live config, using a 3.12 promtool (this is what catches the new queue_config validation before it takes down a pod). In the Helm chart, take the deployed values (they include any --set or extra -f overrides), render, then assert the extraction is non-empty:

    helm repo update
helm get values <release> -n <ns> -o yaml > values.yaml
helm template <release> prometheus-community/prometheus -n <ns> --version 29.13.1 -f values.yaml \
  --show-only templates/cm.yaml > cm.yaml
kubectl annotate --local -f cm.yaml x=y -o jsonpath='{.data.prometheus\.yml}' > prometheus.yml
test -s prometheus.yml || { echo "extraction failed"; exit 1; }

    Then invoke promtool from the release image by overriding the entrypoint; --user runs it as you rather than the image's nobody, so it can read your files:

    docker run --rm --user "$(id -u)" --entrypoint=promtool -v "$PWD:/cfg" \
  quay.io/prometheus/prometheus:v3.12.0 check config --syntax-only /cfg/prometheus.yml

    --syntax-only skips the rule_files and *_file paths, which do not exist inside the container, but still loads the config, where queue_config is validated (config.go at v3.12.0).

  2. Grep for the affected config surfaces.

    grep -nE 'stackit_sd_configs|consul_sd_configs|queue_config|auto-reload-config' prometheus.yml values.yaml
    • Any stackit_sd_configs: rotate every credential that appeared in the config, since it may have been served in plaintext via /api/v1/status/config on 3.11 (release notes, GHSA-39j6-789q-qxvh). Rotate after the upgrade lands so the new credential is never exposed by the old binary.
    • Any consul_sd_configs with filter:: move Health-API predicates into health_filter:, leaving Catalog-only fields such as ServiceTags in filter (#18499). On 3.11.0 or 3.11.1, which reject health_filter, stage this edit for the upgrade apply; 3.11.2 and later accept it now.

  3. Leave the auto-reload-config flag as it is. The feature is promoted to stable in 3.12 (release notes, #18620), but 3.12.0 still accepts --enable-feature=auto-reload-config, logging "This option for --enable-feature is deprecated. Use --config.auto-reload instead." (main.go at v3.12.0). Its replacement --config.auto-reload defaults to false, so deleting the old entry from server.extraFlags/server.extraArgs without adding it turns auto-reload off. Switch only once you no longer need a 3.11 rollback: 3.11.3 defines no --config.auto-reload flag (main.go at v3.11.3). Note server.extraFlags defaults to [web.enable-lifecycle] in chart 29.13.1 (values.yaml).

  4. Check Remote Write sender batch sizes if this Prometheus is a receiver, so that snappy-compressed requests do not declare a decoded length above 32MB (release notes, #18642). Non-Prometheus senders (Grafana Agent/Alloy, Mimir distributors, custom clients) need the same review.

  5. Check OTLP collector batch sizes if you use the OTLP write endpoint (release notes, #18408).

  6. Capture pre-upgrade baselines:

    kubectl -n <ns> get deploy,sts -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=server
kubectl -n <ns> port-forward deploy/<name> 9090:9090 &   # sts/<name> when server.statefulSet.enabled
until curl -sf localhost:9090/-/ready >/dev/null; do sleep 1; done
curl -sfG localhost:9090/api/v1/query --data-urlencode 'query=count by (job) (up)' \
  > baseline-up.json
curl -sfG localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (config) (prometheus_sd_discovered_targets)' \
  > baseline-sd.json
test -s baseline-up.json && test -s baseline-sd.json || echo "FAIL: baseline capture"

    Then take an EBS snapshot of the Prometheus PVC. kubectl get pvc does not expose the EBS volume id — resolve the bound PV and read spec.csi.volumeHandle (spec.awsElasticBlockStore.volumeID for in-tree gp2 volumes), then wait for the snapshot to complete:

    PVC=$(kubectl -n <ns> get pvc -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=server -o jsonpath='{.items[0].metadata.name}')
PV=$(kubectl -n <ns> get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
VOL=$(kubectl get pv "$PV" -o jsonpath='{.spec.csi.volumeHandle}{.spec.awsElasticBlockStore.volumeID}'); VOL=${VOL##*/}
: "${VOL:?PV $PV has no EBS volume id}"
SNAP=$(aws ec2 create-snapshot --volume-id "$VOL" --description "prom pre-3.12" \
  --query SnapshotId --output text)
aws ec2 wait snapshot-completed --snapshot-ids "$SNAP"

    Record $PVC and $SNAP. Do not proceed to step 7 unless aws ec2 wait exits 0.

  7. Upgrade — image bump and the staged config edits in one atomic apply.

    • First, re-render prometheus.yml from the edited values.yaml (Consul health_filter moves included) and re-run the 3.12 promtool check config --syntax-only on it. Do not apply if it fails. For Helm, diff the release against the new chart and stop on changes outside the Prometheus server:
      helm get manifest <release> -n <ns> > old.yaml
helm template <release> prometheus-community/prometheus -n <ns> --version 29.13.1 -f values.yaml --no-hooks > new.yaml
diff old.yaml new.yaml
    • Helm: helm upgrade <release> prometheus-community/prometheus -n <ns> --version 29.13.1 -f values.yaml with the staged edits already in values.yaml, so the new ConfigMap and the new image land together.
    • Plain manifests: apply the new ConfigMap and the image bump to quay.io/prometheus/prometheus:v3.12.0 in a single kubectl apply.
    • The chart's server.image.tag defaults to empty, in which case the appVersion field from Chart.yaml is used (values.yaml), which for 29.13.1 is v3.12.0 (Chart.yaml). If values.yaml sets server.image.tag, set it to v3.12.0 or delete it; otherwise the upgrade keeps the pinned 3.11 image and rollout status still passes.
    • The chart default deployment strategy is Recreate with replicaCount: 1 and server.statefulSet.enabled: false (values.yaml), so expect a short scrape gap during the roll.
    • Gate before Verify: kubectl -n <ns> rollout status deployment/<name> --timeout=10m (statefulset/<name> when server.statefulSet.enabled: true). If it does not reach Ready, read kubectl -n <ns> logs <pod> -c prometheus-server --previous and go to Rollback.

  8. Rotate STACKIT credentials if you use STACKIT SD and /api/v1/status/config was reachable by anyone untrusted (release notes, GHSA-39j6-789q-qxvh).

  9. Do not enable experimental start-timestamp flags in the same change. use-start-timestamps changes rate()/irate()/increase()/resets() (#18344, #18627), st-synthesis is new (#18279), and st-storage changes WAL contents (#18221) (release notes). Ship the version bump first, adopt flags separately.

Gotchas

EKS-specific notes

Verify after upgrade

First resolve the real workload name and open a local port; every check below runs against that forwarded port from your own machine.

kubectl -n <ns> get deploy,sts -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=server
kubectl -n <ns> port-forward deploy/<name> 9090:9090 &   # sts/<name> when server.statefulSet.enabled

  1. Version and startup:

    curl -sf localhost:9090/api/v1/status/buildinfo

    Expect "version":"3.12.0". If you prefer to exec into the pod, resolve the container name first and use whatever HTTP client the image provides (wget or curl):

    POD=$(kubectl -n <ns> get pod -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=server -o jsonpath='{.items[0].metadata.name}')
kubectl -n <ns> get pod "$POD" -o jsonpath='{.spec.containers[*].name}'
kubectl -n <ns> exec "$POD" -c <container-name> -- \
  wget -qO- localhost:9090/api/v1/status/buildinfo

  2. Config actually loaded (not just pod Running):

    curl -s localhost:9090/api/v1/status/config | head -c 400
curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_config_last_reload_successful'

    Expect the reload metric to be 1.

  3. STACKIT redaction (only if you use it). Fail closed — a failed request or empty body must not read as "redacted":

    BODY=$(curl -sf localhost:9090/api/v1/status/config) || { echo "FAIL: config endpoint unreachable"; exit 1; }
[ -n "$BODY" ] || { echo "FAIL: empty config body"; exit 1; }
printf '%s' "$BODY" | jq -r .data.yaml | grep -i -A10 stackit

    The endpoint returns JSON with the whole config in the data.yaml string, so decode it before grepping. Assert service_account_key and private_key render as <secret>, and fail the step if any credential value appears verbatim: #18649 (GHSA-39j6-789q-qxvh) makes them config.Secret, which marshals as <secret> (config.go in prometheus/common v0.67.5).

  4. Target counts unchanged — compare against the baselines captured in Migration step 6, which were produced by:

    curl -sG localhost:9090/api/v1/query --data-urlencode 'query=count by (job) (up)' > baseline-up.json
curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (config) (prometheus_sd_discovered_targets)' > baseline-sd.json

    Now re-run both queries against the upgraded server and diff against baseline-up.json / baseline-sd.json. If those files are absent (e.g. you were handed only this section), capture them from a still-running 3.11 instance, or compare in-place against a lookback query such as:

    curl -sG localhost:9090/api/v1/query --data-urlencode 'query=count by (job) (up offset 1h)'

    Pay special attention to Consul and EC2 jobs.

  5. Remote Write health. The two queries below are sender-side (remote-write queue manager) metrics — they say nothing about this server's role as a receiver:

    curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(prometheus_remote_storage_samples_failed_total[5m])'
curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(prometheus_remote_storage_samples_dropped_total[5m])'

    For the receiver role — where the new 32MB snappy declared-decoded-length rejection (release notes, #18642) and the OTLP gzip decompressed-size limit (#18408) apply — inspect HTTP status codes per handler on this server and the server logs:

    curl -sG localhost:9090/api/v1/query --data-urlencode \
  'query=sum by (code) (rate(prometheus_http_requests_total{handler=~"/api/v1/write|/api/v1/otlp/v1/metrics"}[5m]))'
kubectl -n <ns> logs <pod> -c prometheus-server --since=15m | grep -iE 'decompress|too large|snappy|otlp'

    Also check failure/retry counters on each remote sender (Alloy/Agent, Mimir distributor, OTel collector). Confirm the metric names used here still exist in 3.12 before treating an all-zero result as proof of health — a query returning no series is not the same as a rate of zero.

  6. OTLP ingest (if used): watch for failed gzip-encoded OTLP writes and rising drop counters on the sending collector (release notes, #18408).

  7. TSDB health after WAL replay:

    curl -sG localhost:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_head_chunks'
curl -sG localhost:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_wal_corruptions_total'
curl -sG localhost:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_head_series'

    head_chunks should be non-negative (release notes, #18401).

  8. Rules and alerts still evaluating:

    curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_rule_evaluation_failures_total'
curl -sG localhost:9090/api/v1/query --data-urlencode 'query=count(ALERTS)'

    Watch for rules using last_over_time/first_over_time over subqueries with abs()-style functions producing differently-named output series (release notes, #18409).

  9. New endpoint smoke test:

    curl -s localhost:9090/api/v1/status/self_metrics | head -c 200

    should return JSON (release notes, #18411).

Rollback

First re-resolve the objects this section acts on — do not rely on values from earlier sections:

kubectl -n <ns> get deploy,sts -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=server
helm get values <release> -n <ns>              # current (3.12) values, incl. staged edits
helm history <release> -n <ns>                 # previous revision number
PVC=$(kubectl -n <ns> get pvc -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=server -o jsonpath='{.items[0].metadata.name}')
PV=$(kubectl -n <ns> get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
VOL=$(kubectl get pv "$PV" -o jsonpath='{.spec.csi.volumeHandle}{.spec.awsElasticBlockStore.volumeID}'); VOL=${VOL##*/}
aws ec2 describe-snapshots --filters "Name=volume-id,Values=$VOL" \
  --query 'Snapshots[].{Id:SnapshotId,State:State,Time:StartTime}'

The Consul health_filter: edit reverts only for a rollback to 3.11.0 or 3.11.1. health_filter shipped in v3.11.2 (#18499), so 3.11.2 and later accept it; 3.11.0 and 3.11.1 lack the field and load config with yaml.UnmarshalStrict (config.go at v3.11.0), so for them move those predicates back into filter:.

If you did not enable st-storage, use-start-timestamps or st-synthesis, rollback is straightforward:

  1. Helm: helm rollback <release> <previous-revision> -n <ns> — or re-run helm upgrade pinning server.image.tag: v3.11.x and the previous chart version. This reverts image and config together.
  2. Plain manifests: revert the ConfigMap before, or in the same kubectl apply as, the image revert. kubectl -n <ns> rollout undo deployment/<name> (or rollout undo statefulset/<name> when server.statefulSet.enabled: true) reverts only the pod template — not the ConfigMap — so if you use it, apply the reverted ConfigMap first and then verify the 3.11 pod starts and reloads its config successfully (prometheus_config_last_reload_successful == 1, pod not in CrashLoopBackOff). The chart default revisionHistoryLimit: 10 retains prior revisions (values.yaml).
  3. Apply the health_filter: → filter: revert named above, if it applies, as part of that same rollback apply.

If you enabled st-storage: 3.11 skips the histogram records 3.12 wrote with the flag on, and scaling down or snapshotting first does not change that. Roll back only if losing those histogram samples is acceptable; otherwise stay on 3.12. To return to the pre-upgrade data instead, create a volume from the pre-upgrade snapshot in the old volume's zone, bind it with a PV and PVC per the EBS CSI driver's static-provisioning example, and set server.persistentVolume.existingClaim to that PVC. Everything written since the snapshot is lost.

Rollback is not a remediation for the two security fixes. 3.12 fixes the Remote Write decompression DoS (#18642) and the STACKIT config-endpoint plaintext secret exposure (GHSA-39j6-789q-qxvh) (release notes); the cited sources do not describe rollback behaviour, but a 3.11 binary does not contain those fixes. If you must roll back, restrict network access to /api/v1/write, /api/v1/status/config and /config until you can move forward again.