Skip to main content

Canary analysis gates

Analysis gates judge canaries on real production metrics before every weight advancement and before final promotion. A canary that increases the error rate never reaches 100%.

Defining a gate​

Gates live in the config bucket at analysis/<name>.yaml:

# analysis/error-rate.yaml — 5xx ratio on the canary track
query: |
  sum(rate(nginx_ingress_controller_requests{exported_namespace="{{namespace}}",service="{{canary}}",status=~"5.."}[2m]))
  /
  sum(rate(nginx_ingress_controller_requests{exported_namespace="{{namespace}}",service="{{canary}}"}[2m]))
maxValue: 0.01      # fail when the result exceeds this
failureLimit: 3     # consecutive failures before abort

Queries are PromQL returning a single value. These variables are substituted before execution:

VariableValue
{{app}}application name
{{canary}}canary workload/service name (<app>-canary)
{{namespace}}tenant namespace
{{tenant}}tenant
{{env}}environment

Use minValue instead of maxValue for metrics that must stay above a floor (e.g. success ratio).

Using gates​

Projects (or, better, environment profiles) reference gates by name:

deploy:
  strategy: canary
  canary:
    steps: [10, 50]
    stepInterval: 5m
    analysis: [error-rate, latency-p95]

Named gates resolve from the platform library into the Release snapshot at upload — recorded in templateVersions, so every release documents exactly which gate definitions judged it. Inline gates with an explicit query are also accepted for one-off cases.

Evaluation semantics​

Every gate is evaluated after the canary has been fully ready for a whole stepInterval:

  • All gates pass → advance to the next step (or promote past the last one); the failure budget resets.
  • A gate fails → the canary holds its current weight and re-evaluates every 30 seconds. Nothing is torn down.
  • Consecutive failures reach the limit (strictest configured failureLimit, default 3) → abort: the canary track is deleted, the application points back at the old release (which never stopped serving), and the Release records the failing gate and value: gate error-rate: 0.031 exceeds max 0.01.
  • A gate cannot be measured (query error, no data) → counts as a failure. A canary that cannot be judged must not advance — but because it consumes the same budget, a brief Prometheus outage does not abort a healthy rollout.
  • No metrics backend configured → analysis is skipped with a loud AnalysisSkipped warning event. Platform misconfiguration degrades to readiness-only canaries instead of blocking every deploy.

Wiring Prometheus​

# Helm values
prometheusURL: http://prometheus.monitoring.svc:9090

The operator queries the instant-query API; any Prometheus-compatible endpoint (Thanos, Mimir, VictoriaMetrics) works. Additional backends can implement the same single-method client interface.