WAF-REL-060 – Incident Response & Runbook Readiness

Pillar: Reliability | Severity: High | Category: Logging Monitoring | Automatable: High

Description

Every production workload MUST have a documented Incident Response (IR) plan with defined severity levels, escalation paths, and on-call rotations. Runbooks MUST exist for all critical alerts and MUST be version-controlled. Runbooks MUST be linked directly from alert notifications. Post-incident reviews MUST be conducted within 5 business days and findings MUST be tracked as actionable items. Mean Time to Recovery (MTTR) MUST be measured per incident.

Rationale

Risk	Description
Risk	Missing incident-response plans lead to panicked ad-hoc reactions and longer MTTR.
Risk	Unclear escalation paths delay involving the right experts.

Risk

Description

Risk

Missing incident-response plans lead to panicked ad-hoc reactions and longer MTTR.

Risk

Unclear escalation paths delay involving the right experts.

Requirements

Every production workload MUST have a documented Incident Response (IR) plan with defined severity levels, escalation paths, and on-call rotations.
Runbooks MUST exist for all critical alerts and MUST be version-controlled.
Runbooks MUST be linked directly from alert notifications.
Post-incident reviews MUST be conducted within 5 business days and findings MUST be tracked as actionable items.
Mean Time to Recovery (MTTR) MUST be measured per incident.

Implementation Guidance

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).
Implement the required configuration as code in Terraform or equivalent IaC.

Maturity Levels

Level	Name	Criteria
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Criteria

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-rel-060.tf.aws.sns-topic-alarm-action

Engine: terraform | Provider: aws | Automated: Yes

AWS CloudWatch alarms must route to SNS topics for on-call notification

CloudWatch alarms for production services must have alarm_actions pointing to SNS topics that are connected to on-call notification systems.

Scope: resource — aws_cloudwatch_metric_alarm

Assertions: * alarm_actions not empty — CloudWatch alarm must have alarm_actions configured for on-call notification. * ok_actions not empty — CloudWatch alarm must have ok_actions to notify when alert resolves.

Remediation: Add alarm_actions and ok_actions pointing to an SNS topic. The SNS topic should be subscribed to by PagerDuty, OpsGenie, or email for on-call notification.

Compliant

resource "aws_cloudwatch_metric_alarm" "api_errors" {
  alarm_name          = "payment-api-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "5XXError"
  namespace           = "AWS/ApiGateway"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  alarm_actions       = [aws_sns_topic.oncall_critical.arn]
  ok_actions          = [aws_sns_topic.oncall_critical.arn]
  alarm_description   = "SEV2: Payment API error rate > 1%. Runbook: https://wiki/runbooks/payment-api-errors"
}

Non-Compliant

resource "aws_cloudwatch_metric_alarm" "api_errors" {
  alarm_name          = "payment-api-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "5XXError"
  namespace           = "AWS/ApiGateway"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  # Missing: alarm_actions – alert fires but no one is notified
  # WAF-REL-060 violation
}

waf-rel-060.tf.azurerm.action-group-configured

Engine: terraform | Provider: azurerm | Automated: Yes

Azure Monitor action groups must have webhook or email receivers configured

Azure Monitor action groups used by metric alerts must have at least one email_receiver or webhook_receiver configured for on-call notification.

Scope: resource — azurerm_monitor_action_group

Assertions: * short_name not empty — Action group must have a short_name for SMS/voice notifications. * email_receiver attribute exists — Action group must have at least one email_receiver or webhook_receiver.

Remediation: Add email_receiver or webhook_receiver blocks to the azurerm_monitor_action_group. For PagerDuty or OpsGenie integration, use webhook_receiver pointing to their API URL.

Compliant

resource "azurerm_monitor_action_group" "oncall" {
  name                = "oncall-critical"
  resource_group_name = azurerm_resource_group.main.name
  short_name          = "oncall"

  email_receiver {
    name          = "on-call-team"
    email_address = var.oncall_email
  }

  webhook_receiver {
    name        = "pagerduty"
    service_uri = var.pagerduty_webhook_url
  }
}

Non-Compliant

resource "azurerm_monitor_action_group" "alerts" {
  name                = "alerts"
  resource_group_name = azurerm_resource_group.main.name
  short_name          = "alerts"
  # Missing: no receivers configured – alerts fire silently
  # WAF-REL-060 violation
}

waf-rel-060.tf.google.monitoring-notification-channel

Engine: terraform | Provider: google | Automated: Yes

GCP alert policies must have notification channels configured

GCP Monitoring alert policies must reference at least one notification channel to ensure on-call teams are notified during incidents.

Scope: resource — google_monitoring_alert_policy

Assertions: * notification_channels not empty — Alert policy must reference at least one notification channel. * display_name not empty — Alert policy must have a descriptive display name including service context.

Remediation: Add notification_channels to the google_monitoring_alert_policy referencing configured google_monitoring_notification_channel resources.

Compliant

resource "google_monitoring_notification_channel" "pagerduty" {
  display_name = "PagerDuty On-Call"
  type         = "pagerduty"
  labels = {
    service_key = var.pagerduty_service_key
  }
}

resource "google_monitoring_alert_policy" "api_error_rate" {
  display_name          = "Payment API – Error Rate SEV2"
  combiner              = "OR"
  notification_channels = [google_monitoring_notification_channel.pagerduty.name]
  conditions {
    display_name = "Error rate > 1%"
    condition_threshold {
      filter          = "resource.type=\"cloud_run_revision\" AND metric.type=\"run.googleapis.com/request_count\""
      comparison      = "COMPARISON_GT"
      threshold_value = 100
      duration        = "60s"
    }
  }
}

Non-Compliant

resource "google_monitoring_alert_policy" "api_errors" {
  display_name = "Errors"
  combiner     = "OR"
  conditions {
    display_name = "Error count high"
    condition_threshold {
      filter          = "metric.type=\"logging.googleapis.com/user/errors\""
      comparison      = "COMPARISON_GT"
      threshold_value = 100
      duration        = "60s"
    }
  }
  # Missing: notification_channels – WAF-REL-060 violation
}

waf-rel-060.tf.stackit.auto-recovery-enabled

Engine: terraform | Provider: stackit | Automated: Yes

Every production workload for stackit

Every production workload must be documented in IaC via a auto-recovery-enabled tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the auto-recovery-enabled tag/label.

Remediation: Add a auto-recovery-enabled tag/label to the stackit_project resource to show the stackit implementation of WAF-REL-060 is tracked in IaC.

Compliant

resource "stackit_project" "example" {
  labels = {
    auto-recovery-enabled = "true"
  }
}

Non-Compliant

resource "stackit_project" "example" {
  # Missing auto-recovery-enabled labels
}

waf-rel-060.tf.ovh.auto-recovery-enabled

Engine: terraform | Provider: ovh | Automated: Yes

Every production workload for ovh

Every production workload must be documented in IaC via a auto-recovery-enabled tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the auto-recovery-enabled tag/label.

Remediation: Add a auto-recovery-enabled tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-REL-060 is tracked in IaC.

Compliant

resource "ovh_cloud_project" "example" {
  tags = {
    auto-recovery-enabled = "true"
  }
}

Non-Compliant

resource "ovh_cloud_project" "example" {
  # Missing auto-recovery-enabled tags
}

waf-rel-060.tf.hcloud.auto-recovery-enabled

Engine: terraform | Provider: hcloud | Automated: Yes

Every production workload for hcloud

Every production workload must be documented in IaC via a auto-recovery-enabled tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the auto-recovery-enabled tag/label.

Remediation: Add a auto-recovery-enabled tag/label to the hcloud_server resource to show the hcloud implementation of WAF-REL-060 is tracked in IaC.

Compliant

resource "hcloud_server" "example" {
  labels = {
    auto-recovery-enabled = "true"
  }
}

Non-Compliant

resource "hcloud_server" "example" {
  # Missing auto-recovery-enabled labels
}

Evidence

Kind Required Description

Kind	Required	Description
`iac`	Required	Terraform configuration implementing the control.
`governance`	Required	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Required

Terraform configuration implementing the control.

governance

Required

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

Related Controls

Framework	Controls
ISO 27001:2022	* A.16.1 – Management of information security incidents and improvements
ITIL 4	* Incident Management * Problem Management
BSI C5:2020	* IS-09 – Incident Management
GDPR	* Article 33 – Notification of a personal data breach to the supervisory authority

Framework

Controls

ISO 27001:2022

* A.16.1 – Management of information security incidents and improvements

ITIL 4

* Incident Management * Problem Management

BSI C5:2020

* IS-09 – Incident Management

GDPR

* Article 33 – Notification of a personal data breach to the supervisory authority