WAF-OPS-060 – Runbook & Operational Documentation Coverage
Description
Every production workload MUST have a runbook covering all known failure modes, routine operational tasks, and escalation paths. Runbooks MUST be version-controlled, regularly reviewed (at minimum quarterly), and accessible to all on-call engineers without authentication barriers at 3 AM. Alert rules that trigger on-call pages MUST link to the relevant runbook section.
Rationale
| Risk | Description |
|---|---|
Risk |
Runbooks not linked to alerts slow response to recurring disturbances. |
Risk |
Missing escalation instructions lead to uncoordinated ad-hoc reactions. |
Requirements
-
Every production workload MUST have a runbook covering all known failure modes, routine operational tasks, and escalation paths.
-
Runbooks MUST be version-controlled, regularly reviewed (at minimum quarterly), and accessible to all on-call engineers without authentication barriers at 3 AM.
-
Alert rules that trigger on-call pages MUST link to the relevant runbook section.
Implementation Guidance
-
Document the requirement in a version-controlled policy or standard.
-
Assign ownership and establish a review cadence (e.g., quarterly).
-
Implement the required configuration as code in Terraform or equivalent IaC.
Maturity Levels
| Level | Name | Criteria |
|---|---|---|
L1 |
Ad hoc – control not consistently applied |
* Manual evidence exists for some workloads, but coverage is unknown. |
L2 |
Standardised – policy defined and communicated |
* A version-controlled policy, ADR, or standard describes the requirement. |
L3 |
Integrated – automated validation in CI/CD |
* Terraform checks run automatically on pull requests and report results. |
L4 |
Auditable – metrics and exceptions tracked |
* Control coverage, failures, and approved exceptions are reported quarterly. |
L5 |
Optimised – continuous improvement based on data |
* Control effectiveness is measured and iteratively improved. |
Checks
waf-ops-060.tf.aws.cloudwatch-alarm-runbook-annotation
Engine: terraform | Provider: aws | Automated: Yes
CloudWatch alarms must include runbook URL in description
All CloudWatch metric alarms that trigger on-call notifications MUST include a runbook URL in the alarm_description to reduce mean time to resolution.
Scope: resource — aws_cloudwatch_metric_alarm
Assertions:
* alarm_description matches pattern https?:// — CloudWatch alarm description must include a runbook URL (http:// or https://).
Remediation: Include a runbook URL in the alarm_description field: alarm_description = "Error rate exceeded. Runbook: https://wiki/runbooks/service-errors"
resource "aws_cloudwatch_metric_alarm" "error_rate" {
alarm_name = "payment-service-error-rate"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 2
metric_name = "5XXError"
namespace = "AWS/ApplicationELB"
period = 60
statistic = "Sum"
threshold = 10
alarm_description = "Payment service error rate high. Runbook: https://wiki/runbooks/payment-5xx"
alarm_actions = [aws_sns_topic.oncall.arn]
}resource "aws_cloudwatch_metric_alarm" "error_rate" {
alarm_name = "payment-service-error-rate"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 2
metric_name = "5XXError"
namespace = "AWS/ApplicationELB"
period = 60
statistic = "Sum"
threshold = 10
alarm_description = "High error rate"
# No runbook URL – WAF-OPS-060 Violation
alarm_actions = [aws_sns_topic.oncall.arn]
}waf-ops-060.tf.aws.prometheus-alert-runbook-label
Engine: terraform | Provider: aws | Automated: Yes
Prometheus alert rules must include runbook_url annotation
Prometheus alert rules managed in AWS Managed Prometheus MUST include a runbook_url annotation so on-call engineers have immediate access to remediation steps.
Scope: resource — aws_prometheus_rule_group_namespace
Assertions:
* data matches pattern runbook_url — Prometheus alert rules must include runbook_url annotation.
Remediation: Add runbook_url: "https://wiki/runbooks/<service>" annotation to all Prometheus alert rules.
resource "aws_prometheus_rule_group_namespace" "alerts" {
name = "payment-service-alerts"
workspace_id = aws_prometheus_workspace.main.id
data = <<-EOT
groups:
- name: payment-service
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
for: 2m
annotations:
summary: "High error rate on payment service"
runbook_url: "https://wiki/runbooks/payment-errors"
EOT
}resource "aws_prometheus_rule_group_namespace" "alerts" {
name = "payment-service-alerts"
workspace_id = aws_prometheus_workspace.main.id
data = <<-EOT
groups:
- name: payment-service
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
# No runbook_url annotation – WAF-OPS-060 Violation
EOT
}waf-ops-060.tf.azurerm.monitor-alert-runbook
Engine: terraform | Provider: azurerm | Automated: Yes
Azure Monitor alerts must include runbook or knowledge base links
Every azurerm_monitor_metric_alert that pages on-call MUST include a runbook or knowledge base URL in its description or action webhook payload so responders can act immediately.
Scope: resource — azurerm_monitor_metric_alert
Assertions:
* description matches pattern https?:// — Azure Monitor alert description must include a runbook URL.
Remediation: Add a runbook URL to the description field of every azurerm_monitor_metric_alert. Use action groups with webhook payloads that include the runbook link and alert context.
resource "azurerm_monitor_metric_alert" "errors" {
name = "payment-service-errors"
resource_group_name = azurerm_resource_group.main.name
scopes = [azurerm_application_insights.app.id]
criteria {
metric_namespace = "Microsoft.Insights/components"
metric_name = "requests/failed"
aggregation = "Count"
operator = "GreaterThan"
threshold = 10
}
description = "High failed request rate. Runbook: https://wiki/runbooks/payment-errors"
action {
action_group_id = azurerm_monitor_action_group.oncall.id
}
}resource "azurerm_monitor_metric_alert" "errors" {
name = "payment-service-errors"
resource_group_name = azurerm_resource_group.main.name
scopes = [azurerm_application_insights.app.id]
description = "High failed request rate"
# No runbook link provided
}waf-ops-060.tf.google.monitoring-alert-runbook
Engine: terraform | Provider: google | Automated: Yes
GCP Monitoring alert policies must include runbook or documentation links
Every google_monitoring_alert_policy that triggers on-call notifications MUST include a runbook or knowledge base URL in its documentation block so responders have immediate remediation guidance.
Scope: resource — google_monitoring_alert_policy
Assertions:
* documentation attribute exists — GCP alert policy must include a documentation block with runbook link.
Remediation: Set the documentation field of every google_monitoring_alert_policy to include alert context and a runbook URL. Keep documentation under version control alongside the Terraform configuration.
resource "google_monitoring_alert_policy" "errors" {
display_name = "Payment Service High Error Rate"
documentation {
content = "High 5xx error rate on payment service. Runbook: https://wiki/runbooks/payment-errors"
mime_type = "text/markdown"
}
conditions {
display_name = "5xx rate above threshold"
condition_threshold {
filter = "metric.type=\"logging.googleapis.com/user/payment_service_5xx\""
duration = "300s"
comparison = "COMPARISON_GT"
threshold_value = 10
}
}
notification_channels = [google_monitoring_notification_channel.oncall.id]
}resource "google_monitoring_alert_policy" "errors" {
display_name = "Payment Service High Error Rate"
# No documentation block with runbook link
conditions {
display_name = "5xx rate above threshold"
condition_threshold {
filter = "metric.type=\"logging.googleapis.com/user/payment_service_5xx\""
duration = "300s"
comparison = "COMPARISON_GT"
}
}
}waf-ops-060.tf.stackit.runbook-defined
Engine: terraform | Provider: stackit | Automated: Yes
Every production workload for stackit
Every production workload must be documented in IaC via a runbook-defined tag/label on stackit_project. This validates the control is tracked as code.
Scope: resource — stackit_project
Assertions:
* labels key exists — stackit_project is missing the runbook-defined tag/label.
Remediation: Add a runbook-defined tag/label to the stackit_project resource to show the stackit implementation of WAF-OPS-060 is tracked in IaC.
resource "stackit_project" "example" {
labels = {
runbook-defined = "true"
}
}resource "stackit_project" "example" {
# Missing runbook-defined labels
}waf-ops-060.tf.ovh.runbook-defined
Engine: terraform | Provider: ovh | Automated: Yes
Every production workload for ovh
Every production workload must be documented in IaC via a runbook-defined tag/label on ovh_cloud_project. This validates the control is tracked as code.
Scope: resource — ovh_cloud_project
Assertions:
* tags key exists — ovh_cloud_project is missing the runbook-defined tag/label.
Remediation: Add a runbook-defined tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-OPS-060 is tracked in IaC.
resource "ovh_cloud_project" "example" {
tags = {
runbook-defined = "true"
}
}resource "ovh_cloud_project" "example" {
# Missing runbook-defined tags
}waf-ops-060.tf.hcloud.runbook-defined
Engine: terraform | Provider: hcloud | Automated: Yes
Every production workload for hcloud
Every production workload must be documented in IaC via a runbook-defined tag/label on hcloud_server. This validates the control is tracked as code.
Scope: resource — hcloud_server
Assertions:
* labels key exists — hcloud_server is missing the runbook-defined tag/label.
Remediation: Add a runbook-defined tag/label to the hcloud_server resource to show the hcloud implementation of WAF-OPS-060 is tracked in IaC.
resource "hcloud_server" "example" {
labels = {
runbook-defined = "true"
}
}resource "hcloud_server" "example" {
# Missing runbook-defined labels
}Evidence
| Kind | Required | Description |
|---|---|---|
|
Required |
Terraform configuration implementing the control. |
|
Required |
Policy, ADR, or standard documenting the requirement. |
|
Optional |
Cloud console or API output confirming runtime state. |