WAF-REL-090 – Chaos Engineering & Fault Injection

Pillar: Reliability | Severity: Medium | Category: Logging Monitoring | Automatable: High

Description

Production and staging workloads MUST undergo structured chaos engineering experiments at minimum quarterly. Experiments MUST be based on hypothesis-driven testing against defined failure modes. Results MUST be documented and deviations from expected behavior MUST trigger reliability improvements. Chaos experiments MUST be run in staging before production. Production chaos MUST have defined blast radius limits, rollback procedures, and active monitoring during execution. Ad-hoc or unsanctioned "accidental chaos" is not a substitute.

Rationale

Risk	Description
Risk	Untested workloads surprise teams during real outages because known weaknesses were never exercised.
Risk	Missing chaos experiments prevent targeted discovery of resilience gaps.

Risk

Description

Risk

Untested workloads surprise teams during real outages because known weaknesses were never exercised.

Risk

Missing chaos experiments prevent targeted discovery of resilience gaps.

Requirements

Production and staging workloads MUST undergo structured chaos engineering experiments at minimum quarterly.
Experiments MUST be based on hypothesis-driven testing against defined failure modes.
Results MUST be documented and deviations from expected behavior MUST trigger reliability improvements.
Chaos experiments MUST be run in staging before production.
Production chaos MUST have defined blast radius limits, rollback procedures, and active monitoring during execution.

Implementation Guidance

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).

Maturity Levels

Level	Name	Criteria
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Criteria

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-rel-090.tf.aws.fis-experiment-template

Engine: terraform | Provider: aws | Automated: Yes

AWS FIS experiment templates must define stop conditions

AWS Fault Injection Simulator experiment templates must define stop_condition blocks to automatically abort experiments when error budget thresholds are exceeded.

Scope: resource — aws_fis_experiment_template

Assertions: * description not empty — FIS experiment must have a descriptive description with hypothesis. * stop_condition attribute exists — FIS experiment must define at least one stop_condition. * role_arn not empty — FIS experiment must specify an IAM role ARN.

Remediation: Add a stop_condition block to the aws_fis_experiment_template referencing a CloudWatch alarm that triggers when error budget is consumed.

Compliant

resource "aws_fis_experiment_template" "az_failure" {
  description = "Hypothesis: Service recovers from AZ failure within 60s without SLO violation"
  role_arn    = aws_iam_role.fis.arn

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value  = aws_cloudwatch_metric_alarm.slo_burn_rate.arn
  }

  action {
    name      = "terminate-instances-az1"
    action_id = "aws:ec2:terminate-instances"
    target {
      key   = "Instances"
      value = "az1-instances"
    }
  }

  target {
    name           = "az1-instances"
    resource_type  = "aws:ec2:instance"
    selection_mode = "PERCENT(25)"
    resource_tag {
      key   = "availability-zone"
      value = "eu-west-1a"
    }
  }
}

Non-Compliant

resource "aws_fis_experiment_template" "az_failure" {
  description = "AZ failure test"
  role_arn    = aws_iam_role.fis.arn
  # Missing: stop_condition – experiment has no abort mechanism
  # WAF-REL-090 violation
  action {
    name      = "terminate-instances"
    action_id = "aws:ec2:terminate-instances"
    target {
      key   = "Instances"
      value = "all-instances"
    }
  }
  target {
    name           = "all-instances"
    resource_type  = "aws:ec2:instance"
    selection_mode = "ALL"
  }
}

waf-rel-090.tf.azurerm.chaos-studio-experiment

Engine: terraform | Provider: azurerm | Automated: Yes

Azure Chaos Studio experiments must target non-production environments first

Azure Chaos Studio experiments must have a name and location configured and should target resources tagged with the correct environment.

Scope: resource — azurerm_chaos_studio_experiment

Assertions: * name not empty — Chaos Studio experiment must have a descriptive name. * location not empty — Chaos Studio experiment must specify a location. * identity attribute exists — Chaos Studio experiment must define an identity for execution.

Remediation: Configure azurerm_chaos_studio_experiment with name, resource_group_name, location, and identity. Define the experiment steps in the schema_version block.

Compliant

resource "azurerm_chaos_studio_experiment" "pod_failure" {
  name                = "payment-api-pod-failure-staging"
  location            = azurerm_resource_group.staging.location
  resource_group_name = azurerm_resource_group.staging.name

  identity {
    type = "SystemAssigned"
  }

  schema_version = "2.0"
  steps {
    name = "pod-failure-step"
    branches {
      name = "main-branch"
      actions {
        action_urn = "urn:csci:microsoft:azureKubernetesServiceChaosMesh:podChaos/2.2"
        duration   = "PT5M"
      }
    }
  }
}

Non-Compliant

# No chaos experiments configured – WAF-REL-090 violation
# Resilience claims not validated by fault injection

waf-rel-090.tf.google.fault-injection-test

Engine: terraform | Provider: google | Automated: Yes

GCP Cloud Run services should have fault injection testing configured

GCP workloads should use Traffic Director or Cloud Service Mesh fault injection to validate resilience patterns. Alternatively, load testing pipelines should include fault scenarios.

Scope: resource — google_compute_url_map

Assertions: * name not empty — URL map must have a name. * default_route_action attribute exists — URL map should define default_route_action for fault injection testing.

Remediation: Configure fault_injection_policy in the default_route_action of the URL map to inject delays or abort responses for chaos testing.

Compliant

resource "google_compute_url_map" "api" {
  name = "payment-api-url-map"

  default_route_action {
    fault_injection_policy {
      delay {
        fixed_delay {
          seconds = 5
          nanos   = 0
        }
        percentage = 10  # 10% of requests get 5s delay
      }
      abort {
        http_status = 503
        percentage  = 5  # 5% abort for chaos testing
      }
    }
    weighted_backend_services {
      backend_service = google_compute_backend_service.api.id
      weight          = 1000
    }
  }
}

Non-Compliant

resource "google_compute_url_map" "api" {
  name            = "payment-api-url-map"
  default_service = google_compute_backend_service.api.id
  # No fault injection configured – resilience not testable
  # WAF-REL-090 violation
}

waf-rel-090.tf.stackit.rollback-tested

Engine: terraform | Provider: stackit | Automated: Yes

Production and staging workloads for stackit

Production and staging workloads must be documented in IaC via a rollback-tested tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the rollback-tested tag/label.

Remediation: Add a rollback-tested tag/label to the stackit_project resource to show the stackit implementation of WAF-REL-090 is tracked in IaC.

Compliant

resource "stackit_project" "example" {
  labels = {
    rollback-tested = "true"
  }
}

Non-Compliant

resource "stackit_project" "example" {
  # Missing rollback-tested labels
}

waf-rel-090.tf.ovh.rollback-tested

Engine: terraform | Provider: ovh | Automated: Yes

Production and staging workloads for ovh

Production and staging workloads must be documented in IaC via a rollback-tested tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the rollback-tested tag/label.

Remediation: Add a rollback-tested tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-REL-090 is tracked in IaC.

Compliant

resource "ovh_cloud_project" "example" {
  tags = {
    rollback-tested = "true"
  }
}

Non-Compliant

resource "ovh_cloud_project" "example" {
  # Missing rollback-tested tags
}

waf-rel-090.tf.hcloud.rollback-tested

Engine: terraform | Provider: hcloud | Automated: Yes

Production and staging workloads for hcloud

Production and staging workloads must be documented in IaC via a rollback-tested tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the rollback-tested tag/label.

Remediation: Add a rollback-tested tag/label to the hcloud_server resource to show the hcloud implementation of WAF-REL-090 is tracked in IaC.

Compliant

resource "hcloud_server" "example" {
  labels = {
    rollback-tested = "true"
  }
}

Non-Compliant

resource "hcloud_server" "example" {
  # Missing rollback-tested labels
}

Evidence

Kind Required Description

Kind	Required	Description
`iac`	Required	Terraform configuration implementing the control.
`governance`	Required	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Required

Terraform configuration implementing the control.

governance

Required

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

Related Controls

Framework	Controls
ITIL 4	* Problem Management * Service Continuity Management
SRE Book (Google)	* Chapter 33 – Lessons Learned from Other Industries * DiRT (Disaster Recovery Testing)
ISO 27001:2022	* A.17.1.3 – Verify, review and evaluate information security continuity

Framework

Controls

ITIL 4

* Problem Management * Service Continuity Management

SRE Book (Google)

* Chapter 33 – Lessons Learned from Other Industries * DiRT (Disaster Recovery Testing)

ISO 27001:2022

* A.17.1.3 – Verify, review and evaluate information security continuity