Skip to content

Failure policy executor

The failure policy executor runs fallback policies when something goes wrong, for example when a policy fails or a block keeps failing health checks. A failure policy might post to Slack or Microsoft Teams, call a webhook, write a log entry, or trigger a retry. Failure events are logged.

At a glance

Runs on: the management cluster, as failure-notifier (internal: http://failure-notifier.utils.svc.cluster.local:5000). You can also deploy your own.
Typical caller: a health-check policy or any other policy that detects a failure.
Requires: the POLICY_EXECUTOR_API_URL environment variable to be set on the server.

How it works

flowchart LR
    P["A policy detects a failure<br/><small>e.g. a block health checker</small>"] -->|"POST /executeFailurePolicy"| F["Failure policy executor"]
    F --> FP{{"Failure policy<br/>(your Python)"}}
    FP --> A["Slack / Teams / webhook /<br/>log / retry / ..."]

Write the failure policy like any other policy, register it, and call it by ID with the inputs it needs.


API

POST /executeFailurePolicy

Runs the failure policy failure_policy_id with the given inputs and parameters.

{
  "failure_policy_id": "string",    // Required: the failure policy to run
  "inputs": { ... },                // Required: input data for the policy
  "parameters": { ... }             // Required: parameters for the policy
}
Status Meaning
200 OK Success. The result is in data.
400 Bad Request A required field is missing.
500 Internal Server Error Server error or misconfiguration.

Example: notify Slack that a policy timed out:

curl -X POST http://<failure-policy-server>/executeFailurePolicy \
  -H "Content-Type: application/json" \
  -d '{
        "failure_policy_id": "notify-slack",
        "inputs": {
          "error_message": "Policy X failed due to timeout"
        },
        "parameters": {
          "webhook_url": "https://hooks.slack.com/services/XXX/YYY/ZZZ"
        }
      }'

Calling it from a policy

Add this helper to any policy that needs to report failures:

import requests
import logging

def call_failure_policy(api_url: str, failure_policy_id: str, inputs: dict, parameters: dict):
    """
    Calls /executeFailurePolicy to run a failure policy.

    Args:
        api_url (str): Base URL of the failure policy server (e.g., http://<server-url>:5000).
        failure_policy_id (str): ID of the failure policy to run.
        inputs (dict): Input payload for the policy.
        parameters (dict): Parameters for the policy.

    Returns:
        dict: The server's response.
    """
    try:
        url = f"{api_url}/executeFailurePolicy"
        payload = {
            "failure_policy_id": failure_policy_id,
            "inputs": inputs,
            "parameters": parameters
        }
        headers = {"Content-Type": "application/json"}

        response = requests.post(url, json=payload, headers=headers)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as e:
        logging.error(f"Request failed: {e}")
        return {"success": False, "message": str(e)}
    except Exception as ex:
        logging.error(f"Unexpected error: {ex}")
        return {"success": False, "message": str(ex)}

For a complete example, a health-check policy that escalates after repeated failures, see Block runtime: health checker.