Failure policy executor¶
The failure policy executor runs fallback policies when something goes wrong, for example when a policy fails or a block keeps failing health checks. A failure policy might post to Slack or Microsoft Teams, call a webhook, write a log entry, or trigger a retry. Failure events are logged.
At a glance
Runs on: the management cluster, as failure-notifier (internal: http://failure-notifier.utils.svc.cluster.local:5000). You can also deploy your own.
Typical caller: a health-check policy or any other policy that detects a failure.
Requires: the POLICY_EXECUTOR_API_URL environment variable to be set on the server.
How it works¶
flowchart LR
P["A policy detects a failure<br/><small>e.g. a block health checker</small>"] -->|"POST /executeFailurePolicy"| F["Failure policy executor"]
F --> FP{{"Failure policy<br/>(your Python)"}}
FP --> A["Slack / Teams / webhook /<br/>log / retry / ..."]
Write the failure policy like any other policy, register it, and call it by ID with the inputs it needs.
API¶
POST /executeFailurePolicy¶
Runs the failure policy failure_policy_id with the given inputs and parameters.
{
"failure_policy_id": "string", // Required: the failure policy to run
"inputs": { ... }, // Required: input data for the policy
"parameters": { ... } // Required: parameters for the policy
}
| Status | Meaning |
|---|---|
200 OK |
Success. The result is in data. |
400 Bad Request |
A required field is missing. |
500 Internal Server Error |
Server error or misconfiguration. |
Example: notify Slack that a policy timed out:
curl -X POST http://<failure-policy-server>/executeFailurePolicy \
-H "Content-Type: application/json" \
-d '{
"failure_policy_id": "notify-slack",
"inputs": {
"error_message": "Policy X failed due to timeout"
},
"parameters": {
"webhook_url": "https://hooks.slack.com/services/XXX/YYY/ZZZ"
}
}'
Calling it from a policy¶
Add this helper to any policy that needs to report failures:
import requests
import logging
def call_failure_policy(api_url: str, failure_policy_id: str, inputs: dict, parameters: dict):
"""
Calls /executeFailurePolicy to run a failure policy.
Args:
api_url (str): Base URL of the failure policy server (e.g., http://<server-url>:5000).
failure_policy_id (str): ID of the failure policy to run.
inputs (dict): Input payload for the policy.
parameters (dict): Parameters for the policy.
Returns:
dict: The server's response.
"""
try:
url = f"{api_url}/executeFailurePolicy"
payload = {
"failure_policy_id": failure_policy_id,
"inputs": inputs,
"parameters": parameters
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
return response.json()
except requests.RequestException as e:
logging.error(f"Request failed: {e}")
return {"success": False, "message": str(e)}
except Exception as ex:
logging.error(f"Unexpected error: {ex}")
return {"success": False, "message": str(ex)}
For a complete example, a health-check policy that escalates after repeated failures, see Block runtime: health checker.