Adding Slack escalation to an AI MSP voice triage system

NightDesk handles after-hours calls for MSPs. When a P1 comes in — multiple users can't log in, server down, suspected security incident — the AI triage layer creates a ConnectWise ticket and pages the on-call tech.

The original version had three escalation channels: SMS (Twilio), email (SES), and Microsoft Teams. Every MSP conversation I've had this week has ended with "we use Slack." So Slack is now a first-class channel.

The implementation

The paging module dispatches on method from the runbook's escalation_rules:

if method == "slack":
    return _page_slack(req)

The actual send is 20 lines of stdlib HTTP:

def _page_slack(req: PageRequest) -> dict:
    webhook_url = ""
    if req.target and req.target.startswith("https://hooks.slack.com/"):
        webhook_url = req.target
    else:
        webhook_url = os.environ.get("NIGHTDESK_SLACK_WEBHOOK", "")
    if not webhook_url:
        raise RuntimeError("NIGHTDESK_SLACK_WEBHOOK not set and target is not a Slack URL")

    blocks = [
        {"type": "header", "text": {"type": "plain_text", "text": f":red_circle: {req.subject}", "emoji": True}},
        {"type": "section", "text": {"type": "mrkdwn", "text": req.body}},
        {"type": "context", "elements": [{"type": "mrkdwn", "text": "_NightDesk after-hours AI triage_"}]},
    ]
    payload = {"blocks": blocks, "text": req.subject}
    data = json.dumps(payload).encode()
    http_req = urllib.request.Request(url=webhook_url, data=data,
        headers={"Content-Type": "application/json"}, method="POST")
    with urllib.request.urlopen(http_req, timeout=10) as resp:
        return {"sent": True, "status": resp.status}

Two things worth noting:

Target resolution. The page field in the runbook can be either a Slack webhook URL (starts with https://hooks.slack.com/) or an empty string. If it's a URL, use it directly. Otherwise fall back to NIGHTDESK_SLACK_WEBHOOK in the environment. This means you can have a per-customer Slack webhook (useful if you want escalations to go to a customer-specific channel) or a single MSP-wide #alerts channel.

Block Kit message. Using blocks + a fallback text field. The text field shows in notifications and accessibility contexts where blocks don't render. The block structure is header (red circle + subject) + section (body) + context line (_NightDesk after-hours AI triage_). Minimal but readable in Slack's notification preview.

Runbook configuration

escalation_rules:
  - trigger: "site-wide outage / multiple users affected"
    page: ""              # uses NIGHTDESK_SLACK_WEBHOOK env var
    method: slack

  - trigger: "P1 — direct Slack per customer"
    page: "https://hooks.slack.com/services/T/B/your-per-customer-webhook"
    method: slack

  - trigger: "security incident"
    page: oncall@msp.com  # fallback to email for security escalations
    method: email

The page field value for Slack is the webhook URL, not an email address or phone number. The validator in triage_agent.py accepts any non-empty string for page — the channel module knows how to interpret it.

Tests

The failure modes worth testing for a webhook channel:

1. No webhook URL configured → error returned, not exception raised (paging failures shouldn't break the call)
2. Target is a Slack URL → uses it directly
3. Target is not a Slack URL → falls back to env var
4. Payload contains the subject line
5. Payload is valid Block Kit structure (list of blocks)

All five pass. The broader paging test suite covers Teams and email as well — 141 tests total in the msp-voice-triage test suite after this addition.

What the MSP sees in Slack

🔴 P1 ESCALATION — Acme Manufacturing
Server has been down for 20 minutes, multiple users can't log in.
Ticket #45234 created. CW → Service Desk → P1.

NightDesk after-hours AI triage

The ticket number is in the body text (req.body) which the triage handler builds from the CW ticket creation response before dispatching the page.

The red circle is an emoji in the header — :red_circle: with emoji: true. Minor flourish but it makes P1 pop in a busy #alerts channel at 2am.