PagerDuty Alerts

PagerDuty Alerts

Overview

Alerts are the raw signals from monitoring tools that flow into PagerDuty. When an alert is received, PagerDuty creates or updates an incident based on the service's alert grouping configuration. Multiple alerts can be grouped into a single incident to reduce noise. Alerts can also be suppressed via event rules to prevent unnecessary notifications.

Key Concepts

Alert vs. Incident

Concept	Description
Alert	A single signal from a monitoring tool (e.g., one Datadog alert)
Incident	A PagerDuty object that groups one or more related alerts; this is what responders interact with

An incident may contain many alerts. Resolving an incident resolves all its alerts. Resolving individual alerts does not resolve the incident unless all alerts are resolved.

Alert Statuses

Status	Description
triggered	Alert is active and contributing to an incident
resolved	Alert has been resolved (manually or via resolve event)

Alert Grouping

Alerts on a service can be automatically grouped into a single incident:

Mode	Description
intelligent	ML-based grouping of related alerts
time	Alerts within a configurable time window are grouped
content_based	Alerts with matching field values are grouped

Event Rules & Suppression

Event rules process incoming events before they create alerts:

• Routing rules -- Route events to specific services
• Suppression rules -- Suppress known-noisy events to prevent alert creation
• Severity mapping -- Map event severity to PagerDuty urgency

Deduplication

The dedup_key in the Events API controls deduplication:

• Events with the same dedup_key and routing_key are deduplicated
• A trigger event with an existing dedup_key adds an alert to the existing incident
• An acknowledge or resolve event with a dedup_key updates the existing alert

API Patterns

List Incident Alerts

pagerduty_list_incident_alerts

Parameters:

• incident_id -- The incident ID
• statuses[] -- Filter by status (triggered, resolved)
• sort_by -- Sort field (created_at)
• include[] -- Include related resources
• limit / offset -- Pagination

Example response:

{
  "alerts": [
    {
      "id": "PALERT01",
      "type": "alert",
      "status": "triggered",
      "created_at": "2026-03-27T08:15:00Z",
      "severity": "critical",
      "summary": "CPU usage at 98% on web-server-01",
      "body": {
        "type": "alert_body",
        "details": {
          "metric": "cpu.usage",
          "value": 98,
          "threshold": 90,
          "host": "web-server-01"
        }
      },
      "incident": {
        "id": "P1234ABC",
        "type": "incident_reference"
      },
      "service": {
        "id": "PSVC123",
        "summary": "Web Application"
      }
    }
  ],
  "limit": 25,
  "offset": 0,
  "total": 1,
  "more": false
}

Get Alert Details

pagerduty_get_alert

Parameters:

• incident_id -- The incident ID
• alert_id -- The alert ID

Update Alert

pagerduty_update_alert

Parameters:

• incident_id -- The incident ID
• alert_id -- The alert ID
• status -- New status (resolved)

Events API v2 (Sending Events)

The Events API v2 is used by monitoring tools to send events to PagerDuty. Events are sent to https://events.pagerduty.com/v2/enqueue.

Trigger Event:

{
  "routing_key": "INTEGRATION_KEY",
  "event_action": "trigger",
  "dedup_key": "unique-alert-key",
  "payload": {
    "summary": "High CPU on web-server-01",
    "severity": "critical",
    "source": "monitoring-tool",
    "component": "web-server-01",
    "group": "production",
    "class": "cpu",
    "custom_details": {
      "cpu_percent": 98,
      "threshold": 90
    }
  }
}

Acknowledge Event:

{
  "routing_key": "INTEGRATION_KEY",
  "event_action": "acknowledge",
  "dedup_key": "unique-alert-key"
}

Resolve Event:

{
  "routing_key": "INTEGRATION_KEY",
  "event_action": "resolve",
  "dedup_key": "unique-alert-key"
}

Event Severity Mapping

Events API Severity	PagerDuty Urgency
critical	High
error	High
warning	Low
info	Low (may be suppressed)

Common Workflows

Review Alerts for an Incident

1. Get incident details with pagerduty_get_incident
2. List alerts with pagerduty_list_incident_alerts
3. Review each alert's summary, severity, and details
4. Identify the root cause from alert details
5. Resolve individual alerts that are no longer relevant

Reduce Alert Noise

1. Review incident frequency by service with analytics
2. Identify services with high alert-to-incident ratios
3. Enable intelligent alert grouping on noisy services
4. Create suppression rules for known false positives
5. Tune monitoring thresholds at the source

Investigate Alert Details

1. List alerts for the incident
2. Get detailed alert body for each alert
3. Review custom_details for monitoring data
4. Cross-reference with monitoring tool dashboards
5. Add investigation notes to the incident

Error Handling

Alert Not Found

Cause: Invalid alert ID or alert belongs to a different incident Solution: List alerts for the incident to find the correct ID

Cannot Resolve Acknowledged Alert

Cause: Alert is in an unexpected state Solution: Check current alert status before updating

Events API Rate Limit

Cause: Exceeded 120 events per minute per integration key Solution: Batch events or distribute across multiple integration keys

Best Practices

• Use meaningful dedup_key values for proper deduplication
• Include rich custom_details in events for faster investigation
• Configure intelligent alert grouping to reduce incident noise
• Create suppression rules for planned maintenance or known issues
• Set appropriate severity levels at the event source
• Monitor alert-to-incident ratios to identify noisy services
• Use the Events API v2 (not v1) for all new integrations
• Include source, component, and group fields for better context

Related Skills

• api-patterns - Pagination and error handling
• incidents - Incidents that group alerts
• services - Services where alerts are routed
• analytics - Alert and incident volume metrics