Understanding VictorOps Architecture

Incident Ingestion and Routing

VictorOps receives alerts via REST API or integrations (e.g., Datadog, Prometheus, Nagios) using routing keys to direct incidents to teams. Routing failures often stem from missing keys or inactive rules.

On-Call Scheduling and Escalation Policies

Users are assigned to schedules that rotate responsibility and define escalation paths. Notification delivery depends on accurate user contact methods and escalation logic configured in the timeline.

Common VictorOps Issues

1. Alerts Not Triggering Incidents

Usually caused by invalid or missing routing keys, misformatted payloads, or filters suppressing the alert in the alert rule configuration.

2. On-Call Users Not Receiving Notifications

Triggered when user contact methods (SMS, email, mobile push) are disabled or notification rules do not match alert severity and type.

3. Escalation Policy Not Advancing

Occurs when responders do not acknowledge alerts, but escalation timers or fallback teams are not configured correctly.

4. API Events Not Appearing in Timeline

Often due to malformed JSON payloads, unverified API tokens, or network connectivity issues from external monitoring tools.

5. Schedule Sync Fails With External Systems

Triggered by calendar integration failures (e.g., Google Calendar, Outlook), timezone misalignment, or unauthorized access to linked calendars.

Diagnostics and Debugging Techniques

Validate Routing Key Configuration

In the VictorOps web UI, navigate to Settings → Routing Keys. Ensure incoming alerts reference a valid and active key mapped to a team.

Test User Notification Paths

Go to Users → Contact Methods and click "Test Notification" to verify each channel. Confirm push notifications are enabled in the mobile app.

Review Incident Timeline Logs

Use the Incident Timeline to trace alert flow, acknowledgments, reroutes, and integrations. It provides visibility into alert processing and user response.

Debug API Payloads

Use curl or Postman to send sample alerts. Ensure payloads include required fields such as message_type, entity_id, and routing_key:

curl -X POST https://alert.victorops.com/integrations/generic/20131114/alert/YOUR_API_KEY/YOUR_ROUTING_KEY \
  -H "Content-Type: application/json" \
  -d '{"message_type": "CRITICAL", "entity_id": "server-123", "state_message": "Disk space low"}'

Inspect Calendar Integration Logs

Check Settings → On-Call Schedule for sync status and errors. Confirm that the linked calendar account has granted full access permissions.

Step-by-Step Resolution Guide

1. Fix Missing Incident Creation

Ensure alerts include a valid routing key and required payload fields. Avoid filters in alert rules that may suppress incoming alerts.

2. Restore Notification Delivery

Update or re-enable user contact methods. Adjust notification rules to trigger on all desired alert severities and types.

3. Repair Escalation Path Issues

Verify that escalation steps are configured with fallback users and timers. Test automatic escalation by simulating alert acknowledgement delays.

4. Debug API Event Delivery

Log responses from the API endpoint. Use VictorOps integration logs to confirm receipt and processing of each request.

5. Sync Schedule Calendars

Re-authorize calendar accounts. Manually refresh sync and check timezone consistency across VictorOps and external calendar platforms.

Best Practices for Reliable VictorOps Workflows

  • Standardize routing keys across monitoring tools for traceability.
  • Test all user contact methods monthly to ensure availability.
  • Design escalation paths with redundancy and clear fallback steps.
  • Use dynamic alert rules to route incidents based on metadata like severity or team.
  • Regularly review and update on-call schedules to prevent gaps in coverage.

Conclusion

VictorOps empowers DevOps teams with real-time incident response capabilities, but maintaining consistent performance requires robust configuration of routing, escalation, notifications, and integrations. By methodically troubleshooting alert ingestion, verifying user paths, and managing schedules, teams can ensure rapid and reliable alert delivery and resolution in high-stakes production environments.

FAQs

1. Why isn’t my alert creating an incident in VictorOps?

The routing key may be invalid, inactive, or missing from the alert payload. Confirm that the key maps to a team with active rules.

2. How do I ensure on-call users get notified?

Verify that all users have at least one active contact method and that their notification rules match alert types and severity levels.

3. Why isn’t my escalation policy progressing?

Ensure escalation steps include timers and designated fallback responders. Simulate failures to test policy flow.

4. What causes API alerts to silently fail?

Malformed JSON, invalid API keys, or missing fields will cause silent rejections. Monitor integration logs for feedback.

5. My Google Calendar isn’t syncing with the schedule—why?

Permissions may not allow access, or timezones may conflict. Reauthorize the calendar integration and verify sharing settings.