Openinc-1027·HHHarbourfront Health·started 12m ago
Harbourfront reports endpoint slow — CPU spike on prod-3
Azure CPU alert correlated with Datadog latency degradation on the same Harbourfront VM. Currently investigating.
IMPACT
Harbourfront Health · reports endpoint · 8% of clinic users
CORRELATED ACROSS
AAzure MonitorDdDatadogPPagerDutySSlack
AI-suggested fix · based on 2 similar past incidents across your MSP
Likely root cause
Likely a long-running report query pinning a single VM. Similar to inc-892 (last month) — a report export against a clinic whose patient dataset grew 12x.
Recommended actions
- Identify the clinic currently hitting /api/reports on prod-3 (likely one of the larger sites).
- Trigger VMSS scale-out manually — auto-scale threshold not yet met.
- Add per-clinic concurrency limit on the report export path.
- Consider offloading long-running exports to a background worker.
ProOne-click actions
Timeline
- Alert: VM CPU > 92% for 5m · harbourfront-api-prod-3CPU sustained above 92% on VMSS instance in Harbourfront's Azure subscription. Scale-out threshold reached but not yet firing.
- Monitor: /api/reports latency > 2s (5m)p95 latency on Harbourfront's reporting endpoint climbed 380ms → 2.3s. Concentrated on the CPU-pinned instance.
- PagerDuty page — platform on-callCorrelator merged Azure + Datadog signals; paged Northern Edge platform-primary. Ack in 3m.
- Slack channel created — #harbourfront-inc-1027Rich incident card posted; on-call added, Harbourfront IT lead cc'd.