Copperfield dispatch-app: error spike after deploy v4.19.0

Copperfield · dispatch.copperfield.ca — ~1,100 failed job assignments

The new assignment normalization layer assumed every job had a foremanId, but emergency-callout jobs don't populate foreman until dispatch. On production traffic the code path fired on ~1,100 jobs in 4 minutes before the page.

Rolled back to v4.18.3 via GitHub Actions workflow_dispatch. Latency and error rates returned to baseline within 6 minutes of rollback.

• rachel.wu (Copperfield) · PR with regression test for empty-foreman emergency-callout — due in 2 days
• Northern Edge SRE · Upload source maps in CI so future Sentry issues resolve to line numbers — due this week
• Copperfield eng-mgmt · Pilot progressive rollout for dispatch-app changes — due next sprint

1. GGitHub Audit·18:35·1h ago·workflows.completed
   Deploy to production · dispatch-app
   Commit c9f4a2 by rachel.wu merged to main and deployed via GitHub Actions. 14 files, 322 lines changed.
2. SySentry·18:39·1h ago·issue.first_seen
   New issue: TypeError in dispatch.assignJob
   'Cannot read properties of undefined (reading foremanId)'. First seen on release v4.19.0. 411 events in 4 minutes.
3. DdDatadog·18:40·1h ago·monitor.triggered
   Monitor: p99 latency /api/dispatch > 3s (10m)
   Latency spiked from 420ms to 4.8s after the deploy. Error rate jumped 0.2% → 11.6%.
4. PPagerDuty·18:41·1h ago·incident.trigger
   PagerDuty page — Copperfield on-call
   SLO burn rate: 14x fast. On-call paged; ack in 2m.
5. SSlack·18:42·1h ago·conversations.create
   Slack incident channel created — #copperfield-inc-1025
   Auto-invited on-call + rachel.wu (deploy author). Card links Sentry issue + Datadog dashboard + GitHub diff.
6. GGitHub Audit·18:52·58m ago·workflows.dispatch
   Rollback deploy · dispatch-app → v4.18.3
   workflow_dispatch triggered by rachel.wu. Rollback deploy completed in 3m.
7. DdDatadog·18:58·52m ago·monitor.resolved
   Monitor resolved: p99 latency /api/dispatch
   p99 back to 470ms. Error rate 0.3%. Recovery confirmed.