INCIDENT
The Day the Relay Went Down
What happened
Four of our products lean on large language models at runtime — SpeakEasy Speech drafts speeches, ScribeFlow polishes transcripts, and two more tools call the same backend. To keep keys out of clients and billing in one place, those calls go through a single relay endpoint. One day that relay went down. Not slowly, not partially — requests started failing, and within the day every AI-dependent feature across four products was degraded at once.
That is the uncomfortable part to write down: we had turned four independent products into one shared point of failure, and we found out the way users find out.
The cutover
The fix was not clever. We audited which features actually needed the relay, took DeepSeek's OpenAI-compatible API as the replacement provider, and switched the backends over product by product — new base URL, new key handling, same request shape. The compatible interface is what made a same-day cutover possible: no client changes, no request rewrites, just configuration and careful verification that each product's prompts still behaved.
By the end of the day the four products were answering again. The relay never came back; we no longer needed it to.
What we changed after
- No silent single points of failure. A shared dependency is fine; an unexamined shared dependency is not. Every runtime dependency now gets written down with a named fallback.
- Failover is configuration, not surgery. Provider base URLs and keys live in config, so the next cutover is an edit, not a rewrite.
- Check the thing users feel. Uptime checks now hit the AI-dependent paths, not just the homepage — a product that loads but cannot think is still down.
Why we publish this
We say we build in the open, and the open includes the bad days. An incident writeup is the cheapest honest artifact a studio can ship: it costs a page, and it tells you more about how we work than any landing page could. If one of our tools ever fails you, this is how we intend to behave.
Written by the Zalize studio
-- zalize.com