Incident March 30th, 2026 – Accidental CDN Caching
by cebert on 3/31/2026, 1:28:18 AM
https://blog.railway.com/p/incident-report-march-30-2026-accidental-cdn-caching
Comments
by: varun_chopra
The status page [1] has the actual root cause (enabling "Surrogate Keys" silently bypassed their CDN-off logic). The blog post doesn't. That's backwards.<p>"0.05% of domains" is a vanity metric -- what matters is how many requests were mis-served cross-user. "Cache-Control was respected where provided" is technically true but misleading when most apps don't set it because CDN was off. The status page is more honest here too: they confirmed content without cache-control was cached.<p>They call it a "trust boundary violation" in the last line but the rest of the post reads like a press release. No accounting of what data was actually exposed.<p>[1] <a href="https://status.railway.com/incident/X0Q39H56" rel="nofollow">https://status.railway.com/incident/X0Q39H56</a>
3/31/2026, 3:05:39 AM
by: stingraycharles
This write up doesn’t make sense. Authenticated users are the ones without a Set-Cookie? Surely the ones with the cookie set are the authenticated ones?<p>There are dozens of contradictions, like first they say:<p>“this may have resulted in potentially authenticated data being served to unauthenticated users”<p>and then just a few sentences later say<p>“potentially unauthenticated data is served to authenticated users”<p>which is the opposite. Which one is it?<p>Am I missing something, or is this article poorly reviewed?
3/31/2026, 2:31:37 AM
by: rileymichael
pretty hard to find this on their blog, looks like incidents are tucked away at the bottom. an issue of this size deserve a higher spot.<p>(also looks like two versions of the 'postmortem' are published at <a href="https://blog.railway.com/engineering" rel="nofollow">https://blog.railway.com/engineering</a>)
3/31/2026, 4:13:12 AM
by: muragekibicho
Does Stripe use Railway? The dashboard was down today and this is the only incident report I've encountered and the timeline matches Stripe's downtime.
3/31/2026, 4:06:31 AM
by: sublinear
I'm curious if having unique URLs per user session would mitigate this.<p>I think that's already best practice in most API designs anyway?
3/31/2026, 2:33:34 AM
by: wokgr3t4
[dead]
3/31/2026, 2:46:26 AM
by: sebmellen
Almost three years ago now, Railway poached one of our smartest engineers. They were smart to do so. I have a lot of respect for the Railway team and I’m impressed with their execution.<p>I think this is their first major security incident. Good that they are transparent about it.<p>If possible (@justjake) it would be helpful to understand if there was a QA/test process before the release was pushed. I presume there was, so the question is why this was not caught. Was this just an untested part of the codebase?
3/31/2026, 2:57:03 AM