“It worked in UAT.” Famous last words.
I’ve written a lot of non-functional requirements over the years: availability, performance, load thresholds, email deliverability, concurrency. All the things that sit quietly behind the scenes until something breaks. And I’ll admit, sometimes I knew while writing them that testing would be a challenge.
We talk about non-functionals, but we rarely give them the same airtime as the functional stuff. It’s easy to focus on what the system does. But it’s the hidden requirements, the ones people only notice when they fail, that turn your Monday morning into a mess. Especially when you’ve just spent your entire weekend on a go-live.
When the real world arrives
A couple of public sector go-lives made the news recently for exactly this reason. In one, important notices were created and sent but never reached people, because a major email provider’s spam filters blocked them. In another, a cloud transition left staff in one office unable to work, even though the core systems were up and remote users were fine.
It’s easy to critique from the sidelines, but I’ve been in those project teams. It’s not from lack of effort. Everyone stresses about these eventualities. I remember many a day-one go-live, refreshing the help desk inbox every five seconds, silently praying a P1 doesn’t drop in.
Why test environments let you down
Even if you want to test non-functionals properly, chances are your test environment isn’t close to production, because:
- it costs too much to mirror production
- real-world concurrency is too hard to replicate
- someone forgot to align configuration across environments
Even when you use a copy of production data, you’ve stripped out anything sensitive and changed every email address to a dummy one, because you don’t want to email real clients by accident. That makes sense, but it also means the very thing you’re trying to test, like deliverability through real spam filters, doesn’t match reality.
You can’t performance test a system on a half-sized instance and expect anything useful. You can’t validate email deliverability without sending real emails to real domains. And you can’t simulate user load without also exposing what comes with it: slowdowns, timeouts and session clashes.
Survive contact with the real world
Non-functionals don’t break silently. They break loudly, publicly, and usually in business hours. Good systems don’t just work in the lab. They survive contact with the real world: odd configurations, flaky internet, unexpected user behaviour, spam filters and peak traffic. That’s the test, not the happy path on a dev box.
If you’ve ever gone into a weekend deployment crossing your fingers and hoping production behaves like UAT, you know exactly what I’m talking about.
Need analysts and testers who think beyond the happy path? Talk to Talexa.
