Why we shouldn’t hide complexity
We work really hard to hide complexity behind beautiful abstractions, but that creates a problem with "invisible complexity." Here's how to save your team a lot of headache.

What’s the secret to keeping your customers and leadership happy? Delivering reliably, at a steady cadence? Always putting what your customer expects in their hands? Making sure they get the most “bang for their buck,” without any nasty surprises? And doing it all at a nice, steady, expected pace that builds trust?
If I had to choose one thing, it would be communication.
Ok, but so what? Why a blog post on the obvious topic of communicating well? How does this matter?
Years ago I worked with a social media company to deliver a new user experience. The idea was to share memes that would be expressed on a disc (call it a coin), with a positive message on one side and a personalized note on the other. You’d get these messages on your phone, could appreciate the face of the coin, and flip it over and see a message, picture, or video from your friend.
The team did all the usual things: design an architecture, itemize the work, express underlying complexity with pretty diagrams, hold demos and retros. And we set goals for a minimum viable product to test with a small audience.
We hit all those targets. Every aspirational goal was met — but the CEO immediately berated the team for “not going fast enough.” He couldn’t understand why the app wasn’t finished. He had different timelines in his head despite reams of documentation, tasks, sprint retros, and plans to the contrary.
Where was the gap in understanding? Where did such a huge misunderstanding come from?
I’ve found engineers have a hard time expressing complex problems in a way non-engineers can absorb. I don’t mean that engineers can’t write. Quite the opposite: they’re very good at distilling incredibly complex problems into simple, elegant, well-expressed images.
But therein lies the problem. Those simple abstractions hide all the complexity.
We do this because it makes thinking about the problem space easy. But it also means all that complexity gets hidden away, and is therefore forgotten.
It turns out that we need both. We need to simplify complex problems so we can reason about them, but also communicate that complexity to stakeholders without hiding too much of it. It’s the key to a smooth project and avoiding hard conversations about failed expectations.
I’ll talk about how we can do that, but first let’s discuss why holding that complexity in our head is so important.
Every project ultimately pins its success on collaboration:
Communicating cost and effort with a reasonable level of reliability aids decision making dramatically (like, prioritizing what’s next).
Creating rational expectations about progress with your customer helps establish trust.
Doing the same with your leadership and business stakeholders is a necessary step toward predicting and protecting budgets.
Planning a confluence of events without any of them going off the rails is essential — things like launch dates, marketing, publication, much needed revenue expectations.
Not disappointing any of your stakeholders with nasty surprises (especially those in charge of budgets).
Avoiding frustration, stemming from any of the above, because of a lack of visibility into where things stand now and what’s left to accomplish.
These problems crop up all the time (not just in technology projects). The root cause is communication, chiefly because what we’re doing is inherently complicated — and successfully expressing that complexity and its implications is hard.
If you’re new, welcome to Customer Obsessed Engineering! Every week I publish a new article, direct to your mailbox if you’re a subscriber. As a free subscriber you can read about half of every article, plus all of my free articles.
Anytime you'd like to read more, you can upgrade to a paid subscription.
Complexity and invisibility combined are awful
A lot of missteps stem from the confluence of complexity and invisibility. Complexity, because every technology project is going to have its intricacies. Invisibility, because so many of those intricate details are hidden from view.
Large projects are particularly vulnerable to the “invisible complexity” problem. As multiple teams start to collaborate, they’ll naturally divide and conquer — different teams taking on different problems. Each team starts simplifying their complex problem space.
Despite best efforts, silos begin to form. One team might focus on account management and IAM while another team dives into loan processing. These two teams start to operate in their private little bubbles.
In between those “bubbles,” we rely on simple models to convey complex ideas. It’s effective, conveying an entire problem space in a few words or pictures — but we also lose fidelity. Outside the bubble, we don’t see the internal intricacy.
Business leaders, stakeholders, your customer are all looking in from the outside. They become yet another silo, often separated from what’s going on — running, without “boots on the ground” intelligence, without being exposed to all that complexity on a day-to-day basis. Not understanding why timelines and budgets are what they are.
Why we hide complexity
Extremely complex problems are hard for us to reason about as a whole. Even the brightest engineer can only keep so much of a design in their head at any one time.
Something we’ve probably all dealt with is just logging in. I’ve talked about this before: all we really want is to safely login to our application. But that means storing some personally identifiable information (“PII”), which we need to handle carefully. We’ll need to encrypt that, at the very least. And if we want GDPR compliance, for the European Union, customers must have the “right to be forgotten.” Not trivial, it seems, but easy enough, right?
We get started and realize we don’t want customers to keep track of yet another password. So that probably means having a single sign on (“SSO”) feature. And while we’re at it, we better support two-factor authentication, probably SMS and authenticator apps, as well as passkeys. We already mentioned encrypting data, so that means talking about our data store — what’s going to be “secure enough,” given our industry space? If it’s banking or healthcare, this is on a whole other level. We’ll probably have to use multiple data stores for a “separation of concerns” (necessary for, say, HIPAA compliance).1
Speaking of compliance, the GDPR mandates that, effectively, we have to be able to “erase” a customer’s data upon their request. But wait a minute, if we’re writing information into our event log that we need for transactional integrity or tax records, how do we do that? (Turns out there is a technique, called cryptographic shredding, but we’ll have to build it into our architecture — we’ll need a safe place to store cryptographic keys, and we’ll need to encrypt all personally related data).23
We’re not done yet. What about choosing the right encryption protocol? Or dealing with token expiration? How about expiry of authentication methods? DNS based threat monitoring to, say, prevent a denial of service attack or detect rapid login hacking attempts? Maybe we should just be looking at a product — but which one? There are so many.
I could go on, and that’s kind of the point. Just being able to login, probably one of the most basic requirements we’ll have, can be complicated.
Simplicity does not precede complexity, but follows it. — Alan Perlis
We deal with complexity by creating abstractions. By choosing to use AWS Cognito as our IAM product, we don’t need to keep all that knowledge about encryption, security protocols, expiry, PII, authentication, scalability strategies, different protocols, threat monitoring, and more in our head. Cognito abstracts away a lot of it, letting us focus on building business logic. At least, for the most part, though we’ll have plenty of decisions to make.
We deal with this growth of complexity by building models. For example, we use event storming to model business processes, and domain modeling to describe our functional silos. We wrap it all up with a pretty target state architecture diagram that shows the whole product in a pretty picture.
In so doing, we’re abstracting away a lot of complexity too. We’re boiling a massive, multi-team effort down to a couple pages of summary information. We put a lot of effort into creating that apparent simplicity so that we can talk about it.
We need a way to remind ourselves that there’s a proverbial massive glacier floating under that ice cube on the surface. Make it visible, and communicate it so that everyone knows there’s more to it than “pick an IAM product.”
Visibility is the solution
Abstraction is good. It gives us the ability to reason about complex problems. But we also need to express that underlying complexity — in a way that doesn’t break our abstractions or manifest a high barrier to understanding.
We know that adding identity and access management to an application isn’t quite as simple as “just pick an IAM tool.” But is it adequate to just write “IAM” on a whiteboard?
We need a bit more. We need to expose enough of the underlying complexity so it isn’t forgotten.
Contexts and decomposition
As we design our functional architecture, and our iterative design and delivery models, we’ll decompose complex problems into constituent pieces. In the Delivery Playbook we do this with domain driven design — chiefly, by breaking contexts into smaller components. Those could be aggregates, entities, or entire sub-contexts.
At some point we’ll arrive at a comfortable description of our target state. One that expresses what we have to build in a reasonable number of steps.
For example, in the case of setting up an IAM solution, we might end up with this:
It’s still pretty abstract, but there’s enough there to tell us it’s not as simple as flipping the switch on AWS Cognito. We have to integrate with it. We’ve got to build a couple of APIs and make some decisions — it’s not a single step.
If you use the referral button below, you’ll earn free premium access to Customer Obsessed Engineering. Just three referrals will earn a free month!
Conveying progress without disappointing
With our overall architecture in hand, we need a way to express progress toward the whole. It’s got to be easy to consume, so probably a single picture — humans are bad at synthesizing multiple, disparate data sources accurately.
It should also be easy to maintain — so we don’t have to spend a lot of time fussing with it.
Keep reading with a 7-day free trial
Subscribe to Customer Obsessed Engineering to keep reading this post and get 7 days of free access to the full post archives.