I was chatting with a friend about team dynamics. We’d been talking about restructuring around product teams — the kind of empowered, durable, cross-functional teams I’ve written about before. He’s nodding along, asking sharp questions. Halfway through, he pushed back.

I’m paraphrasing, but the gist was, “Look, I get the theory. But where’s your proof? We’ve all heard the pitch. Cross-functional teams, durable squads, outcome metrics, two-pizza this, empowered that. Show me the data. Show me that any of this actually beats what we already do.”

It was a fair question. A great question, actually. I’d written the case in Building a product oriented team and the prequel, Moving away from the Monolith — faster, better with a product mindset. But those pieces argued from reasoning and experience. They didn’t bring the math.

So today I want to bring it. Because the math is there — it’s been there for over a decade — and once you line it up, the conclusion is hard to escape: product teams, the real ones, ship faster, build less waste, deliver higher quality and produce dramatically better business outcomes than the alternatives. The evidence isn’t a single bombshell study. It’s a stack of independent research, all converging on the same answer.

Let me lay out the receipts.

One note before we start. None of the studies I cite below were designed to answer the question “do product teams outperform?” They were designed to answer narrower, related questions — about deployment, about feature usage, about team effectiveness, about developer velocity. What’s below is my synthesis: the act of lining up findings from different research traditions and showing where they converge. The convergence is the argument. Each citation is a thread; the rope is the pattern across them.

A quick refresher on “product team”

Before the data, let’s get precise. “Product team” gets thrown around so loosely it’s often meaningless.

Marty Cagan and the Silicon Valley Product Group draw the cleanest distinction. A product team is a small, durable, cross-functional unit — typically a product manager, a designer and a handful of engineers — that owns a problem and is held accountable for outcomes. They are given problems to solve, not features to ship. The product manager owns value and viability. The designer owns usability. The tech lead owns feasibility. Together they own the result.1

A feature team looks similar on paper — same disciplines around the table — but is handed a prioritized list of features to build. Outcomes belong to a stakeholder upstream. The team is judged on output: did you ship what was on the roadmap? Velocity, story points, throughput.

A project team is even further from product orientation. It’s pulled together to execute a discrete project, then dissolved. Specialists rotate in and out. There’s no durable home for product knowledge.

The labels matter less than the operating model. As I wrote in Building a product oriented team, the real distinction is durability, ownership and empowerment — long-lived teams that own the whole product end-to-end and have the authority to figure out the how.

Now — what the research says.

Receipt #1: they ship dramatically faster

Start with DORA — the DevOps Research and Assessment program, founded by Dr. Nicole Forsgren and now part of Google. Every year since 2014, DORA has surveyed practitioners — more than 3,000 in 2024 alone, and tens of thousands across the decade-long program — and statistically grouped them into performance tiers based on four metrics: deployment frequency, lead time for changes, change failure rate and mean time to restore service. The methodology, the math and the conclusions are documented in Accelerate — the book that emerged from the research and won the Shingo Publication Award.2

The 2024 Accelerate State of DevOps report is the latest. Elite-tier teams deploy 182 times more frequently than low performers, with 127× faster lead time for changes. Elite teams hit lead time under a day, deploy on demand and are roughly 19% of the surveyed population. Low performers measure lead time in months.3

That’s not a marginal advantage. It’s not “20% better.” It’s two orders of magnitude.

You might be tempted to dismiss this as a DevOps story rather than a team-structure story. But every time DORA digs into what differentiates elite teams, the answers are the same attributes that define product teams: small, cross-functional, owning the whole pipeline from code to production, empowered to make their own technical decisions. The opposite — handoffs across functional silos with stage-gate approvals from external groups — correlates strongly with low performance. Accelerate presents one specific finding bluntly: external change advisory boards do not improve stability. They make lead time worse without making anything safer.4

McKinsey’s “Beyond Agile” work, which I cited in How waterfall is wrong for software, puts a number on the same effect from a different angle. Across their consulting engagements, McKinsey observed that companies could compress the average time from “code complete” to live production from 89 days to 15 days when they reorganised around DevOps and product-team principles. Same basic architectures, same kind of features. Six times faster, just from changing how the team takes work from “done” to “deployed.”5

The simplest summary: product teams own the value stream end-to-end. Feature and project teams hand off across silos. Handoffs cost time. Time turns into distance from the customer. Distance from the customer means everything you ship is stale.

Receipt #2: they get more done — but redefine what “done” means

“More done” is the wrong metric. The right metric is: did anyone use what we shipped?

In 2002, Jim Johnson — chairman of the Standish Group — presented a finding at the XP conference in Sardinia that’s been quoted ever since: across enterprise applications, only 7% of features are “always” used, 13% are “often” used and 16% are used “occasionally.” That leaves 64% of features as either rarely or never used. Standish later extended the analysis to claim that roughly 80% of features deliver low or no value. On large projects, Standish data shows organizations deliver only 42% of the features they originally scoped. The Standish numbers have drawn methodology critiques over the years — fair enough — but more recent work points the same direction. Pendo’s 2019 Feature Adoption Report, drawing on telemetry from hundreds of B2B SaaS products, found roughly 80% of features are rarely or never used.6

Sit with that for a moment.

Now, “rarely used” isn’t synonymous with “no value.” Some rare features are quietly essential — compliance flags, account recovery, the audit trail your CISO will ask about exactly once. Both Pendo and Standish gloss over that distinction. Even granting it generously, though, the ratios still point at a meaningful misallocation: a lot of effort going into features customers rarely or never engage with. (I’m looking at you, Microsoft Office).

A feature team measured on roadmap output is, by these numbers, building a sizable wedge of low-value work. They look terrific on burndown charts and terrible on customer impact.

I’ve watched this play out enough times to be cynical about it. A feature team gets a quarterly roadmap. They estimate the work in story points, deliver on schedule, demo a sleek end-to-end flow and get applauded. Six months later, telemetry shows the feature gets touched by 3% of users. By then the team has moved on to the next item on the roadmap. Nobody is accountable for the gap between “shipped” and “mattered,” because shipped is the goal.

A product team would have caught that earlier. They wouldn’t have built the full feature on a quarterly roadmap; they’d have built the cheapest possible version that tested the hypothesis — a prototype, an A/B variant, a deliberately ugly first cut — and watched the data. If the data said 3%, they’d have killed the feature in week two and gone after a more promising hypothesis. That’s the discovery discipline Marty Cagan and Teresa Torres write about extensively, and it’s the operational reason product teams build less and deliver more. Not because they’re slower. Because they aim differently. They invest in discovery before commitment. They kill a feature that doesn’t earn its keep. They redefine “done” from “shipped” to “moved the metric.” And the metric they move is customer behavior, not story points. (I’ve made the same argument from a measurement angle in Stop measuring effort.)

There’s a parallel finding in McKinsey’s What makes product teams effective? — a five-year study of more than 1,700 teams across 75 organizations. McKinsey identified two structural conditions that consistently predict throughput and velocity improvements: dedicated team membership (no context switching across projects) and team persistence of three to six months minimum. Teams meeting both conditions develop reliable estimates and sustained throughput. Teams that don’t, can’t.7

That finding isn’t surprising if you think about how knowledge accumulates. Product orientation depends on knowledge durability — context that only forms when the same people work on the same problem long enough to internalise it (more on this below). Project teams dissolve before that happens. Feature teams reorganise around the next quarterly priority. Product teams stay put. That’s why their throughput compounds.

Read more

A junior developer told me they’re using Claude as a mentor. They liked the patience that never wore thin, the availability that never closed, the way they’re never made to feel doltish.

Here’s what worried me: they said mentor. Not tutor. Mentor.

I don’t think they’re wrong about what they’re getting. But mentorship isn’t about answers. It’s about being seen — watched while you struggle, nudged when you’re avoiding hard conversations, sent on projects that stretch you in ways you didn’t ask for. Mentors have the nerve to tell you the truth in private because they’re invested in who you’re becoming.

AI can’t do that. Because it isn’t invested in you. Because it’s not “in the room.”

What AI is genuinely good at

Let’s start with what’s actually working, because this isn’t a piece about being suspicious of the tools. The tutoring story is real. It’s the part of the AI hype that’s actually earned.

Benjamin Bloom’s famous 1984 paper described what he called the “2 sigma problem:” students who received one-on-one tutoring performed two standard deviations better than students in conventional classrooms — that’s the difference between an average student and the top 2 percent. The catch was scale. Personal tutors are expensive, and most students will never have one. For forty years that gap has been the unsolved promise of education technology.1

A well-configured AI is one way we’re closing it. When Khan Academy rolled out Khanmigo, built on GPT-4, the pitch was exactly this — a patient, infinitely available one-on-one tutor for every learner. In specific narrow tasks, the early returns are hard to dismiss. Harvard published a physics study in 2024 that found students using a properly designed AI tutor learned over twice as much in less time than their peers. They reported feeling more engaged and motivated.23

I see the same thing in software. With the right model, a serious system prompt and a few well-built skills, AI can:

• Walk a junior through a debugger session, naming what each frame means and why it matters.

• Explain why a piece of code is the way it is — not just what it does — and adjust the explanation when the question lands a level too high.

• Run unfamiliar code, watch it fail and reason about the failure with the learner in real time.

• Drill someone on a foreign language, a SQL idiom or a regex until it’s automatic.

• Take the same question for the third time without sighing.

That last one matters more than people admit. A great deal of human learning gets blocked by the social cost of asking again. Removing that cost is genuinely useful. It’s what the junior in my opening responded to. They felt safe.

How tutoring goes wrong

It’s useful, but not magical. The same AI cheerfully invents APIs that don’t exist, pattern-match on the wrong context and confidently produce code that looks right but isn’t. Apple researchers showed last year that current LLMs aren’t doing genuine reasoning at all — performance on math problems dropped by as much as 65 percent when the team added a single irrelevant clause to the prompt. Andrew Zuo’s piece on the “-10X developer” makes the harder point: when an LLM writes code at or above your level and introduces a subtle bug, you may be the worst possible person to find it.45

A newer concern is harder to wave off. An MIT Media Lab study published in mid-2025 used EEG to compare brain activity across people writing essays with no help, with a search engine and with an LLM. The LLM group showed the weakest neural connectivity and the worst ability to recall their own writing minutes after producing it. The researchers called it “cognitive debt.” That tracks. Outsource the struggle and the learning never lands.6

So a serviceable tutor, yes. A replacement for thinking, no. None of that, though, is the central point. The central point is that tutoring and mentoring are not the same job, and the public conversation keeps blurring them.

What a mentor actually does

If you ask me what my mentors did for me over the last forty years, almost none of it was answering technical questions. The technical content I could get from books, from peers, eventually from Stack Overflow and now from Claude (as long as I’m careful and look for fantasy hallucinations). What my mentors did was something else entirely.

Read more

In 3.6 Specification we worked through the first two stages of Behavior Driven Development — Discovery and Formulation. We surfaced scenarios in three-amigos conversations and shaped them into Gherkin “Given-When-Then” statements. That gave us the outline of every feature in our product increment.

Elaboration is where we move from outline to depth.

A very important takeaway from chapter 3.6 bears repeating here, because the entire purpose of this activity hangs on it:

The specifications you write now are living documents. They will evolve, change and be added to. That’s why it’s so important to create a live link to the implementation — the code that will validate the specifications. As the specifications change that live link will preserve our value chain from customer desires to delivered code.

Hold on to that idea. Elaboration isn’t about freezing requirements before the team writes a single line of code. It’s about building the connections — between the spec and the marble, between the marble and the test, between the test and the production code — that let the spec keep evolving without the value chain ever going slack.

Introduction

This activity has three jobs.

First, we expand each scenario across the levels at which the system is tested: component and assembly. A login feature has UX scenarios about what the customer sees and experiences. It also has assembly scenarios about which events fire, which commands route through which aggregates and what payload comes back. Both layers need specifications. Both layers need to be testable. Skipping a layer means leaving a gap through which defects love to slip.

Second, we create a live link between every specification and the engineering model that implements it. Each Gherkin feature points back to a marble. Each marble points forward to a Gherkin feature. That bi-directional link is what makes our specifications living documents — when the spec changes, the marble updates with it, and vice versa.1

Read more

Picture this: a program manager wants to confirm a design decision before a release. She asks, innocently enough, “Can someone point me at the current version of the architecture document?”

A pause. Then: “I think it’s in the Teams channel — the one from the February workshop.” Someone else chimes in, “No, there’s a newer one. Sriram emailed it last week.” A third voice: “That’s the draft. The signed-off version is in the SharePoint site. But the link in the meeting invite is broken because they restructured the folders.” A fourth: “I have a copy in OneDrive. It might be the latest, I’m not sure.”

Five people. Five different versions. Zero confidence that any of them is authoritative. And this is a team that cares about documentation.

The common denominator is always the same: a collaboration stack built on SharePoint, Teams and Microsoft’s various Office products. Sometimes with Azure DevOps (ADO) bolted on to handle tickets and code.

Here’s an argument that will sound harsh to anyone who’s invested in Microsoft 365 as their delivery platform: the Microsoft collaboration stack is diabolically deficient for serious product delivery. It wasn’t built for the problem delivery teams actually face. It has, piece by piece, gone after what enterprise IT wanted to solve in 2007: replacing the file share.

That’s not the same problem.

What a delivery team actually needs

There are five table stakes a team needs. I describe the first three in Getting started: Using the Delivery Playbook. I’ll restate those briefly here, along with the other two:

1. Live collaboration. Multiple people must be able to work on the same artifact at the same time, editing simultaneously, seeing each other’s cursors, throwing virtual sticky notes on a board in real time. Design work is a collaborative act. The moment one person has to collect and collate feedback, fidelity drops off a cliff.

2. Version control. Every change, by every author, is always captured and reversible. Not just “saved revisions” — the full, granular history, searchable, with diffs, always available. This is table stakes. Git has had it for two decades. Any tool that tracks less than Git doesn’t take the goal seriously.

3. Immutable linking. A document’s URL must never change. Not when the file is renamed. Not when it’s moved to another folder. Not when the folder is reorganized. Not when the author leaves the company. A reference made today should still resolve in five years. Without this, every link in every other document becomes a ticking time bomb.

4. Workflow integration. The documentation platform has to talk, fluently and deeply, to the work-tracking platform. Specs have to link to tickets. Tickets have to link to pages. Status changes in one system should not only be visible in the other, but trigger cross-tool workflow changes and render live embeds, not stale URLs. When that integration is tight, your specs, documentation and tickets stop drifting apart — they become a single living view of the work, not several systems the team has to reconcile.

5. Universal, web-first access. One consistent experience for everyone, on any device. Not a degraded web version and a feature-rich desktop app — a single canonical surface, served through the browser, that everybody gets equally no matter what hardware they’re on.

These five criteria are the measuring stick. Let’s hold SharePoint — and Teams and ADO — up against them.

Links that rot

This is the most damning, and the one I want to spend the most time on, because it’s the one almost nobody takes seriously until they’re knee-deep in the damage.

In SharePoint, a document’s URL is a function of its name and its location in the folder hierarchy. Rename the file, and the URL changes. Move it to another folder, and the URL changes. Move the whole folder, and the URLs of every file inside it change. You can turn on “managed metadata” and “content types” and a bunch of other gears that promise to fix this, but in practice — across every tenant I’ve ever touched — they don’t. The URLs change. References break.

Microsoft has shipped “durable links” and “document IDs” — features that are off by default and rarely enabled. What a team actually encounters is: you renamed the file, the link in the meeting invite from three weeks ago is now dead. The problem is pervasive enough that a cottage industry of third-party tools has emerged just to detect broken SharePoint links.1

Compare that to Atlassian Confluence. Every page has a numeric ID assigned at creation. The ID is embedded in the URL (/pages/12345) and it does not change. You can rename the page, move it between spaces, reorganize the whole tree — the ID-based link still works. You can also use the human-readable URL (which does change when pages get renamed), but the ID-based URL is your forever-address. The system gives you both, and the tight URL is the one you automatically get when you cite the page in another document.2

This seems like a small, technical detail. It is not. It is the single most consequential design choice a documentation platform makes, because it determines whether cross-references decay or persist.

Here’s what happens in practice when links rot: people stop trusting links.

Think about that for a second. The entire promise of the web is that a URL is a durable reference — it says “here is the canonical thing, go fetch it when you need it.” That promise is what lets us build systems on top of systems. Every meeting invite, every wiki page, every code comment, every email that cites a document relies on that promise holding.

When SharePoint breaks the promise — which it does, routinely — teams develop a defensive response. They stop linking. Instead, they attach. They paste a copy of the document into the meeting notes. They email a copy to the distribution list. They download the file and re-upload it to their own folder “in case the original moves.” Multiply this by every team, every document, every month, and within a year you have a document landscape that looks like a hoarder’s basement — dozens of near-duplicates, none of them canonical, each one owned by a different person who genuinely believes theirs is the latest.

It’s rational response. Given that SharePoint links are not reliable, hoarding copies is the locally optimal strategy. However, the result is catastrophic: information silos full of stale documents and an immense toll in lost time and miscommunication.

The identity model that a platform chooses shapes the behavior of the people using it. SharePoint chose the wrong model — one that breeds inefficiency and defensive silo-building.

The version control mirage

SharePoint technically has version control. Open a file, click the ellipsis, choose “Version history,” and you’ll see a list of saves. This sounds like what you want. It is not.

The version history in SharePoint is coarse, inconsistent across product surfaces and frankly untrustworthy. Here are the problems I’ve personally run into, repeatedly:

• Coauthored Office documents roll up edits from multiple people into “a version” at intervals that are not deterministic. If Alice and Bob are editing at the same time, you may end up with one version of their joint work and inaccurate attribution. Contrast with Git: every commit is atomic, attributed and reversible.

• The version history for a file in OneDrive, Teams and elsewhere varies. There is no single source of truth. If you open a file with Teams, you see one history. In SharePoint, you may see a different history — a consequence of Office 365 evolving from a constellation of separately-designed products Microsoft is still trying to stitch together.3

• Restoring a prior version might overwrite the current one, rather than branching. If you discover at 4pm that something important got lost at 10am, “restore” could mean losing everything done between 10am and 4pm. Until recently, this was the default behavior — so be careful.

Confluence, again, does this properly. Every save is a version. Every version compares cleanly against any other. You see what changed, who changed it and when. You can restore a prior version as a new version, preserving full history. This is how version control is supposed to work, and it has been the standard behavior in software since RCS shipped in 1982.

Why does this matter for delivery? Because design artifacts evolve, and the history of how they evolved is often as valuable as the current state. When the security auditor asks, “When did we add this API endpoint to the threat model?” — the answer lives in the diff. When a customer says, “The contract we signed described a different data retention policy,” — the answer lives in the history. When a post-mortem asks, “When did we know about this failure mode?” — the answer lives in the version log.

A platform that can’t answer these questions reliably is a platform that is actively hostile to engineering rigor.

Teams, Teams Sites and the organizing layer

If the first two failures are about individual artifacts, this one is about the landscape those artifacts live in.

Microsoft Teams is, at heart, a chat application. When it was first released in 2017, that’s primarily what it did. Over the years, Microsoft has tried to turn Teams into a “hub for teamwork.” Marketing-speak for bolting everything else Microsoft makes onto the chat app. The Files tab in a Teams channel is a SharePoint document library wearing a trenchcoat. A “Teams Site” is a SharePoint site with a slightly different navigation chrome. Meeting recordings live in OneDrive, not in the channel. Meeting transcripts live in the meeting artifact, which sometimes appears in the channel and sometimes doesn’t. Planner tasks live in a separate product surface that is also “integrated.”

If someone in your team wants to find something, they have to know, in advance, through which product surface it was shared. Was the spec posted in a channel? Check the chat history. Attached to a meeting? Check the meeting recap. Uploaded to the Files tab? That’s SharePoint under the hood, but the folder structure might only be reachable through the Teams UI. Created as a Teams Site? Different URL. Shared in a meeting chat? That goes to OneDrive, in a folder you might not have permission to browse.

There is no single “where did we put the thing” surface. There are five or six of them, depending on how the team has been using the tools, and the mapping between “what kind of content” and “which surface” is convention rather than enforced policy.

In contrast, Confluence is the document surface. Jira is the work surface. Each is a single, distinct product doing one thing well, and the integration points between them are narrow and explicit and they work. No guessing where a document lives — it’s in Confluence. Tickets live in Jira. No impedance mismatch, no cognitive dissonance.

Teams wants to be the union of every Office 365 capability. But integration isn’t about putting everything in the same UI. It’s about making the right connections between distinct tools, so that each one stays simple and focused, and the connections carry exactly the semantics you need. Teams never understood that distinction.

Desktop is king, the browser is second-class

SharePoint and the rest of Office 365 are web-accessible, but they aren’t web-first. Their heritage is Microsoft Office — a suite designed for the desktop and perfunctorily ported to the browser. The difference is not academic.

Open a Word doc in the browser and compare it to the desktop app. The feature set diverges. Advanced formatting, compare-versions, certain macro and formula capabilities, add-ins — these are missing or degraded on the web.4

The consequences cascade. “The copy you want is on my laptop” is a phrase I hear almost every week. Co-authoring gets flakey or breaks entirely because of diverging copies. Changes sit in someone’s sync queue for hours. A document that looks current in SharePoint may be stale. Information silos reappear — not because anyone intended it, but because the tool keeps pulling work off the web and onto individual machines.

Contrast with Confluence, Jira and Miro. Every one of them is web-first by design. There’s no desktop app with a superset of features. The browser is the product. Everyone gets the same experience — Windows, Mac, Linux, iPad, phone, Chromebook — because there’s only one experience to get. One canonical home, universally accessible, identical for everyone.

This is what the web was built for. Atlassian and Google stacks lean into that. Microsoft fights it.

The ADO integration gap

This one is specific to teams that pair SharePoint with Azure DevOps for their ticket tracking and code hosting. It’s a common combo, because “we already have Microsoft” is an easy procurement story, and ADO does a lot of things adequately.

But the integration between SharePoint and ADO is thin. Thin to the point of “here is a hyperlink from the work item to the SharePoint page, good luck.” There is no live embed. There is no bidirectional status update, rich preview or shared workflow trigger. If I link to a SharePoint page from an ADO work item, you see a URL. If that URL breaks, the work item becomes an orphan.

In contrast, a Jira issue in a Confluence page renders a live, interactive issue card with issue key, title, status, assignee and priority — and that card updates when the issue does. If I embed a Jira query in a Confluence page, the page shows a table of live results — as of right now. A Confluence page can be the dashboard for a delivery effort: status pulled from Jira, specs authored inline, risks tracked in a live table, all rendering from the underlying systems of record.

You can build most of this in the Microsoft stack with enough Power Automate flows, Graph API queries, custom connectors and a patient engineer. I’ve seen it attempted. I’ve yet to see it done well enough to matter. The effort required to get SharePoint + ADO to mimic what Confluence + Jira does out of the box is roughly the effort of a mid-size integration project — a project that has to be maintained, re-tested with every platform update, and reverse-engineered when the person who built it leaves the company. That’s not an integration. That’s a liability.

Workflow is an afterthought

The last failure is workflow management — specifically the modeling and enforcement of state transitions for work items.

Jira gets this right. The workflow editor is a first-class product, with a first-class visual flow designer, transition conditions, validators, post-functions, custom statuses and permission checks. A non-engineer on the delivery team can design a workflow that enforces “a ticket cannot transition from ‘In Review’ to ‘Done’ unless the code review is merged with clean quality gates, the responsible stakeholder has reviewed the feature and all acceptance criteria are passing green.” Express it in the GUI, save it, extend with code as needed and the tool enforces it across every ticket that follows that workflow. The team controls its own process — and the workflow itself is a first-class artifact: auditable, exportable, evolvable.

ADO has “process customization.” It is possible to model custom workflows. In practice the feature is buried behind an admin gate, its conditions are a shallow subset of Jira’s, and most teams I’ve worked with never meaningfully touch it. They either live with the default — generic, not their actual process — or they reinvent the enforcement in spreadsheets, PowerBI dashboards, or human oversight routines that have to be re-explained to every new hire.

The reason this matters: workflow is the mechanism by which a team’s process becomes automatic. If your workflow tool doesn’t let you encode “we agreed code reviews are mandatory before a feature branch is merged,” then code reviews are not mandatory — they’re optional, governed only by vigilance, and skipped the moment someone is under pressure. Jira’s workflow editor is a force multiplier for team discipline, because it turns “we agreed to do this” into “the tool won’t let you skip it.” ADO gives you a significantly weaker version of that lever, and most teams don’t reach for it at all.

The cost is reliability. Teams using Jira-style workflow enforcement ship with fewer process escapes — missing reviews, missed gates, skipped steps. Teams using ADO’s default setup rely on human memory and goodwill — both fail at scale. The difference shows up in incident rates, audit findings and the quality of the artifacts the team produces.

What all this costs the team

If you’re keeping a tally, the SharePoint/Teams/ADO stack fails on every criterion. Links don’t persist. Version control is a mirage. The organizing layer is fragmented. The browser is a second-class citizen. Integrations are absent or thin. Workflow is an afterthought.

What does this actually cost a delivery team?

It costs time. A lot of it. The 2024 Stack Overflow Developer Survey (65,000 respondents) found that 61% of developers spend more than 30 minutes every day just searching for answers — and 30% say knowledge silos hit their productivity ten or more times per week.5

It costs trust. When the documentation landscape is chaotic, people stop trusting it as a source of truth. They start relying on oral tradition — “ask Alice, she knows” — and on personal copies. Key decisions stop being recorded in a place anyone can find. Onboarding becomes a six-month apprenticeship rather than a process a new hire can self-serve. Organizational memory degrades.

It costs quality. DORA's research (2022 State of DevOps Report) found that documentation quality is one of the most powerful amplifiers of engineering performance: teams with above-average documentation quality see 12.8x the organizational performance lift from the same technical practices compared to teams with poor documentation.6

It costs morale. Engineers are, as a group, unusually sensitive to tool quality. I’ve seen engineers turn down offers, quit, or burn out over the cumulative drag of bad tooling. A team that feels its own stack is working against them never reaches its potential, however talented the individuals. Tool quality is retention strategy.

Stuck with the stack? Fight back.

If you’re committed to SharePoint, Teams and ADO for the next few years, none of the above changes that. But a handful of disciplines can reduce the damage. None will turn the Microsoft stack into Confluence — they just stop the bleeding.

Here’s six specific strategies you can follow:

Read more

I was recently asked to critique a new decision tree that helps teams decide between waterfall and agile when starting a new project. The intended outcome is to improve long-term results in the organization by giving teams the ability to make critical decisions quickly and accurately.

Overall, “good shift-left stuff” in my book. Anyhow, here’s the diagram I was asked to comment on:

Read more

I once joined a team that had been struggling with delivery for the better part of two years. Demos kept failing. Deployments kept crashing production. Rollbacks were so routine that the ops team had gotten eerily efficient at them. (I’ll refer back to the team later — let’s call the company “Acme”).

I remember one in particular. The team was demoing a reactive search feature — the kind of thing where you start typing a product name and suggestions appear in real time. Same idea as typing “penguins” into Google and getting “Penguins of Madagascar” as a suggestion.

The demo cratered. Someone typed a product name, and nothing happened. Twenty minutes of poking around later, the demo was cancelled. What struck me wasn’t the failure itself — it was the reaction. Nobody was surprised. Nobody expected it to work. “Maybe next time.”¹

Later that afternoon, a planned deployment seemed to have crashed a key service. All hands on deck for a rollback. The team executed quickly. Rollback done. They’d had a lot of practice.

Here’s what immediately jumped out at me: in both cases the team had zero operational intelligence. No telemetry. No metrics. No centralized logs. No traceability. No way to quickly zero in on a root cause. They were flying blind — and they’d been flying blind so long they’d forgotten what visibility even looked like.

And that’s why the reaction was the same each time. Abort. Rollback. “Afterward, we’ll figure out what’s wrong.”

The irony is that both failures had quick fixes. Each time, queries weren’t reaching a service because the wrong port was configured. Instead of changing the port and continuing, the team burned hours — even days — backtracking, diagnosing and trying again.

That experience crystallized something I’d been feeling for decades: observability isn’t a tool you add. It’s an architectural decision you make — or don’t. And the cost of not making it compounds in ways that are genuinely difficult to measure, precisely because you don’t have the instrumentation to measure them.

The “flying blind penalty”

In the old days — before cloud, before microservices, before systems became truly distributed — opacity was tolerable. You had a monolith on a few servers. If something broke, you SSH’d in, looked at logs, poked around. It might take hours, but there wasn’t that much territory to cover.

Modern systems don’t afford that luxury. A single user request could traverse dozens of services, touching databases, message queues, caches and external APIs along the way. Just today I participated in a sprint review discussing a database architecture that was 29 views deep. A latency spike could originate in any one of those — or in the invisible interplay between the services that call them.

Without observability, you’re operating on instinct and prayer.

Here’s what that looks like in practice. A bug ships. It doesn’t manifest right away — it surfaces under load, or in a specific user segment, or under certain timing conditions. By the time anyone notices, users are already suffering. Now the team has to diagnose the problem. If they’re lucky, someone complains and provides enough detail to be helpful. They narrow down which service is involved, reproduce the issue, trace the flow of data through the system, find the root cause, fix it, deploy and verify.

The whole time, sweating over how many more customers are running into the problem.

Every single step without observability compounds in difficulty. You’re not just debugging — you’re doing archaeology at midnight, piecing together behavior from scattered artifacts across dozens of services.

With observability designed in from the start, you proactively see which service is degraded. You can drill into specific request traces. You can correlate logs across service boundaries. Mean time to detect and repair drops from hours to minutes — even becoming predictive.2

DORA.dev research bears this out. Elite-performing teams — the ones deploying multiple times per day with sub-hour recovery times — aren’t elite because they have better engineers. They’re elite because visibility closes a critical gap: it’s a prerequisite for every other capability DORA measures. They can detect and diagnose problems before those problems cascade. Their systems are instrumented to answer questions the team hasn’t even thought to ask yet.3

What matters is how you think about the problem

Monitoring and observability are not the same thing.

Monitoring answers questions you anticipated. You watch CPU utilization, disk space, error rates. You set thresholds and alert when they’re exceeded. “Is the database up? Is the error rate spiking?” These are known questions about known conditions.

Observability answers questions you haven’t thought of yet. It’s the ability to interrogate your system’s behavior without adding new instrumentation every time you have a new question. “Why is this specific user’s checkout so slow today when it was fine yesterday? What changed in the request path between those two deployments?”

Charity Majors — co-founder of Honeycomb and one of the sharpest voices on this topic — frames it as the difference between known-unknowns and unknown-unknowns. Monitoring handles the known-unknowns: the things you anticipated might go wrong. Observability handles the unknown-unknowns: the things you couldn’t have predicted. It empowers you to follow the breadcrumbs wherever they lead.4

Monitoring is reactive. Observability is generative — it creates the conditions for discovery.

Daniela Miao, CTO of Momento, put it well in a Dev Interrupted episode: “By giving more visibility into what’s happening with your software, your application, you actually do enable development to be faster as well. It’s not just insurance for when something goes wrong.”5

That’s critical. Most teams think of observability as insurance — you invest in it to cushion the blow when things break. But the real dividend is velocity. Teams with strong observability ship faster because they can see the impact of their changes in real time. They don’t have to guess whether a deployment worked or a tweak to a query ran faster. They know.

Why we don’t instrument on day one

Early stage. You’re shipping fast. Three or four services, maybe. Errors go to a text file. You’re small enough that you can hold the whole system in your head. Observability feels like overkill. “It’s in the logs, just read ‘em.”

Growth. Suddenly there are ten services, and logs are scattered across different servers. You can’t correlate them anymore. Someone says “we should get a logging tool.” You pick one — Datadog, Splunk, whatever — and bolt it on. The instrumentation is inconsistent. Some services are covered, others aren’t. Metrics are sparse and don’t tell a coherent story. You’re reading fragmented logs from disparate systems and trying to correlate events in your head. It’s starting to fall apart.

Pain. Now you have fifty services. Traffic has increased tenfold. Something breaks, and you’re still flying blind — not because you lack tools, but because the observability layer wasn’t designed. It was stitched on, piecemeal, after the fact. Different teams instrumented different things using different conventions. Tracing is incomplete. Metrics don’t correlate with logs. You’re paying more and more for tools that don’t give you the insight you need.

Here’s what’s easy to miss: that team invested in observability. They bought the tools. They checked the box. But because every investment was reactive — a logging platform when logs got painful, a metrics dashboard when someone asked for one, a tracing system when debugging got unbearable — they ended up with three separate systems that don’t talk to each other. Logs in one silo, metrics in another, traces in a third. Every decision about what to capture was made at write time, under pressure, for a specific fire they were fighting that week.

Charity Majors, co-founder of Honeycomb, calls this Observability 1.0: many pillars, many sources of truth, many tools. It’s what you get by default when you bolt observability on after the fact. The fragmentation isn’t a failure of effort — it’s a consequence of timing. You can’t design a unified telemetry model when you’re patching gaps reactively.6

And it matters — because a fragmented model can answer the questions you anticipated (is the database up? is the error rate spiking?) but collapses when you need to ask something new. “Why are checkouts slow today when they were fine yesterday? What changed between those two deployments?” Those questions require slicing across dimensions you didn’t pre-define. They require the kind of open-ended interrogation that siloed tools simply can’t support.

Insight requires design. That’s the part teams miss.

Why does this happen? Because observability isn’t a feature. It’s not something you built first, it’s what got built last. It doesn’t generate revenue — at least not visibly. It feels expensive up front. And teams, under pressure to deliver, don’t demand the time they need for planning a solid foundation.

I’ve written about the same false economy with security and testing. The things that feel expensive to build at the start turn out to be astronomically more expensive to retrofit.7

Why it has to be architectural

You can’t add observability with a library import. You can add logging and metrics — that’s the easy part. But genuine observability, the kind that lets you answer arbitrary questions about system behavior, requires architectural forethought.

Consider what’s involved:

1. How do requests get identified as they flow across boundaries? Trace IDs need to propagate through HTTP headers, message queue metadata, database session context — everywhere a request touches. If you don’t design this in, you can’t correlate activity across services.

2. What events merit recording? Not every line of code should produce a log. That’s noise. But the meaningful transitions — state changes, boundary crossings between services, external calls and their response characteristics — these are the signals that matter.

3. What metrics actually reflect system health? Not just CPU and memory, but application-level indicators. How long do payment transactions take? What’s the cache hit ratio? What’s the queue depth trend? These are the metrics that tell you whether your system is behaving — or trending toward overload.

4. How does context propagate? When a request carries metadata — a customer ID, a feature flag, a request source — how does that context flow through the system in a way that key transitions are captured seamlessly?

This is the shape of how you structure services, how you handle requests, how you organize code. And capturing it after the fact is brutally hard.

For example: if you decide later that you want end-to-end tracing, you need to add trace ID generation at your entry points, thread those IDs through every service-to-service call, correlate logs using those IDs, and store traces in a specialized backend. That means touching every API, every RPC, every async boundary. If you’d built it in from the start, it’s a few lines of middleware.

Consider structured logging: if you’ve been writing freeform string logs for two years, migrating to structured JSON events means auditing every logging statement, retrofitting context and rewriting every query your team depends on. If you start with structured logs, it’s just how things work from day one.

It’s not expensive when you build it in. It’s a handful of decisions, many of which improve things by introducing consistency. OpenTelemetry standardizes the plumbing — trace propagation, metric collection, structured logging — across languages and frameworks. You focus on what to instrument, not how to propagate context.8

But retrofitting: that’s an ordeal. Orders of magnitude more expensive — in engineering time, in tool sprawl and in the debugging sessions that never needed to happen.

What to actually build

Here’s how to do it right — whether you’re making the investment in an old project or starting fresh. The key is to think in events, not pillars.

Start with events, not tools

The most consequential decision you’ll make is how you model your telemetry. Do you emit three separate streams — logs here, metrics there, traces somewhere else — and try to correlate them after the fact? Or do you start with a single primitive: one structured event per unit of work?

The event-first approach works like this: when a request enters your system, you begin composing a structured event — a wide, context-rich record that accumulates everything meaningful about that request as it flows through your services. By the time the request completes, the event carries the trace ID, service name, duration, user ID, feature flags, error state, dependency latencies, cache hits, queue depths — whatever context matters.

You store that event once. From that single store, you can derive everything else: aggregate events over time and you get metrics; filter and read individual events and you get logs; stitch events together by trace ID and you get traces. One source of truth, many views.

This is what Charity Majors calls Observability 2.0 — and it’s the architecture that actually delivers on the promise of answering questions you haven’t thought of yet. Because the events are wide (carrying dozens or hundreds of fields), you can slice by any dimension after the fact. You didn’t have to anticipate the question at write time. You just ask it: “show me all events where cache_hit=false and region=eu-west and latency_p99 > 2s.” If those fields are in the event, you get your answer.

Compare that to the three-pillars approach, where you'd need to have predefined that specific metric, logged that specific combination, and hoped your tracing tool captured the right spans. Three tools, three queries, three partial answers — if you’re lucky.

The event model is the architectural decision. Everything else follows from it.

Establish a trace ID convention

Every request gets a unique identifier the moment it enters your system. That ID propagates through every downstream call — HTTP headers, message metadata, database session context. Every event includes it.

This is foundational. Without it, you cannot correlate activity across service boundaries. With it, a single ID connects every event across every service for a given request. It’s what lets you stitch individual events into a complete trace — the full journey of a request through your system.

OpenTelemetry handles those mechanics gracefully. Use it. It’s language-agnostic, framework-aware and it solves the propagation problem so you don’t have to.

Enrich your events at every boundary

Not every line of code should produce telemetry. That’s noise. The moments that matter are the boundaries and transitions — the points where a request crosses from one domain to another or where state changes meaningfully:

1. Request entry — what arrived, who initiated it, what context it carried.

2. External calls — what you called, what came back, how long it took.

3. State changes — meaningful transitions: a payment processed, a queue depth shifted, a circuit breaker tripped.

4. Errors — always, with full context. Stack traces, request metadata, the works.

At each of these boundaries, you’re not firing off a disconnected log line — you’re enriching the event that represents this unit of work. Add fields: payment_gateway_latency_ms=302, cache_hit=false, feature_flag_new_checkout=true. The event gets wider and richer as the request progresses. When the request completes, it captures the full story.

Make every field structured and discrete. Not a freeform string like “User 12345 logged in from 192.168.1.1” — but named fields: user_id=12345, source_ip=192.168.1.1, session_id=abc, duration_ms=47. Structured fields are queryable. “Show me all login failures for user 12345 in the last hour” becomes a single query instead of a regex excavation.

Derive your metrics from the same data

Once you have rich, structured events flowing through a single store, metrics become a view — not a separate system. You aggregate events to spot trends:

Read more

You’re working on an automotive assembly line. Your job is to install the steering column assembly and steering wheel. The next car rolls down the line. You grab the parts and lean in, tools in hand and try to fit the column. It won’t fit. Something’s off.

You look closely and realize the mounting point isn’t where it should be — it’s offset by a few inches. You figure, you’ll try anyway. No good. Worse, the chassis is aluminum and your steel column will cause galvanic corrosion on contact. You think to yourself — maybe this car is supposed to have something else entirely — a joystick, perhaps? You’re going to need time to research the platform and figure out what belongs here. But you’re out of time. The line grinds to a halt. Delivery stops. Something unforeseen has surfaced, and you need time to research the right solution.

That’s not how automotive assembly lines work. But it is how software works. Building software is about research, discovery and development.

Read more

Introduction

This activity is about making sure we’ve taken the time to think about happy paths as well as failure scenarios. It’s about adding fidelity to our use cases so that our finished product is a quality product — one that our customers will enjoy using.

As part of that process, we’ll explore variations on the scenarios in our design. For example, we’ve defined a scenario to make a purchase — but what about when the credit authorization fails? Or the product isn’t available in the warehouse? Or our customer adds a duplicate product to the shopping cart? What should we do if the customer abandons their shopping cart?

Working with your team to think through these scenarios — poking each scenario, prodding to see if it stands up under pressure, adding new paths if it doesn’t — will help ensure our design is solid.

Building better specifications

This activity isn’t just about testing our assumptions — it’s about creating better designs. We want to be sure we’ve thought about how code will be used before we build it. This outside-in pressure tends to produce interfaces that are cleaner, more focused and easier to consume, as well as code that is more reliable.

Read more

The incident

On June 4, 1996, the maiden flight of the European Space Agency's Ariane 5 rocket ended in catastrophic failure. About 40 seconds into its flight sequence, at an altitude of about 3,700 meters, the launcher veered off its flight path, broke up and exploded.1

The cost for this failure was estimated to be $370 million. The rocket was carrying four satellites intended to study the earth’s magnetosphere. Because of the payload’s destruction, building new satellites delayed scientific research into the magnetosphere for four more years.23

An elaborate and careful investigation into the cause of the failure was conducted.

While several factors led to a cascade of events, it came down to a data conversion error — an error that would have been caught in testing.

Unfortunately, the internal SRI software exception that caused the problem was never tested. Or, more specifically, it wasn’t tested in the Ariane 5 configuration. The failure resulted because Ariane 4-era software that used 16-bit signed integers had been plugged into a more modern system — a system that used 64-bit floating point data. The older Ariane 4 SRI software couldn’t cope with the new data format.

That’s the technical reason.

There’s a more important lesson here. The real reason for the failure was human error. Somewhere along the line, a decision was made: the decision that the Ariane 4 SRI module didn’t need to be retested in the new Ariane 5 configuration.

The investigatory board pointed to an underlying theme in Ariane 5 development as a root cause, that theme being, “that software should be considered correct until it is shown to be at fault.” The board took another view. Their belief was, “that software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct.”

Systems must be tested to prove correct behavior. We cannot simply assume they are correct.

Preventable failure

The key takeaways from the Ariane 5 disaster are:

1. We cannot rely on software to be reliable unless currently accepted best practices demonstrate that it is correct.

2. Continuous integration and testing for correctness is necessary — even when correctness has previously been proven.

The Ariane 5 case is widely taught in software engineering courses as the canonical example of how a simple, preventable defect — discovered late in the lifecycle (during operation) — can have catastrophic and costly consequences.

In fact, we know that the longer we wait to discover a defect, the more costly it is to fix. Research dating back to the 1970’s says we can fix a defect in design for nearly zero additional cost — or we can wait until production and spend north of 100 times the cost to fix it later (Boehm, 1979).

This exponential curve isn’t arbitrary. It reflects the reality that as software progresses through development stages, each defect becomes increasingly entangled with other code. Downstream systems begin to depend on it.

A requirements misunderstanding caught in design costs a conversation to fix. The same misunderstanding caught after integration means code rework, integration testing, release cycles, product updates and coordination across downstream dependencies. It can cascade into updating multiple products.

In concrete terms, that $25 defect can be fixed early, “for free,” or we can wait until later and spend $2,500 or more.

And there’s evidence that a 100X cost factor is by no means the upper limit. More recent research, in some cases, point to a factor of 1,500X cost to correct defects in operations. That $25 design defect just turned into a $37,500 cost to correct. There’s no upper limit — as the Ariane 5 team learned. (If you’re interested in reviewing more recent data there are references to NIST, NASA and Westland research in the footnotes).456

But there’s good news. We know how to prevent failures: By applying software quality assurance policies and practices — and pairing them with rigorous software testing.

Applying best practices

Test Driven Development (TDD) and Behavior Driven Development (BDD) are fundamental strategies to move defect detection left on the graph — ideally all the way to the design phase where cost is lowest.

Both practices create proofs that prevent defects. What’s more, they rely on executable specifications that prevent regression. Even after initial development, the test suite continues to catch defects early. A change that inadvertently breaks existing behavior is caught in seconds during the next test run — not discovered in production by users.

This ongoing protection is particularly valuable as systems age and original developers move on — the tests preserve institutional knowledge about intended behavior. Imagine how powerful it would have been if the Ariane 4 SRI module had included a continuously run regression test sequence.

The fundamental promise isn’t about testing — it’s about creating better designs. Both practices demand writing tests first. That forces thinking about how code will be used before thinking about how it will be built. This outside-in pressure tends to produce interfaces that are cleaner, more focused, and easier to consume, as well as code that is more reliable.

The deeper point

TDD and BDD aren’t really about testing at all. They’re about feedback loops and thinking clearly about intent before implementation. The tests are a byproduct — valuable, but secondary to the discipline of thoroughly articulating what you want before you build it.

Critics often attack the testing artifacts while missing this point entirely. It’s not always about going faster or whether every scenario needs to be test-driven. It’s whether your development process includes mechanisms that force clarity of thought and protect against regression. Both TDD and BDD are proven ways to achieve that.

TDD or BDD?

TDD catches implementation errors immediately. When you write a failing test before writing code, you discover mismatches between intent and implementation within minutes, not days or weeks.

The feedback loop is so tight that “debugging” often becomes unnecessary. When a test fails you know exactly what change caused it.

Compare this to traditional development where bugs lurk undetected until integration, verification or even user experience — by which point the original context has faded. Investigation becomes archaeological, time consuming and costly.

What TDD does well is clarify the boundaries of your system. Code that’s hard to test is often code with unclear responsibilities or excessive coupling. Even if you don’t achieve full coverage, attempting TDD reveals where your architecture needs attention.

But TDD has a limitation: it doesn’t “shift left” enough on our error discovery graph.

TDD is largely a technical exercise, focused on functional specification. As such it can get deep into the weeds of a feature — but may fail to rise to the level of product behavior. In other words, we might be applying our test-driven thinking to a conceptual design that is already defective.

That’s why I prefer BDD.

Behavior Driven Development

Behavior Driven Development is a collaborative approach that evolved from Test Driven Development. It bridges the gap between technical and business team members by expressing specifications as concrete examples, written in plain language everyone can understand (the “Given-When-Then” style, or Gherkin format).

Read more

A daily standup is a great idea — provided it’s done right.

A well-run standup:

1. Is a place to foster collaboration and mindshare.

2. Provides a forum to get help.

3. Makes sure the team doesn’t spiral.

Unfortunately, most standups that I join don’t do any of these things. Quite the opposite. These standups impede productive work, acting as a needless interruption that offers almost no value.

More often than not what I see is a team forced to stand together while each team member robotically itemizes their work since the last sprint. In some miserable cases, the “PM” and one or two team members hold the entire team hostage while a specific problem is diagnosed.

The standup stretches to 30 or 40 minutes. Your eight person team takes a cumulative five hour hit to their productivity.

Create value in your standups

If you Google “questions to ask at a standup” you’ll find the worst culprit sitting right at the top of the list: “what did you do since yesterday?”

Ban this question.

Almost just as bad is, “wha…

Read more

In the previous article I introduced a foundational architecture for our data pipeline, checking off several of our requirements. Here’s a quick recap:

1. We leveraged an internal cache for isolation to break our dependency on external data sources, improving our resiliency and scalability options.

2. We introduced the idea of an asynchronous message architecture, so we aren’t bound by lengthy, sequential processes.

3. We keep our annotations long-term with internal persistence so we don’t reprocess data when unnecessarily.

4. Source data as treated as an immutable source, thus protecting our data integrity.

5. Our visualization layer is responsive, running off our internal cache so that it’s independent from other components.

Here’s a sketch of the architecture as it stands now:

It’s a good start, but we aren’t finished.

There are still two requirements that don’t have a clear solution:

1. We want to be able to reprocess data, effectively “rewinding history” and applying new analytics. For instance, if we add a new analysis and insight to our pipeline, how are we going to go back and view that new analysis on old data?

2. Finally, we want to be able to model our insights — that is, visualize it in a presentation layer. But we haven’t talked about how that’s going to happen.

Once we tackle those last few requirements, we’ll fill in some details — such as, deciding on a cache technology to use. We’ll also take an end-to-end look at what we’ve designed to see if we missed something important.

Finishing our foundation

One of the most powerful and underused capabilities in data analytics is the ability to reprocess historical data with new logic. When you fix a bug in your analysis or refine a calculation, you often want to know: “what would the last three months have looked like with this new logic?”

Fortunately, our design thus far aligns pretty well with this goal.

Essentially, all we are saying we need to do is throw out information that is no longer accurate and then reprocess the source data and rebuild a new and improved picture of the world. We might do this for a couple of reasons:

1. We changed our analytics. Consequently, we need to throw out all of our old findings and recalculate our insights.

2. The source data changed. Perhaps there was a historical data update, invalidating our findings. Again, we want to throw out our findings and reprocess.

Looking at the problem from that perspective, the architectural answer is simple: we need a way to delete our annotations.

Once we wipe our annotations, we’ll need to make sure they are recalculated. There are different strategies we could use, but for the most part our options are:

1. Rebuild our annotations on-demand. This is fine if we’re OK with a bit of a delay while we wait for analytics to run. Depending on our visualization strategy we might be able to break the work up and trade one big delay for a few small delays.

2. Alternatively, rebuild our annotations preemptively. We might do it as soon as we realize there’s data to analyze, or perhaps we have some kind of schedule or external trigger. The point here is, we run the analysis before we think a user will want to visualize it.

We’ll add these capabilities to our architecture diagram. That means responding to both external data changes as well as internally sourced changes.

Rewinding history

To “rewind and reprocess,” we’ll just delete our annotations and then make sure there’s a mechanism to rebuild them. Here’s an example chain of events we’ll model in our architecture:

1. Administrative user invalidates some (or all) of the annotations we’ve stored in our cache:

   1. Optionally, if the data was invalidated because source data changed, the system loads fresh data from the external data source.

   2. The system deletes or otherwise invalidates the annotations in our cache.

   3. An event such as “analysis invalidated” (with some metadata, perhaps a date range) is emitted.

2. The analytics service is listening for events, and receives the “analysis invalidated” event:

   1. The event includes the date range of deleted data. The service checks the date range to see if annotations are missing from the cache.

   2. If the data is still missing, it runs the analysis and rebuilds the missing annotations.

   3. As annotations are rebuilt, they are written to the cache.

   4. An event such as “analysis completed” (with some metadata) is emitted.

3. Separately, a visualization service might be running where a user asks to display the visualization for some missing data:

   1. The system checks the cache, realizes annotations are still missing, and sends a “run analysis” message directly to the analytics service.

      1. The analytics service is already working on analyzing the data so it can ignore the message, or respond with an “analytics in process” message.

   2. The visualization presents some kind of “waiting for analysis” display while it listens for the “analysis completed” message. When it receives the message, it loads the annotations and displays them.

This is one way to handle a specific chain of events. We’ll need to handle other events and commands as well — such as an event to tell us external data has changed.

Since we’re talking about responding to events that happen, it makes sense to use event storming to model those events and commands. We can model “things that have happened” (the events) as yellow cards, and “what we want to happen” (the commands) as blue cards.

If you need a refresher, Chapter 2.3 Strategic event storming and Chapter 3.4 Tactical event storming provide in-depth explanations on modeling event driven architectures.

Here’s an updated architectural drawing showing how our events and commands interact with the system:

Persistent annotation cache

One of the things we haven’t decided yet is exactly how we’ll store our annotation data. There are quite a few options, and certainly more than one strategy that will work well.

In part two I hinted at using a relational database to store a copy of the external source data. This makes a lot of sense, but it isn’t the only option.

We could store all of our data in a persistent data store such as MongoDB, a noSQL document-oriented database. Document databases offer some fantastic search capabilities and, perhaps most interesting, don’t force us into a predefined schema — we can, essentially, attach anything as our annotations.

That’s a nifty idea, because we want flexibility to change our data analysis and come up with new data sets to store, freely associating them with source data.

We could also stick with a purely event-driven architecture. In this kind of system, we would emit annotations as events into a persistent queue (such as Kafka). The persistent queue becomes our source of truth — and annotations are just messages on the queue, like, “sprint X: total out-of-sprint hours: 500.”

With many options, it’s probably a good idea to decide on our business requirements:

1. Obviously, we want to cache the original source data, so we aren’t tightly coupled to an external system. (We know the external system is a relational database).

2. We want to easily create new annotation data sets. If we dream up a new calculation (maybe, “number of activities completed after hours on Fridays”), we want to be able to store it easily.

3. We know we’re using DataFrames for analysis. Storing our DataFrames without translating them into something else would be convenient (but not necessary).

4. Our annotation data should be persistent (to avoid unnecessary delays rerunning an analysis) — but, once a particular report or visualization is no longer being used, we probably don’t need to keep it around.

5. Finally, we’ll want to think about performance and wait times. We definitely don’t want users to experience a slow system.

The relational cache for source data still makes a lot of sense.

For our internal annotation data, something more flexible is desirable. We don’t want to be constrained by a relational schema, forced to update the schema every time we think up a new report or annotation.

While a noSQL solution seems like a good fit, it may be an unnecessary complication. It would afford long-term persistence, but unless we run into performance problems that long-term persistence may be pointless.

Designing for flexibility makes sense.

We’ll implement our internal annotation cache as a service. The service will:

1. Allow us to store annotations (data structures) in a persistent fashion (but, not necessarily permanently).

2. Provide a shared data service (allowing other services to leverage the shared data).

3. Give us scalability (we can boost processing power or memory, start up multiple copies, or upgrade to a more powerful data store technology).

4. Allow us to start simple, just storing annotation data in memory. A long-running service might give us all we need (after all, we’d only rebuild our annotations if we restart the server).

Let’s update our architecture diagram one more time:

Read more

Introduction

This activity creates fidelity in our engineering models — the diagrams and specifications that give us enough technical depth so we can proceed to coding.

Everything we’re going to build should be modeled. That might sound like of lot of work, but with the right drawings and specification style, it’s not — plus, our models should be reusable. For instance, once I’ve modeled my implementation of a circuit breaker, I don’t have to model it again. I can just use that circuit breaker design everywhere I need it.

The approach I’ll introduce here increases the fidelity of your technical designs using data inputs and outputs, paired with very clear and specific functional results. It builds on your existing engineering artifacts and further validates your value stream, making sure alignment with business objectives is maintained. We capture our control processes and at the same time enhance communication among those designing the solution.

One of the challenges with modeling is deciding on what’s effective yet not overly constraining or complex. We want to define some clear standards of practice and make sure the engineered product is fit for purpose. At the same time, teams should have the flexibility to take a different path if that different path is more fit for purpose.

The modeling approach we use in the Delivery Playbook meets those goals:

• We define engineering artifacts using two specific diagram types, neither of which are difficult to understand or use.

• Clear objectives are defined while giving teams enough freedom to follow a different path — so long as those objectives are met using whatever approach they choose to follow.

Read more

In the previous article I introduced data pipelines, highlighting their ubiquity and a few of the challenges you’ll face architecting a good data pipeline. Here’s a quick recap of those requirements:

1. Moving data efficiently. Pipelines are all about making sure the right data gets to the right place at the right time, in the right format. For many applications, like air traffic control or the stock market, timing is everything. The architecture has to scale, meeting demand without degrading — often with life-saving implications.

2. Data integrity. This is about ensuring the source data is what we think it is — incorruptible, unchanged as it moves through the pipeline. We have to make sure there’s no data loss, no translation errors, no dropped information. Yet, at the same time, pipelines are fundamentally about processing that data, deriving new insights from it.

3. Non-destructive annotation. Being able to add to the data in the pipeline efficiently and flexibly is key. This makes it possible to create new insights from the source data — but it implies a schema that can grow in unpredictable ways.

4. Rewinding history. Pipelines evolve, refining their algorithms and adding new processing steps. When we fix a bug or refine a calculation, we need to know, “what would the last three months have looked like with this new logic?” If we can’t reprocess the data we’ll never know if an inflection point in our analysis was due to a different calculation or new business conditions.

5. Surfacing discoveries and creating actionable insight. All of these steps need to culminate in actionable intelligence. Getting those pieces to work together, then correlating all that disparate data and presenting it clearly, logically, can present its own challenges. Does the data processing sequence matter? How does one element depend on another? What happens when your analysis depends on external data or fundamentally different data structures?

In this article, we’ll develop a foundational data pipeline architecture. I’ll approach the architecture iteratively — beginning with something simple, exploring whether it fits our purpose, and refining it. Our final architecture will provide a foundation that can grow into a scalable, powerful data pipeline platform.

For our use case, we’ll design an application that tracks team activity across projects in an organization. Our goal in tracking all this information is to deliver new business intelligence that gives useful, actionable insights into team efficiency across the organization. That’s intelligence we can use to remove bottlenecks, improve team productivity and velocity, eliminate wasteful process and more.

What are we measuring?

Before we get started, it’s a good idea to know what we’re trying to influence. We want to surface insights, not just tables and charts. The difference between data and intelligence is whether it impacts our decisions.

Actionable intelligence changes what you do next.

For our test case we want to develop a better understanding of team efficiency. There’s an existing body of work that covers this topic, and we’ll leverage it.

Our plan is pretty simple. Following the principles of a waste walk, we’ll have team members tell us whether their day-to-day activities are creating bottlenecks or inefficiencies or draining their effectiveness. They’ll do this by capturing activities and marking them as “in sprint” or “out of sprint.” (If you aren’t familiar with those terms, take a look at part one of this series).

We’re going to assume this data is available — perhaps coming from a JIRA dashboard or other time tracking application. I’ve actually built a mobile-friendly application that does this. It’s part of a waste walk analytics platform in use today. For our purposes, what matters is the data — not the mobile app.1

The data schema is straight-forward. Team members are Users, they track their individual Activities, and each Activity belongs to a Sprint.

If we diagram out the basic data model, it would look like this:

As you can see, Teams are also modeled (so we can manage sprints and their team members as a group), and every activity has an Activity Stereotype. These stereotypes are basically the kinds of activities team members can track in the system. They can’t just record a random event; instead, they pick an activity from a list of “stereotypes” like “meeting,” “writing code” or “story refinement.”

We’ll take a look at the specific fields in a little while. For now, we have a conceptual understanding of the data — that’s what we need to move forward.

Data versus insight

Given the data we have available, we need to ask ourselves: what kind of insights can we derive from the source data? How might those insights change our behavior?

Here are a few ideas:

1. If team members spend too much time doing activities that could be automated, we’d want to prioritize automation — letting those team members get back to more important work.

2. If a lot of time goes into investigating defects, we would probably want to know about it so we could investigate why we have so many defects (and fix the root cause).

3. It would also be informative to know when the team works on fixing defects; is it after the product ships, early in design, or somewhere in-between? Early defect correction is much better than late defect correction.

4. A lot of planning late in the product lifecycle — for instance, planning after a sprint has started — might indicate opportunities to improve coordination or design process. Late-stage planning often creates bottlenecks for team members.

5. If individuals feel their time in sprint ceremonies and meetings is not productive, we should probably investigate why they feel that way — especially if it’s a lot of time.

6. If we knew how many different things team members had to juggle, it might influence how we prioritize work and focus time for the team.

7. Probably most important: we want to know, across the organization, exactly how much time is going into inefficient or wasteful activity. A small amount is fine, but in aggregate, the time and effort lost could be costing us more than we realize.

I’ve done these kinds of analyses for many teams — and the insights are almost always transformative. Just one example: one team found they were spending close to 23% of their time on activities that could be entirely automated. The effort was spread out, of course — an hour here, another there. At an individual level, it was about a day’s effort each week, tweaking manual deployments, doing environment tuning, “sanity testing” in staging and production.

Now scale it up. For a team of eight, that’s 150 hours per sprint. Stated another way, by investing in automation, the team would get back 150 productive hours every single sprint and their quality of life would be improved at the same time. Knowing those numbers, prioritizing Infrastructure as Code (“IaC”) for deployments, automating those repetitive “sanity tests” — suddenly makes a lot of sense.

Transforming raw data into insight

We know what data is available to us. But how do we transform it into actionable intelligence we can use to create better outcomes?

The raw data doesn’t really get us there. That’s where the data pipeline comes in — by transforming that data into something we can use:

1. First we need to “intake” our source data. Once we can access it, we can analyze it, of course.

2. Then, we transform the data into what we need. We aren’t planning anything overly complicated (at least, not yet). Here’s the gist of what we need to do:

   1. Separate “in sprint” versus “out of sprint” activity. We’ll be focusing on the out of sprint activities or, in Lean Engineering terms, the “non-value add” effort. This “non-value add” work is where we’ll find our biggest gains.

   2. Summarize data about activities. For example, we’ll need a way to add up all the activities that could be automated. That tells us how much effort is spent here.

   3. We also want to be able to count up and somehow tally the time between activities (the “context switching time”) for each team member — probably as some kind of aggregate or average number.

3. Of course, we’ll want to visualize the results.

That all seems very straight forward. The first logical step is to think about how we’re getting our source data.

Start with the source data

The first thing we need is our source data. Fortunately for us, it’s readily available and easy to access.

Our tracking application stores all this activity data in a SQL database, using a traditional relational format. That makes a lot of sense — it’s a relatively static data structure, and the relationships between teams, users, sprints, and activities are clearly hierarchical.

One approach is to just read the data directly from the SQL database. This would be the easiest solution. Once we read the data, we can perform our calculations, add in the context switching data, and display it:

This does raise a few interesting questions, though:

1. What happens if the source data changes in some way or if we can’t access the SQL database when we want to display our data?

2. How, exactly, are we going to store the results of the calculations and annotations?

3. Does this mean that every time we display the results we have to run through the entire pipeline, reading the data, performing calculations and then displaying our findings?

4. How does constantly accessing an external database impact our performance?

Data integrity and availability

We want our application to be highly available and reliable.

The problem is, we’ve created a dependency on an external system — the SQL database. If the external system changes or just isn’t available when we need it, we’ve got a problem. It would break our data pipeline. We need a solution that ensures both the integrity of the data as well as its availability.

The simplest solution is to just copy the data and keep a local cache of it. Whenever the source data is updated, we can update our local cache. If something goes wrong — for example, the data format changes or we can’t contact the SQL database — we can keep using our local cache until we fix the problem. That gives us high availability.

This also gives us a degree of certainty about the data format.

While the external SQL database might change, our internal cache can potentially remain constant. For example, let’s say the external database is using an Integer data type for activity duration, but at some point they decide to change it and start using a Time type instead. Since our local cache is under our own control, we have a few options:

1. Update our local cache to use Time as well, and change our pipeline code accordingly.

2. Convert the incoming data from Time to an Integer and store that. Leave our code as-is.

3. Decide to use a completely different type, like Duration, in our internal cache and make similar changes to our code.

Our strategy would depend on the situation — but we have options. We’ll examine a few different ones in a bit, but for now we’ll just note the internal cache in our architecture.

Adding a cache to the architecture might look something like this:

I’ve added an “ACL,” or “Anti-Corruption Layer,” to the intake process. ACLs are used to protect the data integrity of our domain — in other words, we don’t control the external SQL database but we do control our internal cache. That cache can, for instance, use a Duration to represent the passage of time regardless of how the SQL database represents time. The ACL is where we translate from the external format to our internal, domain-specific format.

It gives us a nice layer of isolation from external change. Generally speaking, creating isolation between services is a great strategy for improving reliability (as well as scalability — more on that later).

Benefits of isolation

I pointed out that by copying external data into our own system we are creating isolation. This is an important architecture topic for a few reasons:

1. Isolation means we are not inexorably dependent on an external system. If that external SQL database goes down, or the data changes, we can still function.

2. We also get architectural independence. If we decide to change the structure of our internal data cache, we can. It’s under our own control.

3. Being in control of our own data architecture also means we’re in control of our scalability. If our data analysis product becomes the hottest thing since sliced bread, we can scale up. We might add more servers to our cache, or we might evolve our architecture toward CQRS patterns — we’re in control, and it’s all hidden behind our product API.

From this perspective, copying data is a good idea. We need to remember there’s an external source of truth. But we remain in control of when we run data intake, how we change the data on the way in, and how our own data is structured.

Powerful, non-destructive annotation

Our original requirements include making sure that we maintain our source data integrity. The best possible approach in that regard requires two things.

First, we have to capture every record, in its entirety. That ensures we won’t miss anything important (including the possibility that something becomes important later, even though we thought it wasn’t important at first).

Second, we can’t change what we capture — we can only add to it. This means our analysis and data annotation can’t alter the source data — it’s immutable. We have to find a way to piggy-back or pair our new data with the raw source data.

We’ll leverage that in our code. Let’s say, for instance, you wanted to find out how many seconds of “out of sprint” work each user tracked in a sprint and you also want to make sure paid time off is not included in that number. Here’s a very simple example, using a snippet of Explorer code:

data
|> DF.group_by([“user_id”])
|> DF.filter(stereotype_code != "pto")
|> DF.summarise(out_of_seconds_sum: sum(out_of_sprint_seconds))

Data is just an array of raw data records. The group_by line groups all records with their respective team member. Then, with filter we make sure paid time off is not being included. And finally, we sum up all the out_of_sprint_seconds and put the result in a new variable, out_of_seconds_sum.

Two important things to note here: first, we haven’t altered the source data in any way. Second, we’ve annotated our data set by associating that new variable, out_of_sprint_sum, with our DataFrame.

We’ll explore this in depth soon. For now, I just wanted to give a quick taste of what we can do and explain why it’s valuable.

Tackling data analysis

We’ve decided how we’ll get the data, but there’s more work to do. How are we going to perform our calculations and store the results?

Read more

These past few months were very busy. I had two projects compounded on top of each other — one, a client modernization effort; the other, a new release of an analytics tool we use with our customers. It was more than I like to take on at one time and, yes, I got a bit burned out.

Which is a good reminder — prioritize your well being!

Work will always be there. Life will pass you by.

Recharge in Malaga

After getting all the work out of the way, we took some time to recharge: a few weeks off in Malaga.

Read more

Why data pipelines are interesting

A data pipeline moves data to a destination, transforming it along the way, turning it into something useful.

It’s like a factory assembly line. But, imagine you’re assembling data. Raw data is your source material entering at one end. It goes through various processes and comes out the other end as finished product (clean, structured data ready to use).

For instance, air traffic control systems process data from multiple radar stations, transponders, weather sensors, and flight plan databases to build a real-time picture of airspace. A picture humans can interpret in a split second. An air traffic control system:

• Ingests position data from hundreds of aircraft every few seconds.

• Correlates radar returns with flight plans and transponder signals.

• Filters out noise, birds and ground clutter.

• Predicts flight paths and potential conflicts.

• Integrates weather data to route around storms.

• Handles sensor failures gracefully — if one source goes down, seamlessly switching to backup sources.

Controllers need accurate data within seconds, and failures can have catastrophic consequences. They need real-time data, rather than batches, with real-time redundancy and validation.

Data pipelines are fundamental to modern data infrastructure. A good pipeline makes sure the right data gets to the right place at the right time, in the right format.

If you think about it — that’s the foundation of just about everything in our data-driven modern society.

When it comes to projects that will evolve your mastery over data management, building a data pipeline is one of the most challenging and rewarding. That’s why, in this multi-part series, I wanted to do a deep dive into a data pipeline — including design, architecture and code. In this series, we’ll get hands-on and explore how to build an app that analyzes complex streams of data.

Creating actionable intelligence from data

Data pipelines start with the raw, source data — from wearables, instrumentation, monitors, databases, mobile devices. An Airbus A380 is fitted with up to 25,000 individual sensors. Each one of those is sending raw data that may need to be acted upon quickly. That raw data goes through different processes — cleaning, filtering, combining, reformatting — before landing in its final destination, which might be an instrument, display, data warehouse, analytics tool, or application.

Clearly, copying and interpreting all that data by hand is impossible. Instead, the data pipeline does it automatically, often as a real-time feed that transforms information on the fly. It ensures data moves reliably and consistently.

Raw data is often messy or in the wrong format. The pipeline transforms it into something useful — removing duplicates, standardizing formats, calculating metrics, or joining information from multiple sources, as with the air traffic control example.

Often, data pipelines are event driven. This just means they consume a sequence of events from the real world — some fact that already happened — then turn it into actionable intelligence.

“Actionable intelligence” just means some kind of information you can use. For instance, the fact that a stock price dropped could be turned into an automated buy order for that stock. An e-commerce company might have a pipeline that pulls daily sales data from its sales database, combines it with inventory data, calculates key metrics like revenue per product, and loads everything into a dashboard where executives can see performance at a glance.

There are a lot of moving pieces. Getting those pieces to work together, then correlating all that disparate data and presenting it clearly, logically, is hard. There are challenges to doing it well, which I’ll talk about below.

Preserving data integrity

One of the most important things a good data analysis tool — and data pipeline — needs is a guarantee of data integrity. In this case, data integrity means the data is exactly the data we think we are analyzing. This sounds obvious, but it’s surprisingly easy to get wrong.

Here are some common pitfalls, all of which we need to design against:

• Silent data loss: Information dropped during joins, filtering, or transformations without warning. For example, only capturing summary data or a subset of available data instead of recording (and keeping) everything.

• Type coercion errors: Numbers becoming strings, dates and time zones handled incorrectly, or precision lost in conversions.

• Encoding issues: Data corruption caused when moving between systems — something as simple as assuming an input field uses Latin characters instead of Unicode.

• Partial updates: Only processing some records, possibly ignoring “uninteresting data,” or failing to capture records that don’t process correctly.

• Implicit assumptions: Null handling, unexpected data volume, unexpected inputs or missing parameters, duplicate treatment, or sort order that changes behavior unexpectedly.

Good data pipelines preserve integrity.

In the second article in this series we’ll dive deep into data pipeline architecture and address each of these potential pitfalls.

Annotating data

We also need a way to evolve our data over time — that is, add to it in a way that we don’t end up breaking our data integrity rules.

Raw data rarely speaks for itself. A transaction table tells you what happened, but not why. A spike in errors shows something broke, but not what you were doing at the time. Good data tools let you add context and turn numbers into understanding.

Annotations work at multiple levels. At the record or event level, they document what the data represents, adding fidelity. A data analyst might add a “review” tag to what looks like an accidental double-entry, or a prescription drug regulatory pipeline might link together other drugs that treat similar symptoms.

At the macro level, annotation can build on a core data set, adding new information. Data from multiple sources can be combined to create a more complete picture. Hard-to-perform calculations can be run and attached to existing data, making it easy to extract more meaning later. Later, in our example use case, we’ll do this by annotating our data set for further exploration.

Annotations compound over time. Six months later, when someone investigates why something looks unusual, annotations lead to new institutional knowledge. Without them, context can be lost and every new analyst has to rediscover what happened.

Surfacing insight

Data analysis tools should surface insights, not just tables and charts. The difference between data and intelligence is whether it changes what you do next.

We need the right level of aggregation and annotation: too granular and patterns disappear in noise; too macro and you can’t identify root causes.

Start by focusing on decisions. Before building a dashboard, a report or a graph, ask what decision it will inform. If the answer is vague or simply, “it’s good to know,” it’s not actionable. Every result should express some clear, specific action.

Make the analysis do the work, not the reader. Instead of showing a table of conversion rates by channel, highlight which channels are underperforming, by how much and why. Instead of a trend line, show whether or not execution is deviating from planned goals and exactly how it changes your delivery plan.

Focus on metrics that lead to human action. Also, tune thresholds to minimize false positives. For example, it may be useful to “clean” a data set by removing anomalies and peculiar outliers. We’ll do that in our example app, effectively removing data that is suspicious and likely just “noise.”

The goal is to inform clear action. If it requires tribal knowledge, domain expertise, further exploration or twenty minutes of interpretation, it’s not yet actionable.

“Rewinding” history

One of the most powerful and underused capabilities in data analytics is the ability to reprocess historical data with new logic. When you fix a bug in your analysis or refine a calculation, you often want to know: “what would the last three months have looked like with this new logic?”

This is harder than it sounds. If your pipeline only processes new data, historical numbers are frozen with old limitations or bugs. You end up with charts where the methodology changes partway through, making trends impossible to interpret. “Revenue increased 20% in March,” but we’ll never know if it was growth or if it was due to a changed calculation.

Point-in-time correctness makes this possible: storing raw events or source data permanently, without changing it or pre-processing it. When you need to revise analysis, reprocess the historical data through the updated pipeline. The results of that processing becomes annotation to the data — ultimately, annotations that we can throw out and do again using a new approach. We look at data as an entire time series, a record of facts that happened in the past — and are, therefore, immutable.

But this means treating our analysis and transformation logic as code, not ad-hoc formulas or manual actions. The source data must never be altered — only our code, and the annotations and analysis it produces. That gives us a documented data model, not tribal knowledge about which column means what. The goal is reproducibility: given raw data from any date and a specific version of your data pipeline, you can create an exact analysis for any timeframe.

Building a data pipeline

I promised a deep dive into the design, architecture and code of a data pipeline. There are a lot of use cases — stock trade analysis, weather data, genomic research, geologic tremor research, the list is probably endless.

For this deep dive, we’ll explore an application that tracks team activity across all the projects an organization is working on. Our goal in tracking all this information is to deliver new business intelligence that gives useful, actionable insights into team efficiency across an entire organization.

We’ll build the application in stages, showing its evolution and developing a solid foundation for a data pipeline.

Using the app, team members will track their activities across various tasks, but unlike most “time trackers” they’ll be able to annotate their time as “in sprint” or “out of sprint.” These are defined as:

1. In sprint — essentially, “this is work that contributed direct value to my sprint goals.”

2. Out of sprint — this represents everything else, essentially, “this is work that I had to do, but it didn’t contribute value directly to my goals — it may have even slowed me down.”

We’ll scale this up across an entire organization. For a large organization, that means hundreds of teams, thousands of team members and tens of thousands of individual tasks. It’s a sizeable volume of data to work with, but in terms of what data pipelines are capable of it’s still pretty small (think about the volume of data flowing through the stock market, in comparison).

We’ll need a way for team members to capture day-to-day work, quickly and easily. This will become our data source, pushing data into the pipeline in real-time:

All the activities will flow into our data pipeline. We’ll capture those individual activities, but we need to look at the data in aggregate — from an organizational point of view. This means being able to annotate and analyze the data from different perspectives.

Looking at the data as a stream of events over time, we can “slice and dice” it. We can also compose the data with other analyses, all to produce actionable intelligence. That means being able to aggregate the data and run on-the-fly analysis across different segments of data (for example, being able to take a “slice” of data across four teams out of the entire organization):

We also want to perform detailed analysis. In our activity tracking app, we’ll show time the team spent on manual activity that could have been completely automated — letting the humans on the team spend their brain power on more valuable tasks:

As we discussed above, a good analytics platform also supports annotation — that is, being able to combine new data with existing data.

In our case, we’ll calculate how much time is lost to switching from one task to another. We’ll combine that analysis with pre-existing data to create new insights. Here we show the impact of giving the team more focus time:

Summarizing our goals

From a business perspective what we want to achieve with the activity tracking app doesn’t seem complicated:

1. Give team members a place to easily and quickly record their activities (what they did, how long it took).

2. Let them tag work that falls “outside” the sprint — representing work that, while potentially necessary, didn’t really contribute direct sprint value or move their goals forward.

3. Aggregate data across the entire organization to discover meaningful trends.

4. Expose action that we can take — action that will have an immediate impact on efficiency, productivity — improving “ways of working” across teams.

This is something that most organizations are working on — in fact, most are working on efficiency and productivity goals right now.

But the reality is, it’s complicated. There are competing concerns, unclear cross-organizational goals, a lot of noise to filter out, activity that may be necessary but might not seem necessary. How do we accurately analyze and filter it down into clear, actionable intelligence?

There are solutions. Ideas built on decades of rich, scientific research and business process evolution. Perhaps most famously, Toyota empowered their employees to do whatever it took to streamline automotive production lines. Toyota’s concepts regarding just in time production are deeply embedded in the foundations of Lean Manufacturing. Taiichi Ohno is widely regarded as having pioneered effective techniques for eliminating “wasteful action” in day-to-day work (Muda, a Japanese word meaning “wasteful”).12

Our application will build on that, gathering activity data and turning it into an actionable, live stream of insight about organizational health.

We’ll need to stay true to the critical concerns outlined in this article: preserve data integrity, focus on annotation not transformation, build a system that always delivers actionable intelligence and, perhaps most important, make it possible to “rewind and replay.” As our algorithms evolve and mature, we’ll want those insights to be applied retroactively.

Coming next

We’ve outlined our requirements above. Next up is defining the architecture, the “blueprinting” phase. That’ll be the topic of the next article in this series. By the end of the third article, we’ll have a concrete roadmap as well as an architectural foundation — a steel thread that we can build on to deliver a proof of concept in one or two sprints worth of work.

If you’re following along in the Delivery Playbook, we’ve just about finished our mobilization phase. We’ve got a product vision and we know what the product’s capabilities and functions will be.

If you find Customer Obsessed Engineering and the Delivery Playbook valuable, please share it with your friends and coworkers.

Share Customer Obsessed Engineering

Most teams are building on quicksand and calling it solid ground.

You’ve got 95% test coverage. Your CI pipeline is green. Your unit tests pass every time. You ship with confidence, and then — crash — production burns down because of an edge case nobody thought to test. The user entered a 37-character string instead of 35. Someone sent a negative timestamp. A list came back empty when your code expected at least one item.

Sound familiar? Here’s an uncomfortable truth: traditional unit testing is fundamentally inadequate for building reliable software.

Unit tests are like taking a single photograph of a mountain and claiming you understand the entire landscape. They capture one specific moment, one exact scenario, but they miss the infinite variations that real users will throw at your system. They’re missing the crevasses, caves, and sudden unexpected features. They’re necessary, but they’re not sufficient.

After decades of building and shipping software — from startup MVPs to mission-critical systems handling millions of transactions — I’ve learned that the teams who ship bulletproof software have one thing in common: They know that testing specific use cases is inadequate.

They don’t buy into the illusion of safety that unit testing offers. They use property-based testing.

Property-based testing doesn’t just find bugs. It finds entire categories of bugs you never knew existed. It’s the difference between checking that your bridge can hold one specific car versus proving it can fulfill its design parameters by handling a constant, unpredictable flow of traffic.

The fundamental flaw in how we think about testing

Let me paint you a picture. You’re building a user activity tracking system (let’s call it Tracky). Your traditional testing approach might look something like this:

test "creates activity with valid data" do
  user = insert(:user)
  sprint = insert(:sprint)
  
  assert {:ok, activity} = Activities.new_activity(
    sprint.id, 
    user.id, 
    ~D[2024-01-15], 
    ~T[02:30:00], 
    "Working on feature X"
  )
  
  assert activity.duration == ~T[02:30:00]
  assert activity.description != nil
end

This test passes. You ship. Then production explodes because:

• Someone enters a duration of 25 hours

• A date falls outside the sprint boundaries

• The description is 500 characters long and breaks your UI

• Someone passes a nil sprint_id

• The timestamp has microseconds that cause floating-point precision issues

Your unit test checked one path through the code. But software doesn’t operate in the controlled environment of your test suite. It operates in the chaos of production, where users do unexpected things, data comes in unexpected formats, and Murphy’s Law reigns supreme.

Instead of testing specific examples, we should be testing universal properties that must always hold true.

Property-based testing builds a foundation for quality by exercising all the behavioral boundaries of your code. This is exactly how we approach development using a “steel thread.” A steel thread creates an architecturally complete foundation by touching all the major components of a system.

The truth about test coverage

Here’s something that might challenge a few of your assumptions: high unit test coverage can actually make your software less reliable.

I’ve seen teams with 98% test coverage ship catastrophic bugs because they confused coverage with correctness. They tested their happy paths thoroughly but ignored the vast space of possible inputs that could break their system.

Property-based testing breaks through this limitation. Instead of measuring how much of your code is exercised, you’re measuring how much of your input space is validated. Instead of asking, “did we test this line of code?” you’re asking, “did we prove this business rule is always true?”

A single property test might only exercise 20% of your codebase but validate behavior across millions of potential inputs — including edge cases. Which approach gives you more confidence in production?

The takeaway here: coverage metrics optimize for the wrong thing. They optimize for testing effort rather than testing effectiveness.

What is a “property?”

A “property” is a universal law or attribute. Gravity is a property that dictates how objects are attracted to each other. Fluid dynamics define a set of properties about how fluids flow and interact. And, our Tracky application has certain properties, too:

1. Activities must fall inside the date boundary of a sprint.

2. An activity is limited in time (for our purposes, it’s a business rule that “activities must be less than 8 hours in duration).

3. Activities must be owned by a user, and that user can view, update, and delete only their own activities (not someone else’s activities).

4. Activities can, optionally, have a description.

Of course there’s more, but this is sufficient for the examples we’ll look at in this article.

These properties define how Tracky behaves; what you are allowed to do with it, and what you are not allowed to do. They are universal laws — in all conditions, an activity must fulfill each of these laws — and, very importantly, the system must gracefully prevent the violation of these laws.

What property-based testing actually does

We’re all familiar with unit testing and testing very specific “success criteria,” like I described above. The problem with that approach is the inherent limitation of testing a single path at a time. It’s an impossible burden to test every edge case using that approach.

Property based testing expands our test horizon. Instead of writing specific test cases, you define properties — universal truths about your system — and let the testing framework try to break them using hundreds or thousands of generated test cases.

In short, you write one generalized test scenario, and the system uses it to create thousands of potential test paths for you.

Let’s start with a simple example.

In this example I use a “generator” to seed my test with test data — if you aren’t familiar with generators, don’t worry, we’ll come back to them. For now, just understand that a generator creates an infinite, essentially random stream of test data. They are the powerhouse behind property based testing.

Let’s start by replacing my naive creates activity with valid data with a property-based test. Here’s how it might look:

property "activities must have reasonable durations and valid dates within sprint bounds" do
  check all sprint <- sprint_generator(),
            user <- user_generator(),
            duration <- valid_activity_duration_generator(),
            date <- member_of(Date.range(sprint.start, sprint.end)),
            description <- lorem_style_phrase_generator_or_empty() do

    result = Activities.new_activity(
      sprint.id,
      user.id,
      date,
      duration,
      description
    )

    # Property 1: Valid inputs should always succeed
    assert {:ok, activity} = result
    
    # Property 2: Output duration should match input duration
    assert activity.duration == duration
    
    # Property 3: Activity date must be within sprint bounds
    assert Date.compare(activity.date, sprint.start_date) != :lt
    assert Date.compare(activity.date, sprint.end_date) != :gt
    
    # Property 4: Description, sprint, and user must all match
    assert activity.description == description
    assert activity.sprint == sprint
    assert activity.user == user
  end
end

I’m using Elixir here, but that doesn’t matter. There are property-based test frameworks available for just about every language.1

At first glance, this might not seem to be much different. I’m still creating activities, and comparing the properties of those created activities with expected values.

But, this single property test does the work of thousands of individual unit tests. It generates valid sprints with random date ranges, creates users with different roles, generates durations up to 8 hours, picks dates within the sprint bounds, and creates descriptions of varying lengths. It does this many times, creating a myriad of combinations and edge cases that we would never think to code up by hand.

This means you’ll see a lot of combinations:

1. An administrator creating an activity with a minute-long duration.

2. Another user creating a 5-hour activity, mid-sprint, with a long description filled with emoji and Unicode characters.

3. Activities with nil, empty, very short, and very long descriptions with a variety of punctuation.

4. Activities that belong to sprints far in the future, or that started (and ended) a month ago.

And on it goes, with many, many combinations of users, sprints, dates, durations and descriptions.

And, it gets more powerful: With property testing we also test the boundaries. What happens when the duration is exactly 8 hours? What about when it’s one second over? What about when the date is exactly on the sprint start date? Or the end date? It’s easy to write negative scenario tests as well, just by introducing a generator that creates bad inputs.

The testing framework explores the entire input space systematically, looking for the edge cases and combinations of edge cases that will break your system in production.

The architecture of unbreakable software

Property-based testing forces you to think architecturally about your code. When you write properties, you’re not thinking about specific scenarios — you’re thinking about the fundamental invariants that must hold true regardless of input.

Thinking in terms of property-based testing makes architecture easier. We’re shifting away from, “let’s make this one specific test case succeed” and toward, “let’s define conditions that prove these fundamental truths are always true, no matter what we throw at it.”

Here are some of those “fundamental truths” property-based testing helps to verify:

1. Data integrity properties.

   1. Entities must always maintain appropriate relationships.

   2. Temporal properties (time and dates) must fall within suitable boundaries.

   3. Access and ownership must be part of the system.

Read more

The most productive teams I’ve worked with never argue about implementation details.

That’s probably surprising if you’ve spent years in rooms with engineers propounding about the merits of different algorithms, frameworks, or architectural patterns. What I’ve learned is that the teams who spend their time arguing about how to build something are the same teams that consistently miss deadlines, accumulate technical debt, and burn out their best people.

The highest-performing teams focus on what they want to achieve, then let declarative systems handle the how.

This isn’t just about programming paradigm or tool selection. It’s a fundamental change in how we think about building software, managing teams, and delivering customer value.

It’s the difference between organizations that scale effortlessly and those that collapse under their own complexity.

The hidden cost of procedural thinking

Most of us were trained to think procedurally. We write step-by-step instructions. “First, do this, then do that, and if this happens, handle it this way.” It feels natural because that’s how we’ve been taught to solve problems since childhood.

But procedural thinking carries a hidden tax on everything we build.

Here’s an example I bet sounds familiar: your team finds themselves inheriting deployment tooling for a complex product. I ran into this recently, to the tune of well over 1,000 lines of disparate scripts, loosely tied together across different systems. It handled environment setup, dependency installation, configuration management, and service startup. The scripts had been lovingly maintained by just one or two engineers for years. And it worked really well — most of the time.

The problem wasn’t that the script failed (it did, in spectacular ways, often completely derailing a new product release). The problem was what that script represented: years of institutional knowledge understood by one or two people, and requiring constant maintenance as the underlying infrastructure evolved (which was pretty often).

After those two people left the company, deployment became a bottleneck. When AWS updated their base images, someone had to trace through the script line by line to figure out what broke. When the team wanted to deploy to a different region, they discovered hardcoded assumptions baked into dozens of conditional statements. Little incompatibilities or unforeseen eventualities that the script just couldn’t handle. Sometimes it took days to reverse engineer a fix.

Here’s my key insight about this: every procedural system you build becomes a maintenance burden that grows exponentially with complexity.

Declarative systems are simple. They describe a clear end-state, not a series of complex instructions that hopefully get you there. Consequently, they become more valuable as they scale.

This is the fundamental difference between telling a system how to do something versus telling it what you want accomplished.

What “declarative” really means

Declarative programming isn’t just about using Infrastructure as Code (“IaC”) instead of writing for-loops and if-then statements (though that’s part of it). It’s about expressing intent rather than implementation.

When I tell Kubernetes “I want three replicas of this service running with these resource constraints,” I’m being declarative. The system figures out how to make that happen, how to recover when nodes fail, how to scale when load increases. I describe the desired state, the “what I want,” not the “how.” The system maintains it, figuring out the “how” for me.

CSS is declarative: “I want this text to be bold, pulsing dark blue on a midnight background.” It describes what I want to appear on my web page. The browser handles the complex rendering pipeline, layout algorithms, and cross-platform compatibility.

But “being declarative” is also about thinking intentionally, a way to reason about systems so they mirror my intention. By “systems,” I don’t mean code, I mean organizational systems. How your team operates. How you design your architecture, your product, your business.

That’s what most people miss: declarative thinking applies far beyond code.

When I create a Definition of Done that specifies the characteristics of completed work, I’m being declarative about quality standards. When I establish value stream maps that define customer outcomes, I’m being declarative about business objectives. When I build a playbook that specifies artifacts and quality gates for each delivery phase, I’m being declarative about process.

The teams that master this distinction operate at a fundamentally different level than their peers.

In using steel threads, I discovered that successful teams naturally gravitate toward declarative approaches. They don’t argue about how to implement the “right to be forgotten” for GDPR compliance — they declare what needs to happen and let the architecture emerge from that constraint.

Those declarations became architectural constraints guiding every decision. Teams stop bikeshedding about microservice boundaries and started building systems that provably meet requirements.

How procedural programming fails at scale

Procedural systems fail for the same reason that manually managing server configurations fail: they don’t account for the complexity explosion from system growth.

Every time you write a procedure, you’re making assumptions about the environment, the inputs, the available resources and the desired outcomes. You rely on expected side effects, assumptions about what’s happening outside the procedure. Those assumptions become technical debt the moment any condition changes.

I’ve seen this pattern repeatedly:

1. In DevOps: you start with simple deployment scripts, but as requirements evolve, scripts grow more complex. Eventually, it’s an unmanageable forest of conditional logic that nobody wants to touch.

2. With code complexity: systems designed procedurally become tightly coupled. When one piece changes, ripple effects cascade. What should be a simple update becomes a multi-team coordination effort.

3. During testing: procedural code demands testing every possible execution path, every external dependency. Side effects cause complications. Teams must choose between inadequate tests or spending too much time maintaining those tests.

4. With knowledge bottlenecks: procedural systems distill complex knowledge into internal implementation detail. When the code author departs so does the knowledge. It takes months to reverse-engineer implicit, complex logic.

The declarative alternative

Declarative systems solve these problems. Instead of encoding how something should happen (which brings with it all the fragility of a procedural approach), you specify what should be true, then rely on the system to maintain those conditions.

There are a few arguments against declarative code that I hear now and then. In my experience these are based on misinformation:

“Declarative code is hard to think about.”

I couldn’t disagree more. Declarative systems are concise, clear, and consolidate all of your logic in a single place. It’s easier to comprehend, and far easier to test. This is really about comfort and changing how we were taught to solve problems — we’re more comfortable thinking procedurally.

“You lose control with declarative approaches.”

You gain control. Instead of controlling implementation details, you control outcomes. When you declare “this service should always have three healthy replicas,” the system ensures that’s true even when nodes fail, traffic spikes, or deployments occur.

“Declarative systems are harder to debug.”

Procedural systems require tracing through execution paths to understand what went wrong. Declarative systems let you compare current state to desired state. The difference between current and desired state is the bug.

When a Kubernetes deployment fails, you don’t debug the container orchestration logic. You check if your declared resources can be satisfied and fix the mismatch.

Remember that deployment script example above? 1,000-line deployment script that sometimes failed spectacularly and required days of diagnosis?

We replaced it with IaC in Terraform, declaring the desired infrastructure state. The declarative specification is just a few hundred lines long. More importantly, we now deploy to any region. We change environment settings with a few variables, not by rewrite procedures. Deployments that previously took days (or, just outright failed) are now done with a click, via a dashboard, and it’s utterly reliable.

I mentioned that declarative thinking applies far beyond code. So how do we “scale up” from declarative code to declarative thinking?

Building declarative teams

The most profound realization is that declarative thinking extends beyond code to how we structure teams and processes.

Traditional project management is procedural: detailed project plans with dependencies, specific task assignments, and rigid timelines. When requirements change (and they always do), the entire plan requires revision.

Declarative project management focuses on outcomes: “We need a user authentication system that handles 100,000 concurrent users with 99.9% uptime.” The team figures out how to deliver that outcome, adapting their approach as they learn more about the problem space.

Goals and emphasis for the team shift away from activities and toward specific, measurable, provable outcomes.

This is exactly what the Delivery Playbook enables. Instead of prescriptive methodologies that tell teams how to work, the playbook declares what good delivery looks like, providing a framework to achieve it reliably.

Here’s a practical framework for moving your team toward declarative thinking:

1. Start with outcomes, not activities. Instead of “implement user authentication,” declare “users can securely access their data from any device.” The first focuses on building something; the second focuses on enabling something. This is the shift away from procedural thinking. It takes time, but with practice you’ll get there.

2. Express constraints, not solutions. Instead of “use microservices with REST APIs,” declare “components must be independently deployable and testable.” Let the architecture emerge from the constraints.

3. Define interfaces, not implementations. Instead of detailed API documentation explaining how to use your service, create OpenAPI specifications that declare what your service provides. Generate the documentation, client APIs, and validation from the specifications and code.

4. Build systems that maintain state. Instead of scripts that perform actions, build systems that ensure conditions remain true. Think about Kubernetes’ declarative nature: “there must always be 3 healthy replicas running.” Use Terraform, Kubernetes and frameworks like Ash Framework.

5. Test behaviors, not code paths. Instead of unit testing internal implementation details, write tests that assert observable behaviors and outcomes. Focus on policies, for instance, “forbid non-administrators from creating a new account.”

6. Use design thinking. Build organization processes that model your declarative “way of working.” Adopt inputs and specific outputs for project phases, relying on statements about target state (not focusing on “what activities everyone should be doing”). The playbook is a great example to use as a model.

7. Only measure progress. Focus on evidence-based delivery. What matters is achieving the declared target state. Progress is measured only by validating that your target state — your feature — is working and deployed.

Deep dive: a declarative security API by contract

I’ve written a lot about declarative thinking in abstract. An example will be helpful to crystalize what it really means. Let’s say we’re building a project tracking system for our agile team. We want to manage sprints and their activities.

One thing we’ll need is a way to track those sprints. Here’s one approach — a fairly traditional, procedural one:

Read more

Be uncompromising about your craft

Steve Jobs was notorious for rejecting products that were nearly complete if they didn't meet his exacting standards. He would send teams back to redesign products multiple times, sometimes scrapping months or years of work.

The iPhone launched without a physical keyboard when BlackBerry dominated with keyboards. Jobs believed in making hard choices about what to leave out, saying “innovation is saying no to 1,000 things.”

He believed that true craftsmanship meant caring about every aspect of a product, not just what was visible.

Caring about your craft matters. It gives you passion. It drives exceptionalism.

Your customers care about what you build, too. They want to be delighted. Delighted customers become evangelists (just like Apple customers and the “cult of Mac”). This is your “value stream.” Understanding what really matters, and sticking to it.

Making compromises won’t lead to delight. Compromise leads to mediocrity.

Instead, be uncompromising. Seek excellence, say “no” to 1,000 things. Choose the best possible path forward — even if it takes longer, even if it’s the harder path.

Have really deep conversations with people that matter

It’s hard to build a shared understanding of a tough problem. But, only in-depth mutual understanding makes learning and joint decision-making possible. It how we discover better paths forward.

Where many teams go wrong is not understanding how important this is. They lose patience, try to explain a technical necessity with hand-wavy jargon, or tell their customer it’s “best practice.”

That creates opaque dependencies where your customer, your boss, your coworkers have no power. The only way forward would be blind trust. It’s just not realistic to believe anyone enjoys those conditions.

Plus, what if they know something you don’t?

No matter how experienced I think I am, I always have enough self-doubt to ask, “wait, am I missing something here?”

Have those deep conversations. Explain the tough problems. See if there’s a better way forward.

It means sharing decision making, because you’re opening yourself up. Admitting that maybe, there’s a better solution. At first it feels like you’re giving up control — but you’re not. You are sharing control by having those important conversations.

Test your assumptions

It’s not always easy to know if we made the right decisions, especially when we casually make assumptions. We make assumptions all the time. This feature will be great. That security policy is good enough.

Fortunately, in software, it’s pretty straight-forward to test those assumptions: establish your key results, and then measure them.

Pick the results you care about. Choose a metric to measure, like number of failures, latency to reply, or anything else that proves your feature does what it’s supposed to do.

Then, measure. That just means test the results. In software, I love Test Driven Development (TDD). I wouldn’t code any other way.

But what if we’re talking about doing what’s right? Cutting through the nonsense and deciding if this or that will leave our planet or our society in a better place, or a worse place?

The same strategy applies. Choose your key results and measure. Check the facts.

Maybe you’ve heard the argument, “we’re in a cooling period, the earth is actually cooler now than any time in 485 million years!” That sounds great. But don’t assume it is. Being cooler now than ever before could be a sign of fragility, a statistical anomaly. Outliers become normalized — plus, if you look closely, you’ll see other patterns. The sudden, rapid uptick in temperature that’s been happening for about 100 years? It happened once before, almost 250 million years ago — just before a mass extinction event that wiped out about 90% of life.

In the real world, don’t blindly accept some numbers you hear, instead check to see if they add up. Measure your results, and see if you were right.

It can be hard though. We’re basically trying to catch ourselves in making a mistake. Nobody likes to make mistakes — but at least this way, you catch it first.

Anticipate mistakes

I assume I’m going to make mistakes.

In fact, I assume I’ll make more mistakes when I feel more comfortable, more knowledgeable about something. Because it’s easy to get lazy.

But I don’t actually make that many mistakes. Why?

Because I’ve put systems in place to protect against them.

Those systems are my secret weapon. My personal edge that makes sure I don’t make mistakes — while sometimes, it seems like everyone else is figuring it out as they go along.

All it takes is stopping yourself from charging blindly ahead:

Read more

Being intentional about your drawings

I’ve seen a lot of technical drawings. I’ve been witness to religious wars about one style over another. I’ve seen engineering teams melt down over which arrow directionality means “input” versus “output.”

It can be a contentious topic — perhaps understandably so.

Adopting one kind of drawing style over another imposes a particular style on our architecture as well. Our drawings influence our architecture. If we use an object oriented notation, we’ll likely design a system based on objects. A style that’s heavily relational, we’ll probably put business logic in the database.

Our drawing style shouldn’t lock us into a predetermined technical approach.

That’s why we need to be really careful about our technical drawings.

I like technical drawings that are unambiguous, easy to read and easy to maintain. The drawings need to be understandable by everyone, including your customer. That’s essential — because the moment part of your team can’t follow along, it’s useless.

What’s more, the right style is going to add value to your project by:

1. Adding fidelity to business processes, in a way that the customer, product owner, and business can validate. It can’t be indecipherable technical jargon.

2. Being compatible with how we model business events, extending ideas about event modeling and contextual boundaries.

3. Extending validation criteria; in other words, it should model the acceptance criteria that proves we delivered the right thing.

4. Working well with all of our engineering requirements (not creating an impedance mismatch as we try to translate into a different domain or technology).

For the past decade or so, I’ve been using two specific types of diagrams. This is all I’ve needed across many domains and many project sizes.

I’ve chosen these two drawing types to meet two specific needs:

1. The first adds functional detail to complex business processes, so we can see exactly how these processes will be implemented and tested.

2. The second makes it easier to reason about complex interaction sequences between disparate components in complex systems.

#1. Architectural marble diagrams

Marble diagrams are a visual way to represent behavior. They align well with event modeling because they model streams of information using a timeline-based format. They use symbols to depict the emission of values, completions, and errors. A marble diagram represents an “observable source” with a timeline and demonstrates how information is transformed and ultimately emitted.

Marble diagrams are event oriented, so they fit well with event streaming architecture. Here’s a quick synopsis of terminology that’s frequently used in marble diagrams:

1. “Observable.” A thing that can be observed to be true or false; basically, something we can validate as correct. An “observable source” is just where an observable thing comes from.

2. “Operation.” Some kind of a process or function that we can apply.

3. “Value” and “event” are, respectively, discrete properties and some concrete thing that has happened and can be recorded.

It sounds a lot like an event driven architecture, making marble diagrams a great way to model a business process. It essentially takes an input, shows how the input changes over time, and what is ultimately output. If you’re following my playbook that formula should sound familiar.

A marble diagram, in a nutshell, looks like this:

Here’s a quick explanation. Each of the numbered post-its in the diagram correspond to the explanation below:

1. A business event from your tactical event map. If your diagraming tool supports it (Miro does), create a bidirectional link — allowing you to navigate between your tactical map and marble diagram with a click.

2. The input to a function is identified with a letter. In this case, the data set “A” is an input to the “4.0 Exploded Event” function.

3. Each function is represented as a sequence of “marbles” along a timeline. Individual marbles represent values, functions or “observables,” and emitted values. An emitted value could be a result, a signal or an error.

4. We can add another layer to explode further complexity — it’s “marbles all the way down.” You can think of each layer like another layer in the call stack. But, not necessarily literally; you model added complexity as you need by increasing depth.

5. Once the input has been transformed to a result, it’s no longer an “A” data type. It becomes a new type, here noted with a “B.” The “B” simply represents a new data set, in this case output from the “4.0 Exploded Event” function.

6. One of the things I love about marble diagrams is the representation of the result in your diagram. The output value, “B,” is your acceptance criteria. It connects your technical drawings directly to the specifications.

As you can see marble diagrams fit very well into an event driven architectural model.

The drawing style is easy to adapt and expand upon. You can add annotations and, of course, other shapes and icons to add specific meanings to your design.

And the value of point number 6 cannot be understated: the representation of your data in your design specifications is powerful. We’ve clearly carried our requirements forward, showing business inputs, how we will transform those inputs and then showing the output. The output, then, is your acceptance criteria — it effectively shows us exactly what we must test to validate our work.

Let’s walk through a real-world example. Here’s a simple event map that I’ve annotated by numbering the events — a better alternative would be to use clickable links, which would make numbering unnecessary:

What we want to do is explode the “4.0 Loan prequalified” event, adding more fidelity to that specific card.

I’ve added some annotations to help explain the example diagram below, along with a legend:

1. The marbles now have two styles. The bold style indicates an event has been emitted, for instance, writing the output value to a Kafka event queue.

2. Notations for logging, blocking function calls and retries have been added.

3. Other shapes are shown in the legend. Marble diagrams are flexible; you should feel free to add annotations whenever it adds clarity to your designs.

This example shows the specific steps performed in prequalification:

Read more