Marty Cagan’s decade-old article can be a real rabbit hole. It’s genuinely hard to stop reading his blog, especially when digging up old gems like this.

Here’s something that resonates right now: “Good teams are skilled in the many techniques to rapidly try out product ideas to determine which ones are truly worth building. Bad teams hold meetings to generate prioritized roadmaps.”

Most teams I meet are funded, staffed and measured entirely around output. Hit the date, close the ticket, ring the bell for delivering on a roadmap.

Too few are paying attention to what matters: did what we shipped really make a difference for our customer?

The reorg that actually changes delivery isn’t new ceremonies. It’s moving the moment you celebrate from “we shipped” to “it delighted our customer, it actually moved what matters.”

Your team has probably spent more hours this year waiting on pull requests than it spent writing the code marooned inside them. Andrea Laforgia opens his March 2026 essay Stop Using Pull Requests with numbers from across the industry that count the cost. The piece is the most thorough reckoning I’ve seen of why the modern pull request workflow is mis-designed for the teams that use it. Go read it. Then come back here — because if you’re using the Delivery Playbook, Laforgia is describing the practice you already know. He just gives it a name. And I like it.1

That name is T*D — Test-Driven Development, Trunk-Based Development and Team-Focused Development, fused into one operating model. Hold on to it. We’ll come back to it.

Don’t break the flow

Chapter 2.8 Delivery processes & tools lays out what a well-wrought delivery pipeline does: ephemeral environments, test batteries, zero trust, release orchestration, observability, canary releases. The argument is that your pipeline is part of your product, and its job is reliability and repeatability through automation. The motto I’ve handed to more than one team — the only path to higher environments is through automation — comes straight out of that chapter.

Here’s the thing. If you map a typical team’s workflow against that pipeline, there is exactly one step where a human might still be gating the flow. Tests run automatically. Security scans run automatically. Infrastructure is provisioned automatically. Releases orchestrate automatically. And then — somewhere between “I pushed” and “it merged” — a colleague has to find time, open the diff, scroll through it and click Approve. That manual gate is the pull request. Everything else in the chapter is built around eliminating exactly this kind of step.

Pull requests are the routine human gate that survived a pipeline whose sole purpose is to remove them. Fix that, and the pipeline finally runs at the speed its design intended.

Three findings that should bother you more

Laforgia’s piece marshals the research that counts the cost. Three findings worth carrying into your next retro:

Code review finds fewer bugs than you think. In Bacchelli & Bird’s ICSE 2013 study, the authors manually classified roughly 570 review comments from Microsoft engineers and found that about one in eight — call it 12.5% — were about defects. The rest was structure, naming and style. A separate Microsoft study by Bosu and colleagues built a 1.5-million-comment dataset and asked a different question — were the reviews useful? — and concluded 64–68% were. Two distinct findings, often conflated. The honest read is that code review’s value lives in knowledge transfer and improving the code’s shape, not in catching bugs. And a blocking async queue is one of the clumsiest possible vessels for sharing knowledge.23

Most of the lead time is waiting. A change that takes ten minutes to write but waits four hours for review spends 96% of its existence in queue. Martin Fowler recounts one client whose team logged 130,000 hours waiting on pull requests in 2020 — 91% of which got no comment in return. The process extracted enormous delay and returned almost nothing.4

The lever is the speed of review, not review itself. The 2023 State of DevOps report, drawn from more than 36,000 professionals, found that accelerating code review alone improves software delivery performance by 50%. Elite performers who meet their reliability targets are 2.3 times more likely to use trunk-based development. DORA explicitly names heavyweight code review as a brake on delivery. The methodology that produced this question — the four DORA metrics — comes from Forsgren, Humble and Kim’s Accelerate; the 2023 numbers cited here come from the State of DevOps report that builds on it.56

A caveat that Laforgia treats carefully and I want to carry into the body rather than burying in the close: this evidence speaks loudest for intra-team work, where colleagues know each other and the codebase. Trust is a gradient. For distributed teams collaborating across time zones, open source contributions or any genuinely cross-team change, the case for a gate is stronger. The mistake is importing the gatekeeping pattern wholesale from those high-trust-gap contexts into the daily work of a co-located team that already trusts each other.

T*D is what the Playbook teaches as muscle memory

This is where the Playbook and Laforgia line up. Each leg of T*D slots directly into something you have already been building.

TDD/BDD fills the test batteries slot — properly. Chapter 2.8 says: “If tests fail, the build fails, and the code can’t be merged (no exceptions).” Test-driven and behavior-driven development are how you make that sentence true in practice. The tests exist before the feature exists. They encode the acceptance criteria the team agreed to. They run on every commit, not just at the end. When the gate that decides whether code reaches the next environment is automated and trustworthy, the human gate becomes redundant — there is no class of defect a tired reviewer at 4pm is going to catch that a green CI run did not. This is the same argument I made in The balanced power of TDD: defects must be prevented, not removed. Deming wrote it down in 1982 — Point Three: cease dependence on inspection to achieve quality — and the software industry has been re-discovering it ever since.7

Trunk-based development matches the Playbook’s stance on small increments. Chapter 3.0 Delivery is explicit: “Delivery is intended to move quickly. That means working in small increments, not ‘feature trains’ or monolithic releases.” That is trunk-based development by another name. A long-lived feature branch is a feature train waiting to happen. The Playbook already prefers the alternative — ephemeral environments per branch are fine, but the branches themselves should be short-lived, with code returning to trunk daily and incomplete work hidden behind feature flags. The point is: code the gate has to inspect must be small enough that automated checks will validate it in seconds.

Team-focused development is what you already started building when you killed the tester role. Pair programming and ensemble (mob) programming move review from a post-hoc inspection into a continuous one. In a 2005 controlled experiment, Müller and Tichy found that pair programming and solo-plus-review achieved comparable cost when quality was held constant — pairing did not cost more, it just moved the review work earlier. That experiment is small (38 students in one university), so don’t lean on it alone; Hannay and colleagues’ 2009 meta-analysis casts a wider net across the practitioner record — with the caveat that pairing’s benefits depend strongly on task complexity and developer experience. In both lines of evidence, what changes is when review happens, not whether. I’ve argued the structural version of this case in I don’t hire testers anymore: quality belongs inside the team, not bolted on at the boundary. Pairing is the same logic applied to review.89

This is what I’d like you to carry away: T*D is not three new things to adopt. For anyone running the Playbook, it is a name for the convergence of practices you have been heading toward. The pull request workflow is the last manual gate left after you’ve automated everything else. T*D is what you put in its place.

What to do Monday

Most teams cannot drop pull requests overnight. They shouldn’t try. The path is gradual, and every step buys real flow:

1. Optimise the PRs you have. Cap them at two or three hundred lines. Set a four-hour review SLA. Automate every style and lint check. Reduce required approvers to one. This is housekeeping, not transformation, but it removes the worst of the queue cost while you build the alternative.

2. Adopt Ship / Show / Ask. Rouan Wilsenach’s three categories give a team a way to graduate routine changes off the gated workflow first. Routine changes Ship — straight to trunk, no PR. Notable changes Show — they go to trunk immediately, with a PR opened in parallel as the venue for post-merge discussion. Genuinely uncertain or risky changes Ask — they get the blocking review treatment. Most work, once you look honestly, is Ship or Show.10

3. Pair on production code; merge to trunk daily. Feature flags carry incomplete work; automated tests are non-negotiable. This is also where the Playbook’s Quality Engineer rotation pays off — the engineer in that role this sprint is the one making sure the test batteries and the pipeline can carry the weight you’re about to put on them.

4. Move to ensemble programming where the team is ready. Continuous review during creation. No post-hoc inspection. This is where most teams will need the most patience — it’s a cultural shift, not a tooling change, and it rewards a coach in the room for the first few weeks.

Charity Majors gives you the lodestar: speed is safety. Small changes by single owners reach production fast, the blast radius stays small, and the fix is obvious because the intent is still fresh.11

The harder question

The question for your team is not how do we do pull requests better? It is why do we still need them? If the answer is fear — of regressions, of juniors, of complexity, of the codebase itself — address the fear directly through skills, automation and pairing rather than institutionalizing it as policy. The Playbook has been quietly pointing at this answer the whole time. Laforgia’s piece is the receipt for the empirical cost of ignoring it.

Read the source

Laforgia’s full essay carries the academic citations, the DORA data, the practitioner consensus and the honest caveats I didn’t hit here. He treats correlation versus causation carefully, draws the trust-as-gradient distinction in more depth than I did, and gives the transition path more nuance than four bullets allow. If you lead a team that still gates every change behind an async queue, read his piece end to end and bring it to your next retro. Then open the Playbook to chapter 2.8 and ask which manual gates are still left in your pipeline. There won’t be many, and that’s the point.12

This newsletter grows by word of mouth… I’d really, truly appreciate it if you could refer a friend. Your referrals make it worthwhile.

Refer a friend

Six people. One sticky.

A SWOT analysis at the top of a planning workshop. Strengths quadrant — a dozen sticky notes inside ten minutes. Opportunities, fine. Threats, fine. Weaknesses — one sticky.

The team has been together a year. Mixed group, some senior, some junior, some remote and some local. They’re smart — and obviously not weak in just one way. But one is all they were willing to write down, in front of each other, with their names attached. So the Weaknesses quadrant sat there, embarrassed and underfed, while the rest of the wall filled out around it.

Most retrospectives end this way. Most planning meetings, the same. The team knows. The team won’t say.

There’s a 30-minute exercise that breaks this open. It’s not subtle. It’s not high-tech. It’s a borrowed word from Soviet engineering, an inversion trick from a 19th-century mathematician. It takes a team from laughter to honesty to commitment in less time than your average standup. It’s called TRIZ — the Liberating Structures version, which I’ll come back to — and it’s the cheapest unlock I know for a team that won’t speak its own dysfunctions out loud.

Here’s how it works, why it works and how to run it next week.

The wrong question

The silence in that Weaknesses quadrant is not random. It’s the predictable shape teams take when the question goes the wrong way.

“What are we doing wrong?” is one of the worst questions you can ask. It demands that someone you work with every day be the source of bad news, of blame — in a setting where their answer goes on the record. The forces working against honesty are stacked: the social cost of singling out a behavior, the political cost of singling out a person, the personal cost of being the one who said the thing nobody else was willing to say. So people hedge. They write harmless stickies. They give safe answers and stay quiet about the rest.

This isn’t about psychological safety being too low, exactly. It’s about requiring more safety than most teams I’ve known. An unusually safe team — the rare ones — can handle the direct question. Most teams can’t, even good ones. The Goldilocks zone for direct critique is narrow, and most teams sit just below it.

The wrinkle: the same team, asked the question a different way, will answer it. Same people, same room, same dysfunctions. The information is there. The mechanism for getting it out is what’s broken.

That’s the gap TRIZ fills.

Inversion

The 19th-century Prussian mathematician Carl Jacobi had a habit, when stuck on a problem, of saying man muss immer umkehren — “one must always invert.” If you can’t see how to make a thing succeed, see if you can describe how to make it fail. Charlie Munger, Warren Buffett’s late partner, popularized the move a century and a half later: “Invert, always invert,” he’d repeat, crediting Jacobi each time.1

Inversion is a cognitive shortcut. The forward question — “how do we make this succeed?” — runs into the usual social and intellectual walls. People hedge. People defer to whoever spoke first. People say what they think they’re supposed to say. The inverse question — “how do we guarantee it fails?” — jumps every one of those walls, because there’s no commitment. They’re describing a failure. They can be specific, vivid and sometimes uncomfortably accurate, and the room laughs instead of getting defensive.

Once the failure is on the wall, naming the parts of it that already exist is a much smaller step. You’re not confessing. You’re confirming.

That’s the whole engine of TRIZ.

Why it works on teams that won’t speak up

If you’ve never run TRIZ you’re probably thinking, “why does this work at all?” Why does asking a team to describe failure produce honesty when asking them to describe weakness doesn’t? It’s the same people in the same room talking about the same problem.

Here’s four reasons I keep coming back to. They aren’t independent — pull on one and the others move — and there are probably more I haven’t named. Their common thread: all four lower the cost of saying the thing.

Hypotheticals are cheap. Saying, “we could skip estimation entirely” carries no social cost. Nobody is suggesting it. Saying, “we do skip estimation” costs you — especially if the person who set up the no-estimation pattern is in the room. TRIZ collects the cheap version first. It builds a list of behaviors the team is willing to discuss because the team isn’t yet committing to whether they’re real.

Confirmation is easier than confession. By the time Movement 2 starts, the behavior is already on the wall. The team isn’t introducing a new criticism — that’s the hardest move. It’s confirming an existing one — that’s a much smaller move. “Ah, yea, we do that,” is far easier to say than, “we have a problem with X.” The board already said it. The team is just agreeing.

Laughter changes the room. Ten minutes of laughing at exaggerated worst-case behaviors changes the social register. The same conversation about the same behaviors, started cold, would feel like an accusation round. Started after the comedy, it feels like camaraderie. Defensiveness drops. People interrupt themselves. The room is in a different mode.

The quiet people speak. This is the move I see most reliably. In nearly every TRIZ I’ve facilitated, the first person to circle a behavior in Movement 2 — the first to commit to, “yea, that one” — has been the person who hadn’t said much yet. The senior engineer who watches more than she talks. The new hire who’d been listening for three months. They’d seen the dysfunction the whole time. They didn’t have a frame to say it. TRIZ gave them one.

You can put psychological safety theory on this if you want. You can talk about face-saving, about politeness norms, about the asymmetry between criticism and praise. All of it holds. But the practical version is simpler: it’s a lot easier to say things about hypotheticals than about reality, and TRIZ exploits that gap to get a team to say what it already knows.

That’s why a 30-minute exercise can do work that an hours-long retro can’t.

The three movements

The exercise has three movements. Together, they take 30 minutes if you keep the timing tight; you can stretch to 45 on a hard problem or compress to 20 with a small confident team. Each movement has a single question, and each question is doing different psychological work.

Movement 1: the worst-possible-outcome brainstorm (10 minutes). Frame the outcome the team cares about — high-functioning sprint planning, a retrospective that actually changes things, a release process you’d be willing to demo. Then ask the inverted question: what could we do to guarantee the worst possible version of this? Wild, exaggerated, funny answers welcome. Suppress nothing. Two minutes of silent sticky-writing first, in 1-2-4-All style, then a round-robin share.

The laughter is the point. A team that’s laughing at “we could just cancel the retro and send an email instead” is a team becoming willing to say real things. Don’t rush this part — the comedy is what unlocks the next movement.

Read more

I was chatting with a friend about team dynamics. We’d been talking about restructuring around product teams — the kind of empowered, durable, cross-functional teams I’ve written about before. He’s nodding along, asking sharp questions. Halfway through, he pushed back.

I’m paraphrasing, but the gist was, “Look, I get the theory. But where’s your proof? We’ve all heard the pitch. Cross-functional teams, durable squads, outcome metrics, two-pizza this, empowered that. Show me the data. Show me that any of this actually beats what we already do.”

It was a fair question. A great question, actually. I’d written the case in Building a product oriented team and the prequel, Moving away from the Monolith — faster, better with a product mindset. But those pieces argued from reasoning and experience. They didn’t bring the math.

So today I want to bring it. Because the math is there — it’s been there for over a decade — and once you line it up, the conclusion is hard to escape: product teams, the real ones, ship faster, build less waste, deliver higher quality and produce dramatically better business outcomes than the alternatives. The evidence isn’t a single bombshell study. It’s a stack of independent research, all converging on the same answer.

Let me lay out the receipts.

One note before we start. None of the studies I cite below were designed to answer the question “do product teams outperform?” They were designed to answer narrower, related questions — about deployment, about feature usage, about team effectiveness, about developer velocity. What’s below is my synthesis: the act of lining up findings from different research traditions and showing where they converge. The convergence is the argument. Each citation is a thread; the rope is the pattern across them.

A quick refresher on “product team”

Before the data, let’s get precise. “Product team” gets thrown around so loosely it’s often meaningless.

Marty Cagan and the Silicon Valley Product Group draw the cleanest distinction. A product team is a small, durable, cross-functional unit — typically a product manager, a designer and a handful of engineers — that owns a problem and is held accountable for outcomes. They are given problems to solve, not features to ship. The product manager owns value and viability. The designer owns usability. The tech lead owns feasibility. Together they own the result.1

A feature team looks similar on paper — same disciplines around the table — but is handed a prioritized list of features to build. Outcomes belong to a stakeholder upstream. The team is judged on output: did you ship what was on the roadmap? Velocity, story points, throughput.

A project team is even further from product orientation. It’s pulled together to execute a discrete project, then dissolved. Specialists rotate in and out. There’s no durable home for product knowledge.

The labels matter less than the operating model. As I wrote in Building a product oriented team, the real distinction is durability, ownership and empowerment — long-lived teams that own the whole product end-to-end and have the authority to figure out the how.

Now — what the research says.

Receipt #1: they ship dramatically faster

Start with DORA — the DevOps Research and Assessment program, founded by Dr. Nicole Forsgren and now part of Google. Every year since 2014, DORA has surveyed practitioners — more than 3,000 in 2024 alone, and tens of thousands across the decade-long program — and statistically grouped them into performance tiers based on four metrics: deployment frequency, lead time for changes, change failure rate and mean time to restore service. The methodology, the math and the conclusions are documented in Accelerate — the book that emerged from the research and won the Shingo Publication Award.2

The 2024 Accelerate State of DevOps report is the latest. Elite-tier teams deploy 182 times more frequently than low performers, with 127× faster lead time for changes. Elite teams hit lead time under a day, deploy on demand and are roughly 19% of the surveyed population. Low performers measure lead time in months.3

That’s not a marginal advantage. It’s not “20% better.” It’s two orders of magnitude.

You might be tempted to dismiss this as a DevOps story rather than a team-structure story. But every time DORA digs into what differentiates elite teams, the answers are the same attributes that define product teams: small, cross-functional, owning the whole pipeline from code to production, empowered to make their own technical decisions. The opposite — handoffs across functional silos with stage-gate approvals from external groups — correlates strongly with low performance. Accelerate presents one specific finding bluntly: external change advisory boards do not improve stability. They make lead time worse without making anything safer.4

McKinsey’s “Beyond Agile” work, which I cited in How waterfall is wrong for software, puts a number on the same effect from a different angle. Across their consulting engagements, McKinsey observed that companies could compress the average time from “code complete” to live production from 89 days to 15 days when they reorganised around DevOps and product-team principles. Same basic architectures, same kind of features. Six times faster, just from changing how the team takes work from “done” to “deployed.”5

The simplest summary: product teams own the value stream end-to-end. Feature and project teams hand off across silos. Handoffs cost time. Time turns into distance from the customer. Distance from the customer means everything you ship is stale.

Receipt #2: they get more done — but redefine what “done” means

“More done” is the wrong metric. The right metric is: did anyone use what we shipped?

In 2002, Jim Johnson — chairman of the Standish Group — presented a finding at the XP conference in Sardinia that’s been quoted ever since: across enterprise applications, only 7% of features are “always” used, 13% are “often” used and 16% are used “occasionally.” That leaves 64% of features as either rarely or never used. Standish later extended the analysis to claim that roughly 80% of features deliver low or no value. On large projects, Standish data shows organizations deliver only 42% of the features they originally scoped. The Standish numbers have drawn methodology critiques over the years — fair enough — but more recent work points the same direction. Pendo’s 2019 Feature Adoption Report, drawing on telemetry from hundreds of B2B SaaS products, found roughly 80% of features are rarely or never used.6

Sit with that for a moment.

Now, “rarely used” isn’t synonymous with “no value.” Some rare features are quietly essential — compliance flags, account recovery, the audit trail your CISO will ask about exactly once. Both Pendo and Standish gloss over that distinction. Even granting it generously, though, the ratios still point at a meaningful misallocation: a lot of effort going into features customers rarely or never engage with. (I’m looking at you, Microsoft Office).

A feature team measured on roadmap output is, by these numbers, building a sizable wedge of low-value work. They look terrific on burndown charts and terrible on customer impact.

I’ve watched this play out enough times to be cynical about it. A feature team gets a quarterly roadmap. They estimate the work in story points, deliver on schedule, demo a sleek end-to-end flow and get applauded. Six months later, telemetry shows the feature gets touched by 3% of users. By then the team has moved on to the next item on the roadmap. Nobody is accountable for the gap between “shipped” and “mattered,” because shipped is the goal.

A product team would have caught that earlier. They wouldn’t have built the full feature on a quarterly roadmap; they’d have built the cheapest possible version that tested the hypothesis — a prototype, an A/B variant, a deliberately ugly first cut — and watched the data. If the data said 3%, they’d have killed the feature in week two and gone after a more promising hypothesis. That’s the discovery discipline Marty Cagan and Teresa Torres write about extensively, and it’s the operational reason product teams build less and deliver more. Not because they’re slower. Because they aim differently. They invest in discovery before commitment. They kill a feature that doesn’t earn its keep. They redefine “done” from “shipped” to “moved the metric.” And the metric they move is customer behavior, not story points. (I’ve made the same argument from a measurement angle in Stop measuring effort.)

There’s a parallel finding in McKinsey’s What makes product teams effective? — a five-year study of more than 1,700 teams across 75 organizations. McKinsey identified two structural conditions that consistently predict throughput and velocity improvements: dedicated team membership (no context switching across projects) and team persistence of three to six months minimum. Teams meeting both conditions develop reliable estimates and sustained throughput. Teams that don’t, can’t.7

That finding isn’t surprising if you think about how knowledge accumulates. Product orientation depends on knowledge durability — context that only forms when the same people work on the same problem long enough to internalise it (more on this below). Project teams dissolve before that happens. Feature teams reorganise around the next quarterly priority. Product teams stay put. That’s why their throughput compounds.

Read more

A junior developer told me they’re using Claude as a mentor. They liked the patience that never wore thin, the availability that never closed, the way they’re never made to feel doltish.

Here’s what worried me: they said mentor. Not tutor. Mentor.

I don’t think they’re wrong about what they’re getting. But mentorship isn’t about answers. It’s about being seen — watched while you struggle, nudged when you’re avoiding hard conversations, sent on projects that stretch you in ways you didn’t ask for. Mentors have the nerve to tell you the truth in private because they’re invested in who you’re becoming.

AI can’t do that. Because it isn’t invested in you. Because it’s not “in the room.”

What AI is genuinely good at

Let’s start with what’s actually working, because this isn’t a piece about being suspicious of the tools. The tutoring story is real. It’s the part of the AI hype that’s actually earned.

Benjamin Bloom’s famous 1984 paper described what he called the “2 sigma problem:” students who received one-on-one tutoring performed two standard deviations better than students in conventional classrooms — that’s the difference between an average student and the top 2 percent. The catch was scale. Personal tutors are expensive, and most students will never have one. For forty years that gap has been the unsolved promise of education technology.1

A well-configured AI is one way we’re closing it. When Khan Academy rolled out Khanmigo, built on GPT-4, the pitch was exactly this — a patient, infinitely available one-on-one tutor for every learner. In specific narrow tasks, the early returns are hard to dismiss. Harvard published a physics study in 2024 that found students using a properly designed AI tutor learned over twice as much in less time than their peers. They reported feeling more engaged and motivated.23

I see the same thing in software. With the right model, a serious system prompt and a few well-built skills, AI can:

• Walk a junior through a debugger session, naming what each frame means and why it matters.

• Explain why a piece of code is the way it is — not just what it does — and adjust the explanation when the question lands a level too high.

• Run unfamiliar code, watch it fail and reason about the failure with the learner in real time.

• Drill someone on a foreign language, a SQL idiom or a regex until it’s automatic.

• Take the same question for the third time without sighing.

That last one matters more than people admit. A great deal of human learning gets blocked by the social cost of asking again. Removing that cost is genuinely useful. It’s what the junior in my opening responded to. They felt safe.

How tutoring goes wrong

It’s useful, but not magical. The same AI cheerfully invents APIs that don’t exist, pattern-match on the wrong context and confidently produce code that looks right but isn’t. Apple researchers showed last year that current LLMs aren’t doing genuine reasoning at all — performance on math problems dropped by as much as 65 percent when the team added a single irrelevant clause to the prompt. Andrew Zuo’s piece on the “-10X developer” makes the harder point: when an LLM writes code at or above your level and introduces a subtle bug, you may be the worst possible person to find it.45

A newer concern is harder to wave off. An MIT Media Lab study published in mid-2025 used EEG to compare brain activity across people writing essays with no help, with a search engine and with an LLM. The LLM group showed the weakest neural connectivity and the worst ability to recall their own writing minutes after producing it. The researchers called it “cognitive debt.” That tracks. Outsource the struggle and the learning never lands.6

So a serviceable tutor, yes. A replacement for thinking, no. None of that, though, is the central point. The central point is that tutoring and mentoring are not the same job, and the public conversation keeps blurring them.

What a mentor actually does

If you ask me what my mentors did for me over the last forty years, almost none of it was answering technical questions. The technical content I could get from books, from peers, eventually from Stack Overflow and now from Claude (as long as I’m careful and look for fantasy hallucinations). What my mentors did was something else entirely.

Read more

In 3.6 Specification we worked through the first two stages of Behavior Driven Development — Discovery and Formulation. We surfaced scenarios in three-amigos conversations and shaped them into Gherkin “Given-When-Then” statements. That gave us the outline of every feature in our product increment.

Elaboration is where we move from outline to depth.

A very important takeaway from chapter 3.6 bears repeating here, because the entire purpose of this activity hangs on it:

The specifications you write now are living documents. They will evolve, change and be added to. That’s why it’s so important to create a live link to the implementation — the code that will validate the specifications. As the specifications change that live link will preserve our value chain from customer desires to delivered code.

Hold on to that idea. Elaboration isn’t about freezing requirements before the team writes a single line of code. It’s about building the connections — between the spec and the marble, between the marble and the test, between the test and the production code — that let the spec keep evolving without the value chain ever going slack.

Introduction

This activity has three jobs.

First, we expand each scenario across the levels at which the system is tested: component and assembly. A login feature has UX scenarios about what the customer sees and experiences. It also has assembly scenarios about which events fire, which commands route through which aggregates and what payload comes back. Both layers need specifications. Both layers need to be testable. Skipping a layer means leaving a gap through which defects love to slip.

Second, we create a live link between every specification and the engineering model that implements it. Each Gherkin feature points back to a marble. Each marble points forward to a Gherkin feature. That bi-directional link is what makes our specifications living documents — when the spec changes, the marble updates with it, and vice versa.1

Read more

Picture this: a program manager wants to confirm a design decision before a release. She asks, innocently enough, “Can someone point me at the current version of the architecture document?”

A pause. Then: “I think it’s in the Teams channel — the one from the February workshop.” Someone else chimes in, “No, there’s a newer one. Sriram emailed it last week.” A third voice: “That’s the draft. The signed-off version is in the SharePoint site. But the link in the meeting invite is broken because they restructured the folders.” A fourth: “I have a copy in OneDrive. It might be the latest, I’m not sure.”

Five people. Five different versions. Zero confidence that any of them is authoritative. And this is a team that cares about documentation.

The common denominator is always the same: a collaboration stack built on SharePoint, Teams and Microsoft’s various Office products. Sometimes with Azure DevOps (ADO) bolted on to handle tickets and code.

Here’s an argument that will sound harsh to anyone who’s invested in Microsoft 365 as their delivery platform: the Microsoft collaboration stack is diabolically deficient for serious product delivery. It wasn’t built for the problem delivery teams actually face. It has, piece by piece, gone after what enterprise IT wanted to solve in 2007: replacing the file share.

That’s not the same problem.

What a delivery team actually needs

There are five table stakes a team needs. I describe the first three in Getting started: Using the Delivery Playbook. I’ll restate those briefly here, along with the other two:

1. Live collaboration. Multiple people must be able to work on the same artifact at the same time, editing simultaneously, seeing each other’s cursors, throwing virtual sticky notes on a board in real time. Design work is a collaborative act. The moment one person has to collect and collate feedback, fidelity drops off a cliff.

2. Version control. Every change, by every author, is always captured and reversible. Not just “saved revisions” — the full, granular history, searchable, with diffs, always available. This is table stakes. Git has had it for two decades. Any tool that tracks less than Git doesn’t take the goal seriously.

3. Immutable linking. A document’s URL must never change. Not when the file is renamed. Not when it’s moved to another folder. Not when the folder is reorganized. Not when the author leaves the company. A reference made today should still resolve in five years. Without this, every link in every other document becomes a ticking time bomb.

4. Workflow integration. The documentation platform has to talk, fluently and deeply, to the work-tracking platform. Specs have to link to tickets. Tickets have to link to pages. Status changes in one system should not only be visible in the other, but trigger cross-tool workflow changes and render live embeds, not stale URLs. When that integration is tight, your specs, documentation and tickets stop drifting apart — they become a single living view of the work, not several systems the team has to reconcile.

5. Universal, web-first access. One consistent experience for everyone, on any device. Not a degraded web version and a feature-rich desktop app — a single canonical surface, served through the browser, that everybody gets equally no matter what hardware they’re on.

These five criteria are the measuring stick. Let’s hold SharePoint — and Teams and ADO — up against them.

Links that rot

This is the most damning, and the one I want to spend the most time on, because it’s the one almost nobody takes seriously until they’re knee-deep in the damage.

In SharePoint, a document’s URL is a function of its name and its location in the folder hierarchy. Rename the file, and the URL changes. Move it to another folder, and the URL changes. Move the whole folder, and the URLs of every file inside it change. You can turn on “managed metadata” and “content types” and a bunch of other gears that promise to fix this, but in practice — across every tenant I’ve ever touched — they don’t. The URLs change. References break.

Microsoft has shipped “durable links” and “document IDs” — features that are off by default and rarely enabled. What a team actually encounters is: you renamed the file, the link in the meeting invite from three weeks ago is now dead. The problem is pervasive enough that a cottage industry of third-party tools has emerged just to detect broken SharePoint links.1

Compare that to Atlassian Confluence. Every page has a numeric ID assigned at creation. The ID is embedded in the URL (/pages/12345) and it does not change. You can rename the page, move it between spaces, reorganize the whole tree — the ID-based link still works. You can also use the human-readable URL (which does change when pages get renamed), but the ID-based URL is your forever-address. The system gives you both, and the tight URL is the one you automatically get when you cite the page in another document.2

This seems like a small, technical detail. It is not. It is the single most consequential design choice a documentation platform makes, because it determines whether cross-references decay or persist.

Here’s what happens in practice when links rot: people stop trusting links.

Think about that for a second. The entire promise of the web is that a URL is a durable reference — it says “here is the canonical thing, go fetch it when you need it.” That promise is what lets us build systems on top of systems. Every meeting invite, every wiki page, every code comment, every email that cites a document relies on that promise holding.

When SharePoint breaks the promise — which it does, routinely — teams develop a defensive response. They stop linking. Instead, they attach. They paste a copy of the document into the meeting notes. They email a copy to the distribution list. They download the file and re-upload it to their own folder “in case the original moves.” Multiply this by every team, every document, every month, and within a year you have a document landscape that looks like a hoarder’s basement — dozens of near-duplicates, none of them canonical, each one owned by a different person who genuinely believes theirs is the latest.

It’s rational response. Given that SharePoint links are not reliable, hoarding copies is the locally optimal strategy. However, the result is catastrophic: information silos full of stale documents and an immense toll in lost time and miscommunication.

The identity model that a platform chooses shapes the behavior of the people using it. SharePoint chose the wrong model — one that breeds inefficiency and defensive silo-building.

The version control mirage

SharePoint technically has version control. Open a file, click the ellipsis, choose “Version history,” and you’ll see a list of saves. This sounds like what you want. It is not.

The version history in SharePoint is coarse, inconsistent across product surfaces and frankly untrustworthy. Here are the problems I’ve personally run into, repeatedly:

• Coauthored Office documents roll up edits from multiple people into “a version” at intervals that are not deterministic. If Alice and Bob are editing at the same time, you may end up with one version of their joint work and inaccurate attribution. Contrast with Git: every commit is atomic, attributed and reversible.

• The version history for a file in OneDrive, Teams and elsewhere varies. There is no single source of truth. If you open a file with Teams, you see one history. In SharePoint, you may see a different history — a consequence of Office 365 evolving from a constellation of separately-designed products Microsoft is still trying to stitch together.3

• Restoring a prior version might overwrite the current one, rather than branching. If you discover at 4pm that something important got lost at 10am, “restore” could mean losing everything done between 10am and 4pm. Until recently, this was the default behavior — so be careful.

Confluence, again, does this properly. Every save is a version. Every version compares cleanly against any other. You see what changed, who changed it and when. You can restore a prior version as a new version, preserving full history. This is how version control is supposed to work, and it has been the standard behavior in software since RCS shipped in 1982.

Why does this matter for delivery? Because design artifacts evolve, and the history of how they evolved is often as valuable as the current state. When the security auditor asks, “When did we add this API endpoint to the threat model?” — the answer lives in the diff. When a customer says, “The contract we signed described a different data retention policy,” — the answer lives in the history. When a post-mortem asks, “When did we know about this failure mode?” — the answer lives in the version log.

A platform that can’t answer these questions reliably is a platform that is actively hostile to engineering rigor.

Teams, Teams Sites and the organizing layer

If the first two failures are about individual artifacts, this one is about the landscape those artifacts live in.

Microsoft Teams is, at heart, a chat application. When it was first released in 2017, that’s primarily what it did. Over the years, Microsoft has tried to turn Teams into a “hub for teamwork.” Marketing-speak for bolting everything else Microsoft makes onto the chat app. The Files tab in a Teams channel is a SharePoint document library wearing a trenchcoat. A “Teams Site” is a SharePoint site with a slightly different navigation chrome. Meeting recordings live in OneDrive, not in the channel. Meeting transcripts live in the meeting artifact, which sometimes appears in the channel and sometimes doesn’t. Planner tasks live in a separate product surface that is also “integrated.”

If someone in your team wants to find something, they have to know, in advance, through which product surface it was shared. Was the spec posted in a channel? Check the chat history. Attached to a meeting? Check the meeting recap. Uploaded to the Files tab? That’s SharePoint under the hood, but the folder structure might only be reachable through the Teams UI. Created as a Teams Site? Different URL. Shared in a meeting chat? That goes to OneDrive, in a folder you might not have permission to browse.

There is no single “where did we put the thing” surface. There are five or six of them, depending on how the team has been using the tools, and the mapping between “what kind of content” and “which surface” is convention rather than enforced policy.

In contrast, Confluence is the document surface. Jira is the work surface. Each is a single, distinct product doing one thing well, and the integration points between them are narrow and explicit and they work. No guessing where a document lives — it’s in Confluence. Tickets live in Jira. No impedance mismatch, no cognitive dissonance.

Teams wants to be the union of every Office 365 capability. But integration isn’t about putting everything in the same UI. It’s about making the right connections between distinct tools, so that each one stays simple and focused, and the connections carry exactly the semantics you need. Teams never understood that distinction.

Desktop is king, the browser is second-class

SharePoint and the rest of Office 365 are web-accessible, but they aren’t web-first. Their heritage is Microsoft Office — a suite designed for the desktop and perfunctorily ported to the browser. The difference is not academic.

Open a Word doc in the browser and compare it to the desktop app. The feature set diverges. Advanced formatting, compare-versions, certain macro and formula capabilities, add-ins — these are missing or degraded on the web.4

The consequences cascade. “The copy you want is on my laptop” is a phrase I hear almost every week. Co-authoring gets flakey or breaks entirely because of diverging copies. Changes sit in someone’s sync queue for hours. A document that looks current in SharePoint may be stale. Information silos reappear — not because anyone intended it, but because the tool keeps pulling work off the web and onto individual machines.

Contrast with Confluence, Jira and Miro. Every one of them is web-first by design. There’s no desktop app with a superset of features. The browser is the product. Everyone gets the same experience — Windows, Mac, Linux, iPad, phone, Chromebook — because there’s only one experience to get. One canonical home, universally accessible, identical for everyone.

This is what the web was built for. Atlassian and Google stacks lean into that. Microsoft fights it.

The ADO integration gap

This one is specific to teams that pair SharePoint with Azure DevOps for their ticket tracking and code hosting. It’s a common combo, because “we already have Microsoft” is an easy procurement story, and ADO does a lot of things adequately.

But the integration between SharePoint and ADO is thin. Thin to the point of “here is a hyperlink from the work item to the SharePoint page, good luck.” There is no live embed. There is no bidirectional status update, rich preview or shared workflow trigger. If I link to a SharePoint page from an ADO work item, you see a URL. If that URL breaks, the work item becomes an orphan.

In contrast, a Jira issue in a Confluence page renders a live, interactive issue card with issue key, title, status, assignee and priority — and that card updates when the issue does. If I embed a Jira query in a Confluence page, the page shows a table of live results — as of right now. A Confluence page can be the dashboard for a delivery effort: status pulled from Jira, specs authored inline, risks tracked in a live table, all rendering from the underlying systems of record.

You can build most of this in the Microsoft stack with enough Power Automate flows, Graph API queries, custom connectors and a patient engineer. I’ve seen it attempted. I’ve yet to see it done well enough to matter. The effort required to get SharePoint + ADO to mimic what Confluence + Jira does out of the box is roughly the effort of a mid-size integration project — a project that has to be maintained, re-tested with every platform update, and reverse-engineered when the person who built it leaves the company. That’s not an integration. That’s a liability.

Workflow is an afterthought

The last failure is workflow management — specifically the modeling and enforcement of state transitions for work items.

Jira gets this right. The workflow editor is a first-class product, with a first-class visual flow designer, transition conditions, validators, post-functions, custom statuses and permission checks. A non-engineer on the delivery team can design a workflow that enforces “a ticket cannot transition from ‘In Review’ to ‘Done’ unless the code review is merged with clean quality gates, the responsible stakeholder has reviewed the feature and all acceptance criteria are passing green.” Express it in the GUI, save it, extend with code as needed and the tool enforces it across every ticket that follows that workflow. The team controls its own process — and the workflow itself is a first-class artifact: auditable, exportable, evolvable.

ADO has “process customization.” It is possible to model custom workflows. In practice the feature is buried behind an admin gate, its conditions are a shallow subset of Jira’s, and most teams I’ve worked with never meaningfully touch it. They either live with the default — generic, not their actual process — or they reinvent the enforcement in spreadsheets, PowerBI dashboards, or human oversight routines that have to be re-explained to every new hire.

The reason this matters: workflow is the mechanism by which a team’s process becomes automatic. If your workflow tool doesn’t let you encode “we agreed code reviews are mandatory before a feature branch is merged,” then code reviews are not mandatory — they’re optional, governed only by vigilance, and skipped the moment someone is under pressure. Jira’s workflow editor is a force multiplier for team discipline, because it turns “we agreed to do this” into “the tool won’t let you skip it.” ADO gives you a significantly weaker version of that lever, and most teams don’t reach for it at all.

The cost is reliability. Teams using Jira-style workflow enforcement ship with fewer process escapes — missing reviews, missed gates, skipped steps. Teams using ADO’s default setup rely on human memory and goodwill — both fail at scale. The difference shows up in incident rates, audit findings and the quality of the artifacts the team produces.

What all this costs the team

If you’re keeping a tally, the SharePoint/Teams/ADO stack fails on every criterion. Links don’t persist. Version control is a mirage. The organizing layer is fragmented. The browser is a second-class citizen. Integrations are absent or thin. Workflow is an afterthought.

What does this actually cost a delivery team?

It costs time. A lot of it. The 2024 Stack Overflow Developer Survey (65,000 respondents) found that 61% of developers spend more than 30 minutes every day just searching for answers — and 30% say knowledge silos hit their productivity ten or more times per week.5

It costs trust. When the documentation landscape is chaotic, people stop trusting it as a source of truth. They start relying on oral tradition — “ask Alice, she knows” — and on personal copies. Key decisions stop being recorded in a place anyone can find. Onboarding becomes a six-month apprenticeship rather than a process a new hire can self-serve. Organizational memory degrades.

It costs quality. DORA's research (2022 State of DevOps Report) found that documentation quality is one of the most powerful amplifiers of engineering performance: teams with above-average documentation quality see 12.8x the organizational performance lift from the same technical practices compared to teams with poor documentation.6

It costs morale. Engineers are, as a group, unusually sensitive to tool quality. I’ve seen engineers turn down offers, quit, or burn out over the cumulative drag of bad tooling. A team that feels its own stack is working against them never reaches its potential, however talented the individuals. Tool quality is retention strategy.

Stuck with the stack? Fight back.

If you’re committed to SharePoint, Teams and ADO for the next few years, none of the above changes that. But a handful of disciplines can reduce the damage. None will turn the Microsoft stack into Confluence — they just stop the bleeding.

Here’s six specific strategies you can follow:

Read more

I was recently asked to critique a new decision tree that helps teams decide between waterfall and agile when starting a new project. The intended outcome is to improve long-term results in the organization by giving teams the ability to make critical decisions quickly and accurately.

Overall, “good shift-left stuff” in my book. Anyhow, here’s the diagram I was asked to comment on:

Read more

I once joined a team that had been struggling with delivery for the better part of two years. Demos kept failing. Deployments kept crashing production. Rollbacks were so routine that the ops team had gotten eerily efficient at them. (I’ll refer back to the team later — let’s call the company “Acme”).

I remember one in particular. The team was demoing a reactive search feature — the kind of thing where you start typing a product name and suggestions appear in real time. Same idea as typing “penguins” into Google and getting “Penguins of Madagascar” as a suggestion.

The demo cratered. Someone typed a product name, and nothing happened. Twenty minutes of poking around later, the demo was cancelled. What struck me wasn’t the failure itself — it was the reaction. Nobody was surprised. Nobody expected it to work. “Maybe next time.”¹

Later that afternoon, a planned deployment seemed to have crashed a key service. All hands on deck for a rollback. The team executed quickly. Rollback done. They’d had a lot of practice.

Here’s what immediately jumped out at me: in both cases the team had zero operational intelligence. No telemetry. No metrics. No centralized logs. No traceability. No way to quickly zero in on a root cause. They were flying blind — and they’d been flying blind so long they’d forgotten what visibility even looked like.

And that’s why the reaction was the same each time. Abort. Rollback. “Afterward, we’ll figure out what’s wrong.”

The irony is that both failures had quick fixes. Each time, queries weren’t reaching a service because the wrong port was configured. Instead of changing the port and continuing, the team burned hours — even days — backtracking, diagnosing and trying again.

That experience crystallized something I’d been feeling for decades: observability isn’t a tool you add. It’s an architectural decision you make — or don’t. And the cost of not making it compounds in ways that are genuinely difficult to measure, precisely because you don’t have the instrumentation to measure them.

The “flying blind penalty”

In the old days — before cloud, before microservices, before systems became truly distributed — opacity was tolerable. You had a monolith on a few servers. If something broke, you SSH’d in, looked at logs, poked around. It might take hours, but there wasn’t that much territory to cover.

Modern systems don’t afford that luxury. A single user request could traverse dozens of services, touching databases, message queues, caches and external APIs along the way. Just today I participated in a sprint review discussing a database architecture that was 29 views deep. A latency spike could originate in any one of those — or in the invisible interplay between the services that call them.

Without observability, you’re operating on instinct and prayer.

Here’s what that looks like in practice. A bug ships. It doesn’t manifest right away — it surfaces under load, or in a specific user segment, or under certain timing conditions. By the time anyone notices, users are already suffering. Now the team has to diagnose the problem. If they’re lucky, someone complains and provides enough detail to be helpful. They narrow down which service is involved, reproduce the issue, trace the flow of data through the system, find the root cause, fix it, deploy and verify.

The whole time, sweating over how many more customers are running into the problem.

Every single step without observability compounds in difficulty. You’re not just debugging — you’re doing archaeology at midnight, piecing together behavior from scattered artifacts across dozens of services.

With observability designed in from the start, you proactively see which service is degraded. You can drill into specific request traces. You can correlate logs across service boundaries. Mean time to detect and repair drops from hours to minutes — even becoming predictive.2

DORA.dev research bears this out. Elite-performing teams — the ones deploying multiple times per day with sub-hour recovery times — aren’t elite because they have better engineers. They’re elite because visibility closes a critical gap: it’s a prerequisite for every other capability DORA measures. They can detect and diagnose problems before those problems cascade. Their systems are instrumented to answer questions the team hasn’t even thought to ask yet.3

What matters is how you think about the problem

Monitoring and observability are not the same thing.

Monitoring answers questions you anticipated. You watch CPU utilization, disk space, error rates. You set thresholds and alert when they’re exceeded. “Is the database up? Is the error rate spiking?” These are known questions about known conditions.

Observability answers questions you haven’t thought of yet. It’s the ability to interrogate your system’s behavior without adding new instrumentation every time you have a new question. “Why is this specific user’s checkout so slow today when it was fine yesterday? What changed in the request path between those two deployments?”

Charity Majors — co-founder of Honeycomb and one of the sharpest voices on this topic — frames it as the difference between known-unknowns and unknown-unknowns. Monitoring handles the known-unknowns: the things you anticipated might go wrong. Observability handles the unknown-unknowns: the things you couldn’t have predicted. It empowers you to follow the breadcrumbs wherever they lead.4

Monitoring is reactive. Observability is generative — it creates the conditions for discovery.

Daniela Miao, CTO of Momento, put it well in a Dev Interrupted episode: “By giving more visibility into what’s happening with your software, your application, you actually do enable development to be faster as well. It’s not just insurance for when something goes wrong.”5

That’s critical. Most teams think of observability as insurance — you invest in it to cushion the blow when things break. But the real dividend is velocity. Teams with strong observability ship faster because they can see the impact of their changes in real time. They don’t have to guess whether a deployment worked or a tweak to a query ran faster. They know.

Why we don’t instrument on day one

Early stage. You’re shipping fast. Three or four services, maybe. Errors go to a text file. You’re small enough that you can hold the whole system in your head. Observability feels like overkill. “It’s in the logs, just read ‘em.”

Growth. Suddenly there are ten services, and logs are scattered across different servers. You can’t correlate them anymore. Someone says “we should get a logging tool.” You pick one — Datadog, Splunk, whatever — and bolt it on. The instrumentation is inconsistent. Some services are covered, others aren’t. Metrics are sparse and don’t tell a coherent story. You’re reading fragmented logs from disparate systems and trying to correlate events in your head. It’s starting to fall apart.

Pain. Now you have fifty services. Traffic has increased tenfold. Something breaks, and you’re still flying blind — not because you lack tools, but because the observability layer wasn’t designed. It was stitched on, piecemeal, after the fact. Different teams instrumented different things using different conventions. Tracing is incomplete. Metrics don’t correlate with logs. You’re paying more and more for tools that don’t give you the insight you need.

Here’s what’s easy to miss: that team invested in observability. They bought the tools. They checked the box. But because every investment was reactive — a logging platform when logs got painful, a metrics dashboard when someone asked for one, a tracing system when debugging got unbearable — they ended up with three separate systems that don’t talk to each other. Logs in one silo, metrics in another, traces in a third. Every decision about what to capture was made at write time, under pressure, for a specific fire they were fighting that week.

Charity Majors, co-founder of Honeycomb, calls this Observability 1.0: many pillars, many sources of truth, many tools. It’s what you get by default when you bolt observability on after the fact. The fragmentation isn’t a failure of effort — it’s a consequence of timing. You can’t design a unified telemetry model when you’re patching gaps reactively.6

And it matters — because a fragmented model can answer the questions you anticipated (is the database up? is the error rate spiking?) but collapses when you need to ask something new. “Why are checkouts slow today when they were fine yesterday? What changed between those two deployments?” Those questions require slicing across dimensions you didn’t pre-define. They require the kind of open-ended interrogation that siloed tools simply can’t support.

Insight requires design. That’s the part teams miss.

Why does this happen? Because observability isn’t a feature. It’s not something you built first, it’s what got built last. It doesn’t generate revenue — at least not visibly. It feels expensive up front. And teams, under pressure to deliver, don’t demand the time they need for planning a solid foundation.

I’ve written about the same false economy with security and testing. The things that feel expensive to build at the start turn out to be astronomically more expensive to retrofit.7

Why it has to be architectural

You can’t add observability with a library import. You can add logging and metrics — that’s the easy part. But genuine observability, the kind that lets you answer arbitrary questions about system behavior, requires architectural forethought.

Consider what’s involved:

1. How do requests get identified as they flow across boundaries? Trace IDs need to propagate through HTTP headers, message queue metadata, database session context — everywhere a request touches. If you don’t design this in, you can’t correlate activity across services.

2. What events merit recording? Not every line of code should produce a log. That’s noise. But the meaningful transitions — state changes, boundary crossings between services, external calls and their response characteristics — these are the signals that matter.

3. What metrics actually reflect system health? Not just CPU and memory, but application-level indicators. How long do payment transactions take? What’s the cache hit ratio? What’s the queue depth trend? These are the metrics that tell you whether your system is behaving — or trending toward overload.

4. How does context propagate? When a request carries metadata — a customer ID, a feature flag, a request source — how does that context flow through the system in a way that key transitions are captured seamlessly?

This is the shape of how you structure services, how you handle requests, how you organize code. And capturing it after the fact is brutally hard.

For example: if you decide later that you want end-to-end tracing, you need to add trace ID generation at your entry points, thread those IDs through every service-to-service call, correlate logs using those IDs, and store traces in a specialized backend. That means touching every API, every RPC, every async boundary. If you’d built it in from the start, it’s a few lines of middleware.

Consider structured logging: if you’ve been writing freeform string logs for two years, migrating to structured JSON events means auditing every logging statement, retrofitting context and rewriting every query your team depends on. If you start with structured logs, it’s just how things work from day one.

It’s not expensive when you build it in. It’s a handful of decisions, many of which improve things by introducing consistency. OpenTelemetry standardizes the plumbing — trace propagation, metric collection, structured logging — across languages and frameworks. You focus on what to instrument, not how to propagate context.8

But retrofitting: that’s an ordeal. Orders of magnitude more expensive — in engineering time, in tool sprawl and in the debugging sessions that never needed to happen.

What to actually build

Here’s how to do it right — whether you’re making the investment in an old project or starting fresh. The key is to think in events, not pillars.

Start with events, not tools

The most consequential decision you’ll make is how you model your telemetry. Do you emit three separate streams — logs here, metrics there, traces somewhere else — and try to correlate them after the fact? Or do you start with a single primitive: one structured event per unit of work?

The event-first approach works like this: when a request enters your system, you begin composing a structured event — a wide, context-rich record that accumulates everything meaningful about that request as it flows through your services. By the time the request completes, the event carries the trace ID, service name, duration, user ID, feature flags, error state, dependency latencies, cache hits, queue depths — whatever context matters.

You store that event once. From that single store, you can derive everything else: aggregate events over time and you get metrics; filter and read individual events and you get logs; stitch events together by trace ID and you get traces. One source of truth, many views.

This is what Charity Majors calls Observability 2.0 — and it’s the architecture that actually delivers on the promise of answering questions you haven’t thought of yet. Because the events are wide (carrying dozens or hundreds of fields), you can slice by any dimension after the fact. You didn’t have to anticipate the question at write time. You just ask it: “show me all events where cache_hit=false and region=eu-west and latency_p99 > 2s.” If those fields are in the event, you get your answer.

Compare that to the three-pillars approach, where you'd need to have predefined that specific metric, logged that specific combination, and hoped your tracing tool captured the right spans. Three tools, three queries, three partial answers — if you’re lucky.

The event model is the architectural decision. Everything else follows from it.

Establish a trace ID convention

Every request gets a unique identifier the moment it enters your system. That ID propagates through every downstream call — HTTP headers, message metadata, database session context. Every event includes it.

This is foundational. Without it, you cannot correlate activity across service boundaries. With it, a single ID connects every event across every service for a given request. It’s what lets you stitch individual events into a complete trace — the full journey of a request through your system.

OpenTelemetry handles those mechanics gracefully. Use it. It’s language-agnostic, framework-aware and it solves the propagation problem so you don’t have to.

Enrich your events at every boundary

Not every line of code should produce telemetry. That’s noise. The moments that matter are the boundaries and transitions — the points where a request crosses from one domain to another or where state changes meaningfully:

1. Request entry — what arrived, who initiated it, what context it carried.

2. External calls — what you called, what came back, how long it took.

3. State changes — meaningful transitions: a payment processed, a queue depth shifted, a circuit breaker tripped.

4. Errors — always, with full context. Stack traces, request metadata, the works.

At each of these boundaries, you’re not firing off a disconnected log line — you’re enriching the event that represents this unit of work. Add fields: payment_gateway_latency_ms=302, cache_hit=false, feature_flag_new_checkout=true. The event gets wider and richer as the request progresses. When the request completes, it captures the full story.

Make every field structured and discrete. Not a freeform string like “User 12345 logged in from 192.168.1.1” — but named fields: user_id=12345, source_ip=192.168.1.1, session_id=abc, duration_ms=47. Structured fields are queryable. “Show me all login failures for user 12345 in the last hour” becomes a single query instead of a regex excavation.

Derive your metrics from the same data

Once you have rich, structured events flowing through a single store, metrics become a view — not a separate system. You aggregate events to spot trends:

Read more

You’re working on an automotive assembly line. Your job is to install the steering column assembly and steering wheel. The next car rolls down the line. You grab the parts and lean in, tools in hand and try to fit the column. It won’t fit. Something’s off.

You look closely and realize the mounting point isn’t where it should be — it’s offset by a few inches. You figure, you’ll try anyway. No good. Worse, the chassis is aluminum and your steel column will cause galvanic corrosion on contact. You think to yourself — maybe this car is supposed to have something else entirely — a joystick, perhaps? You’re going to need time to research the platform and figure out what belongs here. But you’re out of time. The line grinds to a halt. Delivery stops. Something unforeseen has surfaced, and you need time to research the right solution.

That’s not how automotive assembly lines work. But it is how software works. Building software is about research, discovery and development.

Read more

Introduction

This activity is about making sure we’ve taken the time to think about happy paths as well as failure scenarios. It’s about adding fidelity to our use cases so that our finished product is a quality product — one that our customers will enjoy using.

As part of that process, we’ll explore variations on the scenarios in our design. For example, we’ve defined a scenario to make a purchase — but what about when the credit authorization fails? Or the product isn’t available in the warehouse? Or our customer adds a duplicate product to the shopping cart? What should we do if the customer abandons their shopping cart?

Working with your team to think through these scenarios — poking each scenario, prodding to see if it stands up under pressure, adding new paths if it doesn’t — will help ensure our design is solid.

Building better specifications

This activity isn’t just about testing our assumptions — it’s about creating better designs. We want to be sure we’ve thought about how code will be used before we build it. This outside-in pressure tends to produce interfaces that are cleaner, more focused and easier to consume, as well as code that is more reliable.

Read more

The incident

On June 4, 1996, the maiden flight of the European Space Agency's Ariane 5 rocket ended in catastrophic failure. About 40 seconds into its flight sequence, at an altitude of about 3,700 meters, the launcher veered off its flight path, broke up and exploded.1

The cost for this failure was estimated to be $370 million. The rocket was carrying four satellites intended to study the earth’s magnetosphere. Because of the payload’s destruction, building new satellites delayed scientific research into the magnetosphere for four more years.23

An elaborate and careful investigation into the cause of the failure was conducted.

While several factors led to a cascade of events, it came down to a data conversion error — an error that would have been caught in testing.

Unfortunately, the internal SRI software exception that caused the problem was never tested. Or, more specifically, it wasn’t tested in the Ariane 5 configuration. The failure resulted because Ariane 4-era software that used 16-bit signed integers had been plugged into a more modern system — a system that used 64-bit floating point data. The older Ariane 4 SRI software couldn’t cope with the new data format.

That’s the technical reason.

There’s a more important lesson here. The real reason for the failure was human error. Somewhere along the line, a decision was made: the decision that the Ariane 4 SRI module didn’t need to be retested in the new Ariane 5 configuration.

The investigatory board pointed to an underlying theme in Ariane 5 development as a root cause, that theme being, “that software should be considered correct until it is shown to be at fault.” The board took another view. Their belief was, “that software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct.”

Systems must be tested to prove correct behavior. We cannot simply assume they are correct.

Preventable failure

The key takeaways from the Ariane 5 disaster are:

1. We cannot rely on software to be reliable unless currently accepted best practices demonstrate that it is correct.

2. Continuous integration and testing for correctness is necessary — even when correctness has previously been proven.

The Ariane 5 case is widely taught in software engineering courses as the canonical example of how a simple, preventable defect — discovered late in the lifecycle (during operation) — can have catastrophic and costly consequences.

In fact, we know that the longer we wait to discover a defect, the more costly it is to fix. Research dating back to the 1970’s says we can fix a defect in design for nearly zero additional cost — or we can wait until production and spend north of 100 times the cost to fix it later (Boehm, 1979).

This exponential curve isn’t arbitrary. It reflects the reality that as software progresses through development stages, each defect becomes increasingly entangled with other code. Downstream systems begin to depend on it.

A requirements misunderstanding caught in design costs a conversation to fix. The same misunderstanding caught after integration means code rework, integration testing, release cycles, product updates and coordination across downstream dependencies. It can cascade into updating multiple products.

In concrete terms, that $25 defect can be fixed early, “for free,” or we can wait until later and spend $2,500 or more.

And there’s evidence that a 100X cost factor is by no means the upper limit. More recent research, in some cases, point to a factor of 1,500X cost to correct defects in operations. That $25 design defect just turned into a $37,500 cost to correct. There’s no upper limit — as the Ariane 5 team learned. (If you’re interested in reviewing more recent data there are references to NIST, NASA and Westland research in the footnotes).456

But there’s good news. We know how to prevent failures: By applying software quality assurance policies and practices — and pairing them with rigorous software testing.

Applying best practices

Test Driven Development (TDD) and Behavior Driven Development (BDD) are fundamental strategies to move defect detection left on the graph — ideally all the way to the design phase where cost is lowest.

Both practices create proofs that prevent defects. What’s more, they rely on executable specifications that prevent regression. Even after initial development, the test suite continues to catch defects early. A change that inadvertently breaks existing behavior is caught in seconds during the next test run — not discovered in production by users.

This ongoing protection is particularly valuable as systems age and original developers move on — the tests preserve institutional knowledge about intended behavior. Imagine how powerful it would have been if the Ariane 4 SRI module had included a continuously run regression test sequence.

The fundamental promise isn’t about testing — it’s about creating better designs. Both practices demand writing tests first. That forces thinking about how code will be used before thinking about how it will be built. This outside-in pressure tends to produce interfaces that are cleaner, more focused, and easier to consume, as well as code that is more reliable.

The deeper point

TDD and BDD aren’t really about testing at all. They’re about feedback loops and thinking clearly about intent before implementation. The tests are a byproduct — valuable, but secondary to the discipline of thoroughly articulating what you want before you build it.

Critics often attack the testing artifacts while missing this point entirely. It’s not always about going faster or whether every scenario needs to be test-driven. It’s whether your development process includes mechanisms that force clarity of thought and protect against regression. Both TDD and BDD are proven ways to achieve that.

TDD or BDD?

TDD catches implementation errors immediately. When you write a failing test before writing code, you discover mismatches between intent and implementation within minutes, not days or weeks.

The feedback loop is so tight that “debugging” often becomes unnecessary. When a test fails you know exactly what change caused it.

Compare this to traditional development where bugs lurk undetected until integration, verification or even user experience — by which point the original context has faded. Investigation becomes archaeological, time consuming and costly.

What TDD does well is clarify the boundaries of your system. Code that’s hard to test is often code with unclear responsibilities or excessive coupling. Even if you don’t achieve full coverage, attempting TDD reveals where your architecture needs attention.

But TDD has a limitation: it doesn’t “shift left” enough on our error discovery graph.

TDD is largely a technical exercise, focused on functional specification. As such it can get deep into the weeds of a feature — but may fail to rise to the level of product behavior. In other words, we might be applying our test-driven thinking to a conceptual design that is already defective.

That’s why I prefer BDD.

Behavior Driven Development

Behavior Driven Development is a collaborative approach that evolved from Test Driven Development. It bridges the gap between technical and business team members by expressing specifications as concrete examples, written in plain language everyone can understand (the “Given-When-Then” style, or Gherkin format).

Read more

A daily standup is a great idea — provided it’s done right.

A well-run standup:

1. Is a place to foster collaboration and mindshare.

2. Provides a forum to get help.

3. Makes sure the team doesn’t spiral.

Unfortunately, most standups that I join don’t do any of these things. Quite the opposite. These standups impede productive work, acting as a needless interruption that offers almost no value.

More often than not what I see is a team forced to stand together while each team member robotically itemizes their work since the last sprint. In some miserable cases, the “PM” and one or two team members hold the entire team hostage while a specific problem is diagnosed.

The standup stretches to 30 or 40 minutes. Your eight person team takes a cumulative five hour hit to their productivity.

Create value in your standups

If you Google “questions to ask at a standup” you’ll find the worst culprit sitting right at the top of the list: “what did you do since yesterday?”

Ban this question.

Almost just as bad is, “wha…

Read more

In the previous article I introduced a foundational architecture for our data pipeline, checking off several of our requirements. Here’s a quick recap:

1. We leveraged an internal cache for isolation to break our dependency on external data sources, improving our resiliency and scalability options.

2. We introduced the idea of an asynchronous message architecture, so we aren’t bound by lengthy, sequential processes.

3. We keep our annotations long-term with internal persistence so we don’t reprocess data when unnecessarily.

4. Source data as treated as an immutable source, thus protecting our data integrity.

5. Our visualization layer is responsive, running off our internal cache so that it’s independent from other components.

Here’s a sketch of the architecture as it stands now:

It’s a good start, but we aren’t finished.

There are still two requirements that don’t have a clear solution:

1. We want to be able to reprocess data, effectively “rewinding history” and applying new analytics. For instance, if we add a new analysis and insight to our pipeline, how are we going to go back and view that new analysis on old data?

2. Finally, we want to be able to model our insights — that is, visualize it in a presentation layer. But we haven’t talked about how that’s going to happen.

Once we tackle those last few requirements, we’ll fill in some details — such as, deciding on a cache technology to use. We’ll also take an end-to-end look at what we’ve designed to see if we missed something important.

Finishing our foundation

One of the most powerful and underused capabilities in data analytics is the ability to reprocess historical data with new logic. When you fix a bug in your analysis or refine a calculation, you often want to know: “what would the last three months have looked like with this new logic?”

Fortunately, our design thus far aligns pretty well with this goal.

Essentially, all we are saying we need to do is throw out information that is no longer accurate and then reprocess the source data and rebuild a new and improved picture of the world. We might do this for a couple of reasons:

1. We changed our analytics. Consequently, we need to throw out all of our old findings and recalculate our insights.

2. The source data changed. Perhaps there was a historical data update, invalidating our findings. Again, we want to throw out our findings and reprocess.

Looking at the problem from that perspective, the architectural answer is simple: we need a way to delete our annotations.

Once we wipe our annotations, we’ll need to make sure they are recalculated. There are different strategies we could use, but for the most part our options are:

1. Rebuild our annotations on-demand. This is fine if we’re OK with a bit of a delay while we wait for analytics to run. Depending on our visualization strategy we might be able to break the work up and trade one big delay for a few small delays.

2. Alternatively, rebuild our annotations preemptively. We might do it as soon as we realize there’s data to analyze, or perhaps we have some kind of schedule or external trigger. The point here is, we run the analysis before we think a user will want to visualize it.

We’ll add these capabilities to our architecture diagram. That means responding to both external data changes as well as internally sourced changes.

Rewinding history

To “rewind and reprocess,” we’ll just delete our annotations and then make sure there’s a mechanism to rebuild them. Here’s an example chain of events we’ll model in our architecture:

1. Administrative user invalidates some (or all) of the annotations we’ve stored in our cache:

   1. Optionally, if the data was invalidated because source data changed, the system loads fresh data from the external data source.

   2. The system deletes or otherwise invalidates the annotations in our cache.

   3. An event such as “analysis invalidated” (with some metadata, perhaps a date range) is emitted.

2. The analytics service is listening for events, and receives the “analysis invalidated” event:

   1. The event includes the date range of deleted data. The service checks the date range to see if annotations are missing from the cache.

   2. If the data is still missing, it runs the analysis and rebuilds the missing annotations.

   3. As annotations are rebuilt, they are written to the cache.

   4. An event such as “analysis completed” (with some metadata) is emitted.

3. Separately, a visualization service might be running where a user asks to display the visualization for some missing data:

   1. The system checks the cache, realizes annotations are still missing, and sends a “run analysis” message directly to the analytics service.

      1. The analytics service is already working on analyzing the data so it can ignore the message, or respond with an “analytics in process” message.

   2. The visualization presents some kind of “waiting for analysis” display while it listens for the “analysis completed” message. When it receives the message, it loads the annotations and displays them.

This is one way to handle a specific chain of events. We’ll need to handle other events and commands as well — such as an event to tell us external data has changed.

Since we’re talking about responding to events that happen, it makes sense to use event storming to model those events and commands. We can model “things that have happened” (the events) as yellow cards, and “what we want to happen” (the commands) as blue cards.

If you need a refresher, Chapter 2.3 Strategic event storming and Chapter 3.4 Tactical event storming provide in-depth explanations on modeling event driven architectures.

Here’s an updated architectural drawing showing how our events and commands interact with the system:

Persistent annotation cache

One of the things we haven’t decided yet is exactly how we’ll store our annotation data. There are quite a few options, and certainly more than one strategy that will work well.

In part two I hinted at using a relational database to store a copy of the external source data. This makes a lot of sense, but it isn’t the only option.

We could store all of our data in a persistent data store such as MongoDB, a noSQL document-oriented database. Document databases offer some fantastic search capabilities and, perhaps most interesting, don’t force us into a predefined schema — we can, essentially, attach anything as our annotations.

That’s a nifty idea, because we want flexibility to change our data analysis and come up with new data sets to store, freely associating them with source data.

We could also stick with a purely event-driven architecture. In this kind of system, we would emit annotations as events into a persistent queue (such as Kafka). The persistent queue becomes our source of truth — and annotations are just messages on the queue, like, “sprint X: total out-of-sprint hours: 500.”

With many options, it’s probably a good idea to decide on our business requirements:

1. Obviously, we want to cache the original source data, so we aren’t tightly coupled to an external system. (We know the external system is a relational database).

2. We want to easily create new annotation data sets. If we dream up a new calculation (maybe, “number of activities completed after hours on Fridays”), we want to be able to store it easily.

3. We know we’re using DataFrames for analysis. Storing our DataFrames without translating them into something else would be convenient (but not necessary).

4. Our annotation data should be persistent (to avoid unnecessary delays rerunning an analysis) — but, once a particular report or visualization is no longer being used, we probably don’t need to keep it around.

5. Finally, we’ll want to think about performance and wait times. We definitely don’t want users to experience a slow system.

The relational cache for source data still makes a lot of sense.

For our internal annotation data, something more flexible is desirable. We don’t want to be constrained by a relational schema, forced to update the schema every time we think up a new report or annotation.

While a noSQL solution seems like a good fit, it may be an unnecessary complication. It would afford long-term persistence, but unless we run into performance problems that long-term persistence may be pointless.

Designing for flexibility makes sense.

We’ll implement our internal annotation cache as a service. The service will:

1. Allow us to store annotations (data structures) in a persistent fashion (but, not necessarily permanently).

2. Provide a shared data service (allowing other services to leverage the shared data).

3. Give us scalability (we can boost processing power or memory, start up multiple copies, or upgrade to a more powerful data store technology).

4. Allow us to start simple, just storing annotation data in memory. A long-running service might give us all we need (after all, we’d only rebuild our annotations if we restart the server).

Let’s update our architecture diagram one more time:

Read more

Introduction

This activity creates fidelity in our engineering models — the diagrams and specifications that give us enough technical depth so we can proceed to coding.

Everything we’re going to build should be modeled. That might sound like of lot of work, but with the right drawings and specification style, it’s not — plus, our models should be reusable. For instance, once I’ve modeled my implementation of a circuit breaker, I don’t have to model it again. I can just use that circuit breaker design everywhere I need it.

The approach I’ll introduce here increases the fidelity of your technical designs using data inputs and outputs, paired with very clear and specific functional results. It builds on your existing engineering artifacts and further validates your value stream, making sure alignment with business objectives is maintained. We capture our control processes and at the same time enhance communication among those designing the solution.

One of the challenges with modeling is deciding on what’s effective yet not overly constraining or complex. We want to define some clear standards of practice and make sure the engineered product is fit for purpose. At the same time, teams should have the flexibility to take a different path if that different path is more fit for purpose.

The modeling approach we use in the Delivery Playbook meets those goals:

• We define engineering artifacts using two specific diagram types, neither of which are difficult to understand or use.

• Clear objectives are defined while giving teams enough freedom to follow a different path — so long as those objectives are met using whatever approach they choose to follow.

Read more

In the previous article I introduced data pipelines, highlighting their ubiquity and a few of the challenges you’ll face architecting a good data pipeline. Here’s a quick recap of those requirements:

1. Moving data efficiently. Pipelines are all about making sure the right data gets to the right place at the right time, in the right format. For many applications, like air traffic control or the stock market, timing is everything. The architecture has to scale, meeting demand without degrading — often with life-saving implications.

2. Data integrity. This is about ensuring the source data is what we think it is — incorruptible, unchanged as it moves through the pipeline. We have to make sure there’s no data loss, no translation errors, no dropped information. Yet, at the same time, pipelines are fundamentally about processing that data, deriving new insights from it.

3. Non-destructive annotation. Being able to add to the data in the pipeline efficiently and flexibly is key. This makes it possible to create new insights from the source data — but it implies a schema that can grow in unpredictable ways.

4. Rewinding history. Pipelines evolve, refining their algorithms and adding new processing steps. When we fix a bug or refine a calculation, we need to know, “what would the last three months have looked like with this new logic?” If we can’t reprocess the data we’ll never know if an inflection point in our analysis was due to a different calculation or new business conditions.

5. Surfacing discoveries and creating actionable insight. All of these steps need to culminate in actionable intelligence. Getting those pieces to work together, then correlating all that disparate data and presenting it clearly, logically, can present its own challenges. Does the data processing sequence matter? How does one element depend on another? What happens when your analysis depends on external data or fundamentally different data structures?

In this article, we’ll develop a foundational data pipeline architecture. I’ll approach the architecture iteratively — beginning with something simple, exploring whether it fits our purpose, and refining it. Our final architecture will provide a foundation that can grow into a scalable, powerful data pipeline platform.

For our use case, we’ll design an application that tracks team activity across projects in an organization. Our goal in tracking all this information is to deliver new business intelligence that gives useful, actionable insights into team efficiency across the organization. That’s intelligence we can use to remove bottlenecks, improve team productivity and velocity, eliminate wasteful process and more.

What are we measuring?

Before we get started, it’s a good idea to know what we’re trying to influence. We want to surface insights, not just tables and charts. The difference between data and intelligence is whether it impacts our decisions.

Actionable intelligence changes what you do next.

For our test case we want to develop a better understanding of team efficiency. There’s an existing body of work that covers this topic, and we’ll leverage it.

Our plan is pretty simple. Following the principles of a waste walk, we’ll have team members tell us whether their day-to-day activities are creating bottlenecks or inefficiencies or draining their effectiveness. They’ll do this by capturing activities and marking them as “in sprint” or “out of sprint.” (If you aren’t familiar with those terms, take a look at part one of this series).

We’re going to assume this data is available — perhaps coming from a JIRA dashboard or other time tracking application. I’ve actually built a mobile-friendly application that does this. It’s part of a waste walk analytics platform in use today. For our purposes, what matters is the data — not the mobile app.1

The data schema is straight-forward. Team members are Users, they track their individual Activities, and each Activity belongs to a Sprint.

If we diagram out the basic data model, it would look like this:

As you can see, Teams are also modeled (so we can manage sprints and their team members as a group), and every activity has an Activity Stereotype. These stereotypes are basically the kinds of activities team members can track in the system. They can’t just record a random event; instead, they pick an activity from a list of “stereotypes” like “meeting,” “writing code” or “story refinement.”

We’ll take a look at the specific fields in a little while. For now, we have a conceptual understanding of the data — that’s what we need to move forward.

Data versus insight

Given the data we have available, we need to ask ourselves: what kind of insights can we derive from the source data? How might those insights change our behavior?

Here are a few ideas:

1. If team members spend too much time doing activities that could be automated, we’d want to prioritize automation — letting those team members get back to more important work.

2. If a lot of time goes into investigating defects, we would probably want to know about it so we could investigate why we have so many defects (and fix the root cause).

3. It would also be informative to know when the team works on fixing defects; is it after the product ships, early in design, or somewhere in-between? Early defect correction is much better than late defect correction.

4. A lot of planning late in the product lifecycle — for instance, planning after a sprint has started — might indicate opportunities to improve coordination or design process. Late-stage planning often creates bottlenecks for team members.

5. If individuals feel their time in sprint ceremonies and meetings is not productive, we should probably investigate why they feel that way — especially if it’s a lot of time.

6. If we knew how many different things team members had to juggle, it might influence how we prioritize work and focus time for the team.

7. Probably most important: we want to know, across the organization, exactly how much time is going into inefficient or wasteful activity. A small amount is fine, but in aggregate, the time and effort lost could be costing us more than we realize.

I’ve done these kinds of analyses for many teams — and the insights are almost always transformative. Just one example: one team found they were spending close to 23% of their time on activities that could be entirely automated. The effort was spread out, of course — an hour here, another there. At an individual level, it was about a day’s effort each week, tweaking manual deployments, doing environment tuning, “sanity testing” in staging and production.

Now scale it up. For a team of eight, that’s 150 hours per sprint. Stated another way, by investing in automation, the team would get back 150 productive hours every single sprint and their quality of life would be improved at the same time. Knowing those numbers, prioritizing Infrastructure as Code (“IaC”) for deployments, automating those repetitive “sanity tests” — suddenly makes a lot of sense.

Transforming raw data into insight

We know what data is available to us. But how do we transform it into actionable intelligence we can use to create better outcomes?

The raw data doesn’t really get us there. That’s where the data pipeline comes in — by transforming that data into something we can use:

1. First we need to “intake” our source data. Once we can access it, we can analyze it, of course.

2. Then, we transform the data into what we need. We aren’t planning anything overly complicated (at least, not yet). Here’s the gist of what we need to do:

   1. Separate “in sprint” versus “out of sprint” activity. We’ll be focusing on the out of sprint activities or, in Lean Engineering terms, the “non-value add” effort. This “non-value add” work is where we’ll find our biggest gains.

   2. Summarize data about activities. For example, we’ll need a way to add up all the activities that could be automated. That tells us how much effort is spent here.

   3. We also want to be able to count up and somehow tally the time between activities (the “context switching time”) for each team member — probably as some kind of aggregate or average number.

3. Of course, we’ll want to visualize the results.

That all seems very straight forward. The first logical step is to think about how we’re getting our source data.

Start with the source data

The first thing we need is our source data. Fortunately for us, it’s readily available and easy to access.

Our tracking application stores all this activity data in a SQL database, using a traditional relational format. That makes a lot of sense — it’s a relatively static data structure, and the relationships between teams, users, sprints, and activities are clearly hierarchical.

One approach is to just read the data directly from the SQL database. This would be the easiest solution. Once we read the data, we can perform our calculations, add in the context switching data, and display it:

This does raise a few interesting questions, though:

1. What happens if the source data changes in some way or if we can’t access the SQL database when we want to display our data?

2. How, exactly, are we going to store the results of the calculations and annotations?

3. Does this mean that every time we display the results we have to run through the entire pipeline, reading the data, performing calculations and then displaying our findings?

4. How does constantly accessing an external database impact our performance?

Data integrity and availability

We want our application to be highly available and reliable.

The problem is, we’ve created a dependency on an external system — the SQL database. If the external system changes or just isn’t available when we need it, we’ve got a problem. It would break our data pipeline. We need a solution that ensures both the integrity of the data as well as its availability.

The simplest solution is to just copy the data and keep a local cache of it. Whenever the source data is updated, we can update our local cache. If something goes wrong — for example, the data format changes or we can’t contact the SQL database — we can keep using our local cache until we fix the problem. That gives us high availability.

This also gives us a degree of certainty about the data format.

While the external SQL database might change, our internal cache can potentially remain constant. For example, let’s say the external database is using an Integer data type for activity duration, but at some point they decide to change it and start using a Time type instead. Since our local cache is under our own control, we have a few options:

1. Update our local cache to use Time as well, and change our pipeline code accordingly.

2. Convert the incoming data from Time to an Integer and store that. Leave our code as-is.

3. Decide to use a completely different type, like Duration, in our internal cache and make similar changes to our code.

Our strategy would depend on the situation — but we have options. We’ll examine a few different ones in a bit, but for now we’ll just note the internal cache in our architecture.

Adding a cache to the architecture might look something like this:

I’ve added an “ACL,” or “Anti-Corruption Layer,” to the intake process. ACLs are used to protect the data integrity of our domain — in other words, we don’t control the external SQL database but we do control our internal cache. That cache can, for instance, use a Duration to represent the passage of time regardless of how the SQL database represents time. The ACL is where we translate from the external format to our internal, domain-specific format.

It gives us a nice layer of isolation from external change. Generally speaking, creating isolation between services is a great strategy for improving reliability (as well as scalability — more on that later).

Benefits of isolation

I pointed out that by copying external data into our own system we are creating isolation. This is an important architecture topic for a few reasons:

1. Isolation means we are not inexorably dependent on an external system. If that external SQL database goes down, or the data changes, we can still function.

2. We also get architectural independence. If we decide to change the structure of our internal data cache, we can. It’s under our own control.

3. Being in control of our own data architecture also means we’re in control of our scalability. If our data analysis product becomes the hottest thing since sliced bread, we can scale up. We might add more servers to our cache, or we might evolve our architecture toward CQRS patterns — we’re in control, and it’s all hidden behind our product API.

From this perspective, copying data is a good idea. We need to remember there’s an external source of truth. But we remain in control of when we run data intake, how we change the data on the way in, and how our own data is structured.

Powerful, non-destructive annotation

Our original requirements include making sure that we maintain our source data integrity. The best possible approach in that regard requires two things.

First, we have to capture every record, in its entirety. That ensures we won’t miss anything important (including the possibility that something becomes important later, even though we thought it wasn’t important at first).

Second, we can’t change what we capture — we can only add to it. This means our analysis and data annotation can’t alter the source data — it’s immutable. We have to find a way to piggy-back or pair our new data with the raw source data.

We’ll leverage that in our code. Let’s say, for instance, you wanted to find out how many seconds of “out of sprint” work each user tracked in a sprint and you also want to make sure paid time off is not included in that number. Here’s a very simple example, using a snippet of Explorer code:

data
|> DF.group_by([“user_id”])
|> DF.filter(stereotype_code != "pto")
|> DF.summarise(out_of_seconds_sum: sum(out_of_sprint_seconds))

Data is just an array of raw data records. The group_by line groups all records with their respective team member. Then, with filter we make sure paid time off is not being included. And finally, we sum up all the out_of_sprint_seconds and put the result in a new variable, out_of_seconds_sum.

Two important things to note here: first, we haven’t altered the source data in any way. Second, we’ve annotated our data set by associating that new variable, out_of_sprint_sum, with our DataFrame.

We’ll explore this in depth soon. For now, I just wanted to give a quick taste of what we can do and explain why it’s valuable.

Tackling data analysis

We’ve decided how we’ll get the data, but there’s more work to do. How are we going to perform our calculations and store the results?

Read more

These past few months were very busy. I had two projects compounded on top of each other — one, a client modernization effort; the other, a new release of an analytics tool we use with our customers. It was more than I like to take on at one time and, yes, I got a bit burned out.

Which is a good reminder — prioritize your well being!

Work will always be there. Life will pass you by.

Recharge in Malaga

After getting all the work out of the way, we took some time to recharge: a few weeks off in Malaga.

Read more

Why data pipelines are interesting

A data pipeline moves data to a destination, transforming it along the way, turning it into something useful.

It’s like a factory assembly line. But, imagine you’re assembling data. Raw data is your source material entering at one end. It goes through various processes and comes out the other end as finished product (clean, structured data ready to use).

For instance, air traffic control systems process data from multiple radar stations, transponders, weather sensors, and flight plan databases to build a real-time picture of airspace. A picture humans can interpret in a split second. An air traffic control system:

• Ingests position data from hundreds of aircraft every few seconds.

• Correlates radar returns with flight plans and transponder signals.

• Filters out noise, birds and ground clutter.

• Predicts flight paths and potential conflicts.

• Integrates weather data to route around storms.

• Handles sensor failures gracefully — if one source goes down, seamlessly switching to backup sources.

Controllers need accurate data within seconds, and failures can have catastrophic consequences. They need real-time data, rather than batches, with real-time redundancy and validation.

Data pipelines are fundamental to modern data infrastructure. A good pipeline makes sure the right data gets to the right place at the right time, in the right format.

If you think about it — that’s the foundation of just about everything in our data-driven modern society.

When it comes to projects that will evolve your mastery over data management, building a data pipeline is one of the most challenging and rewarding. That’s why, in this multi-part series, I wanted to do a deep dive into a data pipeline — including design, architecture and code. In this series, we’ll get hands-on and explore how to build an app that analyzes complex streams of data.

Creating actionable intelligence from data

Data pipelines start with the raw, source data — from wearables, instrumentation, monitors, databases, mobile devices. An Airbus A380 is fitted with up to 25,000 individual sensors. Each one of those is sending raw data that may need to be acted upon quickly. That raw data goes through different processes — cleaning, filtering, combining, reformatting — before landing in its final destination, which might be an instrument, display, data warehouse, analytics tool, or application.

Clearly, copying and interpreting all that data by hand is impossible. Instead, the data pipeline does it automatically, often as a real-time feed that transforms information on the fly. It ensures data moves reliably and consistently.

Raw data is often messy or in the wrong format. The pipeline transforms it into something useful — removing duplicates, standardizing formats, calculating metrics, or joining information from multiple sources, as with the air traffic control example.

Often, data pipelines are event driven. This just means they consume a sequence of events from the real world — some fact that already happened — then turn it into actionable intelligence.

“Actionable intelligence” just means some kind of information you can use. For instance, the fact that a stock price dropped could be turned into an automated buy order for that stock. An e-commerce company might have a pipeline that pulls daily sales data from its sales database, combines it with inventory data, calculates key metrics like revenue per product, and loads everything into a dashboard where executives can see performance at a glance.

There are a lot of moving pieces. Getting those pieces to work together, then correlating all that disparate data and presenting it clearly, logically, is hard. There are challenges to doing it well, which I’ll talk about below.

Preserving data integrity

One of the most important things a good data analysis tool — and data pipeline — needs is a guarantee of data integrity. In this case, data integrity means the data is exactly the data we think we are analyzing. This sounds obvious, but it’s surprisingly easy to get wrong.

Here are some common pitfalls, all of which we need to design against:

• Silent data loss: Information dropped during joins, filtering, or transformations without warning. For example, only capturing summary data or a subset of available data instead of recording (and keeping) everything.

• Type coercion errors: Numbers becoming strings, dates and time zones handled incorrectly, or precision lost in conversions.

• Encoding issues: Data corruption caused when moving between systems — something as simple as assuming an input field uses Latin characters instead of Unicode.

• Partial updates: Only processing some records, possibly ignoring “uninteresting data,” or failing to capture records that don’t process correctly.

• Implicit assumptions: Null handling, unexpected data volume, unexpected inputs or missing parameters, duplicate treatment, or sort order that changes behavior unexpectedly.

Good data pipelines preserve integrity.

In the second article in this series we’ll dive deep into data pipeline architecture and address each of these potential pitfalls.

Annotating data

We also need a way to evolve our data over time — that is, add to it in a way that we don’t end up breaking our data integrity rules.

Raw data rarely speaks for itself. A transaction table tells you what happened, but not why. A spike in errors shows something broke, but not what you were doing at the time. Good data tools let you add context and turn numbers into understanding.

Annotations work at multiple levels. At the record or event level, they document what the data represents, adding fidelity. A data analyst might add a “review” tag to what looks like an accidental double-entry, or a prescription drug regulatory pipeline might link together other drugs that treat similar symptoms.

At the macro level, annotation can build on a core data set, adding new information. Data from multiple sources can be combined to create a more complete picture. Hard-to-perform calculations can be run and attached to existing data, making it easy to extract more meaning later. Later, in our example use case, we’ll do this by annotating our data set for further exploration.

Annotations compound over time. Six months later, when someone investigates why something looks unusual, annotations lead to new institutional knowledge. Without them, context can be lost and every new analyst has to rediscover what happened.

Surfacing insight

Data analysis tools should surface insights, not just tables and charts. The difference between data and intelligence is whether it changes what you do next.

We need the right level of aggregation and annotation: too granular and patterns disappear in noise; too macro and you can’t identify root causes.

Start by focusing on decisions. Before building a dashboard, a report or a graph, ask what decision it will inform. If the answer is vague or simply, “it’s good to know,” it’s not actionable. Every result should express some clear, specific action.

Make the analysis do the work, not the reader. Instead of showing a table of conversion rates by channel, highlight which channels are underperforming, by how much and why. Instead of a trend line, show whether or not execution is deviating from planned goals and exactly how it changes your delivery plan.

Focus on metrics that lead to human action. Also, tune thresholds to minimize false positives. For example, it may be useful to “clean” a data set by removing anomalies and peculiar outliers. We’ll do that in our example app, effectively removing data that is suspicious and likely just “noise.”

The goal is to inform clear action. If it requires tribal knowledge, domain expertise, further exploration or twenty minutes of interpretation, it’s not yet actionable.

“Rewinding” history

One of the most powerful and underused capabilities in data analytics is the ability to reprocess historical data with new logic. When you fix a bug in your analysis or refine a calculation, you often want to know: “what would the last three months have looked like with this new logic?”

This is harder than it sounds. If your pipeline only processes new data, historical numbers are frozen with old limitations or bugs. You end up with charts where the methodology changes partway through, making trends impossible to interpret. “Revenue increased 20% in March,” but we’ll never know if it was growth or if it was due to a changed calculation.

Point-in-time correctness makes this possible: storing raw events or source data permanently, without changing it or pre-processing it. When you need to revise analysis, reprocess the historical data through the updated pipeline. The results of that processing becomes annotation to the data — ultimately, annotations that we can throw out and do again using a new approach. We look at data as an entire time series, a record of facts that happened in the past — and are, therefore, immutable.

But this means treating our analysis and transformation logic as code, not ad-hoc formulas or manual actions. The source data must never be altered — only our code, and the annotations and analysis it produces. That gives us a documented data model, not tribal knowledge about which column means what. The goal is reproducibility: given raw data from any date and a specific version of your data pipeline, you can create an exact analysis for any timeframe.

Building a data pipeline

I promised a deep dive into the design, architecture and code of a data pipeline. There are a lot of use cases — stock trade analysis, weather data, genomic research, geologic tremor research, the list is probably endless.

For this deep dive, we’ll explore an application that tracks team activity across all the projects an organization is working on. Our goal in tracking all this information is to deliver new business intelligence that gives useful, actionable insights into team efficiency across an entire organization.

We’ll build the application in stages, showing its evolution and developing a solid foundation for a data pipeline.

Using the app, team members will track their activities across various tasks, but unlike most “time trackers” they’ll be able to annotate their time as “in sprint” or “out of sprint.” These are defined as:

1. In sprint — essentially, “this is work that contributed direct value to my sprint goals.”

2. Out of sprint — this represents everything else, essentially, “this is work that I had to do, but it didn’t contribute value directly to my goals — it may have even slowed me down.”

We’ll scale this up across an entire organization. For a large organization, that means hundreds of teams, thousands of team members and tens of thousands of individual tasks. It’s a sizeable volume of data to work with, but in terms of what data pipelines are capable of it’s still pretty small (think about the volume of data flowing through the stock market, in comparison).

We’ll need a way for team members to capture day-to-day work, quickly and easily. This will become our data source, pushing data into the pipeline in real-time:

All the activities will flow into our data pipeline. We’ll capture those individual activities, but we need to look at the data in aggregate — from an organizational point of view. This means being able to annotate and analyze the data from different perspectives.

Looking at the data as a stream of events over time, we can “slice and dice” it. We can also compose the data with other analyses, all to produce actionable intelligence. That means being able to aggregate the data and run on-the-fly analysis across different segments of data (for example, being able to take a “slice” of data across four teams out of the entire organization):

We also want to perform detailed analysis. In our activity tracking app, we’ll show time the team spent on manual activity that could have been completely automated — letting the humans on the team spend their brain power on more valuable tasks:

As we discussed above, a good analytics platform also supports annotation — that is, being able to combine new data with existing data.

In our case, we’ll calculate how much time is lost to switching from one task to another. We’ll combine that analysis with pre-existing data to create new insights. Here we show the impact of giving the team more focus time:

Summarizing our goals

From a business perspective what we want to achieve with the activity tracking app doesn’t seem complicated:

1. Give team members a place to easily and quickly record their activities (what they did, how long it took).

2. Let them tag work that falls “outside” the sprint — representing work that, while potentially necessary, didn’t really contribute direct sprint value or move their goals forward.

3. Aggregate data across the entire organization to discover meaningful trends.

4. Expose action that we can take — action that will have an immediate impact on efficiency, productivity — improving “ways of working” across teams.

This is something that most organizations are working on — in fact, most are working on efficiency and productivity goals right now.

But the reality is, it’s complicated. There are competing concerns, unclear cross-organizational goals, a lot of noise to filter out, activity that may be necessary but might not seem necessary. How do we accurately analyze and filter it down into clear, actionable intelligence?

There are solutions. Ideas built on decades of rich, scientific research and business process evolution. Perhaps most famously, Toyota empowered their employees to do whatever it took to streamline automotive production lines. Toyota’s concepts regarding just in time production are deeply embedded in the foundations of Lean Manufacturing. Taiichi Ohno is widely regarded as having pioneered effective techniques for eliminating “wasteful action” in day-to-day work (Muda, a Japanese word meaning “wasteful”).12

Our application will build on that, gathering activity data and turning it into an actionable, live stream of insight about organizational health.

We’ll need to stay true to the critical concerns outlined in this article: preserve data integrity, focus on annotation not transformation, build a system that always delivers actionable intelligence and, perhaps most important, make it possible to “rewind and replay.” As our algorithms evolve and mature, we’ll want those insights to be applied retroactively.

Coming next

We’ve outlined our requirements above. Next up is defining the architecture, the “blueprinting” phase. That’ll be the topic of the next article in this series. By the end of the third article, we’ll have a concrete roadmap as well as an architectural foundation — a steel thread that we can build on to deliver a proof of concept in one or two sprints worth of work.

If you’re following along in the Delivery Playbook, we’ve just about finished our mobilization phase. We’ve got a product vision and we know what the product’s capabilities and functions will be.

If you find Customer Obsessed Engineering and the Delivery Playbook valuable, please share it with your friends and coworkers.

Share Customer Obsessed Engineering

Most teams are building on quicksand and calling it solid ground.

You’ve got 95% test coverage. Your CI pipeline is green. Your unit tests pass every time. You ship with confidence, and then — crash — production burns down because of an edge case nobody thought to test. The user entered a 37-character string instead of 35. Someone sent a negative timestamp. A list came back empty when your code expected at least one item.

Sound familiar? Here’s an uncomfortable truth: traditional unit testing is fundamentally inadequate for building reliable software.

Unit tests are like taking a single photograph of a mountain and claiming you understand the entire landscape. They capture one specific moment, one exact scenario, but they miss the infinite variations that real users will throw at your system. They’re missing the crevasses, caves, and sudden unexpected features. They’re necessary, but they’re not sufficient.

After decades of building and shipping software — from startup MVPs to mission-critical systems handling millions of transactions — I’ve learned that the teams who ship bulletproof software have one thing in common: They know that testing specific use cases is inadequate.

They don’t buy into the illusion of safety that unit testing offers. They use property-based testing.

Property-based testing doesn’t just find bugs. It finds entire categories of bugs you never knew existed. It’s the difference between checking that your bridge can hold one specific car versus proving it can fulfill its design parameters by handling a constant, unpredictable flow of traffic.

The fundamental flaw in how we think about testing

Let me paint you a picture. You’re building a user activity tracking system (let’s call it Tracky). Your traditional testing approach might look something like this:

test "creates activity with valid data" do
  user = insert(:user)
  sprint = insert(:sprint)
  
  assert {:ok, activity} = Activities.new_activity(
    sprint.id, 
    user.id, 
    ~D[2024-01-15], 
    ~T[02:30:00], 
    "Working on feature X"
  )
  
  assert activity.duration == ~T[02:30:00]
  assert activity.description != nil
end

This test passes. You ship. Then production explodes because:

• Someone enters a duration of 25 hours

• A date falls outside the sprint boundaries

• The description is 500 characters long and breaks your UI

• Someone passes a nil sprint_id

• The timestamp has microseconds that cause floating-point precision issues

Your unit test checked one path through the code. But software doesn’t operate in the controlled environment of your test suite. It operates in the chaos of production, where users do unexpected things, data comes in unexpected formats, and Murphy’s Law reigns supreme.

Instead of testing specific examples, we should be testing universal properties that must always hold true.

Property-based testing builds a foundation for quality by exercising all the behavioral boundaries of your code. This is exactly how we approach development using a “steel thread.” A steel thread creates an architecturally complete foundation by touching all the major components of a system.

The truth about test coverage

Here’s something that might challenge a few of your assumptions: high unit test coverage can actually make your software less reliable.

I’ve seen teams with 98% test coverage ship catastrophic bugs because they confused coverage with correctness. They tested their happy paths thoroughly but ignored the vast space of possible inputs that could break their system.

Property-based testing breaks through this limitation. Instead of measuring how much of your code is exercised, you’re measuring how much of your input space is validated. Instead of asking, “did we test this line of code?” you’re asking, “did we prove this business rule is always true?”

A single property test might only exercise 20% of your codebase but validate behavior across millions of potential inputs — including edge cases. Which approach gives you more confidence in production?

The takeaway here: coverage metrics optimize for the wrong thing. They optimize for testing effort rather than testing effectiveness.

What is a “property?”

A “property” is a universal law or attribute. Gravity is a property that dictates how objects are attracted to each other. Fluid dynamics define a set of properties about how fluids flow and interact. And, our Tracky application has certain properties, too:

1. Activities must fall inside the date boundary of a sprint.

2. An activity is limited in time (for our purposes, it’s a business rule that “activities must be less than 8 hours in duration).

3. Activities must be owned by a user, and that user can view, update, and delete only their own activities (not someone else’s activities).

4. Activities can, optionally, have a description.

Of course there’s more, but this is sufficient for the examples we’ll look at in this article.

These properties define how Tracky behaves; what you are allowed to do with it, and what you are not allowed to do. They are universal laws — in all conditions, an activity must fulfill each of these laws — and, very importantly, the system must gracefully prevent the violation of these laws.

What property-based testing actually does

We’re all familiar with unit testing and testing very specific “success criteria,” like I described above. The problem with that approach is the inherent limitation of testing a single path at a time. It’s an impossible burden to test every edge case using that approach.

Property based testing expands our test horizon. Instead of writing specific test cases, you define properties — universal truths about your system — and let the testing framework try to break them using hundreds or thousands of generated test cases.

In short, you write one generalized test scenario, and the system uses it to create thousands of potential test paths for you.

Let’s start with a simple example.

In this example I use a “generator” to seed my test with test data — if you aren’t familiar with generators, don’t worry, we’ll come back to them. For now, just understand that a generator creates an infinite, essentially random stream of test data. They are the powerhouse behind property based testing.

Let’s start by replacing my naive creates activity with valid data with a property-based test. Here’s how it might look:

property "activities must have reasonable durations and valid dates within sprint bounds" do
  check all sprint <- sprint_generator(),
            user <- user_generator(),
            duration <- valid_activity_duration_generator(),
            date <- member_of(Date.range(sprint.start, sprint.end)),
            description <- lorem_style_phrase_generator_or_empty() do

    result = Activities.new_activity(
      sprint.id,
      user.id,
      date,
      duration,
      description
    )

    # Property 1: Valid inputs should always succeed
    assert {:ok, activity} = result
    
    # Property 2: Output duration should match input duration
    assert activity.duration == duration
    
    # Property 3: Activity date must be within sprint bounds
    assert Date.compare(activity.date, sprint.start_date) != :lt
    assert Date.compare(activity.date, sprint.end_date) != :gt
    
    # Property 4: Description, sprint, and user must all match
    assert activity.description == description
    assert activity.sprint == sprint
    assert activity.user == user
  end
end

I’m using Elixir here, but that doesn’t matter. There are property-based test frameworks available for just about every language.1

At first glance, this might not seem to be much different. I’m still creating activities, and comparing the properties of those created activities with expected values.

But, this single property test does the work of thousands of individual unit tests. It generates valid sprints with random date ranges, creates users with different roles, generates durations up to 8 hours, picks dates within the sprint bounds, and creates descriptions of varying lengths. It does this many times, creating a myriad of combinations and edge cases that we would never think to code up by hand.

This means you’ll see a lot of combinations:

1. An administrator creating an activity with a minute-long duration.

2. Another user creating a 5-hour activity, mid-sprint, with a long description filled with emoji and Unicode characters.

3. Activities with nil, empty, very short, and very long descriptions with a variety of punctuation.

4. Activities that belong to sprints far in the future, or that started (and ended) a month ago.

And on it goes, with many, many combinations of users, sprints, dates, durations and descriptions.

And, it gets more powerful: With property testing we also test the boundaries. What happens when the duration is exactly 8 hours? What about when it’s one second over? What about when the date is exactly on the sprint start date? Or the end date? It’s easy to write negative scenario tests as well, just by introducing a generator that creates bad inputs.

The testing framework explores the entire input space systematically, looking for the edge cases and combinations of edge cases that will break your system in production.

The architecture of unbreakable software

Property-based testing forces you to think architecturally about your code. When you write properties, you’re not thinking about specific scenarios — you’re thinking about the fundamental invariants that must hold true regardless of input.

Thinking in terms of property-based testing makes architecture easier. We’re shifting away from, “let’s make this one specific test case succeed” and toward, “let’s define conditions that prove these fundamental truths are always true, no matter what we throw at it.”

Here are some of those “fundamental truths” property-based testing helps to verify:

1. Data integrity properties.

   1. Entities must always maintain appropriate relationships.

   2. Temporal properties (time and dates) must fall within suitable boundaries.

   3. Access and ownership must be part of the system.

Read more