Your team works hard. The hours are long, the standups are full and everyone looks busy — and yet the thing that should take a day takes three. Velocity feels slower than the effort warrants. Retros surface the same complaints again and again yet nothing moves. If you’ve started to suspect the team is just slow, stop.

It isn’t. The time isn’t evaporating. It’s leaking — escaping through gaps you walk past every day without seeing them. And here’s the part that matters: the gaps aren’t a mystery. There are exactly eight of them, and they were named decades ago, on the factory floors of postwar Japan.

When Dave Brailsford ran British Cycling and then founded Team Sky, he didn’t chase one breakthrough. He hunted one-percent improvements everywhere — the bikes, the riders’ sleep, even the way they washed their hands to dodge a cold before a race. Stack enough of them, the theory went, and a country that had never produced a Tour de France winner takes five of the next six. That’s the arithmetic James Clear made famous: get one percent better every day and you compound to thirty-seven times better in just one year. The deeper idea is older — kaizen, the culture of relentless small improvements that Toyota turned into an industrial empire, one daily fix at a time.12

The catch is that improvement is the work everyone defers. It’s the first thing cut when a deadline looms and the last thing scheduled when one doesn’t — a luxury we’ll get to once the “real” work is handled. That’s exactly backwards. Finding the next one percent is what makes next week cheaper than this one. And the fastest place to find it is wherever your time is leaking.

So before you can plug the leaks, you have to see them. Here they are — the eight ways work leaks time, the list Toyota’s engineers assembled and the rest of the world adopted. Lean remembers it with a mnemonic you won’t forget once you’ve heard it: DOWNTIME. And yes, it applies to software.3

Most teams already have words for some of this. They call it “process overhead” or “tech debt” or “scope creep” — phrases that stay vague on purpose, so everyone can nod without anyone having to fix anything. DOWNTIME is the vocabulary those euphemisms exist to replace. You can’t kill what you can’t name. Eight letters, eight leaks: defect, overproduction, waiting, non-utilized talent, transportation, inventory, motion and extra processing. Read that once and you’ll start seeing it everywhere — which is the entire point.

The ones you know too well

You won’t feel all eight equally. Most teams are dominated by two or three. Here are the ones that tend to draw blood.

Waiting. The pull request that sat for two days while its author moved on to something else. The deploy you watched spin while you refreshed a dashboard and pretended that counted as work. Nobody did anything wrong — which is exactly why waiting is invisible. The work just sat there, and so did you.

Defect — the rework tax. You build the feature and you build it well. Then the acceptance criteria turn out to have been wrong and you build it again. In software, rework is often the most expensive waste you carry, because it’s work you already paid for, now billed a second time. Every “wait, that’s not what we meant” is a withdrawal from an account you didn’t know was still open.4

Motion — death by a thousand switches. Context-switching across tools, the same manual clicks twenty times a day, the hand-rolled deploy sequence you’ve run a hundred times. The cost isn’t the click — it’s the recovery. Gloria Mark’s research clocked the average return from an interruption at twenty-three minutes. Get pulled away every few minutes and you never actually arrive at the work at all.5

Extra processing — the meeting that decided nothing. The ceremony that produces no decision. The low-risk change you tested five ways. The elegant abstraction built for a second use case that never showed up. All of it is real effort. None of it is progress. None of it adds real value.

The quieter four

The other four leak just as quietly — but they leak all the same.

Overproduction — building ahead of demand. The feature shipped before anyone asked for it, the configuration knobs added “just in case,” the speculative endpoints built for a use case that never arrives. It looks like output. It’s effort that earns nothing until someone needs it — and most of it, nobody ever does.

Transportation — the handoff tax. Every time work crosses a border — dev to QA, one team to another, a ticket bounced to a different queue — it stops and waits for someone on the far side to pick it up. The work itself doesn’t change. It just accrues delay at each frontier, like luggage transferring between flights. (This is the cousin of motion: motion is you moving between tools, transportation is the work moving between people.)

Inventory — the column that only grows. Half-finished features, branches waiting to merge, tickets piling up in a “ready for review” lane nobody clears. Unlike a warehouse, software inventory is invisible, so it accumulates without anyone flinching. Every item is capital you’ve already spent and haven’t earned back.

Non-utilized talent — the most expensive leak of all. Your strongest engineer triaging tickets a junior could clear. The designer who never designs because she’s stuck in status meetings. The architect rubber-stamping config changes. It’s the waste that doesn’t just cost hours — it’s the one that quietly walks people out the door.

Notice what just happened as you read those. You’ve been losing time to them for years. The instant they have names, they stop being the weather — just how things are around here — and become a list of things you can go after. That is the whole move. Naming waste is how you stop tolerating it.

Stop guessing, start measuring

Want to know which of the eight is actually costing you the most? Don’t argue about it in a retro — measure it. For one sprint, have everyone jot down where their time went and whether it moved the work forward. Two minutes a day. Add it up and look at the shape of it. That’s a waste walk, and it’s about as complicated as it sounds.6

What comes back will reorder your priorities on the spot. The bottleneck you were certain about turns out to be a rounding error. The “quick” thing nobody ever questioned turns out to be the biggest line on the page. The vague dread that “we’re just slow” snaps into focus as something you can actually attack — this many hours bled to rework, this many to manual steps a script should have saved us. You come in with a hunch. You walk out with a map that has an “X” on it and an action list.

What you actually get back

The obvious prize is the hours. But the hours aren’t what makes this worth doing. Look at the eight wastes again and notice what they share: every one of them is a part of the job you already hate. Rework, waiting, manual drudgery, meetings that end in nothing — that’s the misery. It isn’t the work you signed up for. It’s the sludge that collects around it.

So when you cut the waste, you’re not just going faster. You’re scraping away the worst parts of the week and leaving the work you actually wanted to do — the building, the solving, the craft. A team that kills its waste doesn’t just ship more. It spends its days on better things and gets to go home at a decent hour.

And you trade in something subtler: the shrug. “That’s just how it is here” is the most expensive sentence in software, because it ends the conversation before it starts. Naming and measuring waste ends the shrug instead. It hands the team back the knowledge that “the way things are” is a choice and not a fact — that the team gets a vote.

That’s the one percent. Not a heroic, quarter-long transformation — one leak found and plugged this sprint, then the next one. It’s how Brailsford’s riders went from also-rans to the top of the podium: not a single breakthrough, a hundred small ones. Your team can run the same play, and it starts with a single word for the thing that’s been slowing you down.

Try this tomorrow

You don’t need a platform, a budget or anyone’s permission to start. You need eight words and a little attention.

Tomorrow, just name things as they happen. The bug you’re fixing because the criteria were vague — defect. The forty minutes lost waiting on a deploy — waiting. The fourth manual step in a release you’ve run a hundred times — motion. Don’t fix anything yet. Just see it. By Friday the list will have stopped being a mnemonic and started being a habit — the habit of a team that’s about to get one percent better, and then keep going.

When that habit sticks, take it to the next level: run a waste walk. Give it a single sprint and you stop guessing entirely — the data names your biggest leak for you and hands you the business case to fix the ones that need leadership to move. I’ve written the whole playbook for it in Stop doing things that don’t make us faster, better, more focused.

Word of mouth is how this newsletter grows, and your support means everything.

Share Customer Obsessed Engineering

Just about every large pre-Internet company out there today has some sort of “digital transformation initiative,” but what most of them don’t realize is that the heart of this transformation is moving from project-mindset to product-mindset. — Marty Cagan, Founder, Partner, Product at SVPG

Forty-plus people, a chunk of a week blocked off, a cavernous meeting room commandeered. Sticky notes color-coded by team. Butcher paper taped along the walls. Somewhere near the front, a Release Train Engineer is walking the room with a Sharpie, drawing red strings between dependent features. By the end of day two, every team has its slice of a 10-week plan committed to the wall, and a senior leader is asking each team for a five-finger confidence vote.

I’ve watched that scene play out. I know what comes next. Six weeks from now, three of those red strings will have moved, two features will be quietly cut and one team will be rebuilding a thing nobody asked for. The plan that got committed is not the plan that ships. Everyone in the room knows. Nobody says it.

But the plan got a vote. The Release Train left the station. It’s now truth — until the next PI, when a slightly different plan will become the next truth. The change orders get signed and project budgets run over in the guise of “increased scope.”

That’s PI Planning. And I’m going to argue, carefully and with receipts, that what just happened in that room is waterfall on short cycle — and almost nothing about it is recognizably agile.

The premise we're not going to relitigate

I am not going to relitigate whether product teams outperform. I made the case in Product teams really do outperform: bringing the receipts. DORA, McKinsey, Pendo, Standish and Project Aristotle converge unambiguously. Small, durable, cross-functional teams that own a problem end-to-end deploy 182× more frequently, ship higher quality, redefine “done” around outcomes and drive four to five times faster revenue growth than the alternatives.1

Take that as given. The question for today is narrower: if that is what real agility looks like, where exactly does SAFe land on the map?

The short answer — the one I will spend the rest of this article defending — is that SAFe lands far closer to waterfall than its name implies. Not because its proponents are dishonest. Because its structural unit of planning, funding and authority is the Agile Release Train, not the team. And that single design choice quietly recreates every handoff, gate and silo agile was meant to eliminate.

Before I break the case, let me make it

I owe SAFe and the people who chose it a fair hearing — and a fair hearing means arguing with its strongest version, not its sales deck. So let me make SAFe's case the way its creator makes it.

Dean Leffingwell does not justify SAFe by appeal to certifications or board-deck optics. He grounds it in flow economics — specifically Don Reinertsen's Principles of Product Development Flow, a serious book that the continuous-delivery world draws on too. The argument lives in SAFe's seventh principle, “apply cadence, synchronize with cross-domain planning,” and it runs like this. “Solution development is an inherently uncertain process… This risk conflicts with the need for businesses to manage investments, track progress, and have enough confidence in future outcomes to plan and commit to a reasonable course of action.” The resolution Leffingwell offers is a, “safety zone, where enough uncertainty actually provides the freedom to pursue innovation, while sufficient confidence allows the business to operate.” Cadence and synchronization are how you buy that confidence without freezing the work. PI Planning, in this telling, is not a commitment ritual — it is a periodic re-baselining of the plan against the objective state of working systems, which is SAFe's fifth principle.2

Take that seriously. The coordination problem SAFe addresses is real. Eight teams shipping into the same product surface need to know about each other's plans. Two hundred teams across a portfolio need someone to decide the trade-offs. Lean Budgets — funding value streams instead of approving each project individually — are genuinely better than the alternative most enterprises were running before. Inspect & Adapt at the end of each PI is a retrospective at scale, and that’s not nothing. PI Planning, for all its flaws, beats no planning. At the team level, SAFe leaves Scrum or Kanban in place. The bottom-up practices are still there.

There is also a colder, political version of the case, and it deserves the same hearing. SAFe is, in many enterprises, the only model executive leadership will fund. The CFO can read it. The PMO can map their roles into it (that’s a key point; hold on to it). The risk officer can find their gates. There is a certification track, a vendor, a roadmap. For an organization that needs to tell a board, “we are being agile,” SAFe is the politically survivable option. The choice between SAFe and a Cagan-style product transformation is rarely the choice on the table. The actual choice — for many large organizations — is between SAFe and the project-funded, requirements-thrown-over-the-wall, six-month-release waterfall it replaced.

So the honest steelman has two layers. Intellectually, SAFe is an attempt to operationalize real flow economics at enterprise scale. Politically, it is the floor, not the ceiling — and the floor is higher than what came before. For organizations starting from heavy-handed waterfall, SAFe is an upgrade on both counts.

That is the strongest case I can make, and I built it from SAFe's own foundations. Now let me explain why it isn't enough — and why the upgrade comes at a steeper price than its buyers usually realize.

Five tells that betray the waterfall underneath

If you want to know whether something is waterfall, do not read its marketing. Read its structure. Waterfall has diagnostic attributes — sequential phases, big-batch commitments, multi-team handoffs, stage gates with formal approval, plan-driven coordination above team level — and any process that exhibits those attributes will behave like waterfall whether or not it carries the name.

What follows are five attributes that SAFe exhibits, with the practices that produce each. Some of these will look familiar from the structural critique Marty Cagan put on the record in 2018. Others are mine, drawn from watching the framework operate across enterprise programs. The attributes are independent. Any one of them is a yellow flag. Three of them is a diagnosis. Five of them is what we are looking at.

Tell #1: the Release Train is a waterfall, on rails

Start with the name. Agile Release Train. The naming is not accidental — it is defensive. Strip the adjective and you are left with release train, which is exactly what waterfall organizations have always called their integrated, scheduled, multi-team delivery cycles. Pull a release train out of a 1995 PMO and it would have looked nearly identical to a SAFe ART: 50 to 125 people working toward a synchronized release at a fixed cadence, a release manager coordinating dependencies, a portfolio manager approving the work and a steering committee voting on the plan. SAFe replaced the release manager with a Release Train Engineer, the portfolio manager with a Lean Portfolio Manager and the steering committee with a PI Planning ceremony. The roles got new titles. The meeting got a Sharpie. The cadence shrank from six months to ten weeks. The shape did not change.

Look at what an ART actually does. It locks 50 to 125 people into a shared 10-week commitment. It synchronizes their backlogs into a single Program Backlog. It coordinates their dependencies — through a Solution Train, when the ART is too small to contain the dependencies on its own. It runs ceremonies — PI Planning, ART Sync, Scrum of Scrums, System Demo, Inspect & Adapt — whose entire purpose is to manage the friction created by the ART's own existence. Every one of those ceremonies is overhead a real product team does not pay, because a real product team does not have a 50-to-125-person commitment to coordinate around in the first place.3

Marty Cagan put this plainly enough that I want to quote him directly. In Revenge of the PMO, he writes that, “the critical notion of a dedicated product team (aka squad) has been gutted. In SAFe, this concept of a true product team has been undermined and demoted, and the core concept is now a Program, which has a top-down model of a product manager, an architect, and a release train manager, and these people make all the key decisions, and then some number of engineering teams with a low-level product owner are assigned various parts to build.” That is a verbatim description of a feature team inside a project structure. SAFe just renamed the project a Program and the project manager an RTE.4

Dave Farley — co-author of Continuous Delivery and not a man with a Scrum axe to grind — reached the same place from the engineering side. Writing about the clients he had watched adopt SAFe, he was blunt: “the orgs that I have seen try SAFe look indistinguishable from the orgs that pay lip-service to Scrum to me. In both cases, they look like waterfall orgs to me.” His objection is structural, and it lands on the load-bearing artifact directly. He dislikes the Release Train itself — “I think that they are an anti-Continuous Integration step,” one that, “represent[s] a plateau that halts team's progress towards a more effective flow-based approach.” The author of the canonical book on continuous delivery looked at the Release Train and saw the thing standing between teams and flow.5

Pull the Release Train out and SAFe collapses. Keep it in and you have kept every structural property that makes waterfall what it is — synchronized big-batch commitments, multi-team handoffs, a coordinator above the team, a plan that hardens the moment the room votes on it.

Tell #2: cadence as commitment

A 10-week PI plan is functionally a specification.

SAFe will tell you it is a forecast — soft alignment, not a frozen commitment. Confidence votes are five-finger, not yes-or-no. Teams can replan. Deployment is not locked to the PI. The wording is all soft. Look at what actually happens, though. The room agrees on a set of features for the PI. Those features get pulled into the Program Backlog. Engineers start work against them. Dependencies get committed to other teams' plans. Stakeholders are told what is coming. Mid-PI, the cost of dropping a feature is not zero — it is the team's confidence vote, the dependent teams' rework, the stakeholder's roadmap update, the RTE's coordination tax, the System Demo's missing slot.

So features almost never get dropped mid-PI — the discovery that should happen during the increment rarely does, and when it surfaces a problem, the dependencies make dropping the feature more expensive than shipping it anyway. The plan, framed as soft, hardens into something tighter than waterfall ever was — because waterfall at least admitted to being a commitment. PI Planning insists it is not a commitment, then behaves like one, and punishes anyone who treats it as if it isn't.

Compare this to a real product team. A team starting Monday morning with a hypothesis runs the cheapest possible test by Friday. If the data says no, the hypothesis dies before week two. The team has lost five days. No roadmap vote, no red strings. The stakeholder isn’t told anything because there is nothing to tell. One of these is research. The other is a Gantt chart that happens to be 10 weeks wide instead of 10 months wide.

Royce warned about this in 1970 — and it is worth remembering, because the diagram he drew has been the most consequential misreading in software engineering history. In Managing the Development of Large Software Systems, Royce presented the now-familiar sequential flow of phases and then said in the same paper that an implementation plan keyed only to those steps was “risky and invites failure.” The bulk of his paper argues that any large software system needs to iterate — that every commitment to a plan should be tested against working software before it hardens. People extracted his diagrams and ignored his warnings. PI Planning ships Royce's exact failure mode under a different name, on a shorter cycle. The cadence being shorter is the entire thing that is “agile” about it. Strip the cadence and what is left is a stage-gated portfolio process.6

Tell #3: the decomposition pyramid

Epic → Feature → Story is requirements → design → implementation, in different fonts.

The Epic is approved upstream by Lean Portfolio Management before it lands at the ART. The Feature is decomposed by a Product Manager and a System Architect before it lands at the team. The team's Product Owner — a role title borrowed from Scrum and stripped of Scrum's authority — turns Features into Stories. Each step down the pyramid is a translation from one set of vocabulary to another. Each translation loses information. By the time a Story is in front of a developer, the original customer intent has been refracted through three layers of people who did not sit with the customer and who do not sit on the team that will build the thing.

Where, exactly, does product discovery happen?

Discovery — as Cagan, Torres and the discovery-coaching tradition mean it — is hypothesis-testing with real customers before commitment. In SAFe, that work happens before the Epic enters the funnel, in a separate process, performed by people who do not sit on the team that will build it. The feature team receives a Feature already shaped by a Product Manager, refined by a System Architect, approved by Lean Portfolio Management and committed to by 50 to 125 of their colleagues at PI Planning. The team's role is to decompose the Feature into Stories and execute. That is not discovery. That is a business analyst with new vocabulary.7

I am not the first person to point this out. Jeff Gothelf, the co-creator of Lean UX, put it bluntly:

“All the principles we've built into Lean UX don't seem to exist in SAFe.”

He went on to note that customer-centricity, evidence-based decision-making, experimentation and course correction, “are visibly absent from the SAFe conversation.” He is right because the conversation cannot include them. The framework has placed those activities upstream of the team, and made the team's slot in the value chain a unit of execution, not a unit of inquiry.8

The Delivery Playbook focuses on customer value as a core practice — through value stream engineering and customer-centric design. These are not optional — they are table stakes. SAFe cuts both out of the conversation.

Tell #4: the team-ownership ceiling

A SAFe team does not own a value stream. It owns its slice of an ART.

End-to-end ownership of the customer outcome — what separates product teams from feature teams — lives at the ART or Solution Train level. That is the line Receipts drew with Cagan's framing of product, feature and project teams. SAFe puts the team on the wrong side of that line by design. The framework cannot fix this because the line is the design. The Release Train is the unit of ownership. The team is the unit of execution. Move the unit of ownership down to the team and you do not have an ART anymore — you have a portfolio of small product teams, which is what Cagan, Team Topologies and DORA's elite-tier qualitative findings all converge on.9

This is the structural reason SAFe transformations stall. The ceremonies improve. The ART runs more smoothly. PI Planning gets more efficient. Nobody on the team feels more agile, because nothing about the operating model above the team has changed. The team is still being handed Features it did not discover, against a 10-week commitment it did not negotiate, with deployment dependencies it cannot break. The agile-coach stack is workable. The teams are not.

Teams cannot build ownership in what they don’t own.

Tell #5: Lean Portfolio Management is the PMO with new business cards

The title is mine; the diagnosis is Cagan's — his article is literally called Revenge of the PMO.10

Look at what Lean Portfolio Management actually does. It captures Epics in a Funnel. It moves them through Reviewing and Analyzing states. It requires a Lean Business Case for each Epic before that Epic can advance. It gates approval at a Portfolio Backlog state. It tracks Implementation against the original case.

State the same workflow in 2005 vocabulary: capture projects in an intake queue, move them through analysis with required business cases, gate approval at portfolio review, track implementation against the original case. The verbs are identical. The artifacts are identical. The decision rights are identical.

SAFe's defenders will say Lean Budgets are different — funded to value streams, not projects — and that is fair, in principle. Funding value streams is a real improvement over funding projects. But the governance overlay — the gates, the cases, the approvals, the portfolio reporting — is the project portfolio management discipline that PMOs have run for thirty years. It’s not bad because it’s old. It’s bad because it sits above the team and makes the team's authority subordinate to the portfolio's case-building exercise. Authority for “what to build” still flows top-down. The team gets to decide how. That is exactly the bargain SAFe strikes: the project mindset wearing a costume.

If you have a PMO that has been quietly converting itself into Lean Portfolio Management, the costume is the only thing that has changed. The decisions are still being made by the same people, in the same room, against the same business cases, with the same approval ladder. Agile pushed the PMO to the side; Lean Portfolio Management is its path back into the room.

The DORA test

Here is the test I keep coming back to. It is short. It is fatal.

Read more

Marty Cagan’s decade-old article can be a real rabbit hole. It’s genuinely hard to stop reading his blog, especially when digging up old gems like this.

Here’s something that resonates right now: “Good teams are skilled in the many techniques to rapidly try out product ideas to determine which ones are truly worth building. Bad teams hold meetings to generate prioritized roadmaps.”

Most teams I meet are funded, staffed and measured entirely around output. Hit the date, close the ticket, ring the bell for delivering on a roadmap.

Too few are paying attention to what matters: did what we shipped really make a difference for our customer?

The reorg that actually changes delivery isn’t new ceremonies. It’s moving the moment you celebrate from “we shipped” to “it delighted our customer, it actually moved what matters.”

Read more

Your team has probably spent more hours this year waiting on pull requests than it spent writing the code marooned inside them. Andrea Laforgia opens his March 2026 essay Stop Using Pull Requests with numbers from across the industry that count the cost. The piece is the most thorough reckoning I’ve seen of why the modern pull request workflow is mis-designed for the teams that use it. Go read it. Then come back here — because if you’re using the Delivery Playbook, Laforgia is describing the practice you already know. He just gives it a name. And I like it.1

That name is T*D — Test-Driven Development, Trunk-Based Development and Team-Focused Development, fused into one operating model. Hold on to it. We’ll come back to it.

Don’t break the flow

Chapter 2.8 Delivery processes & tools lays out what a well-wrought delivery pipeline does: ephemeral environments, test batteries, zero trust, release orchestration, observability, canary releases. The argument is t…

Read more

Six people. One sticky.

A SWOT analysis at the top of a planning workshop. Strengths quadrant — a dozen sticky notes inside ten minutes. Opportunities, fine. Threats, fine. Weaknesses — one sticky.

The team has been together a year. Mixed group, some senior, some junior, some remote and some local. They’re smart — and obviously not weak in just one way. But one is all they were willing to write down, in front of each other, with their names attached. So the Weaknesses quadrant sat there, embarrassed and underfed, while the rest of the wall filled out around it.

Most retrospectives end this way. Most planning meetings, the same. The team knows. The team won’t say.

There’s a 30-minute exercise that breaks this open. It’s not subtle. It’s not high-tech. It’s a borrowed word from Soviet engineering, an inversion trick from a 19th-century mathematician. It takes a team from laughter to honesty to commitment in less time than your average standup. It’s called TRIZ — the Liberating Structures version, which I’ll come back to — and it’s the cheapest unlock I know for a team that won’t speak its own dysfunctions out loud.

Here’s how it works, why it works and how to run it next week.

The wrong question

The silence in that Weaknesses quadrant is not random. It’s the predictable shape teams take when the question goes the wrong way.

“What are we doing wrong?” is one of the worst questions you can ask. It demands that someone you work with every day be the source of bad news, of blame — in a setting where their answer goes on the record. The forces working against honesty are stacked: the social cost of singling out a behavior, the political cost of singling out a person, the personal cost of being the one who said the thing nobody else was willing to say. So people hedge. They write harmless stickies. They give safe answers and stay quiet about the rest.

This isn’t about psychological safety being too low, exactly. It’s about requiring more safety than most teams I’ve known. An unusually safe team — the rare ones — can handle the direct question. Most teams can’t, even good ones. The Goldilocks zone for direct critique is narrow, and most teams sit just below it.

The wrinkle: the same team, asked the question a different way, will answer it. Same people, same room, same dysfunctions. The information is there. The mechanism for getting it out is what’s broken.

That’s the gap TRIZ fills.

Inversion

The 19th-century Prussian mathematician Carl Jacobi had a habit, when stuck on a problem, of saying man muss immer umkehren — “one must always invert.” If you can’t see how to make a thing succeed, see if you can describe how to make it fail. Charlie Munger, Warren Buffett’s late partner, popularized the move a century and a half later: “Invert, always invert,” he’d repeat, crediting Jacobi each time.1

Inversion is a cognitive shortcut. The forward question — “how do we make this succeed?” — runs into the usual social and intellectual walls. People hedge. People defer to whoever spoke first. People say what they think they’re supposed to say. The inverse question — “how do we guarantee it fails?” — jumps every one of those walls, because there’s no commitment. They’re describing a failure. They can be specific, vivid and sometimes uncomfortably accurate, and the room laughs instead of getting defensive.

Once the failure is on the wall, naming the parts of it that already exist is a much smaller step. You’re not confessing. You’re confirming.

That’s the whole engine of TRIZ.

Why it works on teams that won’t speak up

If you’ve never run TRIZ you’re probably thinking, “why does this work at all?” Why does asking a team to describe failure produce honesty when asking them to describe weakness doesn’t? It’s the same people in the same room talking about the same problem.

Here’s four reasons I keep coming back to. They aren’t independent — pull on one and the others move — and there are probably more I haven’t named. Their common thread: all four lower the cost of saying the thing.

Hypotheticals are cheap. Saying, “we could skip estimation entirely” carries no social cost. Nobody is suggesting it. Saying, “we do skip estimation” costs you — especially if the person who set up the no-estimation pattern is in the room. TRIZ collects the cheap version first. It builds a list of behaviors the team is willing to discuss because the team isn’t yet committing to whether they’re real.

Confirmation is easier than confession. By the time Movement 2 starts, the behavior is already on the wall. The team isn’t introducing a new criticism — that’s the hardest move. It’s confirming an existing one — that’s a much smaller move. “Ah, yea, we do that,” is far easier to say than, “we have a problem with X.” The board already said it. The team is just agreeing.

Laughter changes the room. Ten minutes of laughing at exaggerated worst-case behaviors changes the social register. The same conversation about the same behaviors, started cold, would feel like an accusation round. Started after the comedy, it feels like camaraderie. Defensiveness drops. People interrupt themselves. The room is in a different mode.

The quiet people speak. This is the move I see most reliably. In nearly every TRIZ I’ve facilitated, the first person to circle a behavior in Movement 2 — the first to commit to, “yea, that one” — has been the person who hadn’t said much yet. The senior engineer who watches more than she talks. The new hire who’d been listening for three months. They’d seen the dysfunction the whole time. They didn’t have a frame to say it. TRIZ gave them one.

You can put psychological safety theory on this if you want. You can talk about face-saving, about politeness norms, about the asymmetry between criticism and praise. All of it holds. But the practical version is simpler: it’s a lot easier to say things about hypotheticals than about reality, and TRIZ exploits that gap to get a team to say what it already knows.

That’s why a 30-minute exercise can do work that an hours-long retro can’t.

The three movements

The exercise has three movements. Together, they take 30 minutes if you keep the timing tight; you can stretch to 45 on a hard problem or compress to 20 with a small confident team. Each movement has a single question, and each question is doing different psychological work.

Movement 1: the worst-possible-outcome brainstorm (10 minutes). Frame the outcome the team cares about — high-functioning sprint planning, a retrospective that actually changes things, a release process you’d be willing to demo. Then ask the inverted question: what could we do to guarantee the worst possible version of this? Wild, exaggerated, funny answers welcome. Suppress nothing. Two minutes of silent sticky-writing first, in 1-2-4-All style, then a round-robin share.

The laughter is the point. A team that’s laughing at “we could just cancel the retro and send an email instead” is a team becoming willing to say real things. Don’t rush this part — the comedy is what unlocks the next movement.

Read more

I was chatting with a friend about team dynamics. We’d been talking about restructuring around product teams — the kind of empowered, durable, cross-functional teams I’ve written about before. He’s nodding along, asking sharp questions. Halfway through, he pushed back.

I’m paraphrasing, but the gist was, “Look, I get the theory. But where’s your proof? We’ve all heard the pitch. Cross-functional teams, durable squads, outcome metrics, two-pizza this, empowered that. Show me the data. Show me that any of this actually beats what we already do.”

It was a fair question. A great question, actually. I’d written the case in Building a product oriented team and the prequel, Moving away from the Monolith — faster, better with a product mindset. But those pieces argued from reasoning and experience. They didn’t bring the math.

So today I want to bring it. Because the math is there — it’s been there for over a decade — and once you line it up, the conclusion is hard to escape: product teams, the real ones, ship faster, build less waste, deliver higher quality and produce dramatically better business outcomes than the alternatives. The evidence isn’t a single bombshell study. It’s a stack of independent research, all converging on the same answer.

Let me lay out the receipts.

One note before we start. None of the studies I cite below were designed to answer the question “do product teams outperform?” They were designed to answer narrower, related questions — about deployment, about feature usage, about team effectiveness, about developer velocity. What’s below is my synthesis: the act of lining up findings from different research traditions and showing where they converge. The convergence is the argument. Each citation is a thread; the rope is the pattern across them.

A quick refresher on “product team”

Before the data, let’s get precise. “Product team” gets thrown around so loosely it’s often meaningless.

Marty Cagan and the Silicon Valley Product Group draw the cleanest distinction. A product team is a small, durable, cross-functional unit — typically a product manager, a designer and a handful of engineers — that owns a problem and is held accountable for outcomes. They are given problems to solve, not features to ship. The product manager owns value and viability. The designer owns usability. The tech lead owns feasibility. Together they own the result.1

A feature team looks similar on paper — same disciplines around the table — but is handed a prioritized list of features to build. Outcomes belong to a stakeholder upstream. The team is judged on output: did you ship what was on the roadmap? Velocity, story points, throughput.

A project team is even further from product orientation. It’s pulled together to execute a discrete project, then dissolved. Specialists rotate in and out. There’s no durable home for product knowledge.

The labels matter less than the operating model. As I wrote in Building a product oriented team, the real distinction is durability, ownership and empowerment — long-lived teams that own the whole product end-to-end and have the authority to figure out the how.

Now — what the research says.

Receipt #1: they ship dramatically faster

Start with DORA — the DevOps Research and Assessment program, founded by Dr. Nicole Forsgren and now part of Google. Every year since 2014, DORA has surveyed practitioners — more than 3,000 in 2024 alone, and tens of thousands across the decade-long program — and statistically grouped them into performance tiers based on four metrics: deployment frequency, lead time for changes, change failure rate and mean time to restore service. The methodology, the math and the conclusions are documented in Accelerate — the book that emerged from the research and won the Shingo Publication Award.2

The 2024 Accelerate State of DevOps report is the latest. Elite-tier teams deploy 182 times more frequently than low performers, with 127× faster lead time for changes. Elite teams hit lead time under a day, deploy on demand and are roughly 19% of the surveyed population. Low performers measure lead time in months.3

That’s not a marginal advantage. It’s not “20% better.” It’s two orders of magnitude.

You might be tempted to dismiss this as a DevOps story rather than a team-structure story. But every time DORA digs into what differentiates elite teams, the answers are the same attributes that define product teams: small, cross-functional, owning the whole pipeline from code to production, empowered to make their own technical decisions. The opposite — handoffs across functional silos with stage-gate approvals from external groups — correlates strongly with low performance. Accelerate presents one specific finding bluntly: external change advisory boards do not improve stability. They make lead time worse without making anything safer.4

McKinsey’s “Beyond Agile” work, which I cited in How waterfall is wrong for software, puts a number on the same effect from a different angle. Across their consulting engagements, McKinsey observed that companies could compress the average time from “code complete” to live production from 89 days to 15 days when they reorganised around DevOps and product-team principles. Same basic architectures, same kind of features. Six times faster, just from changing how the team takes work from “done” to “deployed.”5

The simplest summary: product teams own the value stream end-to-end. Feature and project teams hand off across silos. Handoffs cost time. Time turns into distance from the customer. Distance from the customer means everything you ship is stale.

Receipt #2: they get more done — but redefine what “done” means

“More done” is the wrong metric. The right metric is: did anyone use what we shipped?

In 2002, Jim Johnson — chairman of the Standish Group — presented a finding at the XP conference in Sardinia that’s been quoted ever since: across enterprise applications, only 7% of features are “always” used, 13% are “often” used and 16% are used “occasionally.” That leaves 64% of features as either rarely or never used. Standish later extended the analysis to claim that roughly 80% of features deliver low or no value. On large projects, Standish data shows organizations deliver only 42% of the features they originally scoped. The Standish numbers have drawn methodology critiques over the years — fair enough — but more recent work points the same direction. Pendo’s 2019 Feature Adoption Report, drawing on telemetry from hundreds of B2B SaaS products, found roughly 80% of features are rarely or never used.6

Sit with that for a moment.

Now, “rarely used” isn’t synonymous with “no value.” Some rare features are quietly essential — compliance flags, account recovery, the audit trail your CISO will ask about exactly once. Both Pendo and Standish gloss over that distinction. Even granting it generously, though, the ratios still point at a meaningful misallocation: a lot of effort going into features customers rarely or never engage with. (I’m looking at you, Microsoft Office).

A feature team measured on roadmap output is, by these numbers, building a sizable wedge of low-value work. They look terrific on burndown charts and terrible on customer impact.

I’ve watched this play out enough times to be cynical about it. A feature team gets a quarterly roadmap. They estimate the work in story points, deliver on schedule, demo a sleek end-to-end flow and get applauded. Six months later, telemetry shows the feature gets touched by 3% of users. By then the team has moved on to the next item on the roadmap. Nobody is accountable for the gap between “shipped” and “mattered,” because shipped is the goal.

A product team would have caught that earlier. They wouldn’t have built the full feature on a quarterly roadmap; they’d have built the cheapest possible version that tested the hypothesis — a prototype, an A/B variant, a deliberately ugly first cut — and watched the data. If the data said 3%, they’d have killed the feature in week two and gone after a more promising hypothesis. That’s the discovery discipline Marty Cagan and Teresa Torres write about extensively, and it’s the operational reason product teams build less and deliver more. Not because they’re slower. Because they aim differently. They invest in discovery before commitment. They kill a feature that doesn’t earn its keep. They redefine “done” from “shipped” to “moved the metric.” And the metric they move is customer behavior, not story points. (I’ve made the same argument from a measurement angle in Stop measuring effort.)

There’s a parallel finding in McKinsey’s What makes product teams effective? — a five-year study of more than 1,700 teams across 75 organizations. McKinsey identified two structural conditions that consistently predict throughput and velocity improvements: dedicated team membership (no context switching across projects) and team persistence of three to six months minimum. Teams meeting both conditions develop reliable estimates and sustained throughput. Teams that don’t, can’t.7

That finding isn’t surprising if you think about how knowledge accumulates. Product orientation depends on knowledge durability — context that only forms when the same people work on the same problem long enough to internalise it (more on this below). Project teams dissolve before that happens. Feature teams reorganise around the next quarterly priority. Product teams stay put. That’s why their throughput compounds.

Read more

A junior developer told me they’re using Claude as a mentor. They liked the patience that never wore thin, the availability that never closed, the way they’re never made to feel doltish.

Here’s what worried me: they said mentor. Not tutor. Mentor.

I don’t think they’re wrong about what they’re getting. But mentorship isn’t about answers. It’s about being seen — watched while you struggle, nudged when you’re avoiding hard conversations, sent on projects that stretch you in ways you didn’t ask for. Mentors have the nerve to tell you the truth in private because they’re invested in who you’re becoming.

AI can’t do that. Because it isn’t invested in you. Because it’s not “in the room.”

What AI is genuinely good at

Let’s start with what’s actually working, because this isn’t a piece about being suspicious of the tools. The tutoring story is real. It’s the part of the AI hype that’s actually earned.

Benjamin Bloom’s famous 1984 paper described what he called the “2 sigma problem:” students who received one-on-one tutoring performed two standard deviations better than students in conventional classrooms — that’s the difference between an average student and the top 2 percent. The catch was scale. Personal tutors are expensive, and most students will never have one. For forty years that gap has been the unsolved promise of education technology.1

A well-configured AI is one way we’re closing it. When Khan Academy rolled out Khanmigo, built on GPT-4, the pitch was exactly this — a patient, infinitely available one-on-one tutor for every learner. In specific narrow tasks, the early returns are hard to dismiss. Harvard published a physics study in 2024 that found students using a properly designed AI tutor learned over twice as much in less time than their peers. They reported feeling more engaged and motivated.23

I see the same thing in software. With the right model, a serious system prompt and a few well-built skills, AI can:

• Walk a junior through a debugger session, naming what each frame means and why it matters.

• Explain why a piece of code is the way it is — not just what it does — and adjust the explanation when the question lands a level too high.

• Run unfamiliar code, watch it fail and reason about the failure with the learner in real time.

• Drill someone on a foreign language, a SQL idiom or a regex until it’s automatic.

• Take the same question for the third time without sighing.

That last one matters more than people admit. A great deal of human learning gets blocked by the social cost of asking again. Removing that cost is genuinely useful. It’s what the junior in my opening responded to. They felt safe.

How tutoring goes wrong

It’s useful, but not magical. The same AI cheerfully invents APIs that don’t exist, pattern-match on the wrong context and confidently produce code that looks right but isn’t. Apple researchers showed last year that current LLMs aren’t doing genuine reasoning at all — performance on math problems dropped by as much as 65 percent when the team added a single irrelevant clause to the prompt. Andrew Zuo’s piece on the “-10X developer” makes the harder point: when an LLM writes code at or above your level and introduces a subtle bug, you may be the worst possible person to find it.45

A newer concern is harder to wave off. An MIT Media Lab study published in mid-2025 used EEG to compare brain activity across people writing essays with no help, with a search engine and with an LLM. The LLM group showed the weakest neural connectivity and the worst ability to recall their own writing minutes after producing it. The researchers called it “cognitive debt.” That tracks. Outsource the struggle and the learning never lands.6

So a serviceable tutor, yes. A replacement for thinking, no. None of that, though, is the central point. The central point is that tutoring and mentoring are not the same job, and the public conversation keeps blurring them.

What a mentor actually does

If you ask me what my mentors did for me over the last forty years, almost none of it was answering technical questions. The technical content I could get from books, from peers, eventually from Stack Overflow and now from Claude (as long as I’m careful and look for fantasy hallucinations). What my mentors did was something else entirely.

Read more

In 3.6 Specification we worked through the first two stages of Behavior Driven Development — Discovery and Formulation. We surfaced scenarios in three-amigos conversations and shaped them into Gherkin “Given-When-Then” statements. That gave us the outline of every feature in our product increment.

Elaboration is where we move from outline to depth.

A very important takeaway from chapter 3.6 bears repeating here, because the entire purpose of this activity hangs on it:

The specifications you write now are living documents. They will evolve, change and be added to. That’s why it’s so important to create a live link to the implementation — the code that will validate the specifications. As the specifications change that live link will preserve our value chain from customer desires to delivered code.

Hold on to that idea. Elaboration isn’t about freezing requirements before the team writes a single line of code. It’s about building the connections — between the spec and the marble, between the marble and the test, between the test and the production code — that let the spec keep evolving without the value chain ever going slack.

Introduction

This activity has three jobs.

First, we expand each scenario across the levels at which the system is tested: component and assembly. A login feature has UX scenarios about what the customer sees and experiences. It also has assembly scenarios about which events fire, which commands route through which aggregates and what payload comes back. Both layers need specifications. Both layers need to be testable. Skipping a layer means leaving a gap through which defects love to slip.

Second, we create a live link between every specification and the engineering model that implements it. Each Gherkin feature points back to a marble. Each marble points forward to a Gherkin feature. That bi-directional link is what makes our specifications living documents — when the spec changes, the marble updates with it, and vice versa.1

Read more

Picture this: a program manager wants to confirm a design decision before a release. She asks, innocently enough, “Can someone point me at the current version of the architecture document?”

A pause. Then: “I think it’s in the Teams channel — the one from the February workshop.” Someone else chimes in, “No, there’s a newer one. Sriram emailed it last week.” A third voice: “That’s the draft. The signed-off version is in the SharePoint site. But the link in the meeting invite is broken because they restructured the folders.” A fourth: “I have a copy in OneDrive. It might be the latest, I’m not sure.”

Five people. Five different versions. Zero confidence that any of them is authoritative. And this is a team that cares about documentation.

The common denominator is always the same: a collaboration stack built on SharePoint, Teams and Microsoft’s various Office products. Sometimes with Azure DevOps (ADO) bolted on to handle tickets and code.

Here’s an argument that will sound harsh to anyone who’s invested in Microsoft 365 as their delivery platform: the Microsoft collaboration stack is diabolically deficient for serious product delivery. It wasn’t built for the problem delivery teams actually face. It has, piece by piece, gone after what enterprise IT wanted to solve in 2007: replacing the file share.

That’s not the same problem.

What a delivery team actually needs

There are five table stakes a team needs. I describe the first three in Getting started: Using the Delivery Playbook. I’ll restate those briefly here, along with the other two:

1. Live collaboration. Multiple people must be able to work on the same artifact at the same time, editing simultaneously, seeing each other’s cursors, throwing virtual sticky notes on a board in real time. Design work is a collaborative act. The moment one person has to collect and collate feedback, fidelity drops off a cliff.

2. Version control. Every change, by every author, is always captured and reversible. Not just “saved revisions” — the full, granular history, searchable, with diffs, always available. This is table stakes. Git has had it for two decades. Any tool that tracks less than Git doesn’t take the goal seriously.

3. Immutable linking. A document’s URL must never change. Not when the file is renamed. Not when it’s moved to another folder. Not when the folder is reorganized. Not when the author leaves the company. A reference made today should still resolve in five years. Without this, every link in every other document becomes a ticking time bomb.

4. Workflow integration. The documentation platform has to talk, fluently and deeply, to the work-tracking platform. Specs have to link to tickets. Tickets have to link to pages. Status changes in one system should not only be visible in the other, but trigger cross-tool workflow changes and render live embeds, not stale URLs. When that integration is tight, your specs, documentation and tickets stop drifting apart — they become a single living view of the work, not several systems the team has to reconcile.

5. Universal, web-first access. One consistent experience for everyone, on any device. Not a degraded web version and a feature-rich desktop app — a single canonical surface, served through the browser, that everybody gets equally no matter what hardware they’re on.

These five criteria are the measuring stick. Let’s hold SharePoint — and Teams and ADO — up against them.

Links that rot

This is the most damning, and the one I want to spend the most time on, because it’s the one almost nobody takes seriously until they’re knee-deep in the damage.

In SharePoint, a document’s URL is a function of its name and its location in the folder hierarchy. Rename the file, and the URL changes. Move it to another folder, and the URL changes. Move the whole folder, and the URLs of every file inside it change. You can turn on “managed metadata” and “content types” and a bunch of other gears that promise to fix this, but in practice — across every tenant I’ve ever touched — they don’t. The URLs change. References break.

Microsoft has shipped “durable links” and “document IDs” — features that are off by default and rarely enabled. What a team actually encounters is: you renamed the file, the link in the meeting invite from three weeks ago is now dead. The problem is pervasive enough that a cottage industry of third-party tools has emerged just to detect broken SharePoint links.1

Compare that to Atlassian Confluence. Every page has a numeric ID assigned at creation. The ID is embedded in the URL (/pages/12345) and it does not change. You can rename the page, move it between spaces, reorganize the whole tree — the ID-based link still works. You can also use the human-readable URL (which does change when pages get renamed), but the ID-based URL is your forever-address. The system gives you both, and the tight URL is the one you automatically get when you cite the page in another document.2

This seems like a small, technical detail. It is not. It is the single most consequential design choice a documentation platform makes, because it determines whether cross-references decay or persist.

Here’s what happens in practice when links rot: people stop trusting links.

Think about that for a second. The entire promise of the web is that a URL is a durable reference — it says “here is the canonical thing, go fetch it when you need it.” That promise is what lets us build systems on top of systems. Every meeting invite, every wiki page, every code comment, every email that cites a document relies on that promise holding.

When SharePoint breaks the promise — which it does, routinely — teams develop a defensive response. They stop linking. Instead, they attach. They paste a copy of the document into the meeting notes. They email a copy to the distribution list. They download the file and re-upload it to their own folder “in case the original moves.” Multiply this by every team, every document, every month, and within a year you have a document landscape that looks like a hoarder’s basement — dozens of near-duplicates, none of them canonical, each one owned by a different person who genuinely believes theirs is the latest.

It’s rational response. Given that SharePoint links are not reliable, hoarding copies is the locally optimal strategy. However, the result is catastrophic: information silos full of stale documents and an immense toll in lost time and miscommunication.

The identity model that a platform chooses shapes the behavior of the people using it. SharePoint chose the wrong model — one that breeds inefficiency and defensive silo-building.

The version control mirage

SharePoint technically has version control. Open a file, click the ellipsis, choose “Version history,” and you’ll see a list of saves. This sounds like what you want. It is not.

The version history in SharePoint is coarse, inconsistent across product surfaces and frankly untrustworthy. Here are the problems I’ve personally run into, repeatedly:

• Coauthored Office documents roll up edits from multiple people into “a version” at intervals that are not deterministic. If Alice and Bob are editing at the same time, you may end up with one version of their joint work and inaccurate attribution. Contrast with Git: every commit is atomic, attributed and reversible.

• The version history for a file in OneDrive, Teams and elsewhere varies. There is no single source of truth. If you open a file with Teams, you see one history. In SharePoint, you may see a different history — a consequence of Office 365 evolving from a constellation of separately-designed products Microsoft is still trying to stitch together.3

• Restoring a prior version might overwrite the current one, rather than branching. If you discover at 4pm that something important got lost at 10am, “restore” could mean losing everything done between 10am and 4pm. Until recently, this was the default behavior — so be careful.

Confluence, again, does this properly. Every save is a version. Every version compares cleanly against any other. You see what changed, who changed it and when. You can restore a prior version as a new version, preserving full history. This is how version control is supposed to work, and it has been the standard behavior in software since RCS shipped in 1982.

Why does this matter for delivery? Because design artifacts evolve, and the history of how they evolved is often as valuable as the current state. When the security auditor asks, “When did we add this API endpoint to the threat model?” — the answer lives in the diff. When a customer says, “The contract we signed described a different data retention policy,” — the answer lives in the history. When a post-mortem asks, “When did we know about this failure mode?” — the answer lives in the version log.

A platform that can’t answer these questions reliably is a platform that is actively hostile to engineering rigor.

Teams, Teams Sites and the organizing layer

If the first two failures are about individual artifacts, this one is about the landscape those artifacts live in.

Microsoft Teams is, at heart, a chat application. When it was first released in 2017, that’s primarily what it did. Over the years, Microsoft has tried to turn Teams into a “hub for teamwork.” Marketing-speak for bolting everything else Microsoft makes onto the chat app. The Files tab in a Teams channel is a SharePoint document library wearing a trenchcoat. A “Teams Site” is a SharePoint site with a slightly different navigation chrome. Meeting recordings live in OneDrive, not in the channel. Meeting transcripts live in the meeting artifact, which sometimes appears in the channel and sometimes doesn’t. Planner tasks live in a separate product surface that is also “integrated.”

If someone in your team wants to find something, they have to know, in advance, through which product surface it was shared. Was the spec posted in a channel? Check the chat history. Attached to a meeting? Check the meeting recap. Uploaded to the Files tab? That’s SharePoint under the hood, but the folder structure might only be reachable through the Teams UI. Created as a Teams Site? Different URL. Shared in a meeting chat? That goes to OneDrive, in a folder you might not have permission to browse.

There is no single “where did we put the thing” surface. There are five or six of them, depending on how the team has been using the tools, and the mapping between “what kind of content” and “which surface” is convention rather than enforced policy.

In contrast, Confluence is the document surface. Jira is the work surface. Each is a single, distinct product doing one thing well, and the integration points between them are narrow and explicit and they work. No guessing where a document lives — it’s in Confluence. Tickets live in Jira. No impedance mismatch, no cognitive dissonance.

Teams wants to be the union of every Office 365 capability. But integration isn’t about putting everything in the same UI. It’s about making the right connections between distinct tools, so that each one stays simple and focused, and the connections carry exactly the semantics you need. Teams never understood that distinction.

Desktop is king, the browser is second-class

SharePoint and the rest of Office 365 are web-accessible, but they aren’t web-first. Their heritage is Microsoft Office — a suite designed for the desktop and perfunctorily ported to the browser. The difference is not academic.

Open a Word doc in the browser and compare it to the desktop app. The feature set diverges. Advanced formatting, compare-versions, certain macro and formula capabilities, add-ins — these are missing or degraded on the web.4

The consequences cascade. “The copy you want is on my laptop” is a phrase I hear almost every week. Co-authoring gets flakey or breaks entirely because of diverging copies. Changes sit in someone’s sync queue for hours. A document that looks current in SharePoint may be stale. Information silos reappear — not because anyone intended it, but because the tool keeps pulling work off the web and onto individual machines.

Contrast with Confluence, Jira and Miro. Every one of them is web-first by design. There’s no desktop app with a superset of features. The browser is the product. Everyone gets the same experience — Windows, Mac, Linux, iPad, phone, Chromebook — because there’s only one experience to get. One canonical home, universally accessible, identical for everyone.

This is what the web was built for. Atlassian and Google stacks lean into that. Microsoft fights it.

The ADO integration gap

This one is specific to teams that pair SharePoint with Azure DevOps for their ticket tracking and code hosting. It’s a common combo, because “we already have Microsoft” is an easy procurement story, and ADO does a lot of things adequately.

But the integration between SharePoint and ADO is thin. Thin to the point of “here is a hyperlink from the work item to the SharePoint page, good luck.” There is no live embed. There is no bidirectional status update, rich preview or shared workflow trigger. If I link to a SharePoint page from an ADO work item, you see a URL. If that URL breaks, the work item becomes an orphan.

In contrast, a Jira issue in a Confluence page renders a live, interactive issue card with issue key, title, status, assignee and priority — and that card updates when the issue does. If I embed a Jira query in a Confluence page, the page shows a table of live results — as of right now. A Confluence page can be the dashboard for a delivery effort: status pulled from Jira, specs authored inline, risks tracked in a live table, all rendering from the underlying systems of record.

You can build most of this in the Microsoft stack with enough Power Automate flows, Graph API queries, custom connectors and a patient engineer. I’ve seen it attempted. I’ve yet to see it done well enough to matter. The effort required to get SharePoint + ADO to mimic what Confluence + Jira does out of the box is roughly the effort of a mid-size integration project — a project that has to be maintained, re-tested with every platform update, and reverse-engineered when the person who built it leaves the company. That’s not an integration. That’s a liability.

Workflow is an afterthought

The last failure is workflow management — specifically the modeling and enforcement of state transitions for work items.

Jira gets this right. The workflow editor is a first-class product, with a first-class visual flow designer, transition conditions, validators, post-functions, custom statuses and permission checks. A non-engineer on the delivery team can design a workflow that enforces “a ticket cannot transition from ‘In Review’ to ‘Done’ unless the code review is merged with clean quality gates, the responsible stakeholder has reviewed the feature and all acceptance criteria are passing green.” Express it in the GUI, save it, extend with code as needed and the tool enforces it across every ticket that follows that workflow. The team controls its own process — and the workflow itself is a first-class artifact: auditable, exportable, evolvable.

ADO has “process customization.” It is possible to model custom workflows. In practice the feature is buried behind an admin gate, its conditions are a shallow subset of Jira’s, and most teams I’ve worked with never meaningfully touch it. They either live with the default — generic, not their actual process — or they reinvent the enforcement in spreadsheets, PowerBI dashboards, or human oversight routines that have to be re-explained to every new hire.

The reason this matters: workflow is the mechanism by which a team’s process becomes automatic. If your workflow tool doesn’t let you encode “we agreed code reviews are mandatory before a feature branch is merged,” then code reviews are not mandatory — they’re optional, governed only by vigilance, and skipped the moment someone is under pressure. Jira’s workflow editor is a force multiplier for team discipline, because it turns “we agreed to do this” into “the tool won’t let you skip it.” ADO gives you a significantly weaker version of that lever, and most teams don’t reach for it at all.

The cost is reliability. Teams using Jira-style workflow enforcement ship with fewer process escapes — missing reviews, missed gates, skipped steps. Teams using ADO’s default setup rely on human memory and goodwill — both fail at scale. The difference shows up in incident rates, audit findings and the quality of the artifacts the team produces.

What all this costs the team

If you’re keeping a tally, the SharePoint/Teams/ADO stack fails on every criterion. Links don’t persist. Version control is a mirage. The organizing layer is fragmented. The browser is a second-class citizen. Integrations are absent or thin. Workflow is an afterthought.

What does this actually cost a delivery team?

It costs time. A lot of it. The 2024 Stack Overflow Developer Survey (65,000 respondents) found that 61% of developers spend more than 30 minutes every day just searching for answers — and 30% say knowledge silos hit their productivity ten or more times per week.5

It costs trust. When the documentation landscape is chaotic, people stop trusting it as a source of truth. They start relying on oral tradition — “ask Alice, she knows” — and on personal copies. Key decisions stop being recorded in a place anyone can find. Onboarding becomes a six-month apprenticeship rather than a process a new hire can self-serve. Organizational memory degrades.

It costs quality. DORA's research (2022 State of DevOps Report) found that documentation quality is one of the most powerful amplifiers of engineering performance: teams with above-average documentation quality see 12.8x the organizational performance lift from the same technical practices compared to teams with poor documentation.6

It costs morale. Engineers are, as a group, unusually sensitive to tool quality. I’ve seen engineers turn down offers, quit, or burn out over the cumulative drag of bad tooling. A team that feels its own stack is working against them never reaches its potential, however talented the individuals. Tool quality is retention strategy.

Stuck with the stack? Fight back.

If you’re committed to SharePoint, Teams and ADO for the next few years, none of the above changes that. But a handful of disciplines can reduce the damage. None will turn the Microsoft stack into Confluence — they just stop the bleeding.

Here’s six specific strategies you can follow:

Read more

I was recently asked to critique a new decision tree that helps teams decide between waterfall and agile when starting a new project. The intended outcome is to improve long-term results in the organization by giving teams the ability to make critical decisions quickly and accurately.

Overall, “good shift-left stuff” in my book. Anyhow, here’s the diagram I was asked to comment on:

Read more

I once joined a team that had been struggling with delivery for the better part of two years. Demos kept failing. Deployments kept crashing production. Rollbacks were so routine that the ops team had gotten eerily efficient at them. (I’ll refer back to the team later — let’s call the company “Acme”).

I remember one in particular. The team was demoing a reactive search feature — the kind of thing where you start typing a product name and suggestions appear in real time. Same idea as typing “penguins” into Google and getting “Penguins of Madagascar” as a suggestion.

The demo cratered. Someone typed a product name, and nothing happened. Twenty minutes of poking around later, the demo was cancelled. What struck me wasn’t the failure itself — it was the reaction. Nobody was surprised. Nobody expected it to work. “Maybe next time.”¹

Later that afternoon, a planned deployment seemed to have crashed a key service. All hands on deck for a rollback. The team executed quickly. Rollback done. They’d had a lot of practice.

Here’s what immediately jumped out at me: in both cases the team had zero operational intelligence. No telemetry. No metrics. No centralized logs. No traceability. No way to quickly zero in on a root cause. They were flying blind — and they’d been flying blind so long they’d forgotten what visibility even looked like.

And that’s why the reaction was the same each time. Abort. Rollback. “Afterward, we’ll figure out what’s wrong.”

The irony is that both failures had quick fixes. Each time, queries weren’t reaching a service because the wrong port was configured. Instead of changing the port and continuing, the team burned hours — even days — backtracking, diagnosing and trying again.

That experience crystallized something I’d been feeling for decades: observability isn’t a tool you add. It’s an architectural decision you make — or don’t. And the cost of not making it compounds in ways that are genuinely difficult to measure, precisely because you don’t have the instrumentation to measure them.

The “flying blind penalty”

In the old days — before cloud, before microservices, before systems became truly distributed — opacity was tolerable. You had a monolith on a few servers. If something broke, you SSH’d in, looked at logs, poked around. It might take hours, but there wasn’t that much territory to cover.

Modern systems don’t afford that luxury. A single user request could traverse dozens of services, touching databases, message queues, caches and external APIs along the way. Just today I participated in a sprint review discussing a database architecture that was 29 views deep. A latency spike could originate in any one of those — or in the invisible interplay between the services that call them.

Without observability, you’re operating on instinct and prayer.

Here’s what that looks like in practice. A bug ships. It doesn’t manifest right away — it surfaces under load, or in a specific user segment, or under certain timing conditions. By the time anyone notices, users are already suffering. Now the team has to diagnose the problem. If they’re lucky, someone complains and provides enough detail to be helpful. They narrow down which service is involved, reproduce the issue, trace the flow of data through the system, find the root cause, fix it, deploy and verify.

The whole time, sweating over how many more customers are running into the problem.

Every single step without observability compounds in difficulty. You’re not just debugging — you’re doing archaeology at midnight, piecing together behavior from scattered artifacts across dozens of services.

With observability designed in from the start, you proactively see which service is degraded. You can drill into specific request traces. You can correlate logs across service boundaries. Mean time to detect and repair drops from hours to minutes — even becoming predictive.2

DORA.dev research bears this out. Elite-performing teams — the ones deploying multiple times per day with sub-hour recovery times — aren’t elite because they have better engineers. They’re elite because visibility closes a critical gap: it’s a prerequisite for every other capability DORA measures. They can detect and diagnose problems before those problems cascade. Their systems are instrumented to answer questions the team hasn’t even thought to ask yet.3

What matters is how you think about the problem

Monitoring and observability are not the same thing.

Monitoring answers questions you anticipated. You watch CPU utilization, disk space, error rates. You set thresholds and alert when they’re exceeded. “Is the database up? Is the error rate spiking?” These are known questions about known conditions.

Observability answers questions you haven’t thought of yet. It’s the ability to interrogate your system’s behavior without adding new instrumentation every time you have a new question. “Why is this specific user’s checkout so slow today when it was fine yesterday? What changed in the request path between those two deployments?”

Charity Majors — co-founder of Honeycomb and one of the sharpest voices on this topic — frames it as the difference between known-unknowns and unknown-unknowns. Monitoring handles the known-unknowns: the things you anticipated might go wrong. Observability handles the unknown-unknowns: the things you couldn’t have predicted. It empowers you to follow the breadcrumbs wherever they lead.4

Monitoring is reactive. Observability is generative — it creates the conditions for discovery.

Daniela Miao, CTO of Momento, put it well in a Dev Interrupted episode: “By giving more visibility into what’s happening with your software, your application, you actually do enable development to be faster as well. It’s not just insurance for when something goes wrong.”5

That’s critical. Most teams think of observability as insurance — you invest in it to cushion the blow when things break. But the real dividend is velocity. Teams with strong observability ship faster because they can see the impact of their changes in real time. They don’t have to guess whether a deployment worked or a tweak to a query ran faster. They know.

Why we don’t instrument on day one

Early stage. You’re shipping fast. Three or four services, maybe. Errors go to a text file. You’re small enough that you can hold the whole system in your head. Observability feels like overkill. “It’s in the logs, just read ‘em.”

Growth. Suddenly there are ten services, and logs are scattered across different servers. You can’t correlate them anymore. Someone says “we should get a logging tool.” You pick one — Datadog, Splunk, whatever — and bolt it on. The instrumentation is inconsistent. Some services are covered, others aren’t. Metrics are sparse and don’t tell a coherent story. You’re reading fragmented logs from disparate systems and trying to correlate events in your head. It’s starting to fall apart.

Pain. Now you have fifty services. Traffic has increased tenfold. Something breaks, and you’re still flying blind — not because you lack tools, but because the observability layer wasn’t designed. It was stitched on, piecemeal, after the fact. Different teams instrumented different things using different conventions. Tracing is incomplete. Metrics don’t correlate with logs. You’re paying more and more for tools that don’t give you the insight you need.

Here’s what’s easy to miss: that team invested in observability. They bought the tools. They checked the box. But because every investment was reactive — a logging platform when logs got painful, a metrics dashboard when someone asked for one, a tracing system when debugging got unbearable — they ended up with three separate systems that don’t talk to each other. Logs in one silo, metrics in another, traces in a third. Every decision about what to capture was made at write time, under pressure, for a specific fire they were fighting that week.

Charity Majors, co-founder of Honeycomb, calls this Observability 1.0: many pillars, many sources of truth, many tools. It’s what you get by default when you bolt observability on after the fact. The fragmentation isn’t a failure of effort — it’s a consequence of timing. You can’t design a unified telemetry model when you’re patching gaps reactively.6

And it matters — because a fragmented model can answer the questions you anticipated (is the database up? is the error rate spiking?) but collapses when you need to ask something new. “Why are checkouts slow today when they were fine yesterday? What changed between those two deployments?” Those questions require slicing across dimensions you didn’t pre-define. They require the kind of open-ended interrogation that siloed tools simply can’t support.

Insight requires design. That’s the part teams miss.

Why does this happen? Because observability isn’t a feature. It’s not something you built first, it’s what got built last. It doesn’t generate revenue — at least not visibly. It feels expensive up front. And teams, under pressure to deliver, don’t demand the time they need for planning a solid foundation.

I’ve written about the same false economy with security and testing. The things that feel expensive to build at the start turn out to be astronomically more expensive to retrofit.7

Why it has to be architectural

You can’t add observability with a library import. You can add logging and metrics — that’s the easy part. But genuine observability, the kind that lets you answer arbitrary questions about system behavior, requires architectural forethought.

Consider what’s involved:

1. How do requests get identified as they flow across boundaries? Trace IDs need to propagate through HTTP headers, message queue metadata, database session context — everywhere a request touches. If you don’t design this in, you can’t correlate activity across services.

2. What events merit recording? Not every line of code should produce a log. That’s noise. But the meaningful transitions — state changes, boundary crossings between services, external calls and their response characteristics — these are the signals that matter.

3. What metrics actually reflect system health? Not just CPU and memory, but application-level indicators. How long do payment transactions take? What’s the cache hit ratio? What’s the queue depth trend? These are the metrics that tell you whether your system is behaving — or trending toward overload.

4. How does context propagate? When a request carries metadata — a customer ID, a feature flag, a request source — how does that context flow through the system in a way that key transitions are captured seamlessly?

This is the shape of how you structure services, how you handle requests, how you organize code. And capturing it after the fact is brutally hard.

For example: if you decide later that you want end-to-end tracing, you need to add trace ID generation at your entry points, thread those IDs through every service-to-service call, correlate logs using those IDs, and store traces in a specialized backend. That means touching every API, every RPC, every async boundary. If you’d built it in from the start, it’s a few lines of middleware.

Consider structured logging: if you’ve been writing freeform string logs for two years, migrating to structured JSON events means auditing every logging statement, retrofitting context and rewriting every query your team depends on. If you start with structured logs, it’s just how things work from day one.

It’s not expensive when you build it in. It’s a handful of decisions, many of which improve things by introducing consistency. OpenTelemetry standardizes the plumbing — trace propagation, metric collection, structured logging — across languages and frameworks. You focus on what to instrument, not how to propagate context.8

But retrofitting: that’s an ordeal. Orders of magnitude more expensive — in engineering time, in tool sprawl and in the debugging sessions that never needed to happen.

What to actually build

Here’s how to do it right — whether you’re making the investment in an old project or starting fresh. The key is to think in events, not pillars.

Start with events, not tools

The most consequential decision you’ll make is how you model your telemetry. Do you emit three separate streams — logs here, metrics there, traces somewhere else — and try to correlate them after the fact? Or do you start with a single primitive: one structured event per unit of work?

The event-first approach works like this: when a request enters your system, you begin composing a structured event — a wide, context-rich record that accumulates everything meaningful about that request as it flows through your services. By the time the request completes, the event carries the trace ID, service name, duration, user ID, feature flags, error state, dependency latencies, cache hits, queue depths — whatever context matters.

You store that event once. From that single store, you can derive everything else: aggregate events over time and you get metrics; filter and read individual events and you get logs; stitch events together by trace ID and you get traces. One source of truth, many views.

This is what Charity Majors calls Observability 2.0 — and it’s the architecture that actually delivers on the promise of answering questions you haven’t thought of yet. Because the events are wide (carrying dozens or hundreds of fields), you can slice by any dimension after the fact. You didn’t have to anticipate the question at write time. You just ask it: “show me all events where cache_hit=false and region=eu-west and latency_p99 > 2s.” If those fields are in the event, you get your answer.

Compare that to the three-pillars approach, where you'd need to have predefined that specific metric, logged that specific combination, and hoped your tracing tool captured the right spans. Three tools, three queries, three partial answers — if you’re lucky.

The event model is the architectural decision. Everything else follows from it.

Establish a trace ID convention

Every request gets a unique identifier the moment it enters your system. That ID propagates through every downstream call — HTTP headers, message metadata, database session context. Every event includes it.

This is foundational. Without it, you cannot correlate activity across service boundaries. With it, a single ID connects every event across every service for a given request. It’s what lets you stitch individual events into a complete trace — the full journey of a request through your system.

OpenTelemetry handles those mechanics gracefully. Use it. It’s language-agnostic, framework-aware and it solves the propagation problem so you don’t have to.

Enrich your events at every boundary

Not every line of code should produce telemetry. That’s noise. The moments that matter are the boundaries and transitions — the points where a request crosses from one domain to another or where state changes meaningfully:

1. Request entry — what arrived, who initiated it, what context it carried.

2. External calls — what you called, what came back, how long it took.

3. State changes — meaningful transitions: a payment processed, a queue depth shifted, a circuit breaker tripped.

4. Errors — always, with full context. Stack traces, request metadata, the works.

At each of these boundaries, you’re not firing off a disconnected log line — you’re enriching the event that represents this unit of work. Add fields: payment_gateway_latency_ms=302, cache_hit=false, feature_flag_new_checkout=true. The event gets wider and richer as the request progresses. When the request completes, it captures the full story.

Make every field structured and discrete. Not a freeform string like “User 12345 logged in from 192.168.1.1” — but named fields: user_id=12345, source_ip=192.168.1.1, session_id=abc, duration_ms=47. Structured fields are queryable. “Show me all login failures for user 12345 in the last hour” becomes a single query instead of a regex excavation.

Derive your metrics from the same data

Once you have rich, structured events flowing through a single store, metrics become a view — not a separate system. You aggregate events to spot trends:

Read more

You’re working on an automotive assembly line. Your job is to install the steering column assembly and steering wheel. The next car rolls down the line. You grab the parts and lean in, tools in hand and try to fit the column. It won’t fit. Something’s off.

You look closely and realize the mounting point isn’t where it should be — it’s offset by a few inches. You figure, you’ll try anyway. No good. Worse, the chassis is aluminum and your steel column will cause galvanic corrosion on contact. You think to yourself — maybe this car is supposed to have something else entirely — a joystick, perhaps? You’re going to need time to research the platform and figure out what belongs here. But you’re out of time. The line grinds to a halt. Delivery stops. Something unforeseen has surfaced, and you need time to research the right solution.

That’s not how automotive assembly lines work. But it is how software works. Building software is about research, discovery and development.

Read more

Introduction

This activity is about making sure we’ve taken the time to think about happy paths as well as failure scenarios. It’s about adding fidelity to our use cases so that our finished product is a quality product — one that our customers will enjoy using.

As part of that process, we’ll explore variations on the scenarios in our design. For example, we’ve defined a scenario to make a purchase — but what about when the credit authorization fails? Or the product isn’t available in the warehouse? Or our customer adds a duplicate product to the shopping cart? What should we do if the customer abandons their shopping cart?

Working with your team to think through these scenarios — poking each scenario, prodding to see if it stands up under pressure, adding new paths if it doesn’t — will help ensure our design is solid.

Building better specifications

This activity isn’t just about testing our assumptions — it’s about creating better designs. We want to be sure we’ve thought about how code will be used before we build it. This outside-in pressure tends to produce interfaces that are cleaner, more focused and easier to consume, as well as code that is more reliable.

Read more

The incident

On June 4, 1996, the maiden flight of the European Space Agency's Ariane 5 rocket ended in catastrophic failure. About 40 seconds into its flight sequence, at an altitude of about 3,700 meters, the launcher veered off its flight path, broke up and exploded.1

The cost for this failure was estimated to be $370 million. The rocket was carrying four satellites intended to study the earth’s magnetosphere. Because of the payload’s destruction, building new satellites delayed scientific research into the magnetosphere for four more years.23

An elaborate and careful investigation into the cause of the failure was conducted.

While several factors led to a cascade of events, it came down to a data conversion error — an error that would have been caught in testing.

Unfortunately, the internal SRI software exception that caused the problem was never tested. Or, more specifically, it wasn’t tested in the Ariane 5 configuration. The failure resulted because Ariane 4-era software that used 16-bit signed integers had been plugged into a more modern system — a system that used 64-bit floating point data. The older Ariane 4 SRI software couldn’t cope with the new data format.

That’s the technical reason.

There’s a more important lesson here. The real reason for the failure was human error. Somewhere along the line, a decision was made: the decision that the Ariane 4 SRI module didn’t need to be retested in the new Ariane 5 configuration.

The investigatory board pointed to an underlying theme in Ariane 5 development as a root cause, that theme being, “that software should be considered correct until it is shown to be at fault.” The board took another view. Their belief was, “that software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct.”

Systems must be tested to prove correct behavior. We cannot simply assume they are correct.

Preventable failure

The key takeaways from the Ariane 5 disaster are:

1. We cannot rely on software to be reliable unless currently accepted best practices demonstrate that it is correct.

2. Continuous integration and testing for correctness is necessary — even when correctness has previously been proven.

The Ariane 5 case is widely taught in software engineering courses as the canonical example of how a simple, preventable defect — discovered late in the lifecycle (during operation) — can have catastrophic and costly consequences.

In fact, we know that the longer we wait to discover a defect, the more costly it is to fix. Research dating back to the 1970’s says we can fix a defect in design for nearly zero additional cost — or we can wait until production and spend north of 100 times the cost to fix it later (Boehm, 1979).

This exponential curve isn’t arbitrary. It reflects the reality that as software progresses through development stages, each defect becomes increasingly entangled with other code. Downstream systems begin to depend on it.

A requirements misunderstanding caught in design costs a conversation to fix. The same misunderstanding caught after integration means code rework, integration testing, release cycles, product updates and coordination across downstream dependencies. It can cascade into updating multiple products.

In concrete terms, that $25 defect can be fixed early, “for free,” or we can wait until later and spend $2,500 or more.

And there’s evidence that a 100X cost factor is by no means the upper limit. More recent research, in some cases, point to a factor of 1,500X cost to correct defects in operations. That $25 design defect just turned into a $37,500 cost to correct. There’s no upper limit — as the Ariane 5 team learned. (If you’re interested in reviewing more recent data there are references to NIST, NASA and Westland research in the footnotes).456

But there’s good news. We know how to prevent failures: By applying software quality assurance policies and practices — and pairing them with rigorous software testing.

Applying best practices

Test Driven Development (TDD) and Behavior Driven Development (BDD) are fundamental strategies to move defect detection left on the graph — ideally all the way to the design phase where cost is lowest.

Both practices create proofs that prevent defects. What’s more, they rely on executable specifications that prevent regression. Even after initial development, the test suite continues to catch defects early. A change that inadvertently breaks existing behavior is caught in seconds during the next test run — not discovered in production by users.

This ongoing protection is particularly valuable as systems age and original developers move on — the tests preserve institutional knowledge about intended behavior. Imagine how powerful it would have been if the Ariane 4 SRI module had included a continuously run regression test sequence.

The fundamental promise isn’t about testing — it’s about creating better designs. Both practices demand writing tests first. That forces thinking about how code will be used before thinking about how it will be built. This outside-in pressure tends to produce interfaces that are cleaner, more focused, and easier to consume, as well as code that is more reliable.

The deeper point

TDD and BDD aren’t really about testing at all. They’re about feedback loops and thinking clearly about intent before implementation. The tests are a byproduct — valuable, but secondary to the discipline of thoroughly articulating what you want before you build it.

Critics often attack the testing artifacts while missing this point entirely. It’s not always about going faster or whether every scenario needs to be test-driven. It’s whether your development process includes mechanisms that force clarity of thought and protect against regression. Both TDD and BDD are proven ways to achieve that.

TDD or BDD?

TDD catches implementation errors immediately. When you write a failing test before writing code, you discover mismatches between intent and implementation within minutes, not days or weeks.

The feedback loop is so tight that “debugging” often becomes unnecessary. When a test fails you know exactly what change caused it.

Compare this to traditional development where bugs lurk undetected until integration, verification or even user experience — by which point the original context has faded. Investigation becomes archaeological, time consuming and costly.

What TDD does well is clarify the boundaries of your system. Code that’s hard to test is often code with unclear responsibilities or excessive coupling. Even if you don’t achieve full coverage, attempting TDD reveals where your architecture needs attention.

But TDD has a limitation: it doesn’t “shift left” enough on our error discovery graph.

TDD is largely a technical exercise, focused on functional specification. As such it can get deep into the weeds of a feature — but may fail to rise to the level of product behavior. In other words, we might be applying our test-driven thinking to a conceptual design that is already defective.

That’s why I prefer BDD.

Behavior Driven Development

Behavior Driven Development is a collaborative approach that evolved from Test Driven Development. It bridges the gap between technical and business team members by expressing specifications as concrete examples, written in plain language everyone can understand (the “Given-When-Then” style, or Gherkin format).

Read more

A daily standup is a great idea — provided it’s done right.

A well-run standup:

1. Is a place to foster collaboration and mindshare.

2. Provides a forum to get help.

3. Makes sure the team doesn’t spiral.

Unfortunately, most standups that I join don’t do any of these things. Quite the opposite. These standups impede productive work, acting as a needless interruption that offers almost no value.

More often than not what I see is a team forced to stand together while each team member robotically itemizes their work since the last sprint. In some miserable cases, the “PM” and one or two team members hold the entire team hostage while a specific problem is diagnosed.

The standup stretches to 30 or 40 minutes. Your eight person team takes a cumulative five hour hit to their productivity.

Create value in your standups

If you Google “questions to ask at a standup” you’ll find the worst culprit sitting right at the top of the list: “what did you do since yesterday?”

Ban this question.

Almost just as bad is, “wha…

Read more

In the previous article I introduced a foundational architecture for our data pipeline, checking off several of our requirements. Here’s a quick recap:

1. We leveraged an internal cache for isolation to break our dependency on external data sources, improving our resiliency and scalability options.

2. We introduced the idea of an asynchronous message architecture, so we aren’t bound by lengthy, sequential processes.

3. We keep our annotations long-term with internal persistence so we don’t reprocess data when unnecessarily.

4. Source data as treated as an immutable source, thus protecting our data integrity.

5. Our visualization layer is responsive, running off our internal cache so that it’s independent from other components.

Here’s a sketch of the architecture as it stands now:

It’s a good start, but we aren’t finished.

There are still two requirements that don’t have a clear solution:

1. We want to be able to reprocess data, effectively “rewinding history” and applying new analytics. For instance, if we add a new analysis and insight to our pipeline, how are we going to go back and view that new analysis on old data?

2. Finally, we want to be able to model our insights — that is, visualize it in a presentation layer. But we haven’t talked about how that’s going to happen.

Once we tackle those last few requirements, we’ll fill in some details — such as, deciding on a cache technology to use. We’ll also take an end-to-end look at what we’ve designed to see if we missed something important.

Finishing our foundation

One of the most powerful and underused capabilities in data analytics is the ability to reprocess historical data with new logic. When you fix a bug in your analysis or refine a calculation, you often want to know: “what would the last three months have looked like with this new logic?”

Fortunately, our design thus far aligns pretty well with this goal.

Essentially, all we are saying we need to do is throw out information that is no longer accurate and then reprocess the source data and rebuild a new and improved picture of the world. We might do this for a couple of reasons:

1. We changed our analytics. Consequently, we need to throw out all of our old findings and recalculate our insights.

2. The source data changed. Perhaps there was a historical data update, invalidating our findings. Again, we want to throw out our findings and reprocess.

Looking at the problem from that perspective, the architectural answer is simple: we need a way to delete our annotations.

Once we wipe our annotations, we’ll need to make sure they are recalculated. There are different strategies we could use, but for the most part our options are:

1. Rebuild our annotations on-demand. This is fine if we’re OK with a bit of a delay while we wait for analytics to run. Depending on our visualization strategy we might be able to break the work up and trade one big delay for a few small delays.

2. Alternatively, rebuild our annotations preemptively. We might do it as soon as we realize there’s data to analyze, or perhaps we have some kind of schedule or external trigger. The point here is, we run the analysis before we think a user will want to visualize it.

We’ll add these capabilities to our architecture diagram. That means responding to both external data changes as well as internally sourced changes.

Rewinding history

To “rewind and reprocess,” we’ll just delete our annotations and then make sure there’s a mechanism to rebuild them. Here’s an example chain of events we’ll model in our architecture:

1. Administrative user invalidates some (or all) of the annotations we’ve stored in our cache:

   1. Optionally, if the data was invalidated because source data changed, the system loads fresh data from the external data source.

   2. The system deletes or otherwise invalidates the annotations in our cache.

   3. An event such as “analysis invalidated” (with some metadata, perhaps a date range) is emitted.

2. The analytics service is listening for events, and receives the “analysis invalidated” event:

   1. The event includes the date range of deleted data. The service checks the date range to see if annotations are missing from the cache.

   2. If the data is still missing, it runs the analysis and rebuilds the missing annotations.

   3. As annotations are rebuilt, they are written to the cache.

   4. An event such as “analysis completed” (with some metadata) is emitted.

3. Separately, a visualization service might be running where a user asks to display the visualization for some missing data:

   1. The system checks the cache, realizes annotations are still missing, and sends a “run analysis” message directly to the analytics service.

      1. The analytics service is already working on analyzing the data so it can ignore the message, or respond with an “analytics in process” message.

   2. The visualization presents some kind of “waiting for analysis” display while it listens for the “analysis completed” message. When it receives the message, it loads the annotations and displays them.

This is one way to handle a specific chain of events. We’ll need to handle other events and commands as well — such as an event to tell us external data has changed.

Since we’re talking about responding to events that happen, it makes sense to use event storming to model those events and commands. We can model “things that have happened” (the events) as yellow cards, and “what we want to happen” (the commands) as blue cards.

If you need a refresher, Chapter 2.3 Strategic event storming and Chapter 3.4 Tactical event storming provide in-depth explanations on modeling event driven architectures.

Here’s an updated architectural drawing showing how our events and commands interact with the system:

Persistent annotation cache

One of the things we haven’t decided yet is exactly how we’ll store our annotation data. There are quite a few options, and certainly more than one strategy that will work well.

In part two I hinted at using a relational database to store a copy of the external source data. This makes a lot of sense, but it isn’t the only option.

We could store all of our data in a persistent data store such as MongoDB, a noSQL document-oriented database. Document databases offer some fantastic search capabilities and, perhaps most interesting, don’t force us into a predefined schema — we can, essentially, attach anything as our annotations.

That’s a nifty idea, because we want flexibility to change our data analysis and come up with new data sets to store, freely associating them with source data.

We could also stick with a purely event-driven architecture. In this kind of system, we would emit annotations as events into a persistent queue (such as Kafka). The persistent queue becomes our source of truth — and annotations are just messages on the queue, like, “sprint X: total out-of-sprint hours: 500.”

With many options, it’s probably a good idea to decide on our business requirements:

1. Obviously, we want to cache the original source data, so we aren’t tightly coupled to an external system. (We know the external system is a relational database).

2. We want to easily create new annotation data sets. If we dream up a new calculation (maybe, “number of activities completed after hours on Fridays”), we want to be able to store it easily.

3. We know we’re using DataFrames for analysis. Storing our DataFrames without translating them into something else would be convenient (but not necessary).

4. Our annotation data should be persistent (to avoid unnecessary delays rerunning an analysis) — but, once a particular report or visualization is no longer being used, we probably don’t need to keep it around.

5. Finally, we’ll want to think about performance and wait times. We definitely don’t want users to experience a slow system.

The relational cache for source data still makes a lot of sense.

For our internal annotation data, something more flexible is desirable. We don’t want to be constrained by a relational schema, forced to update the schema every time we think up a new report or annotation.

While a noSQL solution seems like a good fit, it may be an unnecessary complication. It would afford long-term persistence, but unless we run into performance problems that long-term persistence may be pointless.

Designing for flexibility makes sense.

We’ll implement our internal annotation cache as a service. The service will:

1. Allow us to store annotations (data structures) in a persistent fashion (but, not necessarily permanently).

2. Provide a shared data service (allowing other services to leverage the shared data).

3. Give us scalability (we can boost processing power or memory, start up multiple copies, or upgrade to a more powerful data store technology).

4. Allow us to start simple, just storing annotation data in memory. A long-running service might give us all we need (after all, we’d only rebuild our annotations if we restart the server).

Let’s update our architecture diagram one more time:

Read more

Introduction

This activity creates fidelity in our engineering models — the diagrams and specifications that give us enough technical depth so we can proceed to coding.

Everything we’re going to build should be modeled. That might sound like of lot of work, but with the right drawings and specification style, it’s not — plus, our models should be reusable. For instance, once I’ve modeled my implementation of a circuit breaker, I don’t have to model it again. I can just use that circuit breaker design everywhere I need it.

The approach I’ll introduce here increases the fidelity of your technical designs using data inputs and outputs, paired with very clear and specific functional results. It builds on your existing engineering artifacts and further validates your value stream, making sure alignment with business objectives is maintained. We capture our control processes and at the same time enhance communication among those designing the solution.

One of the challenges with modeling is deciding on what’s effective yet not overly constraining or complex. We want to define some clear standards of practice and make sure the engineered product is fit for purpose. At the same time, teams should have the flexibility to take a different path if that different path is more fit for purpose.

The modeling approach we use in the Delivery Playbook meets those goals:

• We define engineering artifacts using two specific diagram types, neither of which are difficult to understand or use.

• Clear objectives are defined while giving teams enough freedom to follow a different path — so long as those objectives are met using whatever approach they choose to follow.

Read more

In the previous article I introduced data pipelines, highlighting their ubiquity and a few of the challenges you’ll face architecting a good data pipeline. Here’s a quick recap of those requirements:

1. Moving data efficiently. Pipelines are all about making sure the right data gets to the right place at the right time, in the right format. For many applications, like air traffic control or the stock market, timing is everything. The architecture has to scale, meeting demand without degrading — often with life-saving implications.

2. Data integrity. This is about ensuring the source data is what we think it is — incorruptible, unchanged as it moves through the pipeline. We have to make sure there’s no data loss, no translation errors, no dropped information. Yet, at the same time, pipelines are fundamentally about processing that data, deriving new insights from it.

3. Non-destructive annotation. Being able to add to the data in the pipeline efficiently and flexibly is key. This makes it possible to create new insights from the source data — but it implies a schema that can grow in unpredictable ways.

4. Rewinding history. Pipelines evolve, refining their algorithms and adding new processing steps. When we fix a bug or refine a calculation, we need to know, “what would the last three months have looked like with this new logic?” If we can’t reprocess the data we’ll never know if an inflection point in our analysis was due to a different calculation or new business conditions.

5. Surfacing discoveries and creating actionable insight. All of these steps need to culminate in actionable intelligence. Getting those pieces to work together, then correlating all that disparate data and presenting it clearly, logically, can present its own challenges. Does the data processing sequence matter? How does one element depend on another? What happens when your analysis depends on external data or fundamentally different data structures?

In this article, we’ll develop a foundational data pipeline architecture. I’ll approach the architecture iteratively — beginning with something simple, exploring whether it fits our purpose, and refining it. Our final architecture will provide a foundation that can grow into a scalable, powerful data pipeline platform.

For our use case, we’ll design an application that tracks team activity across projects in an organization. Our goal in tracking all this information is to deliver new business intelligence that gives useful, actionable insights into team efficiency across the organization. That’s intelligence we can use to remove bottlenecks, improve team productivity and velocity, eliminate wasteful process and more.

What are we measuring?

Before we get started, it’s a good idea to know what we’re trying to influence. We want to surface insights, not just tables and charts. The difference between data and intelligence is whether it impacts our decisions.

Actionable intelligence changes what you do next.

For our test case we want to develop a better understanding of team efficiency. There’s an existing body of work that covers this topic, and we’ll leverage it.

Our plan is pretty simple. Following the principles of a waste walk, we’ll have team members tell us whether their day-to-day activities are creating bottlenecks or inefficiencies or draining their effectiveness. They’ll do this by capturing activities and marking them as “in sprint” or “out of sprint.” (If you aren’t familiar with those terms, take a look at part one of this series).

We’re going to assume this data is available — perhaps coming from a JIRA dashboard or other time tracking application. I’ve actually built a mobile-friendly application that does this. It’s part of a waste walk analytics platform in use today. For our purposes, what matters is the data — not the mobile app.1

The data schema is straight-forward. Team members are Users, they track their individual Activities, and each Activity belongs to a Sprint.

If we diagram out the basic data model, it would look like this:

As you can see, Teams are also modeled (so we can manage sprints and their team members as a group), and every activity has an Activity Stereotype. These stereotypes are basically the kinds of activities team members can track in the system. They can’t just record a random event; instead, they pick an activity from a list of “stereotypes” like “meeting,” “writing code” or “story refinement.”

We’ll take a look at the specific fields in a little while. For now, we have a conceptual understanding of the data — that’s what we need to move forward.

Data versus insight

Given the data we have available, we need to ask ourselves: what kind of insights can we derive from the source data? How might those insights change our behavior?

Here are a few ideas:

1. If team members spend too much time doing activities that could be automated, we’d want to prioritize automation — letting those team members get back to more important work.

2. If a lot of time goes into investigating defects, we would probably want to know about it so we could investigate why we have so many defects (and fix the root cause).

3. It would also be informative to know when the team works on fixing defects; is it after the product ships, early in design, or somewhere in-between? Early defect correction is much better than late defect correction.

4. A lot of planning late in the product lifecycle — for instance, planning after a sprint has started — might indicate opportunities to improve coordination or design process. Late-stage planning often creates bottlenecks for team members.

5. If individuals feel their time in sprint ceremonies and meetings is not productive, we should probably investigate why they feel that way — especially if it’s a lot of time.

6. If we knew how many different things team members had to juggle, it might influence how we prioritize work and focus time for the team.

7. Probably most important: we want to know, across the organization, exactly how much time is going into inefficient or wasteful activity. A small amount is fine, but in aggregate, the time and effort lost could be costing us more than we realize.

I’ve done these kinds of analyses for many teams — and the insights are almost always transformative. Just one example: one team found they were spending close to 23% of their time on activities that could be entirely automated. The effort was spread out, of course — an hour here, another there. At an individual level, it was about a day’s effort each week, tweaking manual deployments, doing environment tuning, “sanity testing” in staging and production.

Now scale it up. For a team of eight, that’s 150 hours per sprint. Stated another way, by investing in automation, the team would get back 150 productive hours every single sprint and their quality of life would be improved at the same time. Knowing those numbers, prioritizing Infrastructure as Code (“IaC”) for deployments, automating those repetitive “sanity tests” — suddenly makes a lot of sense.

Transforming raw data into insight

We know what data is available to us. But how do we transform it into actionable intelligence we can use to create better outcomes?

The raw data doesn’t really get us there. That’s where the data pipeline comes in — by transforming that data into something we can use:

1. First we need to “intake” our source data. Once we can access it, we can analyze it, of course.

2. Then, we transform the data into what we need. We aren’t planning anything overly complicated (at least, not yet). Here’s the gist of what we need to do:

   1. Separate “in sprint” versus “out of sprint” activity. We’ll be focusing on the out of sprint activities or, in Lean Engineering terms, the “non-value add” effort. This “non-value add” work is where we’ll find our biggest gains.

   2. Summarize data about activities. For example, we’ll need a way to add up all the activities that could be automated. That tells us how much effort is spent here.

   3. We also want to be able to count up and somehow tally the time between activities (the “context switching time”) for each team member — probably as some kind of aggregate or average number.

3. Of course, we’ll want to visualize the results.

That all seems very straight forward. The first logical step is to think about how we’re getting our source data.

Start with the source data

The first thing we need is our source data. Fortunately for us, it’s readily available and easy to access.

Our tracking application stores all this activity data in a SQL database, using a traditional relational format. That makes a lot of sense — it’s a relatively static data structure, and the relationships between teams, users, sprints, and activities are clearly hierarchical.

One approach is to just read the data directly from the SQL database. This would be the easiest solution. Once we read the data, we can perform our calculations, add in the context switching data, and display it:

This does raise a few interesting questions, though:

1. What happens if the source data changes in some way or if we can’t access the SQL database when we want to display our data?

2. How, exactly, are we going to store the results of the calculations and annotations?

3. Does this mean that every time we display the results we have to run through the entire pipeline, reading the data, performing calculations and then displaying our findings?

4. How does constantly accessing an external database impact our performance?

Data integrity and availability

We want our application to be highly available and reliable.

The problem is, we’ve created a dependency on an external system — the SQL database. If the external system changes or just isn’t available when we need it, we’ve got a problem. It would break our data pipeline. We need a solution that ensures both the integrity of the data as well as its availability.

The simplest solution is to just copy the data and keep a local cache of it. Whenever the source data is updated, we can update our local cache. If something goes wrong — for example, the data format changes or we can’t contact the SQL database — we can keep using our local cache until we fix the problem. That gives us high availability.

This also gives us a degree of certainty about the data format.

While the external SQL database might change, our internal cache can potentially remain constant. For example, let’s say the external database is using an Integer data type for activity duration, but at some point they decide to change it and start using a Time type instead. Since our local cache is under our own control, we have a few options:

1. Update our local cache to use Time as well, and change our pipeline code accordingly.

2. Convert the incoming data from Time to an Integer and store that. Leave our code as-is.

3. Decide to use a completely different type, like Duration, in our internal cache and make similar changes to our code.

Our strategy would depend on the situation — but we have options. We’ll examine a few different ones in a bit, but for now we’ll just note the internal cache in our architecture.

Adding a cache to the architecture might look something like this:

I’ve added an “ACL,” or “Anti-Corruption Layer,” to the intake process. ACLs are used to protect the data integrity of our domain — in other words, we don’t control the external SQL database but we do control our internal cache. That cache can, for instance, use a Duration to represent the passage of time regardless of how the SQL database represents time. The ACL is where we translate from the external format to our internal, domain-specific format.

It gives us a nice layer of isolation from external change. Generally speaking, creating isolation between services is a great strategy for improving reliability (as well as scalability — more on that later).

Benefits of isolation

I pointed out that by copying external data into our own system we are creating isolation. This is an important architecture topic for a few reasons:

1. Isolation means we are not inexorably dependent on an external system. If that external SQL database goes down, or the data changes, we can still function.

2. We also get architectural independence. If we decide to change the structure of our internal data cache, we can. It’s under our own control.

3. Being in control of our own data architecture also means we’re in control of our scalability. If our data analysis product becomes the hottest thing since sliced bread, we can scale up. We might add more servers to our cache, or we might evolve our architecture toward CQRS patterns — we’re in control, and it’s all hidden behind our product API.

From this perspective, copying data is a good idea. We need to remember there’s an external source of truth. But we remain in control of when we run data intake, how we change the data on the way in, and how our own data is structured.

Powerful, non-destructive annotation

Our original requirements include making sure that we maintain our source data integrity. The best possible approach in that regard requires two things.

First, we have to capture every record, in its entirety. That ensures we won’t miss anything important (including the possibility that something becomes important later, even though we thought it wasn’t important at first).

Second, we can’t change what we capture — we can only add to it. This means our analysis and data annotation can’t alter the source data — it’s immutable. We have to find a way to piggy-back or pair our new data with the raw source data.

We’ll leverage that in our code. Let’s say, for instance, you wanted to find out how many seconds of “out of sprint” work each user tracked in a sprint and you also want to make sure paid time off is not included in that number. Here’s a very simple example, using a snippet of Explorer code:

data
|> DF.group_by([“user_id”])
|> DF.filter(stereotype_code != "pto")
|> DF.summarise(out_of_seconds_sum: sum(out_of_sprint_seconds))

Data is just an array of raw data records. The group_by line groups all records with their respective team member. Then, with filter we make sure paid time off is not being included. And finally, we sum up all the out_of_sprint_seconds and put the result in a new variable, out_of_seconds_sum.

Two important things to note here: first, we haven’t altered the source data in any way. Second, we’ve annotated our data set by associating that new variable, out_of_sprint_sum, with our DataFrame.

We’ll explore this in depth soon. For now, I just wanted to give a quick taste of what we can do and explain why it’s valuable.

Tackling data analysis

We’ve decided how we’ll get the data, but there’s more work to do. How are we going to perform our calculations and store the results?

Read more

These past few months were very busy. I had two projects compounded on top of each other — one, a client modernization effort; the other, a new release of an analytics tool we use with our customers. It was more than I like to take on at one time and, yes, I got a bit burned out.

Which is a good reminder — prioritize your well being!

Work will always be there. Life will pass you by.

Recharge in Malaga

After getting all the work out of the way, we took some time to recharge: a few weeks off in Malaga.

Read more