When should you think about security?

Security doesn't have to be a pain. Don't put your whole project in jeopardy. Adopt a "security first" mindset from day one with this 3 step plan.

Photo by Brook Anderson on Unsplash

I’m working on the next playbook chapter, 2.8 Delivery processes & tools. It’s a challenging topic and one that I intend to revisit, expanding on it with several supporting articles. This is the first, going deep into how front-loading your software security eliminates huge delivery risks. On the other hand, procrastination will lead to flaws and shortcomings that could cost you 10-fold or more downstream. I hope you enjoy the stories I’ve included. Both are examples of real-world projects, one that went off the rails and one that went swimmingly, all because of early security decisions. You’ll also find a step-by-step guide for implementing security controls in your build pipeline.

Sexy or reliable?

I remember working on a sexy new product nearly a decade ago for a large company in the hospitality space. It was at the conjunction of customer experience and AI, or at least the precursor of AI. The whole point was to predict what customers want. Imagine walking up to a bartender at a resort or casino, and she already knows your favorite drink. She greets you by name. Asks if you enjoyed that Old Fashioned you had last Friday evening. Even though she’s never seen you before.

These days, there’s a good chance you’ve experienced something like it, though at the time it was pretty cutting edge. They had a massive team dedicated to the project and even had it working in less than six months.

But here’s the catch: After showing off the fully working product, it took another 15 months to get the product out the door. Over a year, even though it was feature complete and “ready!”

Plus, the finished product didn’t quite live up to expectations. Some of those fancy features that were so exciting in the demo never shipped. That made product management very unhappy. It cost millions, and they lost a nice competitive edge.

So what went wrong? How could it take a year to get a seemingly finished product out the door?

If you’re new, welcome to Customer Obsessed Engineering! Every week I publish a new article, direct to your mailbox if you’re a subscriber. As a free subscriber you can read about half of every article, plus all of my free articles.

Anytime you'd like to read more, you can upgrade to a paid subscription.

It all started with security. More precisely, it started without security. That is to say, security was conspicuously absent from early design and implementation. The reasoning was simple: It was more important to build features, demonstrate the product, and prove viability early. Once the product was proven, security could be added before going into production.

One of the early design decisions had to do with the movement of customer information. Essentially, point of sale devices — say, an iPad where your bartender enters orders — needs to have a lot of information about you, the customer. Information like what you ordered recently, the kinds of drinks you enjoy, and even your method of payment (so you wouldn’t have to provide it each time you placed an order).

In product demos, this worked fabulously. You literally could walk up to the bar and be greeted as a regular. The bartender saw a photo of you, your most recent order, your drink preferences and your open account. All before you could even plop down on a stool and say, “Hi.”

Behind the scenes, the system used a distributed, event driven architecture. With geolocation from your phone, a digital key, or a fob, your location is constantly tracked. In a venue with a lot of bluetooth sensors (for instance, a casino or a bar), your location can be very accurately pinpointed. All of that movement data creates events, broadcast over an event stream, ultimately being listened to by an iPad sitting behind the bar. Then the point of sale, the iPad, just requests a copy of all that information about you from the system.

That’s a lot of interesting, highly personal data. Loaded into a lot of disparate little services. Running all over the place, disconnected from each other, doing things their own way — with lots and lots of copies of information.

And now, with the fabulously successful demo behind the team, comes the tough part: Make sure it’s secure. Lock all that data down. Make sure it can’t be compromised.

The first step toward a secure system was locking down all those different services. Adding a security layer to each one, so they couldn’t get hacked. So information wouldn’t leak out. That means going back to the delivery pipeline and injecting things like code scanning, container scanning, penetration testing. Since each service was unique, that meant a lot of work — and it was just the start.

Securing each service led to another realization — most of the events flowing around the network were not, themselves, secure. Many were unencrypted. But that wasn’t the hard part. In order to be compliant with some security standards, the system had to segregate personal data from identifying information. That meant a complete rearchitecture of the whole event broadcasting system.

It also meant going back to the drawing board for most of those loose lipped and chatty services and thinking really hard about what information they were sharing. When is it actually OK to expose a picture of someone? Their payment profile? Their purchase history? Their location history?

And it didn’t end there.

The toughest part of it all was compliance. GDPR, and the “right to be forgotten,” meaning the right you have to click a button somewhere that says, “I don’t want you to track me, and I want you to forget all about me.” The thing is, the event driven architecture was based on an immutable stream of events. Anything written to the event stream was there, forever. It’s kind of the whole point of an event stream, but it also means information on the event stream can never be erased. Now the team was faced with finding a solution that would fundamentally change the very heart of what made the system work so well.

It took a long time. Well over another year. Some features had to be cut out of the first release because they posed some difficult challenges — and ultimately, shipping something became an ever-increasing priority.

The project became a death march. What had been an exciting, ground-breaking product became a late, over-budget project built on missed promises and features that didn’t actually work.

The flip side

I have another story. It’s a lot brighter.

I recently led a small team — just a handful of developers — building out an application in the healthcare space. Healthcare applications need absolutely, unquestionably strong security from day one.

On day one, the question came up, “How can we, a development team of about 3 people, build a secure architecture and still get a product out the door in a matter of months?”

The answer might surprise you: Front load all the security requirements. Do it right from day one. Take the time to build a solid delivery pipeline that does “all the things,” the scanning and testing. Focus on a secure architecture that properly separates personal data from identity data. Use the right technologies and foundation right from the start.

One of the first things the team did was specify the delivery pipeline requirements: It had to automate the build, stand up ephemeral environments, run code and container scanning, even do basic penetration testing. The reasoning was simple. With a fully automated pipeline, the team would spend more time working on features. The automated pipeline would protect them from mistakes.

Architecturally the product had to be cryptographically secure, support industry grade identity and access management, and protect personal data.

And they did it. The first product went into production in about 3 months. It was secure, it was feature complete, and it worked from day one.

It turns out that building something correctly is a lot less expensive than building it wrong and fixing it later. But it’s also important to leverage everything you can, and focus on building value: Building new ideas and new product that doesn’t already exist. Don’t spend a lot of time recreating, say, identity and access management when you can leverage a perfectly good technology that meets your needs. Don’t try to reinvent threat scanning and prevention when other people make really good tools available to you. Don’t waste your time — be intentional and focus on what matters: your unique product value.

Security from day one

The lessons to take away from both stories are these:

1. Never compromise and settle on what you know is an inadequate design. Build it correctly from the start.

2. Don’t reinvent something that already exists. Invent something new, while leveraging existing technology as much as you can.

The first rule is about intentional design. The idea that you need to carefully consider all aspects of your product, and keep all of those different aspects in mind as you design it. That doesn’t mean designing and building every single feature from day one. It means considering all of the important implications, the foundation, your goals and factoring in anything that makes a long-term difference. For example, GDPR and the “right to be forgotten” is foundational when it comes to data architecture design. It’s too big a piece to ignore until later.

The second rule means, focus on your strengths. If you’re building a healthcare application, focus on creating value in healthcare. It doesn’t make sense to design, build, and maintain an identity and access management solution from scratch when there are plenty of fit for purpose products readily available. It’s not realistic to think, “I’m going to build a better solution than companies that specialize in security, like Amazon, Crowdstrike, and Okta.”

Being intentional in your design

Intentional design is about avoiding accidents. To “be intentional” means to do something on purpose, in a calculated, considered manner.

If we take the time to consider all the implications of building software, the labor involved in going back and fixing our work quickly becomes clear. The cost of correcting defects in software has been written about for decades. I was introduced to the concept in college. I remember learning about the long-term cost of correcting mistakes in the 1980’s, when I saw this diagram in my Software Engineering textbook:¹

Reproduction: First appearing in Software Engineering, 1979.

This is why I follow a prescriptive design process. I don’t mean a rigid design process. I mean a thorough and detailed process that takes care to consider all aspects of software design and implementation.

The goal of having such a careful, intentional design process is to make sure no important details are forgotten. One of those details is security — and it’s a critical one. Information security is complex, and often a very deep rabbit hole to explore. It can lead to fundamental changes in how we think about, protect, and handle data, as the team in my first story learned.

Hence, lesson number one: Never compromise and settle on what you know is an inadequate design. Build it correctly from the start.

Leveraging instead of inventing

Leveraging technology is about not reinventing the wheel, and instead pouring your own effort and creativity into something new.

I’ve worked with a lot of organizations that demonstrate a bias against “not built here” technology. It’s particularly prevalent in finance, but not exclusively. Engineers, after all, love to create stuff, and often dislike the way something works out of the box. It’s easy to think, “I could design that better.”

But it’s a slippery slope, one that quickly leads to reinventing technology that simply doesn’t move the needle. It doesn’t contribute value to your own mission. It doesn’t speed you up — instead, it slows you down. In many cases, the idea that “I could design that better” turns out to be a falsehood.

Instead, we need to be intentional about where we invest our time. The right answer is always to invest in building unique, new features that are specifically about our unique business value. Otherwise, we waste a lot of time — and potentially, never even finish that sexy new product.

Hence, lesson number two: Don’t reinvent something that already exists. Invent something new, leverage your unique abilities and values.

Building your security pipeline

Being intentional in our design means not missing the important details, not compromising, and not reinventing. So unless your business model is to actually build a better security solution, you need to find a fit for purpose solution and start using it “day one.”

The good news is, it’s not hard. It really isn’t. I know implementing all of those security constraints sounds like a lot of pain — but it’s not. Not compared to spending a whole year reworking your entire product down the road.

Here’s my three-step plan for building a solid, foundational security pipeline, along with a tech stack that will get you up-and-running on your first build.

If you use the referral button below, you’ll earn free premium access to Customer Obsessed Engineering. Just three referrals will earn a free month!