Software Engineering for Disruptive Innovation

… is different from software engineering for sustaining innovation, but not in the ways you might think!

For the unprepared this often looks like chaos and is far from a systematic process typical in more traditional projects. Yet this is the most fun and inspiring work for me, because the stakes are high, the pace is fast, the enthusiasm and like minded determination are infectious. However adjustments to engineering attitudes and practices may be needed to get at this fun.

tl;dr

1. Disruptive innovation is about discovering a business, not merely about building a product.
2. To offset high business risk, technology choices and architecture need to be a disciplined balance of modern and conservative.
3. Agility is essential in the spirit of the original agile manifesto, but formal “Agile” processes aren’t agile enough.
4. Refactoring happens on every sprint, because development happens in layers as well as in features.

So, What is Disruptive Innovation?

(Dear Reader, So sorry about this section ;) It’s a necessary tangent to tunnel under the hype and hyperbole of “disruption”, and frame the engineering perspective. — John)

Perhaps counter intuitively, engineering isn’t the center of the disruptive innovation universe. Rather engineering is just one of several concurrent work groups of an enterprise trying to find and exploit a market opportunity.

Clayton Christensen’s classic book “The Innovator’s Dilemma” draws a clear distinction between sustaining innovation and disruptive innovation. To be sure most innovation is sustaining and supports market incumbents by improving or creating products for existing markets. As Christensen observes, the majority of technological breakthroughs are actually motivated by and applied as such sustaining innovations. In contrast disruptive innovation is relatively infrequent and happens when three things occur:

1. Technological improvements outpace the market demand in a given value network.
2. Existing technology is used to create or address a new value network where excessive risk or lower margins make it unattractive to the incumbents.
3. Growth of the new value network consumes the surplus technological improvements, which allows them to threaten incumbents in their value networks.

Christensen presents a variety of case studies which follow this pattern. For example in the disk drive industry:

1. The capacity of 14 inch drives surpassed the demand of the mainframe computer market.
2. The advanced technology of the 14 inch drives was applied to creating 8 inch drives which were needed by the infant mini-computer market. And the lower margins of the 8 inch drive market disincentivized the 14 inch drive manufacturers from entering the new market.
3. The mini computer market expanded which created higher margin opportunities for the 8 inch drive manufacturers in the mini computer value network and eventually the mainframe value network.

For us the key observation is that technology improvements were developed for the incumbent mainframe market. But what ultimately made these improvements disruptive was their application to a new mini-computer market. Or to put the lesson most bluntly, disruptive innovators steal technology improvements from sustaining innovators.

This pattern repeats in a wide variety of examples, and in the 20 years since the “Innovator’s Dilemma” was published this story has played out so frequently that the theory has spawned an active area of management research. It should be noted that not all successful start-up enterprises are categorized as disruptive (e.g. Uber), and not all game-changing innovations are from start ups (e.g. iPhone). However the research supports a strong correlation between start-up success and disruptive innovation.

What this Means for Engineers

The key observation is that disruptive innovations are not characterized by technology breakthroughs, but rather characterized by opportunistic applications of existing technologies to expose new value networks.

“opportunistic”

… implies a short window of opportunity. This doesn’t just mean that engineers need to work fast toward a minimal viable product (MVP), but rather all of the business functions: marketing, sales, product , design, and engineering must be stood up concurrently and quickly.

“applications of existing technologies”

Disruptive innovation is inherently risky, so mitigating those risks through modern, established technologies for languages, tools, and frameworks is always a good idea. Even the engineering of basic building components should tilt toward more conservative approaches, since those individual components usually have little business differentiation value for actual disruptive innovation.

“to expose new value networks”

And this is where rubber meets the road and the core business differentiation occurs, but it’s also the source of the challenges. Product, design, and engineering must all start with uncertainty about who the user actually is. Even though product managers construct personas of target users, the personas and their priorities are highly speculative until there are actual paying customers. User persona uncertainty flows downstream into all development processes to create uncertainty in UX design, and UX implementation, and even data structures and system architecture. Furthermore the discovery process of “product market fit” creates uncertainty in marketing, branding, sales channels, and even management of the business. Any of these can deflect the priorities of engineering. Overall the level of uncertainty involved in disruptive innovation is often surprising, uncomfortable, and occasionally intolerable to uninitiated engineers.

Engineering for Disruptive Innovation — Keeping it Fun.

Use modern, established technology for tools, languages, and engineering frameworks. Strategy: Default to common patterns and best practices on decisions that aren’t central to market differentiation. Assume that none of your innovative ideas are particularly original. The fun part of this work isn’t solving “interesting” problems with elegant algorithms, nor is it pioneering cutting edge software, or even promoting the latest fashion in languages, frameworks, or patterns. Rather it’s discovering existing solutions and molding them to our novel use cases. And furthermore it’s far easier to learn from prior art, and absorb the background needed to implement the novel solutions when they are truly needed.

Show progress early and often. Be mindful that other business functions and project sponsors are managing substantial risk. If they perceive engineering risk, either real or imagined, they will become more directly involved to the point of micro management (minus fun), or cut you out of the loop with contingency plans (also minus fun). On the other hand if they perceive that you are working hard, making progress, and directly communicating risks, then you and the team prevail to earn more freedom and collaboration (plus plus fun).

Engineer for flexibility. Expect changes and commit to refactoring. It’s almost always better to create a specialized solution with a path to future refactoring, than to develop a general solution that covers irrelevant use cases. It helps to imagine developing in layers where each new feature is built upon a layer of refactoring. In other words, don’t even try to write perfect code for each feature — get it working first for the immediate use cases. Then you can come back to it (potentially multiple times) for clarity, modularity, generality, or testability. Then each refactoring pass will add value for the use cases that actually matter.

Architect according to the stable dependency principle since you’re committed to refactoring. Keep an eye on component dependencies. Uncouple them when possible, and prioritize stability and testing on components that have many or deep dependencies. Then components with zero or few dependencies can be loose, experimental, incomplete, or even sloppy. My most effective work rhythm is to commit to refactoring in the first half of a sprint, and pile on the dependent features in the second half of the sprint.

Pay attention to Conway’s Law (roughly: software systems mirror the communication patterns of the people that create them). So for example if you have a team member who is overloaded by demands from too many people, you probably have a highly coupled software component that only that person understands. If you can partially uncouple that component, you can then distribute responsibility. Conversely, if you can cleanly disperse responsibility among developers, they will find ways to uncouple the software. Fundamentally, you can use Conway’s Law to your advantage by sync’ing the software architecture and team formation.

Use agile principles but loose “Agile” processes. The original agile principles express these specific values:

Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

There are few circumstances where these values are more apt than in disruptive innovation. Unfortunately modern “Agile” is too often the opposite of these values. And “Agile” practitioners are often surprised by the failure of their processes and tools to manage engineering for disruptive innovation. In particular these techniques aren’t well suited for capturing rapid changes in priorities, expressing risks, surfacing alternatives, and identifying hidden dependencies. Stubbornly forcing them to be used beyond their capabilities is not productive nor fun for anyone. Understanding these limitations and rediscovering the agile principles is the path back to fun.

Avoid optimization and scaling. File these in the “good problems” folder, and save that sort of fun for when they become relevant. If you commit to refactoring, decoupling, and stable dependencies, then you can revisit these problems along the dimensions that actually matter and bring value to the enterprise.

Engineering for Disruptive Innovation is more a Discovery Process than a Construction Process.

This is what really makes our work fun — being part of a team that discovers a unique intersection of need, market, and technology. However the nature of that discovery process is necessarily messy, and almost impossible to plan. Hopefully the above guidance and suggestions on what to expect and what to prioritize will give you the winning edge in this discovery process.

Software Engineering for Disruptive Innovation was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.