Prototype More, Test Often, Pilot Less.

So, let me say right from the start, If you’ve happened upon this page looking for more about this little fella, you’re in the wrong place. This isn’t a blog post about a prototype test pilot, it’s a post about prototypes, tests and pilots. More specifically, the differences between a prototype, test and pilot.

When we took a retrospective Bromford Lab view of 2018, we saw that the most popular post last year was actually one that Tom had written way back in 2015 entitled ’What’s the difference between a test and a pilot?’. Based on the continued popularity of Tom’s post it seems like this is a question a lot of people are asking themselves. In fact, as an in-house design and innovation unit it’s a question close to our hearts, so we thought that to kick off the new year we might revisit the topic and refresh our definition of prototypes, tests and pilots.

One of the principle reasons that the Lab exists is to reduce the number of cases of inititivitis diagnosed at Bromford. Over the past few years, together with colleagues, we’ve been able to achieve this by among other things, the introduction of a structured approach to innovation, and repeat applications of a structured design process consisting of problem definition, prototyping, idea generation, testing and piloting. But despite this, we still often get asked the question — WHY?

‘Why can’t we just get on and do things? And why do we need to test something — isn’t it quicker just to get out and pilot something?’

Cue the flashing red light and claxons. Pull down the shutters and seal off the building — we’ve got an acute case of initiativitis! Whilst it might seem like the quickest way to get results is to jump straight to pilot, in fact doing things this way can often take longer to arrive at the right solution, or in extreme cases it can even lead to bad ideas being scaled. The best approach is to use prototyping and testing to rapidly learn more about a problem, fail safely, kill bad ideas early, and move on quickly.

‘But prototyping, testing and piloting are just different words for the same thing right?’

No, prototyping, testing and piloting are different tools with very different objectives and outcomes:

As Tom set out in his original post, whist on the surface they might feel like they are the same, prototyping and testing are nearly polar opposites to pilots in the way they go about things.

Prototyping

Prototyping is a flexible tool which can be deployed at various stages of the design process. At its most basic, prototyping can be as simple as sketching or modeling during an idea generation session — creating an ’artifact’ to aid creative thinking or help communicate an idea. But prototyping can also be used to test an idea; not by creating a smaller working version of a service or product, but by testing the many different component parts or even thinking abstractly in order to start to uncover what it might feel like to use the service or product. Generally speaking prototyping is about learning quickly and as a design tool it can be split into two distinct categories: low-fidelity and high-fidelity.

Low-Fidelity Prototyping

Low-fi prototyping can take the form of:

• Storyboarding
• Sketching
• Modeling (using lego or craft materials — think Doc Brown’s ‘crude’ models in Back to the Future).
• Screen mockups

Low-fi prototyping is fast, cheap and disposable, making it possible to gather feedback quickly and iterate concepts on the fly. Using Low-fi prototyping allows you to get a feel for how a service or product might work using minimal time and effort; focusing on the detail of a design comes much later and takes a different form of testing.

High-Fidelity Prototyping

High-fi prototyping looks a little more like a finished article, although in terms of testing it’s still a pretty ‘quick and dirty’ way of doing things. A little while ago we worked with co-op digital to help them gather insight into a service they were thinking about developing. We designed a set of artifacts to make the service seem real and spent a day door knocking in one of our Bromford localities. By talking to people about ‘the service’ and essentially asking them to buy-in to something that ‘looked real’, we got a huge amount of learning which resulted in the concept being killed almost instantly. No further testing was required, let alone a pilot. The co-op digital team were able to iterate their thinking and eventually came up with an entirely different service offer.

Test (Proof of Concept)

Testing takes things a little further. Once prototyping has helped shape an idea, testing can be used to gain a proof of concept (POC); testing assumptions about how something might work or be used with the primary objective being to prove (or disprove) that a solution is viable.

Testing is a way of making a more detailed study of a problem with an open brief and the flexibility to rapidly react to changes. Starting with an initial hypothesis, the test team pulls a test plan together and is free to change and iterate the concept until enough evidence is gathered to satisfy the required objectives. The form that evidence can take however varies from softer qualitative insight through to harder facts and figures.

The key to a test is starting with ‘just enough’ insight and gathering the rest as you go. Often, trying to get all of the detail in place at the start of a test means that it never gets off the ground or worse turns into a pilot — and an ill-informed pilot at that.

Testing helps transport you from a position of thinking to a position of knowing.

Testing allows you to take a loose idea based on minimal insight and transform it into a robust proposal through iteration and learning; testing a concept to breaking point in a safe and controlled environment. The result of testing is a new perspective into:

• Systems — What organisational structures and processes will support the service?
• People — What roles, skills and behaviours are needed to deliver the service?
• Information — What information is communicated and in what format?
• Physical — What products or environments are needed to deliver the service?

Pilot (Minimum Viable Product)

Not every prototype progresses to test and not every test progresses to pilot. Prototyping and testing act as gatekeepers, ensuring that only well defined concepts make it to pilot. Pilots are the best way of testing an idea out in real life situations. The primary purpose of a pilot is therefore to gain a better understanding of how a service or product will be used in the field. As Tom explained in his original post, pilots evaluate the whole assembled service and usually take place over a protracted time-frame so you can spot the interactions you might have missed in testing stages; adding ‘the noise’ back in to see if your idea holds up whilst also seeking to uncover strong evidence of outcomes.

The main difference between a test and a pilot is that a pilot is static. The way a service or product works can’t be changed once the pilot starts. Unlike a prototype or test, pilots are less about learning and more about robust evaluation. Changing the way in which a pilot operates half way through risks the integrity of the evaluation.

Some of the biggest takeaways from Tom’s original post remain as relevant today as they did when first written:

• Don’t get caught in the trap of fetishizing about how many pilots you have on the go at any one time.
• Calving more unwieldy pilots into existence is not a badge of honour, it’s a badge of not valuing your own time.
• Pilots should never be implemented or scaled into the business without being evaluated.
• If you’re not going to let the idea fail, there’s no point piloting it in the first place.

Prototype More, Test Often, Pilot Less

Prototyping will help you design better tests and better tests lead to more ‘good failure’, less wasted resources, better outcomes and more informed pilots.

With a new year often comes a new set of resolutions, so this year why not forget wasting money on that gym membership you’ll never use and resolve yourself to doing just three simple things in 2019:

Originally published at www.bromfordlab.com.

Prototype More, Test Often, Pilot Less. was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.