Build-Measure-Learn

INTRO

Recently, I and my colleague Frank Heljebrant (@fhelje on Twitter) held a presentation on parts of the Tradera innovation process for the development teams at Dagens Nyheter and Dagens Industri. This post is a shortened version of the content in the presentation but the key concepts of each of the slides are available. You can also view the presentation  (The presentation was in swedish which is why the slides are in swedish).

PRESENTATION – WHAT AND WHY?

(slide #1) In short, Tradera has been online since 1999 and is now one of Sweden´s largest an online market places with roughly 1,5 million registered users. Up until the summer of 2015 Tradera was part of eBay and was then acquired by PayPal. The topic of the presentation is Build – Measure – Learn as part of the Tradera innovation process.

(Slide #2.1) So, why is the methodology of build-measure-learn [1] used and how did we arrive at where we are now? Roughly 3 years ago it was decided to transform the Tradera experience into a responsive web experience but having a shortage of developers in combination with a highly diverse customer base forced Tradera to think about how to do the transformation in a cost efficient way without upsetting or losing customers. (Slide #2.2) Consequently, in order to reduce the amount of work and simultaneously delivering on customer needs there was a need to put customers in the core of the new development process. By doing so the development could be focused on real needs and thus minimized waste in terms of developing functions that the customers did not need. (Slide #2.3) However, understanding the customer needs is not the same as asking what the customer wants. The focus has been on observing and understanding the “why?” in customer behavior and help the customer to verbalize their real pains and needs [2]. (Slide #2.4)  As mentioned above, the other focal point was to reduce the amount of work and waste in terms of developing features that customers did not need. In order to minimize waste, breaking the development down into smaller pieces which could be released into production with a high frequency allowed Tradera to verify the need for specific features by looking at real customer behavior. Consequently, by realizing many small improvements the improvements can be aggregated into an amazing improvement. (Slide #2.5) Additionally, the build-measure-learn approach creates engagement around- and a feeling of ownership of the product among employees which generates a lot of happiness and a sense of making a difference for the customers using Tradera. (Slide #3) The process has developed over time, in the beginning there was no formalized way of working around the measuring and learnings. Consequently, the teams were not aligned, nor were individuals in the teams always aligned in terms of what the customer need was or how to measure the outcome of whatever feature was in development. In order to mitigate such issues a deductive hypothesis driven methodology [3] was developed where all team members together defined the problem/customer need, the business case, a hypothesis for the solution and metrics to validate the hypothesis.

PRESENTATION – HOW?

(Slide #4.1) If one would like to implement this way of working, what are the fundamental principles and how does one generate suitable hypothesis? (Slide #4.2) All companies have, or should have, a vision. Generally a vision statement can be seen as a direction in which the company is moving [4]. At Tradera the vision is broken down into different strategic initiatives which form the framework for which teams can direct their efforts. (Slide #4.3) Given the framework, the teams gather and discuss what customer needs, business opportunities and potential hypothesis are relevant to focus on in the coming time period. Engaging the whole team facilitates alignment across team members and creates engagement as well as a sense of ownership. Such factors in turn facilitate cooperation and increase team focus, assuring that the team members are going in the same direction. (Slide #4.4 – 4.5) The business opportunities and the potential hypothesizes are grounded in customer behavior analysis, user tests, marketing channels such as social media, competitor analysis, in–house creativity workshops, insights from the customer support department etc.  (Slide #4.6) In order to verify or discard a hypothesis measurements are needed. The issues of verifying a hypothesis is rather difficult so it is easier to focus on measurements which help discard a hypothesis if it would be wrong. Consequently, each hypothesis has custom metrics which have been agreed on within the team based on the assumed outcome of a hypothesis as well as what is possible. It is important that the measurements are put in place before development begins as they are needed to create a baseline before any experiment as well as help developers put metrics in place during development if the tools are unavailable.

PRESENTATION – EXAMPLE

(Slide #5.1 – 5.2) As thousands of items are uploaded for sale on Tradera each day it is clear that customers needed a simple way of filtering the information on the site. At the time, part of the vision was to create a product experience which helped users find items relevant for them. However, Tradera does not have control over the data input of specific items since most sold items are C2C which means that the customers define the data input of each item they want to sell. Never the less, Tradera has a category tree and in the flow of posting an item a seller is required to choose what category the item belongs into. Additionally, shoe or other clothing sellers usually specify the brand, size and other characteristics of the item in a header or in the description of the item. What if the data already provided by customers could be utilized in order to facilitate a simpler filtering than plain search or browsing the category tree? (Slide #5.3 – 5.4) The first hypothesis was to state “We can technically do a simple filtering solution based on text analytics on texts provided by users?” The hypothesis was then tested during a so called “Innovation week”, which can be compared to Google´s 20% of “do whatever you want”-time [5]. (Slide #5.5 – 5.6) The hypothesis was verified and thus a new hypothesis was created which was “Will it work on the Tradera.com?” The next Innovation week was used to build a simple prototype and test it with the employees at the office and the response was unanimously positive. Testing it at the office also created an understanding of the potential of the solution across the organization which allowed “tag navigation” (as the experiment were to be called) to be pushed into the backlog of an entire team and thus also utilizing all resources in that team. (Slide #5.7) Fast forward a couple of months and one of the teams at Tradera are in a meeting formulating a hypothesis and discussing metrcis for the Tag navigation experiment, the hypothesis became “If our users can choose size etc. more easily, they will find what they are looking for faster and that should increase conversion”. In order to test it on customers and simultaneously managing risk, it was decided that the experiment was to be conducted only in a specific category and would be visible for desktop users only. (Slide #5.8) Some weeks later the team saw a 17% increase in the number of bids which then was seen as a validation of the hypothesis. (Slide #5.9) Due to the success of the experiment on desktop the team formulated a new hypothesis “We ought to see the same type of increase on mobile screens”. (Slide #5.10) However, while using the same metrics as the desktop experiment the Tag navigation on mobile was a failure. There was hardly any increase in bids and worse, the filters were barely used. (Slide #5.11 – 5.12) In order to understand “Why?” several mobile users with specific attributes were randomly selected and then invited to conduct an in-house user test. One remarkable finding from those tests was that the users did not use the new filters since they could not find them. Consequently, a new hypothesis based on the new findings was created where the main idea was that a better UI could solve the problem. However, a new UI did not solve the problem which forced the team to radically alter their approach and creating a new hypothesis (still using the same metrics). (Slide #5.13) Based on the core mechanics of the Tag navigation technology and the insights from user tests a new hypothesis was formed which emphasized the importance of making the category selection more prominent and thus guiding the user into the world of Tag navigation. (Slide #5.14) Several UX alternatives were created through an including in-house build-measure-learn process, again including the whole team and testing on other employees at Tradera. Finally two alternatives emerged which were then A/B-tested on real Tradera users. The winning alternative increased the number of bids with 3,3%  in the experiment category.

GENERAL LEARNINGS

(Slides #6.1 – 6.10)

  1. Measurements are difficult and need continuous development
  2. Measurement tools provided by third parties are not fully suitable for a complex product as Tradera, thus internally developed tools were needed.
  3. Putting the focus on customer needs creates an immense pool of insights which can be used to shorten development time of features, minimizing risk or providing insights for breakthrough ideas.
  4. Testing takes time but it also saves costs and minimizes the risk of realizing something that doesn’t “fly” among customers.
  5. Teamwork rules! Teamwork facilitates learning, alignment, increases efficiency and minimizes mistakes.
  6. Since Tradera is a whole system one needs to be careful of optimizing one part without taking other parts into consideration. Be sure to broaden the scope of measurements and include other team members in the development of measurements.
  7. Create a baseline to compare against before conducting the experiment.
  8. Don´t give up!
  9. Comparing qualitative results with quantitative results is difficult, especially when they point in different directions. You need to find a way that works for you.

The theoretical foundations for the concepts explained above can be found in :

[1] Ries, Eric, The Lean Startup: how constant innovation creates radically successful businesses, Portfolio Penguin, London, 2011

[2] Osterwalder, Alexander, Pigneur, Yves, Bernarda, Greg & Smith, Alan (red.),Value proposition design: how to create products and services customers want : get started with…, John Wiley & Sons, Hoboken, 2014

[3] Bryman, A. & Bell, E. (2011). Business research methods. (3. ed.) Oxford: Oxford University Press.

[4] http://www.businessnewsdaily.com/3882-vision-statement.html 2016-05-05 11:18

[5] Steiber, Annika., The Google Model [Elektronisk resurs] : Managing Continuous Innovation in a Rapidly Changing World, 2014

Double looped learning http://www.afs.org/blog/icl/?p=2653 2016-05-05 12:04

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s