Assumption Backlog & Experimentation Board PDF Template
Assumption Backlog & Experimentation Board Miro TemplatePrevious
¶ Interpretation Criteria
¶ What it is
Before going into the experimentation phase itself, it is critical to set out the criteria by which we will evaluate the success or failure of these experiments beforehand. Therefore the Interpretation Criteria help us to define how we agree to assess outcomes before we know them to keep us objective.
¶ Why it is useful
We want to agree on how we will interpret the results with a sober and reflective mind before we have any data to ensure that we don’t start bending interpretation to what will suit us down the line. If an experiment does not go as well as expected, people tend to rationalise post-factum and declare it a success. We must find ways to manage that tendency, and setting the interpretation beforehand is a good way of doing that.
¶ When to use it
It should be governed by clearly set interpretations when you are looking to run an experimentation process.
¶
¶ How is it done
- Prepare a workshop board with the Assumption Backlog boards from the Experiment Design & Selection exercise.
- As a team, go through each experiment - reflect on how you will run it and what kind of data you will collect (e.g., quant vs qual). Also, reflect on the assumption you are addressing with that data.
- Holding those pieces of information in mind, discuss a reasonable threshold for verifying the assumption based on the types of data you will have. For example, what data will confirm your hypothesis? What results will disprove your hypothesis?
- Record the agreed interpretation criteria for each assumption on the board as a final and measurable statement.
¶ Do's & Don't
Do's
- Try to get references for experiments where you do not know what “good” thresholds are - like bounce rates or click-through rates on landing pages. Again, reference data can help set the context right.
- Be critical in setting those thresholds to eliminate the risk associated with the assumption. If you are too lenient, your results might be overly optimistic.
Don't
- Don’t shy away from iterating experiments if the discussion of the interpretation criteria reveals a weakness in experiment design.
- Don’t make interpretation statements too vague, they must be measurable, and there has to be a straightforward way of assigning a true/false value to them after the data is in.
¶ Tools needed
- Whiteboard (physical or virtual)
- Post-it notes (physical or virtual)
- Assumption Backlog Template
¶ Example
