Previous
¶ Experiment Refinement
¶ What it is
As you conduct and run your various market experiments, you will notice that an investigation might not get to the types of data you want to generate or that it only receives negative feedback or no reactions in the market. The team needs to monitor experiments on an ongoing basis. Suppose they identify a problem with the experiment design. In that case, they need to be willing to run an Experiment Refinement, in which a non-performing experiment is changed to improve its design toward generating the correct data.
¶ Why it is useful
We can only sometimes, at the outset, know whether a given experiment is well-designed. We can design it with the best intentions, yet mistakes will be inevitable. Refining them along the way when we see it does not generate data puts us in the position to salvage that work and get back on track.
¶ When to use it
Whenever a market experiment is not delivering the kinds of data we need (NOTE: this does not mean it provides data we do not like, e.g., disagree with, but data that is not what we can use for evaluation or no data at all).
¶ How is it done
- For all experiments, you create and run in the market, set review points where you look at the data they have designed and control whether this data helps evaluate the relevant hypothesis against the interpretation criteria.
- If the data is not helpful or does not generate data, review with the team what is happening and why that is happening. Discuss in what ways the experiments might be refined to serve the purpose.
- After review and agreement, conduct the necessary refinements and restart the experiment. Discard data generated before this point.
¶ Do's & Don't
Do's
- Do stop and iterate experiments that do not deliver data or deliver non-usable data.
- Understand the flaw in the experiment design before iterating.
- Consider changing the type of experiment altogether if necessary or appropriate.
Don't
- Don’t change experiments just because you disagree with the data or don’t like the data - these are results, not mistakes.
¶ Tools needed
- Whiteboard (physical or virtual)
- Post-it notes (physical or virtual)