Testing your way to answers you can defend

I've always learned better by doing something, getting it wrong and figuring out the reason than by studying the theory on its own. That was true when I was breaking layouts in 2007 and it's still true now.

Experimentation gives you a structured way to be wrong cheaply. You put a version in front of real people, you find out, and the argument that would otherwise have run for three weeks in a meeting room ends in a fortnight with a number attached.

It also does something quieter that I value more. It slowly removes seniority from the decision. When there's a test result, the loudest person in the room stops being the deciding factor, which is good for the product and quite good for the team.

The clearest recent example is the guest checkout work at JoyBuy. New customer conversion went from 61.94% to 68.58%, a rise of 6.64 percentage points.

In practical terms that was roughly 383 extra orders a day and an estimated 2.24m euros of incremental value over 90 days. I include the euro figure because it's the number that makes sense to people outside the design team, and translating between those two languages is a large part of what I do now.

None of that came from one clever idea. It came from taking a journey that was asking new customers to commit before they had any reason to, and removing the reasons they were dropping out one at a time.

The most common blocker isn't tooling. It's that nobody has agreed what would count as a good result before the test runs, so afterwards everyone reads the data in a way that supports what they already wanted.

The second is traffic. Plenty of pages simply don't get enough of it to test meaningfully, and pretending otherwise produces confident nonsense. In those cases I'd rather run qualitative work and make a judgement than dress a guess up as significance.

The third is that a lot of tests are boring. Small wins on a checkout field don't make anyone feel creative. But those are usually where the money is, and part of leading this work is being honest that the interesting ideas and the valuable ideas aren't always the same ideas.

If you only ever optimise, you get very good at improving something that may not deserve to exist. I've seen teams spend a year making a flow measurably better while the underlying proposition quietly stopped making sense.

So I try to keep two questions running in parallel. Is this working, and is this the right thing to be doing at all. The second one rarely gets answered by a test.

The Shop in Shop work at JoyBuy is a case where the bigger structural change mattered more than the incremental improvements. That contributed to 14x GMV, a 35% rise in ARPU and a 5.5 percentage point rise in repurchase rate, and no amount of button testing would have got there.