New Zealand's search volume is too small for the tests you are running
Half the performance experiments run in this market cannot reach significance before the conditions change. Most of them are reported as results anyway.
A test that would take a fortnight in Sydney takes a quarter in Auckland, and quarters contain seasonality, competitor activity and at least one platform change. By the time a New Zealand test could tell you something, the thing it was testing has moved.
The failure mode is confident nonsense
Nobody usually declares a winner at 60% confidence on purpose. It happens because the test was set up with a target that the market cannot supply, the deadline arrives, and the direction of the difference gets reported as if it were the finding.
The result is a strategy assembled from noise, and it is unusually hard to unwind because every step of it was measured.
How to test in a small market
- Calculate the required sample before the test, and if the market cannot supply it in the time available, do not run that test. That is a legitimate answer.
- Test bigger differences. A small market can detect a large effect; it cannot detect a 3% lift in button copy, ever.
- Use geo and time-based holdouts rather than user-level splits where volume is thin.
- Borrow from Australia for direction and validate locally for magnitude, while being explicit that the two markets behave differently on price and on brand trust.
The honest report on most New Zealand A/B tests is: we could not tell, and here is the number of weeks it would have taken.
