Statistical significance

A test result is statistically significant when a difference that large would rarely appear by chance alone.

The common bar is 95%: a gap that would turn up less than one time in twenty between two identical versions. It says the difference is probably real, not that it is large.

Peeking at a test and stopping when it looks good inflates false winners. Decide the sample size before starting.