Suppose we do an experiment with 4 subjects. Two subjects are randomly assigned
to treatment A. The other 2 receive treatment B. The response variable is
number right on a 10 point quiz. The subjects that received treatment A had
scores of 7 and 5 on the quiz. The subjects that received treatment B had
scores of 3 and 1 on the quiz. Compute a 2-sided p-value for testing for a
difference between treatment A and treatment B.
Original Data:
A B Difference in Means
7 5 3 1 4
All possible assignments of subjects to treatments:
A B Difference in Means
7 5 3 1 4
7 3 5 1 2
7 1 5 3 0
3 1 7 5 -4
5 1 7 3 -2
5 3 7 1 0
2-sided p-value is 2/6 or about 0.333 because 2 of the 6 possible assignments of
subjects to treatments have a difference as far from 0 as the difference
observed in the original data.
There is no convincing evidence that one treatment is better than the other
because it would not be too unusual (2 out of 6 chance) to see results as
extreme as those we obtained even if the treatment has no effect on the quiz
scores.