Hypothesis Test Process
1. Set a Null Hypothesis
The null hypothesis can be thought of as the opposite of the "guess" the research made (in this example the biologist thinks the plant height will be different for the fertilizers). So the null would be that there will be no difference among the groups of plants.
2. Set the significance level (alpha)
H0 is true | H0 is false | |
Accept | true positive | false positive |
Reject | true negative (alpha) | false negative |
3. Collect Data
Usually we use Permutation sampling to collect the new random data for the following hypothesis test, called hypothesis data. The original data is called observed data.
4. Calculate the test statistic
Could be the mean of differences, pearson correlation r value and so on. Depend on real situation.
5. Calculate the p-value
the term "p-value" is used to indicate a probability that you calculate after a given study. p = "sum(the situations that can prove the null hypothesis is true)/#ofAllSituations"
If your p-value is less than the chosen significance level then you reject the null hypothesis.
A full example:
Null hypothesis: performance in the finals and semifinals are identical
Test statistic: the mean of the fractional improvement. mean(f) where f = (semi_perm - final_perm) / semi_perm
f_mean is calculated from observed data, perm_reps[i] is calculated from hypothesis data.
p-value: sum(perm_reps >= f_mean) / len(perm_reps)
so, if hypothesis is true, the ideal situation is that f_mean is some value close to the average value of perm_reps so that the p-value should be 0.5;
however, the result shows that p-value equals to 0.001, which means that "f_mean" is not gonna happen in this hypothesis. Hence we reject the hypothesis.