Hypothesis
Testing Errors

Review 
The central reason we do hypothesis
testing is to decide whether or not the sample data
are consistent with the null hypothesis.
In the second step of the procedure we identify
the kind of data that is expected if the null hypothesis
is true.
Specifically, we identify the mean we expect if
the null hypothesis is true.
Decisions:
 If the mean of the sample of data obtained
in the experiment is consistent with the mean
under the null hypothesis, we believe the null
hypothesis is true: We "do not reject (we retain)
the null hypothesis".
 If the mean of the sample of data obtained in
the experiment is inconsistent with the null hypothesis,
we decide it is not true: We "reject the null
hypothesis".
We can be wrong in either decision we reach. 
Errors in Hypothesis Testing
We can come to one of two decisions:
 We "do not reject (we retain) the null hypothesis".
 We "reject the null hypothesis".
Since there are two decisions, there are two ways to be wrong
(and two ways to be right).
Errors in Hypothesis Testing 

Actual Situation 

No Effect
Ho True 
Effect Exists
Ho False 
Decision:
Reject Ho 
Type I
Error

Decision
Correct

Decision:
Retain Ho 
Decision
Correct

Type II
Error


 Type I Error: A type I error consists of rejecting
the null hypothesis when it is actually true. This is
a very serious error that we want to seldomly make. We
don't want to be very likely to conclude the experiment
had an effect when it didn't.
The experimental results look really different than
we expect according to the null hypothesis. But it could
come out the way it did just because by chance we have
a wierd sample.
Example 
Control 
We have a control group
in which a random sample of rats receives daily
doses of water during pregnancy. At birth, we measure
the weight of the sample of newborn rats. The weights,
in grams, are shown in the table.The mean weight
is 23 grams.
We observe that the rat pups are really heavy
and conclude that prenatal exposure to water increases
birthweight.
In reality (let's assume we know this) water
has no effect on weight, but what we see
in our sample is that it increases weight.
We conclude, erroneously, that the mothers drinking
increased water causes them to have heavier pups!
There could be another reason. Perhaps the mother
has unusual genes.
We have made a Type I error, but we don't know
it!

Sample Mean: 23

 Type II Error: A type II error consists of failing
to reject the null hypothesis when it is actually false.
This error has less grevious implications, so we are more
willing to err in this direction (of not concluding the
experiment had an effect when it, in fact, did). Of course,
we may be missing an important effect.
The experimental results don't look different than
we expect according to the null hypothesis, but they
are, perhaps because the effect isn't very big, or perhaps
because our sample is too small.
Example 
Experiment 2 
A second experimenter
repeats the experiment, using alcohol, and gets
the data shown at the right.
The rat pups weigh 16.5 grams and we conclude
there is no effect.
We do not reject the null hypothesis.
But "really" (if we only knew!) alcohol does
reduce weight, we just don't have a big enough
effect to see it.
We have not rejected the null hypothesis, but
it was really false and should have been rejected.
We have made a Type II error, but we don't know
it!

Sample Mean = 16.5

