Your Guide to Hypothesis Testing and P-Values

Big chance you came across one of these buzz words and you have some questions. If this is you then you are at the right place. In this article we are going to explain Hypothesis Testing, Confidence Intervals, Statistical Significance and P-Values in a really simple way. So stick around because we are going to dig into each one.

Hypothesis Testing

Hypothesis testing is a scientific process of testing whether or not the hypothesis is plausible (in other words, it’s a process of testing the hypothesis for rejection or approval)

In each hypothesis test, there are null and alternative hypothesis which are represented respectively by H0 and H1 or Ho and Ha.

So … What are Null and Alternative Hypothesis anyway?

Null Hypothesis( H0 or Ho): it’s an assumption made on the population which states that nothing new is happening and the old/default theory is True until there’re evidence to prove otherwise.

Alternative Hypothesis (H1 or Ha): it’s complementing the null hypothesis and it is actually nothing but the opposite to the null hypothesis and it’s accepted when we reject the null.

In the example of courtroom we know that “every man is innocent until proved guilty”

Null Hypothesis( H0 or Ho): a man is innocent.

Alternative Hypothesis (H1 or Ha): a man is guilty.

When we talk about Regression problem, the null hypothesis simply means that there is no relationship between the dependent and independent variable, on the other hand, the opposite is the alternative hypothesis.

Significance level (α)

In hypothesis tests, two errors are possible, Type I and Type II errors:

Type I error (also called alpha α) : Supporting the alternative hypothesis when really the null hypothesis is true.

Type II error ( called beta β): Not supporting the alternative hypothesis when really the alternative hypothesis is true.

In our previous courtroom example, Type I error is judging that a man is guilty and should be in jail while in reality he is nothing but innocent. While Type II error is judging that a man is innocent and should be sat free while in reality he is guilty.

Great thanks, but … What does this have to do with Significance level?

Significance level is the probability of a type I error, usually represented as alpha(α). The most common value for alpha is 5%.

So if a result is statistically significant, that means we can feel confident that it is real, not that we just got lucky (or unlucky) in choosing the sample. It’s unlikely to be explained solely by chance or random factors.

If alpha for example is 0.05. That mean 5% of the time we’d wrongly conclude that we should reject the null hypothesis while we shouldn’t and thus there’s a 5% probability we will suffer a type I error by rejecting a true null hypothesis.

Confidence Interval and Confidence Level

For a given statistic (e.g mean) calculated for a sample, the confidence interval is basically the range of values around that statistic that we believe it contains the true value of that statistic (e.g the population value) with a certain probability (e.g 95%).

Tl;dr: it’s the range of values that are likely to encompass the true value.

Ook … What about Confidence Level? Confidence Level is nothing but the probability or certainty that the confidence interval contains the true population value when we draw a random sample many times.

Its calculated from the standard error of mean which is standard deviation divided by the square root of the sample size. A 95% confidence interval is the most commonly used in hypothesis testing, that is 2 standard deviation either side of the observed value, it is the range of values between which we can be 95% confident the true value lies.

P-Value

p-value is used in hypothesis testing to help support or reject the null hypothesis. It is an evidence against the null hypothesis. If the p-value is less than or equal to alpha then it’s statistically significant and we decide to reject the null hypothesis.

An important note: if we couldn’t reject the null hypothesis then we say “we failed to reject the null” and “Failing to reject” the null hypothesis does not mean “accepting” the null hypothesis. The alternative hypothesis might indeed be true, but we just don’t have enough data to prove that at the time.

So Where does p-value fit in the picture?

The smaller the p-value the stronger the evidence in favor of the alternative hypothesis and against the null hypothesis, e.g a p-value of less than 0.05 tells us that the result is statistically significant.

So really a p-value is nothing but the likelihood or the probability of the observed effect to have occurred by chance. This gives us a measure of statistical significance which helps us decide on our hypothesis.

Hope the article was useful and you liked it.. :)

Until next time✌🏻