YouTip

Significance testing is the process of first making a hypothesis about the parameters of a population (random variable) or the form of its distribution, and then using sample information to judge whether this hypothesis (the alternative hypothesis) is reasonable. In other words, it determines whether the difference between the population's true state and the null hypothesis is significant. Alternatively, significance testing judges whether the difference between the sample and our hypothesis about the population is due to random chance or arises from a mismatch between our hypothesis and the population's true state. Significance testing examines our hypothesis about the population. Its principle is to accept or reject the hypothesis based on the "impossibility of a low-probability event in practice."

Significance testing is used to determine whether there is a difference between the effects of an experimental treatment group and a control group, or between two different treatments, and whether that difference is significant.

SciPy provides the scipy.stats module to perform Scipy significance tests.

Statistical Hypothesis

A statistical hypothesis is a hypothesis about the unknown distribution of one or more random variables. A statistical hypothesis where the form of the distribution is known and only involves one or more unknown parameters within that distribution is called a parametric hypothesis. The process of testing a statistical hypothesis is called hypothesis testing, and testing a parametric hypothesis is called a parametric test.

Null Hypothesis

The null hypothesis (H₀), a statistical term, is the hypothesis established in advance when conducting a statistical test. When the null hypothesis holds, the relevant statistics should follow a known probability distribution.

When the calculated value of the statistic falls into the rejection region, it indicates that a low-probability event has occurred, and the null hypothesis should be rejected.

The hypothesis to be tested is often denoted as H₀, called the null hypothesis. The hypothesis opposing H₀ is denoted as H₁, called the alternative hypothesis.

• When the null hypothesis is true, deciding to reject it is called a Type I error, and its probability is usually denoted as α.
• When the null hypothesis is false, deciding not to reject it is called a Type II error, and its probability is usually denoted as β.
• α + β does not necessarily equal 1.

Typically, only the maximum probability of making a Type I error, α, is specified, without considering the probability of making a Type II error, β. This type of hypothesis test is also called a significance test, and the probability α is called the significance level.

The most commonly used α values are 0.01, 0.05, 0.10, etc. Generally, based on the research problem, if rejecting a true hypothesis incurs a large loss, α is set smaller to reduce this type of error; conversely, α is set larger.

Alternative Hypothesis

The alternative hypothesis (H₁) is one of the fundamental concepts in statistics. It contains all propositions about the population distribution that make the null hypothesis invalid. The alternative hypothesis is also called the opposing hypothesis or the alternative hypothesis.

The alternative hypothesis can replace the null hypothesis.

For example, in evaluating students, we might adopt:

"The student is below average" — as the null hypothesis.
"The student is above average" — as the alternative hypothesis.

One-Sided Test

A one-sided test (also called a one-tailed test or one-sided test) is a method in hypothesis testing that constructs a critical region using the area in one tail of the density curve of the test statistic and the x-axis to perform the test.

When our hypothesis tests only one side of the value, it is called a "one-tailed test."

Example:

For the null hypothesis:
"The mean is equal to k"

We can have the alternative hypothesis:
"The mean is less than k" or "The mean is greater than k"

Two-Sided Test

A two-sided test (also called a two-tailed test or two-sided test) is a method in hypothesis testing that constructs a critical region using the areas in both the left and right tails of the density curve of the test statistic and the x-axis.

When our hypothesis tests both sides of the value.

Example:

For the null hypothesis:
"The mean is equal to k"

We can have the alternative hypothesis:
"The mean is not equal to k"

In this case, both sides where the mean is less than or greater than k must be checked.

Alpha Value

The alpha value is the significance level.

The significance level is the probability of making an error when estimating that the population parameter falls within a certain interval, denoted by α.

It indicates how close the data must be to the extreme to reject the null hypothesis.

It is usually taken as 0.01, 0.05, or 0.1.

P-Value

The P-value indicates how close the data actually is to the extreme.

Compare the P-value and the alpha value to determine the level of statistical significance.

If the p-value <= alpha, we reject the null hypothesis and say the data has statistical significance; otherwise, we accept the null hypothesis.

T-Test

The T-test is used to determine whether there is a significant difference between the means of two variables and to judge if they belong to the same distribution.

This is a two-tailed test.

The function ttest_ind() takes two samples of the same size and returns a tuple containing the t-statistic and the p-value.

Find whether given values v1 and v2 come from the same distribution:

Example

import numpy as np
from scipy.stats import ttest_ind

v1 = np.random.normal(size=100)
v2 = np.random.normal(size=100)

res = ttest_ind(v1, v2)
print(res)

The output result is:

Ttest_indResult(statistic=0.40833510339674095, pvalue=0.68346891833752133)

If you only want to return the p-value, use the pvalue attribute:

Example

import numpy as np
from scipy.stats import ttest_ind

v1 = np.random.normal(size=100)
v2 = np.random.normal(size=100)

res = ttest_ind(v1, v2).pvalue
print(res)

The output result is:

0.68346891833752133

KS Test

The KS test is used to check whether given values conform to a distribution.

The function takes two parameters: the values to test and the CDF.

CDF stands for Cumulative Distribution Function.

The CDF can be a string or a callable function that returns probabilities.

It can be used as a one-tailed or two-tailed test.

By default, it is a two-tailed test. We can pass the parameter alternative as a string, which can be one of 'two-sided', 'less', or 'greater'.

Find whether given values conform to a normal distribution:

Example

import numpy as np
from scipy.stats import kstest

v = np.random.normal(size=100)
res = kstest(v, 'norm')
print(res)

The output result is:

KstestResult(statistic=0.047798701221956841, pvalue=0.97630967161777515)

Data Statistical Description

Using the describe() function, you can view information about an array, which includes the following values:

1. nobs -- Number of observations
2. minmax -- Minimum and maximum values
3. mean -- Arithmetic mean
4. variance -- Variance
5. skewness -- Skewness
6. kurtosis -- Kurtosis

Display the statistical description information of the array:

Example

import numpy as np
from scipy.stats import describe

v = np.random.normal(size=100)
res = describe(v)
print(res)

The output result is:

DescribeResult(nobs=100, minmax=(-2.0991855456740121, 2.1304142707414964), mean=0.11503747689121079, variance=0.99418092655064605, skewness=0.013953400984243667, kurtosis=-0.671060517912661)

Normality Test (Skewness and Kurtosis)

A test that uses observed data to determine whether the population follows a normal distribution is called a normality test. It is an important special type of goodness-of-fit hypothesis test in statistical decision-making.

The normality test is based on skewness and kurtosis.

The normaltest() function returns the p-value for the null hypothesis:
"x comes from a normal distribution"

Skewness

A measure of the symmetry of the data.

For a normal distribution, it is 0.

If negative, it indicates the data is skewed to the left.

If positive, it means the data is skewed to the right.

Kurtosis

A measure of whether the data is heavy-tailed or light-tailed compared to a normal distribution.

Positive kurtosis indicates heavy tails.

Negative kurtosis indicates light tails.

Find the skewness and kurtosis of the values in the array:

Example

import numpy as np
from scipy.stats import skew, kurtosis

v = np.random.normal(size=100)
print(skew(v))
print(kurtosis(v))

The output result is:

0.11168446328610283
-0.1879320563260931

Find whether the data comes from a normal distribution:

Example

import numpy as np
from scipy.stats import normaltest

v = np.random.normal(size=100)
print(normaltest(v))

The output result is:

NormaltestResult(statistic=4.4783745697002848, pvalue=0.10654505998635538)