An Overview of Statistical Tests: Chi-Square, T-Test, Z-Test, and F-Test

Every monitoring and evaluation report eventually reaches the same question: are the numbers telling us something real, or are we just looking at random noise? When a development programme reports that literacy scores improved or that more women joined a community scheme, evaluators need a way to separate genuine effects from chance fluctuations. This is exactly where statistical tests come in. They convert raw data into evidence you can defend. Among the most widely used are the chi-square test, the t-test, the z-test, and the F-test, and each answers a slightly different question about your data.

Why statistical tests matter in data analysis
The chi-square test
Two main types of chi-square test
The t-test and the z-test
When to reach for a t-test
When the z-test fits better
The F-test
Comparing variances and multiple means
Choosing the right test

Why statistical tests matter in data analysis

A statistical test is a formal procedure for deciding whether the patterns you see in a sample are strong enough to say something about the wider population. The whole exercise rests on hypothesis testing. You begin with two competing statements. The null hypothesis (H₀) usually claims there is no difference or no relationship in your data. The alternative hypothesis (H₁) claims that a meaningful difference or relationship does exist.

Suppose a state government runs a new skill-training scheme and wants to know whether it raised employment rates. The null hypothesis would say the scheme had no effect, while the alternative would say it did. The test then weighs the evidence and tells you how likely your observed result would be if the null hypothesis were actually true. That probability is the p-value. By convention, when the p-value falls below 0.05, the result is treated as statistically significant, meaning the pattern is unlikely to be a fluke.

The choice of test depends on three things: the type of data you have, whether you are comparing means or variances, and your sample size. As one comparative study notes, the t-test and z-test are used for continuous data, the chi-square test for categorical data, and the F-test for comparing more than two means. Getting this match right is what keeps an evaluation credible.

The chi-square test

The chi-square test is the go-to tool when you are working with categorical variables, the kind of data that sorts people or things into groups rather than measuring them on a scale. Think of categories like male or female, urban or rural, enrolled or not enrolled. Because you cannot calculate a meaningful average of such labels, the chi-square test works with counts and frequencies instead.

The core idea is simple. The test compares the observed frequencies in your data with the expected frequencies you would see if there were no relationship between the variables. The bigger the gap between what you observed and what you expected, the stronger the signal that the variables are connected. The statistic is calculated as χ² = Σ [(O − E)² / E], summed across every cell of the table.

Two main types of chi-square test

There are two common versions, and they answer different questions. The chi-square test of independence checks whether two categorical variables are related, while the goodness-of-fit test uses a single variable to see whether its distribution matches an expected pattern. A goodness-of-fit test could check whether the age groups visiting a public health camp match the age distribution of the surrounding district.

For a programme evaluator, the test of independence is often the more useful one. Imagine you want to know whether gender influences participation in different community programmes. You collect data on 200 participants and find that men and women join sports and educational programmes in noticeably different proportions. The chi-square test tells you whether that difference is a real association or just sampling chance. One practical safeguard to remember is that the expected frequency in each cell should generally be at least 5 for the result to be reliable. When cells fall below that, evaluators often combine categories or switch to an alternative like Fisher’s exact test.

The t-test and the z-test

When your data is continuous, such as test scores, household income, or crop yield, and you want to compare averages, you move to the t-test and the z-test. Both are built to answer a question about means, but they make different assumptions about how much you already know.

When to reach for a t-test

The t-test compares means and is designed for situations where the sample is small and the population standard deviation is unknown, which is the normal state of affairs in most field studies. It comes in a few flavours. A one-sample t-test checks a single sample mean against a known value. An independent samples t-test compares the means of two separate groups, such as students in two different schools. A paired t-test compares two measurements taken from the same subjects, for example test scores before and after a study programme, where each student’s first score is matched to their second.

Because it relies on the t-distribution, which has fatter tails than the normal curve when the sample is small, the t-test is the more cautious choice. As sample sizes grow, the t-distribution gradually approaches the normal distribution, and the two tests converge. For valid results, the t-test assumes the data is drawn from a roughly normal distribution and collected through random sampling.

When the z-test fits better

The z-test is the close cousin used when conditions are more favourable. You apply it when the sample size is large, usually above 30, or when the population standard deviation is already known from previous studies or census data. Large samples let you invoke the Central Limit Theorem, which says the distribution of sample means becomes approximately normal regardless of the underlying shape of the data.

The practical difference is worth understanding. Because the t-distribution has thicker tails for small samples, the t-test always produces a slightly larger p-value than the z-test, making it the more conservative option. If you wrongly use a z-test on a small sample, you risk concluding there is a significant effect when there really is not. For large samples, the gap between the two becomes negligible, which is why the z-test is reserved for big datasets where its assumptions hold comfortably.

The F-test

The t-test and z-test work beautifully for one or two groups, but real evaluations often involve several groups at once. You might be comparing the impact of three different teaching methods, four irrigation schemes, or multiple districts running the same programme. Running many t-tests in this situation inflates the chance of a false positive. This is where the F-test earns its place.

Comparing variances and multiple means

At its heart, the F-test analyses variances. In its simplest form it compares the variances of two populations to decide whether they are equal, which is a useful check before running other tests that assume equal spread. The statistic is a ratio, the variance of one group divided by the variance of another, and it follows the Snedecor F-distribution under the null hypothesis.

Its most powerful application, however, is inside ANOVA (analysis of variance). ANOVA uses the F-test to compare the means of three or more groups simultaneously by examining how much variation exists between the groups relative to the variation within them. If the between-group variation is large compared to the within-group variation, the F-statistic grows, and a low p-value signals that at least one group mean genuinely differs from the rest.

A clear contrast helps fix the distinction. The t-test compares means between two groups, while the F-test compares variances or tests differences across two or more groups. Picture an agriculture department testing whether three fertilisers produce different average crop yields. A single ANOVA with an F-test handles all three at once. If the result is significant, evaluators then run follow-up, or post-hoc, comparisons to find exactly which fertilisers differ.

Choosing the right test

The four tests are not competitors so much as a toolkit, and the trick is matching the tool to the question. Use the chi-square test for relationships between categorical variables. Use the t-test for comparing one or two means with small samples or unknown population variance. Use the z-test when samples are large or the population variance is known. Use the F-test when you need to compare variances or the means of three or more groups.

One assumption ties most of them together. The t, chi-square, and F distributions all trace back to a normal parent population, which is why checking your data for normality and random sampling is never just a formality. For anyone running a monitoring and evaluation study, these tests turn a stack of survey responses into defensible conclusions, allowing you to say with confidence whether a programme worked, whom it reached, and whether the differences you see are real.

What do you think? If you were evaluating a state-wide scholarship scheme with both continuous data (exam scores) and categorical data (district, gender, scheme enrolment), which combination of these tests would give you the most complete picture? And how might a small sample size change the conclusions you are willing to draw?

References

PDF 📄

An Overview of Statistical Tests: Chi-Square, T-Test, Z-Test, and F-Test

Table of Contents

Why statistical tests matter in data analysis 🔗

The chi-square test 🔗

Two main types of chi-square test 🔗

The t-test and the z-test 🔗

When to reach for a t-test 🔗

When the z-test fits better 🔗

The F-test 🔗

Comparing variances and multiple means 🔗

Choosing the right test 🔗