SAT Evaluating statistical claims
Judge what a study design does and doesn't let you conclude.
How to score it
- You can only generalize to the population that was randomly sampled — pick the most restricted matching group.
- Random assignment → cause is allowed; mere observation → only correlation.
- The SAT never lets an observational study conclude causation.
Common traps
- Generalizes to a broader group than the sample frame.
- Allows “causes” from an observational study.
- Treats a convenience sample as if it were random.
The 3 question types, with real examples
Largest population you can generalize to
“What is the largest population to which the results can be generalized?”
A fitness app company sent a survey to 500 users who had logged workouts for at least 20 days in the previous month. Of those surveyed, 92% reported being satisfied with the app. Based on the results, which of the following is the largest population to which the results can be generalized?
- AAll users of the fitness app
- BAll people who use fitness apps
- CUsers of the fitness app who logged workouts for at least 20 days in the previous month✓
- DAll users who are satisfied with fitness apps
The sample consists only of frequent users who logged at least 20 workouts in the previous month. These active users are likely to have different satisfaction levels than occasional or inactive users. The results can only be generalized to users who logged workouts for at least 20 days in the previous month.
Cause vs. correlation
“Which conclusion is most appropriate?”
A researcher tracked 400 adults over 5 years and found that those who meditated regularly experienced fewer sick days than those who did not meditate regularly. Which conclusion is best supported by the study design?
- ARegular meditation will prevent illness in all adults.
- BAmong the adults tracked, regular meditation was associated with fewer sick days.✓
- CRegular meditation causes a reduction in sick days.
- DMeditation is the only health behavior that affects sick days.
This was an observational study tracking existing behaviors without random assignment. Observational studies can demonstrate associations but cannot establish causation due to potential confounding variables.
Effect of changing sample size / method
“If the same survey were conducted with [larger sample / more random method], what would happen to the margin of error?”
A market research firm surveyed 600 shoppers by interviewing people leaving a single grocery store location. The survey estimated that 55% of shoppers prefer organic produce, with a margin of error %.
If the same survey were conducted by randomly selecting 600 shoppers from customer records across all store locations in the chain instead of interviewing people at one location, what would happen to the margin of error?
- AThe margin of error would remain % because the sample size is still 600 shoppers.
- BThe margin of error would increase because sampling from multiple locations adds geographic variability.
- CThe margin of error would be cut in half because the improved method doubles the effective sample coverage.
- DThe margin of error would decrease because random selection across all locations reduces location-specific bias.✓
Surveying shoppers at only one location creates a biased sample because that location may attract customers with different preferences than the overall customer base. Random selection across all locations provides a more representative sample and decreases the margin of error.
Want to try the hard ones? 25 hard Evaluating statistical claims questions with full explanations — free, no account needed.
Drill Evaluating statistical claims — free trial→