25 hard SAT Evaluating statistical claims questions
Real questions from the SAT Climb bank, all at the hard difficulty tier. Pick an answer before you open the explanation. Every question tells you why the answer is right and why each wrong choice is tempting.
Math · Problem-Solving and Data Analysis~0.4 — rare, but easy points per testHard tier
Generalizes to a broader group than the sample frame.
Allows “causes” from an observational study.
Treats a convenience sample as if it were random.
Question 1Hard
A technology company analyzed productivity patterns by examining data from 380 employees who had voluntarily installed a time-tracking application on their work computers and who worked in the company's headquarters building. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why C is right
The sample has two constraints: voluntary installation of the application (introducing self-selection bias) and working in headquarters. Results can only be generalized to employees meeting both criteria, as those who chose to install tracking software may differ systematically from those who did not.
Why the others are wrong
AThis ignores both the voluntary participation and the headquarters location restrictions that define the sample.
BThis omits the crucial voluntary installation requirement; only headquarters employees who chose to install the app are represented.
DThis extends beyond the company and ignores both the voluntary nature and headquarters restriction of the sample.
Question 2Hard
A researcher wanted to study the relationship between daily coffee consumption and self-reported energy levels among office workers. The researcher surveyed 120 office workers at random from a large company and found that workers who reported drinking more than 3 cups of coffee per day had higher average energy levels than workers who reported drinking 3 or fewer cups per day. Which conclusion is best supported by the study design?
Show the answer and explanation
Why B is right
The study is observational, not experimental, so causation cannot be established. The sample was randomly selected from office workers at one company, so conclusions can only be drawn about that population. The correct answer appropriately limits the conclusion to an association within the sampled population.
Why the others are wrong
AThis choice incorrectly asserts causation from an observational study. Without random assignment to coffee consumption levels, confounding variables could explain the observed relationship.
CThis choice overgeneralizes beyond the study population. The sample was drawn only from office workers at one company, not from all adults.
DThis choice ignores potential confounding variables. The observational design cannot isolate the effect of coffee consumption from other factors that might influence both coffee drinking and energy levels.
Question 3Hard
A market research firm surveyed 196 customers who had filed complaints to estimate overall customer satisfaction. The margin of error was calculated as 7 percentage points. The firm recognizes that surveying only complainants may bias the results.
If the firm were to survey a random sample of 784 customers from all customers instead, what would happen to the margin of error?
Show the answer and explanation
Why C is right
The sample size increases from 196 to 784 (a factor of 4), which reduces the margin of error by a factor of 4=2 (from 7 to 3.5 percentage points). Additionally, switching from a sample of only complainants to a random sample of all customers eliminates selection bias. Both factors contribute to an improved, more reliable estimate with a smaller margin of error.
Why the others are wrong
AThis incorrectly claims that sampling method does not affect margin of error validity, ignoring that the original margin of error from a biased sample is not meaningful.
BThis reverses the relationship, incorrectly suggesting that larger samples or more diversity increase margin of error.
DThis divides by 4 instead of by √4, incorrectly applying the sample size ratio directly rather than taking the square root.
Question 4Hard
A researcher wanted to study the relationship between daily coffee consumption and reported stress levels among office workers. The researcher selected 80 office workers at random from a large corporation and asked each worker to report their average daily coffee consumption and their stress level on a scale from 1 to 10. Workers who reported drinking 3 or more cups of coffee per day had an average stress level of 6.8, while workers who reported drinking fewer than 3 cups per day had an average stress level of 5.2. Which conclusion is best supported by the study design?
Show the answer and explanation
Why B is right
The study is observational—workers self-reported their coffee consumption and stress levels without any random assignment to coffee consumption groups. Therefore, the study can only establish an association, not causation. The sample was randomly selected from office workers at the corporation, so the results can be generalized to that population but only support a correlation claim.
Why the others are wrong
AThis choice incorrectly asserts causation from an observational study. Without random assignment of coffee consumption levels, confounding variables could explain the relationship.
CThis choice overgeneralizes beyond the study population. The sample was drawn only from one corporation, so results cannot be generalized to all office workers.
DThis choice ignores potential confounding variables. Many factors could influence stress levels, and the observational design cannot isolate coffee as the primary factor.
Question 5Hard
A researcher surveyed 200 members of a professional cycling club who regularly participate in competitive races to determine average weekly training hours. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why B is right
The sample consists only of members of one specific professional cycling club who regularly participate in competitive races. Results can only be generalized to the exact population from which the sample was drawn, which is members of this particular club who meet the competitive participation criterion.
Why the others are wrong
AThis population is far too broad; bicycle ownership does not imply competitive racing or membership in the surveyed club.
CThis extends beyond the sample frame, which was limited to members of one specific club, not all cyclists in the region.
DThis generalizes beyond the single club that was surveyed; the sample came from one club only.
Question 6Hard
An economist studied the relationship between college education and income levels. The economist analyzed data from 500 adults selected at random from a national database and found that adults with a college degree had higher average annual income than adults without a college degree. Which conclusion is best supported by the study design?
Show the answer and explanation
Why B is right
This is an observational study with a random sample from the national database. The design supports finding an association within that population but cannot establish causation. The random sampling allows generalization to the full database population.
Why the others are wrong
AThis choice incorrectly asserts causation from an observational study. Without random assignment to college education, confounding variables such as family background could explain the income difference.
CThis choice undergeneralizes. Because the sample was randomly selected from the national database, the results can be generalized to all adults in that database, not just the sample.
DThis choice ignores potential confounding. The observational design cannot isolate the effect of college education from factors like family socioeconomic background that might influence both college attendance and income.
Question 7Hard
A news organization conducted a telephone poll of 625 randomly selected adults to estimate public opinion on a policy issue. The margin of error was reported as 4 percentage points. Due to budget constraints, the organization could only afford a convenience sample of 625 adults for a follow-up poll.
If the follow-up poll were conducted with the convenience sample of 625 adults instead of a random sample, what would happen to the margin of error?
Show the answer and explanation
Why C is right
The margin of error formula applies only to random samples. Convenience sampling introduces systematic bias that cannot be measured by the standard margin of error calculation. While the computed value might remain numerically 4 percentage points, it would not represent true sampling error and would be misleading.
Why the others are wrong
AThis incorrectly assumes that margin of error depends only on sample size, ignoring the requirement that the sample must be random for the margin of error to be valid.
BThis applies an arbitrary doubling factor without basis in statistical theory for how convenience sampling affects margin of error.
DThis incorrectly suggests that convenience sampling reduces margin of error, reversing the correct understanding that non-random sampling undermines the validity of margin of error calculations.
Question 8Hard
A sociologist wanted to study the relationship between hours of weekly volunteer work and reported life satisfaction among retirees. The sociologist surveyed 200 retirees randomly selected from a statewide retirement community organization. Retirees who reported volunteering 5 or more hours per week had an average life satisfaction score of 8.2 on a 10-point scale, while retirees who reported volunteering fewer than 5 hours per week had an average score of 7.1. Which conclusion is best supported by the study design?
Show the answer and explanation
Why B is right
The study is observational—retirees self-reported their volunteer hours and life satisfaction without random assignment to volunteer levels. Therefore, only an association can be established. The sample was randomly selected from the statewide organization, so results can be generalized to that population but cannot support causation.
Why the others are wrong
AThis choice incorrectly asserts causation from an observational study. Volunteer hours were not randomly assigned, so confounding factors such as health status or social connectedness could explain the relationship.
CThis choice ignores potential confounding variables and overstates the finding. The observational design cannot identify volunteer work as the most important factor among many possible influences on life satisfaction.
DThis choice overgeneralizes beyond the study population and incorrectly asserts causation. The sample was from one statewide retirement organization, and the observational design cannot support causal claims.
Question 9Hard
A transportation agency examined commute times by randomly selecting 280 individuals from a list of people who had registered vehicles at downtown parking garages that charge daily fees. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why C is right
The sampling frame is limited to people who registered vehicles at specific downtown parking garages that charge daily fees. This excludes those who park elsewhere, use monthly passes, or take other transportation, so results apply only to this narrow group.
Why the others are wrong
AThis ignores the restriction to those who use paid daily parking garages, which excludes many commuters.
BThis extends beyond the actual sample frame by including car commuters who do not use the specific paid downtown garages.
DThis is broader than the sample, which came only from those using daily-fee parking garages, not all downtown drivers.
Question 10Hard
A public health researcher wanted to study the relationship between daily walking time and resting heart rate. The researcher collected data from 200 volunteers at a community health fair who self-reported their average daily walking time and had their resting heart rate measured. Volunteers who reported walking more than 30 minutes per day had lower average resting heart rates than volunteers who reported walking 30 minutes or less per day. Which conclusion is best supported by the study design?
Show the answer and explanation
Why C is right
The study collected observational data from volunteers at one health fair, which is not a random sample of the broader community. The design supports finding an association within the volunteer group but cannot establish causation or generalize to all community adults.
Why the others are wrong
AThis choice incorrectly asserts causation from observational data. Without random assignment to walking time, the observed relationship could be due to confounding factors.
BThis choice ignores potential confounding. The observational design cannot determine whether the relationship is independent of overall fitness level or other health factors.
DThis choice overgeneralizes beyond the study population. The sample consisted of volunteers at one health fair, not a random sample of all adults in the community.
Question 11Hard
A company surveyed a convenience sample of 225 employees who volunteered to participate, asking about job satisfaction. The reported margin of error was 4 percentage points.
If the company were to conduct the survey using a random sample of 900 employees instead, what would happen to the margin of error?
Show the answer and explanation
Why C is right
Two improvements occur: the sample size increases by a factor of 4, which reduces margin of error by 4=2 (to 2 percentage points), and switching from volunteer sampling to random sampling eliminates self-selection bias. Both changes contribute to a decrease in the margin of error and increase the reliability of the estimate.
Why the others are wrong
AThis reverses the correct relationship, incorrectly suggesting that larger samples increase margin of error.
BThis fails to recognize that both sample size and sampling method affect the validity and magnitude of the margin of error.
DThis divides by 4 instead of by √4, incorrectly applying the ratio of sample sizes directly rather than taking the square root.
Question 12Hard
An economist wanted to study the relationship between years of formal education and annual income among adults in a metropolitan area. The economist randomly selected 250 adults aged 30 to 40 from census records in the metropolitan area and collected data on their years of formal education and their annual income. Adults with 16 or more years of education had an average annual income of 68000 dollars, while adults with fewer than 16 years of education had an average annual income of 43000 dollars. Which conclusion is best supported by the study design?
Show the answer and explanation
Why C is right
The study is observational—the economist measured existing education levels and income without manipulating education. Therefore, only an association can be established. The sample was randomly selected from adults aged 30 to 40 in the metropolitan area, so results can be generalized to that population but cannot support causation.
Why the others are wrong
AThis choice incorrectly asserts causation from an observational study. Education level was not randomly assigned, so confounding factors such as family background or innate ability could explain the relationship.
BThis choice ignores potential confounding variables. Many factors such as work experience, field of study, and economic opportunities affect income beyond years of education.
DThis choice overgeneralizes beyond the study population and incorrectly asserts causation. The sample was limited to adults aged 30 to 40 in one metropolitan area, and the observational design cannot support causal claims.
Question 13Hard
A health organization studied exercise habits by selecting 320 individuals from a database of people who had purchased gym memberships in the past year and tracking their workout frequency. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why A is right
The sample was drawn exclusively from people who had purchased gym memberships in the past year. Results can only be generalized to this specific population, as gym membership purchasers may have different exercise habits than the general population.
Why the others are wrong
BThis is far too broad; the sample came only from gym membership purchasers, not a representative sample of all adults.
CPurchasing a gym membership does not guarantee regular exercise, and this category extends beyond the sampling frame.
DThis is vaguer and broader than the actual sample frame, which was specifically limited to gym membership purchasers.
Question 14Hard
A transportation researcher investigated the relationship between commute time and job satisfaction. The researcher surveyed 250 employees at random from workers at a large technology company and found that employees with commute times under 30 minutes reported higher average job satisfaction than employees with commute times of 30 minutes or more. Which conclusion is best supported by the study design?
Show the answer and explanation
Why B is right
This is an observational study with random selection from one company. The design supports finding an association within that population but cannot establish causation or generalize beyond workers at the technology company.
Why the others are wrong
AThis choice asserts causation and overgeneralizes. The observational study cannot establish causation, and the sample came from only one technology company, not all workers.
CThis choice ignores potential confounding. The observational design cannot determine whether the relationship is independent of factors like salary and work environment that might influence both commute choices and job satisfaction.
DThis choice incorrectly asserts causation from an observational study. Without random assignment to commute times, the observed association could be due to confounding variables.
Question 15Hard
A university conducted an online survey to estimate student support for a campus initiative. The survey link was posted on social media, and 1024 students responded. The computed margin of error was 3 percentage points. University officials realized this convenience sample might not represent all students.
If the university were to conduct the survey using a random sample of 256 students from the official enrollment list instead, what would happen to the margin of error?
Show the answer and explanation
Why B is right
While switching to random sampling improves validity and eliminates self-selection bias, the sample size decreases from 1024 to 256 (a factor of 4), which increases the margin of error by a factor of 4=2, from 3 to 6 percentage points. The random sample provides a more reliable estimate despite the larger margin of error.
Why the others are wrong
AThis fails to account for the fact that reducing sample size increases margin of error, even though the sampling method improves.
CThis incorrectly suggests that margin of error depends on population size rather than sample size and sampling method.
DThis incorrectly applies the inverse relationship and divides by 4 instead of multiplying by √4.
Question 16Hard
An education researcher wanted to study the relationship between class size and student test performance in elementary schools. The researcher randomly selected 120 fourth-grade classrooms from all fourth-grade classrooms in a school district. The researcher recorded the number of students in each classroom and the average test score of students in that classroom on a standardized mathematics assessment. Classrooms with 20 or fewer students had an average test score of 78, while classrooms with more than 20 students had an average test score of 72. Which conclusion is best supported by the study design?
Show the answer and explanation
Why D is right
The study is observational—the researcher measured existing class sizes and test scores without manipulating class size. Therefore, only an association can be established. The classrooms were randomly selected from the school district, so results can be generalized to that district but cannot support causation.
Why the others are wrong
AThis choice incorrectly asserts causation from an observational study. Class size was not randomly assigned, so confounding factors such as teacher experience or school resources could explain the relationship.
BThis choice ignores potential confounding variables. Many factors affect test performance, and the observational design cannot establish class size as the primary determinant.
CThis choice overgeneralizes beyond the study population and incorrectly asserts causation. The sample was from one school district, and the observational design cannot support causal claims.
Question 17Hard
An educational researcher studied reading comprehension by testing 175 students who were enrolled in summer remedial reading programs and whose parents had signed consent forms for the study. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why B is right
The sample has two restrictions: students must be in summer remedial reading programs AND have parents who signed consent forms. The consent requirement introduces potential bias, as only students whose parents engaged with the consent process are included.
Why the others are wrong
AThis omits both the remedial reading focus and the parental consent restriction that defines the sample.
CThis is much broader than the sample and ignores both the summer program enrollment and consent form requirements.
DThis vastly overgeneralizes by ignoring both the remedial program enrollment and parental consent restrictions.
Question 18Hard
A medical researcher wanted to test whether a new medication reduces blood pressure. The researcher recruited 180 adults with high blood pressure from three clinics and randomly assigned 90 participants to receive the new medication and 90 participants to receive a placebo for eight weeks. Participants receiving the new medication showed greater reduction in blood pressure than those receiving the placebo. Which conclusion is best supported by the study design?
Show the answer and explanation
Why D is right
This randomized experiment with random assignment to treatment and control groups supports a causal conclusion. The population is limited to adults with high blood pressure from the three clinics where recruitment occurred.
Why the others are wrong
AThis states association rather than causation. Random assignment in this experiment allows a causal conclusion, making this weaker than warranted by the study design.
BThis incorrectly treats the experiment as observational with self-selection. Random assignment eliminates preference-based selection, so the blood pressure reduction can be attributed to the medication itself.
CThis overgeneralizes to all adults with high blood pressure. Participants were recruited from only three clinics, so results cannot be generalized beyond that recruitment population.
Question 19Hard
A pollster surveyed a random sample of 400 registered voters in a city to estimate the proportion who support a ballot measure. The margin of error for the estimate was 5 percentage points.
If the same survey were conducted with a random sample of 1600 registered voters instead, what would be the margin of error?
Show the answer and explanation
Why B is right
The margin of error is inversely proportional to the square root of the sample size. When the sample size increases from 400 to 1600 (a factor of 4), the margin of error decreases by a factor of 4=2. Therefore, the new margin of error is 5/2 = 2.5 percentage points.
Why the others are wrong
AThis incorrectly divides by 4 instead of by √4, treating the margin of error as inversely proportional to sample size rather than to the square root of sample size.
CThis incorrectly multiplies the original margin of error by 2 instead of dividing, reversing the relationship between sample size and margin of error.
DThis multiplies by 4, the ratio of the sample sizes, without applying the square root relationship.
Question 20Hard
A sports physiologist wanted to determine whether a new training program improves endurance in collegiate athletes. The physiologist recruited 60 collegiate athletes who volunteered for the study. The physiologist randomly assigned 30 athletes to follow the new training program for 8 weeks and assigned the other 30 athletes to follow their usual training routine for 8 weeks. At the end of 8 weeks, the athletes who followed the new program had an average improvement in endurance of 12%, while those who followed their usual routine had an average improvement of 4%. Which conclusion is best supported by the study design?
Show the answer and explanation
Why C is right
This is a randomized experiment where athletes were randomly assigned to treatment groups. Random assignment allows for causal inference. However, since the athletes volunteered rather than being randomly selected from all collegiate athletes, generalization is limited to athletes similar to those in the study.
Why the others are wrong
AThis choice overgeneralizes to all collegiate athletes. The study used volunteers, not a random sample from all collegiate athletes, so broad generalization is not warranted.
BThis choice incorrectly uses 'associated with' when causation can be inferred. The random assignment to treatment groups in this experiment supports a causal conclusion.
DThis choice misrepresents what random assignment accomplishes. Random assignment balances confounding variables across groups but does not eliminate their existence.
Question 21Hard
A university conducted a study by emailing a voluntary survey about campus dining satisfaction to all students, receiving responses from 450 students who chose to participate. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why C is right
The sample is self-selected because participation was voluntary. Only students who chose to respond are represented, so results can only be generalized to those who actually responded to the voluntary survey, not to all students.
Why the others are wrong
AVoluntary response creates bias; students who felt strongly about dining were more likely to respond, so the sample does not represent all students.
BThis extends far beyond the sample frame and compounds the voluntary response bias across multiple institutions.
DThe survey was voluntary, so even among dining facility users, only those who chose to respond are represented.
Question 22Hard
A nutritionist studied the relationship between protein intake and muscle mass in adult athletes. The nutritionist randomly selected 80 athletes from a regional sports league and measured their daily protein intake and muscle mass. Athletes with protein intake above 1.5 grams per kilogram of body weight had greater average muscle mass than athletes with protein intake at or below 1.5 grams per kilogram. Which conclusion is best supported by the study design?
Show the answer and explanation
Why C is right
This is an observational study with random selection from a specific population. The design supports finding an association within that population but cannot establish causation or generalize beyond athletes in the regional sports league.
Why the others are wrong
AThis choice asserts causation and overgeneralizes to all adults, when the study only observed athletes in one regional league without experimental manipulation.
BThis choice incorrectly claims the effect is independent of confounders. The observational design cannot rule out that exercise regimen or other factors explain the relationship.
DThis choice incorrectly asserts causation from an observational study. Without random assignment to protein intake levels, the observed association could be due to confounding variables.
Question 23Hard
A health department estimated the vaccination rate in a county by surveying 144 randomly selected households. The margin of error for this estimate was 8 percentage points. Officials want to reduce the margin of error for their next survey.
If the next survey were conducted with a random sample of 576 households instead, what would be the margin of error?
Show the answer and explanation
Why B is right
The sample size increases from 144 to 576, which is a factor of 4. The margin of error is inversely proportional to the square root of the sample size, so it decreases by a factor of 4=2. The new margin of error is 8/2 = 4 percentage points.
Why the others are wrong
AThis divides by 4 instead of by √4, incorrectly treating the margin of error as inversely proportional to sample size rather than to the square root of sample size.
CThis multiplies the original margin of error by 2 instead of dividing, reversing the correct relationship.
DThis multiplies by 4, the ratio of the sample sizes, without applying the inverse square root relationship.
Question 24Hard
An agricultural scientist wanted to determine whether a new fertilizer increases crop yield in wheat fields. The scientist identified 80 wheat fields owned by farmers who agreed to participate in the study. The scientist randomly assigned 40 fields to receive the new fertilizer and 40 fields to receive the standard fertilizer used in the region. After one growing season, fields that received the new fertilizer had an average yield of 62 bushels per acre, while fields that received the standard fertilizer had an average yield of 54 bushels per acre. Which conclusion is best supported by the study design?
Show the answer and explanation
Why B is right
This is a randomized experiment with random assignment to fertilizer types, which supports causal inference. However, fields were owned by farmers who agreed to participate rather than being randomly selected from all wheat fields, so generalization is limited to fields similar to those in the study.
Why the others are wrong
AThis choice overgeneralizes to all wheat fields. The study used fields from farmers who agreed to participate, not a random sample from all wheat fields.
CThis choice misunderstands random assignment. While random assignment balances confounding variables such as soil quality and weather across groups, it does not mean the fertilizer is the only factor affecting yield.
DThis choice uses association language when causation can be inferred. The random assignment to fertilizer types supports a causal conclusion.
Question 25Hard
A marketing firm analyzed shopping preferences by collecting data from 500 customers who made purchases using a store's mobile app during a promotional weekend event. Based on the results, which of the following is the largest population to which the results can be generalized?
Show the answer and explanation
Why C is right
The sample was restricted to customers who used the mobile app specifically during a promotional weekend event. Behavior during promotional periods may differ from typical behavior, so results can only be generalized to this exact subset.
Why the others are wrong
AThis ignores both the mobile app restriction and the promotional event timing, making it far too broad.
BThis extends beyond the single store's app users and ignores the promotional event constraint.
DThis omits the crucial restriction that the sample came only from a promotional weekend event, when behavior may be atypical.
These 25 are a sample, not a study plan
These 25 come from a bank of 17,599 questions across all 31 SAT skills and three difficulty tiers. Inside SAT Climb you get the rest of Evaluating statistical claims, a diagnostic that finds which skills are actually costing you points, and a grid that shows what to drill next.
Questions are written by SAT Climb and drawn from its own item bank. SAT® is a registered trademark of College Board, which is not affiliated with and does not endorse SAT Climb.