Critical Thinking Across the Curriculum: Empirical Argument Analysis

Critical Thinking
Across the
Curriculum Project

Analyzing Statistical Arguments

Contributed by Michael Connelly, Longview Community College.

Evaluating Statistical Arguments:

Start with "Simple Statistical Statements"
These statements have the following features:

They refer to a group of items as the subject matter of the statement - in short, the POPULATION.
They say something about the members of the Population - they ascribe some PROPERTY to the members.
They claim that some PERCENTAGE of the Population has the Property referred to in the statement.

Thus, the standard form for Simple Statistical Statements is:

Percentage of the Population has Property .

Statistical Statements (Premises) are usually the conclusion of a Statistical or Survey argument. These sorts of arguments rely on a SAMPLE of a larger Population to draw their conclusions. The strongest sort of Statistical Argument is where the SAMPLE POPULATION (the population actually observed or surveyed) is identical to the TARGET POPULATION (the group you are drawing conclusions about). Since this is often impractical, the Sample is often smaller than the Population being generalized about.

We will also need to distinguish between the MEASURED PROPERTY and the TARGET PROPERTY. The Measured Property is the property of the Sample Population that was actually looked at. The Target Property is the property of the Target Population that the generalization represents.

For example, consider the following argument:

The administration wants to find out how much a particular computer lab is being used. They set up a tracking program which records when a computer is used in the lab, and how many are being used. They track the usage for two weeks and find that the data shows that a computer is on in the lab for 75 of the 100 hours a week it is open, and that 20 or more computers are on for 50 of those hours. They conclude that the lab is being used 75% of the time it is open, and that 50% of the time it is open it is being used heavily. (Given a capacity of 30 computers).

In this argument, the Measured property is the same as the Target property. Now consider this argument:

The administration wants to find out how much a particular computer lab is being used. They set up a table in the Campus Center and ask as many students as they can if they use the computer lab, and how often. Of the 200 students who responded, 150 said they used the lab at least once a week, and 50 said they did not use the lab at all. Using this data, They conclude that the lab is used by about 75% of the students at least once a week.

In this argument, the Measured Property is not the same as the Target Property. Saying you use the lab and actually doing so are two different things. Buried in this argument is the assumption that the students are accurately reporting their use of the lab.

With these distinctions, we can generate a preliminary version of a standard pattern for statistical arguments:

Description of sample (the particulars)
Results of sample
Conclusion about sample
Final conclusion (the Generalization)

In evaluating Statistical Arguments, we will focus on three areas:

The accuracy of survey - is the measured property a good indicator of the target property?
The representativeness of the sample - is the sample population truly representative of the target population?
The reporting of the results - are the results of the sample reported in a way which mis-represents the data collected? (what I call gee-whiz graphs)

Accuracy in the survey:

There are at least three ways the accuracy of the survey can be compromised:

Slanted questions
Dishonest answers
Inaccurate tests

Representativeness of the Sample:

There are many ways in which the Sample Population could not represent the Target Population:

Small samples - usually a small sample will not be representative of a larger population
Unrepresentative samples - the sample population is not demographically similar to the target population
random samples - phone surveys
stratified random samples - mail surveys, Nielsens
significant margin of error - if margin of error is +3%, and the difference between groups (candidates) is 4%, then the results are not statistically significant.

Reporting the results:

Many times the results of statistical analysis will be reported using graphs and charts. There are visual ways of presenting information about a sample population which will lead us to one conclusion, while a more accurate presentation of the information would not do so. For some examples of these sorts of graphs, check out the Web site of USA Today under "snapshots", or look at these prime examples of "Gee Whiz Graphs".

Copyright © 1996
Critical Thinking Across the Curriculum Project
Longview Community College , Lee's Summit, Missouri - U.S.A.
One of the Metropolitan Community Colleges
An Equal Opportunity/Affirmative Action Employer

Permission to reproduce these resource pages is granted for
non-profit educational use provided the above information
is retained on all copies.

Inquiries to: michael.connelly@mcckc.edu
Last modified: 03/02/04