Research Methods

Chapter 9: Controls to Reduce Threats to Validity

Graziano and Raulin

Research Methods (8th Edition)

Threats to Validity

Covered in Chapter 8

Validity can be threatened in many ways

Presence of confounding variables
Unrepresentative samples
Inappropriate statistical tests or violations of statistics assumptions
Subject and experimenter effects

All these threats can be controlled

Graziano & Raulin (1997)

Control Procedures

General control procedures (applicable to virtually all research)

Control over subject and experimenter effects

Control through the selection and assignment of participants

Control through experimental design

Graziano & Raulin (1997)

General Control Procedures

Preparation of the setting

Free of distractions that might interfere
A natural setting increases external validity

Response Measurement

Use reliable and valid measures

Replication

Demonstrates that findings are consistent and robust

Graziano & Raulin (1997)

Types of Replication

Exact Replication

Repeating a study using identical procedures to the original

Systematic Replication

Using a theoretical or procedural change

Conceptual Replication

Varying the operational definitions of the variables to get new research hypotheses

Graziano & Raulin (1997)

Subject and Experimenter Effects

Blind procedures

Best control for expectancy effects
Single-blind: The experimenter does not know what condition the participant is in
Double-blind: Neither the experimenter nor the participant knows what condition the participant is in

Graziano & Raulin (1997)

Subject and Experimenter Effects

Automation

Reduces contact between participants and the experimenter
Gives the experimenter less opportunity to affect participants

Using objective measures

Objective measure require less judgment
Provides less opportunity for subtle experimenter biases to affect the data

Graziano & Raulin (1997)

Subject and Experimenter Effects

Multiple observers

Reduces bias because it challenges observers to be as precise and objective as possible
Can measure amount of observer agreement (percent agreement or Kappa)

Using deception

Hides purpose of the study from participants
Balanced placebo design is a good example

Graziano & Raulin (1997)

Balanced Placebo Design

Separates the pharmacological effects from the expectancy effects of alcohol

A two-factor design

Factor 1 is whether the person drinks alcohol
Factor 2 is whether the person thinks he or she is drinking alcohol

Graziano & Raulin (1997)

Balanced Placebo Design

This design crosses the consumption of alcohol with the belief that alcohol is being consumed	People Led to Believe
Drinking Alcohol	Not Drinking Alcohol
Actual Situation	Drinking Alcohol
Not Drinking Alcohol

Graziano & Raulin (1997)

Participant Selection

Can generalize only if your sample is representative

Populations and samples

General population: all potential participants
Target population: those participants you are interested in
Accessible population: portion of target population that is available to the researcher
Sample: drawn from the accessible population

Graziano & Raulin (1997)

Populations and Sampling

This figure shows the relationship between the various populations

General Population
Target Population
Accessible Population
Sample

Graziano & Raulin (1997)

Sampling Procedures

Random sampling

Every participant has an equal chance of being sampled

Stratified random sampling

Random sampling within strata (subgroups)

Ad hoc samples

Random sample from accessible population
Must generalize cautiously
Should describe sample to help define limits of generalization

Graziano & Raulin (1997)

Participant Assignment

Assignment Procedures

Free random assignment
Random assignment of participants to groups
Randomize within blocks
Randomly assign in blocks of one participant per condition
Matched random assignment
Random assignment of participants in matched sets to groups
Other matching procedures
e.g., match groups on key characteristics

Graziano & Raulin (1997)

Matched Random Assignment

Match on relevant variables

Variables likely to affect the dependent measure
Variables that show the largest variability in the population

Procedures

Match in sets on the relevant variable
Set size is the number of groups in the study
Randomly assign participants from the set, one to each group
Keep track of matching data for the statistical analysis

Graziano & Raulin (1997)

Experimental Design

Main focus of Chapters 10 through 13

Experimental design maximizes validity

Need to also include the other control procedures covered in this chapter

Key elements of experiments

One or more control groups
Random assignment of participants to groups

Graziano & Raulin (1997)

Ethical Principles

Balanced Placebo Design raises several ethical issues

Alcohol is a controlled substance
Intoxication poses risks
Some individuals are at especially high risk (e.g., people with certain medical conditions)

Must address these issues

Graziano & Raulin (1997)

Screening Participants

Must assure that participants are legally old enough to consume alcohol

Must screen out people who abuse alcohol and those with no experience with alcohol

Must exclude those with medical problems that might be exacerbated by alcohol

Must inform potential participants that alcohol may be consumed, so those with moral objects can decline

Graziano & Raulin (1997)

Deception

Participants know that, depending on the condition, they may or may not be consuming alcohol

They agree to this in the informed consent
What they don’t know is that they may be deceived about the condition to which they are assigned

Debriefing is required to clear up misconceptions

Graziano & Raulin (1997)

Protecting Participants

Must have medical safeguards available in the event of an adverse reaction

Must assure that the participant does not drive intoxicated

Usually by keeping them in the lab until the blood alcohol level has dropped

Graziano & Raulin (1997)

Summary

Most threats to validity can be minimized with proper use of control procedures

Broad classes of control procedures

General control procedures
Control over subject and experimenter effects
Control through participant selection and assignment
Control through specific experimental design

Graziano & Raulin (1997)

SUPPLEMENTAL SLIDES

Website Resources

Chapter figures

Graziano & Raulin (1997)

Website Resources

9:01 Computational procedures for Kappa

9:02 Why Small Samples May Not be Representative

9:03 Use of the Random Number Program

9:04 Sampling of Participants

9:05 Assigning Participants to Conditions

9:06 Study Guide/Lab Manual

9:07 Related Internet Sites

Graziano & Raulin (1997)

Pretest-Posttest Design

The simple pretest-posttest design fails to control many sources of confounding

History, Maturation, Regression to the Mean, etc.

Graziano & Raulin (1997)

Pretest-posttest Control-group Design

The control-group design controls most sources of confounding PROVIDED the groups are equivalent

Use random assignment to assure equivalence

Graziano & Raulin (1997)

Plagiarism Free Papers

Quick Links

We Accept

Contact Us