Date of Award

2016

Degree Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Education

Abstract

Evaluators need more design options to meet the challenges they face in detecting change or growth. Researchers have offered the retrospective pretest/posttest design as a remedy to curb response-shift bias and better estimate program effects, but few studies have used this approach with youth. After School Matters, a Chicago nonprofit that provides after-school programs to teens tested the retrospective pretest/posttest design using a mixed methods design to determine whether response-shift bias exists. My study provided several findings. First, though my quantitative analysis did not indicate response-shift bias was as prevalent as literature would indicate, my qualitative findings indicated that response-shift bias was in fact an issue. Second, I found a relationship between teens’ self-reported interpersonal skills and response-shift bias. Teens who reported positive interactions with their peers and instructors tended to display large shifts in their responses from traditional pretest to retrospective pretest. Third, teens preferred to see the posttest and retrospective questions in chronological order, which is contrary to the literature. Fourth, I found acquiescence to be the biggest potential bias when using the design with teens. Overall, the retrospective pretest/posttest design is a practical and useful design to evaluate youth self-reported change. The mixed methods design led to dissonance, iterative data analysis, and some inconclusive findings, but also a much deeper understanding of response-shift bias.

Creative Commons License

Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License.

Download

Included in

Education Commons

Share

COinS