Milwaukee Showcase

How Valid Are Self-Report Survey Data Obtained from School District Personnel?

Description

"Most quantitative evaluation endeavors undertaken to assess the effectiveness of large
scale educational reform efforts must rely on self-report data to some extent. For example, the
analytical framework used to assess the impact of the Milwaukee Mathematics Partnership
(MMP) on increasing student achievement in mathematics is largely dependent on data obtained
from an online survey administered to all Milwaukee Public School District (MPS) employees at
the K-8 level who have the potential to positively affect student achievement in mathematics.
This survey was designed to measure various aspects of educators’ daily responsibilities that
would likely be affected by MMP activities and in turn, affect student achievement in
mathematics. However, the validity of self-report data is always suspect due to the possibility of
response bias, which occurs if participants respond to items in a more socially appealing manner.
In the current survey, because respondents are primarily reporting on the quality of their own
work and that of their colleagues it is quite possible that the validity of our data is compromised
by response bias. Moreover, if this is true then the results of any statistical tests conducted on
these data may be biased because these results are wholly dependent on how well the variables
have been measured.


Therefore, this year the evaluation team of the MMP made a concerted effort to assess the
validity of our self-report data. Two approaches were taken to attain this goal: (1) A global
approach that utilized all schools in the district that participated in the quantitative evaluation
efforts; and (2) A case-study approach that only made use of schools that participated in the
qualitative evaluation efforts. For the global approach, overall indicators of the quality and
quantity of a school’s participation in MMP related activities obtained from the self-report
survey data were compared to ratings obtained from the district level Mathematics Teaching
Specialists (MTS) assigned to that school. For the case-study approach, an attempt was made to
triangulate the self-report survey data obtained with other external criteria in a subset of schools
targeted to participate in more in-depth intensive evaluation efforts. The primary purpose of this
paper is to describe the methodology utilized and report the findings for both of these approaches
to assess the validity of our data."