  Score reliability is necessary for establishing a validity argument for an
instrument, and is therefore highly important to investigate. Depending on the
proposed instrument use and score interpretations, differing degrees of
precision in measurement or reliability are required. Researchers sometimes
fail to take a critical stance when investigating this important measurement
property, and default to accepted values of commonly known measures. This study
takes a multi-faceted approach to scrutinizing score reliability from a measure
of STEM teacher strategic knowledge using rater agreement, classical test
theory conceptions of reliability, and Generalizability Theory. This detailed
examination provides insight about where the greatest gains in score
reliability can be realized, given the design of the instrument and the context
of measurement.
