  Guidance on how to validate computational text-based measures of social
constructs is fragmented. While researchers generally acknowledge the
importance of validating text-based measures, they often lack a shared
vocabulary and a unified framework to do so. This paper introduces ValiText, a
new validation framework designed to assist scholars in validly measuring
social constructs in textual data. The framework is built on a conceptual
foundation of validity in the social sciences, strengthened by an empirical
review of validation practices in the social sciences and consultations with
experts. Ultimately, ValiText prescribes researchers to demonstrate three types
of validation evidence: substantive evidence (outlining the theoretical
underpinning of the measure), structural evidence (examining the properties of
the text model and its output) and external evidence (testing for how the
measure relates to independent information). The framework is further
supplemented by a checklist of validation steps, offering practical guidance in
the form of documentation sheets that guide researchers in the validation
process.
