Blog / What Validity Evidence Means: Start With the Intended Use of a Score
What Validity Evidence Means: Start With the Intended Use of a Score
Validity is an evidence-based argument for a specific interpretation and use of scores, not a permanent badge attached to a questionnaire.
October 19, 2026 · 6 min read · By Cepingku Editorial Team
Ready to start?
Start the Career Decision Readiness →A score, chart, or familiar-looking label does not establish what the score can be used to conclude. The first question is more specific: what interpretation and use are being proposed? Using one self-report session to organise reflection on current career-exploration habits is a different claim from predicting job performance, making a diagnosis, or deciding who should be hired. Each claim needs evidence that directly fits that purpose. APA’s Guidelines for Psychological Assessment and Evaluation identify evidence based on test content, response processes, internal structure, relations to other variables, and testing consequences. The Guidelines also say that a valid inference requires scores to be directly, demonstrably, and consistently related to the outcome or purpose for which an instrument is used.
Validity is an argument tied to a use
Validity is not a permanent certificate earned when a questionnaire is written, nor does one statistic validate every later application. It is a reviewable argument: state what the score is intended to mean, assemble evidence appropriate to that interpretation, inspect conflicting evidence, and state what the score does not justify. APA’s Principles for the Validation and Use of Personnel Selection Procedures makes an important distinction: the familiar categories are sources of evidence for a proposed score interpretation and use, rather than interchangeable kinds of validity. A study that supports one link in the argument cannot silently carry every other link.
A fictional questionnaire shows the separate checks
Consider a deliberately fictional 12-item Career Preparation Reflection Questionnaire. Its only proposed use is to help adults review current career-information exploration and planning habits. It is not proposed for hiring, admission, diagnosis, or prediction of future earnings. Suppose its developer names two tentative domains, Career Information Exploration and Planning Action, with six items in each. That narrow claim still needs evidence, but it must not be expanded into a stronger claim without new, directly relevant support.
| Evidence source | Question it can address for the fictional questionnaire |
|---|---|
| Content | Do items, wording, response options, instructions, and scoring cover the two declared domains rather than repeatedly asking only whether someone has a plan? |
| Response processes | When respondents read ‘actively learn about roles,’ do they understand it as reading job descriptions, talking with workers, already holding a role, or something else? |
| Internal structure | With an appropriate sample, do relationships among item responses fit the proposed one-domain or two-domain structure, or a different structure? |
| Relations to other variables | If a relationship with a separately recorded exploration action is predicted in advance, does it appear in the specified direction and time window? |
| Consequences | Could the report lead users or organisations to treat a reflection score as an employment qualification, diagnosis, or guaranteed career outcome? |
Content evidence is more than asking whether a topic name sounds plausible. APA’s Principles describes test content as questions, tasks, formats and wording, response formats, instructions, and administration and scoring guidance. If a score is said to represent Planning Action while its items only ask about interest or emotion, the content argument is already weak. Conversely, topical coverage by itself does not show that a score predicts work performance or is suitable for a high-stakes decision.
Internal consistency answers only part of the question
Suppose the twelve fictional items yield a high internal-consistency coefficient. That may indicate that item responses vary together in some way. It does not alone show that the items cover the right content, that people understood them as intended, that the two proposed domains exist, or that the score can be used for selection. APA’s Principles explicitly says that generic consistency indices such as coefficient alpha do not evaluate a test’s internal structure, and that evidence of internal structure by itself does not establish job relatedness. The point is not that consistency is useless; it is that ‘items are similar’ must not be translated into ‘this score supports any important decision.’
Match each claim to inspectable material
A minimum review record for the fictional tool could include a domain definition and item blueprint for content; cognitive interviews showing how people understand key items; a pre-specified structural analysis in a named sample; a pre-specified external variable, time window, and expected relation if an external relation is claimed; and a review of report language for foreseeable misuse. Any result can strengthen, limit, or weaken the proposed interpretation. Selecting only supportive findings would not complete the argument. Changing the language, target group, item wording, response setting, or score use can also change what needs to be established, because the content, response process, and consequences may have changed.
Career Decision Readiness is a self-reflection entry point for exploring how a person gathers information, makes choices, and turns a possible direction into next steps. This article does not claim that the site’s assessment has content, response-process, structural, external-relation, fairness, norm, or predictive validation from the APA materials. It is not an employment, admission, diagnostic, or major-decision instrument.
Want to see where you land?
Take the Career Decision Readiness and get your results in minutes.
Take the Career Decision Readiness →