Developing validity arguments for artificial intelligence‐based assessment: Balancing affordances and threats
Martin G. Tolsgaard, Lawrence Grierson, Martin V Pusic, Laurah Turner
University of Copenhagen Copenhagen University Hospital Rigshospitalet Copenhagen Academy for Medical Education and Simulation
阅读操作
确认中在文库中上传 PDF 后可生成中文音频讲解。
摘要与影响
BACKGROUND: Artificial intelligence (AI) is increasingly used to generate, score and interpret educational assessment, yet these applications are being adopted in a largely unregulated environment. This creates a paradox: Whereas AI systems used in clinical care are subject to formal scrutiny for safety, performance and monitoring, AI systems used to inform consequential decisions about learner progression and future clinical practice are not. Existing validity frameworks remain useful but may not fully account for AI-specific threats. METHODS: We conducted a state-of-the-science conceptual review examining validity for AI-based assessment through Kane's four inferences: scoring, generalisation, extrapolation and implications. We integrated conceptual and empirical literature from medical education and adjacent AI evaluation fields to identify major threats, evidence needs and practical implications for defensible AI-based assessment use. RESULTS: AI introduces distinct validity threats across the inferential chain. For scoring, key risks include construct contamination, prompt instability and limited explainability. For generalisation, domain shift, rater culture differences and temporal drift threaten reproducibility across settings and time. For extrapolation, AI may alter the construct being assessed, making explicit construct definition essential to defensible interpretation. For implications, risks include subgroup inequities, automation bias, deskilling, weak accountability and erosion of academic integrity. Across these inferences, we identified seven cross-cutting mechanisms that can operate as either threats or affordances depending on design and governance. CONCLUSIONS: Validity for AI-based assessment should be approached as ongoing governance of a sociotechnical system rather than one-time validation of a static instrument or assessment process. Defensible adoption requires explicit validity arguments, auditability, multi-site evaluation and continuous consequence monitoring that is aware of the specific AI affordances and threats.
逐年被引趋势
关键指标
同类平均 = 1
同领域 · 同年份 · 同类型
Google Scholar 与 OpenAlex 的被引统计范围不同,数值存在差异属正常。
AI 辅助阅读
依据:摘要
可就本文提问;依据不足时会说明。
学术脉络
学科主题
生物医学Artificial Intelligence in Healthcare and Education
Innovations in Medical Education · Artificial Intelligence in Education