Assessing the psychometric properties of AI-generated multiple-choice exams in a psychology subject

This study assessed the psychometric properties of AI-generated multiple-choice questions in undergraduate psychology education, specifically focusing on an Experimental Psychology course. Using a mixed-methods approach, we evaluated 80 multiple-choice questions created by ChatGPT-4 through expert content validation, administration to undergraduate psychology students, and comprehensive psychometric analysis. Results indicated that AI-generated items demonstrated reasonable content validity with strongest ratings for clarity and weakest for distractor quality. The assessment showed acceptable internal consistency and moderate test-retest reliability. Item difficulty analysis revealed a slight tendency toward easier items, with 58.75% of questions in the medium difficulty range. Discrimination indices were generally good, though 11.25% of items showed poor discrimination. Cognitive level analysis identified a significant imbalance in the distribution of items across Bloom's taxonomy, with 73.75% targeting lower-order thinking skills (remembering, understanding) and only 8.75% addressing higher-order skills (analyzing, evaluating). These findings suggest that while AI can generate psychometrically sound multiple-choice questions for psychology assessment, significant limitations exist in assessing higher-order cognitive skills. The results support a hybrid approach that leverages AI for generating fundamental knowledge assessment items while preserving human expertise for higher cognitive domain evaluation. This study contributes to the emerging understanding of AI applications in psychology education by providing domain-specific insights into the capabilities and limitations of AI-generated assessment tools.