Investigating the Decision-making Style of Translation Raters in Large-scale Language Tests

Scoring constructed responses in large-scale language tests is crucial, as it affects the reliability, validity, and fairness of test results. While previous research has examined rater-related factors in explaining scoring variations, most studies focused on raters’ background variables, with limited attention to their cognitive decision-making. Although a few studies have examined raters’ decision-making styles in writing assessments, revealing the importance of raters’ cognitive differences in scoring, little is known about how such styles operate in translation assessments. This study employs a two-phase design to investigate the decision-making characteristics of translation raters. First, to assess the applicability of the General Decision-Making Style Inventory (GDMSI) in the College English Test (CET) translation rating context, responses from 469 CET raters were analyzed using exploratory factor analysis, confirmatory factor analysis, and reliability analysis. Second, a convergent parallel mixed-methods approach was used to construct decision-making profiles for ten experienced raters by integrating their GDMSI responses with qualitative think-aloud data. The results showed that, with minor adjustments, the GDMSI is a valid tool for assessing decision-making styles among CET translation raters, and the ten experienced raters exhibited three primary styles: rational, intuitive, and a hybrid of both rational and intuitive. These findings have practical implications for improving rater training and quality control in translation assessment contexts.