Validating a Test to Measure Depth of Vocabulary Knowledge

This study is a contribution to the ongoing validation of the word associates format, a selected-response type of test item devised by the author to measure depth of vocabulary knowledge (i.e., the extent to which learners are familiar with the range of meaning and uses of a set of target words). The present study builds on previous work with an earlier version of the format. A test consisting of word associates items was developed with 40 high-frequency English adjectives as target words. To obtain concurrent validity evidence, two other measures were employed to assess knowledge of the same words: a written test with a matching format and an interview. The two written tests were analyzed using the Rasch partial credit model, and the three measures were intercorrelated. The analysis showed that the word associates test was reliable and generally had a good fit to the Rasch model. However, the other two tests were more highly correlated with each other, and there was evidence that guessing played a significant role in performance on the word associates test. These results are discussed within the framework of Messick’s theory of test validity and, in particular, the kind of inferences that can be drawn from the test scores as well as the nature of the construct being tapped. Some future lines of validity inquiry are also considered.