Natural Language Response Formats for Assessing Depression and Worry With Large Language Models: A Sequential Evaluation With Model Pre-Registration

Large Language Models can transform individuals’ mental health descriptions into scores that correlate with rating scales approaching theoretical upper limits. However, such analyses have combined word- and text-responses with little known about their differences. We develop response formats ranging from closed-ended to open-ended: 1) select words from lists, write 2) descriptive words, 3) phrases, or 4) texts. Participants answered questions about their depression/worry using the response formats and related rating scales. Language responses were transformed into word embeddings and trained to rating scales. We compare the validity (concurrent, incremental, face, discriminant, and external validity) and reliability (prospective sample and test-retest reliability) of the response formats. Using the Sequential Evaluation with Model Pre-registration design, machine-learning models were trained on a development dataset (N=963), and then pre-registered before tested on a prospective sample (N=145). The pre-registered models demonstrate strong validity and reliability, yielding high accuracy in the prospective sample (r=.60–.79). Additionally, the models demonstrated external validity to self-reported sick-leave/healthcare visits, where the text-format yielded the strongest correlations (being higher/equal to rating scales for 9 of 12 cases). The overall high validity and reliability across formats suggest the possibility of choosing formats according to clinical needs.