Towards a Benchmark Dataset for Legal Language Understanding in Vietnamese

Large Language Models have demonstrated promising capabilities in professional domains; however, their application in law requires rigorous adherence to precision and logical reasoning. While benchmarks have been established for high-resource languages such as English and Chinese, the Vietnamese legal domain still lacks a structured framework for evaluating legal reasoning beyond surface-level question answering. Towards addressing this gap, we introduce VLaw158, an initial benchmark dataset for assessing legal language understanding in the Vietnamese legal context, with a focus on the Civil Law system. Rather than relying on monolithic evaluation metrics, our dataset is organized according to a hierarchical cognitive framework inspired by Bloom’s Taxonomy, covering three levels: Memorization, Understanding, and Application. The benchmark consists of 1,580 expert-verified instances spanning Civil, Administrative, and Criminal law. We conduct zero-shot evaluations using both state-of-the-art general-purpose models (e.g., GPT-4o-mini) and region-specific open-source models. Our results reveal a consistent performance gap across cognitive levels: while models perform relatively well on memorization-oriented tasks, their accuracy declines notably on tasks requiring statutory interpretation and application. These observations suggest limitations in current models’ ability to perform deeper legal reasoning, and indicate that smaller, language-specialized models may be competitive in certain Vietnamese legal settings. We release this dataset to support future research on legal language understanding and low-resource legal AI at https://github.com/diephoangngocc/VieLaw.