CLER: A benchmark for Chinese litigation evidence reasoning

Evidence prediction is the cornerstone of litigation, which requires complex legal reasoning to bridge the gap between facts and the parties’ assertions. However, most existing large language model (LLM) benchmarks do not offer a systematic assessment for evidence planning. Therefore, we introduce CLER, a large-scale Chinese Litigation Evidence Reasoning benchmark. CLER is built through a four-stage pipeline from millions of real civil and criminal judgments. The pipeline includes (1) semi-structured data collection, (2) LLM-based evidence extraction, (3) purpose refinement through decomposition into legal elements, and (4) validation through a tri-party adversarial review method that imitates courtroom debate. To this end, we obtain over 100,000 samples and conduct extensive experiments on 23 baselines, including open-weight, commercial, domain-specific LLMs and retrieval-augmented generation (RAG) systems. The findings indicate that the task poses a major challenge for current LLMs. RAG strategies, particularly a hybrid method retrieving similar cases, achieve the best performance with an F1 score of 54.4%, surpassing state-of-the-art proprietary models and improving the base model by at least 22%. Our analysis reveals several key insights. There is a significant performance gap with an average difference F1 of 16.2% exists between element checking in civil cases and narrative completeness in criminal cases. There is an inverted U-shaped correlation between case length and model performance, as insufficient or overloaded information leads to abandonment of precise reasoning, causing an average drop of approximately 10% in F1.