🔍 Read the full analysis: How To Read Z.ai’s Legal AI Benchmark Results And Ranking on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai says GLM-5.3-FlashX-Legal achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s archived leaderboard records 19.17%, placing it seventh among 81 models, while its criteria-level pass rate was 92.5%. The ranking measures completion of benchmark tasks under a strict all-or-nothing rule; it does not establish that the model is reliable for unsupervised legal work or that it is open source.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh of 81 listed models in the archived Version 1 table. The result measures how often the model met every requirement in a task—not the share of individual legal criteria it handled correctly—and Z.ai’s broader claim that it is state of the art among open-source models is not established by the ranking alone.
Z.ai describes the system as a model for professional legal workflows, including research, contract review and drafting. Its documentation says the model underwent legal-domain mid-training and post-training, with reinforcement learning in tens of thousands of simulated task environments. Z.ai recommends using it as an agent connected to services for laws, cases and company data. Those are company descriptions, not independent findings about performance in live legal practice.
The benchmark score is an all-or-nothing task pass rate. Harvey’s benchmark instructions say a task counts as complete only when every rubric criterion passes. Vals’ archived leaderboard lists GLM Legal at 19.17% across a 120-task private holdout, with a reported cost of $0.15 per test. Z.ai rounds the score to 19.2% in its claim. The leaderboard also gives the model a 92.5% criteria pass rate, using its stated rule that a criterion counts only when both grading judges pass it.
The distinction changes how the result should be read. A model can pass most individual checks across the benchmark while still failing many complete tasks because a task may require correct facts, conclusions, citations, deadlines, recommendations and formatting together. The task score means GLM Legal missed at least one required criterion on roughly four out of five tasks. It does not mean the model got only 19.2% of all legal questions right, nor does the 92.5% criteria score mean that 92.5% of complete work products were usable.
What the Score Means for Legal Teams
The result matters because benchmark rankings can shape procurement decisions, especially when buyers compare legal AI products on a single headline number. Here, the task-level score and criteria-level score describe different forms of performance. The high criteria figure may indicate that the model often satisfies individual rubric items, while the low all-pass rate captures the difficulty of completing every component of a multi-part assignment correctly.
Neither number, by itself, establishes how the product performs on a particular firm’s documents, practice area or workflow. The benchmark uses a defined set of tasks and grading rules; it is not a guarantee of accuracy on client matters. Legal teams considering the model still need to check citations and legal conclusions, test it against their own requirements, and review how client information is handled. The reported $0.15 per benchmark test is also not a full measure of deployment cost: Z.ai separately lists token prices of $0.37 per million input tokens and $1.25 per million output tokens, and connected tools may add costs.
Products Worth Considering
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public set contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction and client matter materials with a requested work product, assessed against more than 75,000 expert-written, binary criteria. The criteria can cover details such as facts, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Harvey’s all-pass rule is strict by design: one missed requirement makes the entire task a failure. Harvey has said that a deal-team report identifying eight of ten risks is materially incomplete, rather than 80% complete. Vals’ public leaderboard uses a 120-task private holdout modeled on the benchmark’s public mix. Internet access is disabled, and two language-model judges—GPT-5.5 and Claude Sonnet 4.6—grade submissions. Vals says the task pass rate is averaged across those judges.
On the same archived Vals table, the base GLM-5.3-Flash scored 6.67%, compared with 19.17% for GLM Legal. That is a 12.50 percentage-point difference, matching Z.ai’s stated improvement. Other models ranked above GLM Legal in that version, including Muse Spark 1.2 at 25.42% and Gemini 4 Argon at 19.58%. Harvey’s separate May baseline found a 7.1% best all-pass result among the frontier models it tested; because that was a different run, it should not be treated as a direct comparison.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.””
— Z.ai
Limits Behind the Ranking
The available leaderboard establishes a score under Vals’ stated protocol, but it does not show whether the model is dependable across specific legal settings or how often its errors would be material in practice. The source material also describes a small four-task spot check in which the reviewer said GLM Legal cited rulings they could not find; that check is not a benchmark and cannot establish an overall error rate. In the same spot check, the reviewer reported no invented rulings from OpenAI and Claude models but said two Claude models cited a real ruling for the opposite of its holding. These observations are limited and should not be generalized to the products as a whole.
Z.ai calls the result state of the art among open-source models, but the cited material does not specify the comparison set, and the source review found no published weights for the legal model. The company’s label therefore cannot be independently assessed from the ranking alone. The leaderboard is also a dated snapshot: its archived Version 1 table was updated October 18, while another Vals benchmark page was updated October 30. The supplied information does not establish whether that later version changes GLM Legal’s score or position. The terms governing client-data processing and the details of production safeguards are not established here either.
What Buyers Should Verify
Readers comparing legal AI systems should check the leaderboard version, task definition and scoring method before interpreting a rank. They should also confirm whether a comparison includes only models with published weights, how the test set and judges were selected, and whether the displayed costs cover the same workload. A ranking can help identify candidates for evaluation; it cannot replace review of the underlying tasks or testing on representative work.
For firms considering GLM Legal, the next steps are practical: test it on controlled examples, require qualified review of legal conclusions and every citation, and ask Z.ai who processes client data, where it is handled and under what terms. Z.ai’s October 13 AutoClaw post described a beta available by application. Further information about access, benchmark updates, model availability and data handling may clarify the product’s status, but no later release or revised score is confirmed in the available source material.
Key Questions
What does GLM Legal’s 19.17% score measure?
It is the share of tasks on Vals AI’s 120-task holdout for which the model met every required rubric criterion, averaged across two judges. A task with even one failed criterion does not count as a pass.
Why is the criteria pass rate 92.5%?
The criteria rate counts individual rubric checks rather than complete tasks. A model can pass many separate checks but still fail a whole task if it misses one required item. Vals says a criterion counts only when both judges pass it.
Was GLM Legal the top-ranked model?
No. In Vals AI’s archived Version 1 leaderboard, GLM Legal ranked seventh of 81 models. The table listed several models above it, including Muse Spark 1.2 and Gemini 4 Argon.
Does the benchmark prove the model is safe for legal work?
No. The score measures performance on a defined benchmark, not reliability on every firm’s matters or approval for unsupervised legal use. Firms would need to test the model, verify its work and assess client-data terms.
Source: ThorstenMeyerAI.com
