🔍 Read the full analysis: Zero Legal Score: The Result Of Our Home AI Weight-Editing Trial on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used directional ablation to reduce looping in the local model Ornith 1.5 35B-A3B. On the same setup, it completed all 10 published Lexassay tasks, compared with four before editing, but scored zero on legal accuracy and produced 55 wrong or invented citations.
A home trial of direct weight editing made the local AI model Ornith 1.5 35B-A3B complete all 10 published Lexassay legal benchmark tasks, up from four on the same setup, but its legal score remained zero. The result suggests the edit addressed the model’s looping behavior, not its ability to answer legal questions accurately.
The experiment, reported by ThorstenMeyerAI.com and updated November 9, used a technique called directional ablation. A Claude Opus 5.5 session ran the model, measured an internal direction associated with looping, and removed that direction at half strength from the output weights of 27 of the model’s 40 layers. The trial used no new legal training data and did not have an AI inspect the weights directly as text.
On the 10 published benchmark tasks, the edited model answered 10 of 10, compared with four on the same experimental setup before editing. But its score stayed at 0: the edited responses met seven of 50 checked criteria, up from three, and contained 55 wrong or invented citations, according to the report. The score measures the benchmark’s assessment of legal content and citations, not whether the model completed a response.
The report says a separate run with the model card’s recommended sampling settings, without editing, answered nine of the 10 published tasks. That comparison was on a different setup, however, and the author cautions that the two setups are not directly comparable. An independent review by OpenAI’s GPT-6 Astra, running in Codex, identified faults in the experiment’s code, figures and claims, the report says.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Completion Did Not Improve Legal Accuracy
The result highlights a distinction that matters for anyone evaluating AI systems: finishing an answer is not the same as getting it right. A model that stops repeating itself may appear more capable, but a higher completion rate does not establish that its legal reasoning or citations are reliable.
That distinction is especially relevant in law, where a fabricated statute or court decision can make a plausible-sounding response unusable. In this trial, the edit addressed a specific behavioral problem, while the benchmark found persistent weaknesses in the substance of the answers. The report therefore does not support using the edited model for legal guidance.
The findings also illustrate why benchmark scores need their task count and evaluation method alongside them. The figures here cover 10 published tasks, not the full 15-question Lexassay set described by the author. Five tasks are kept hidden, and the reported results should not be read as a broad measure of legal competence.
Products Worth Considering
Samsung Odyssey G9 QD-OLED 49 Dual QHD Curved Gaming Monitor
LG UltraGear 32-inch Curved QHD Gaming Monitor (2560×1440, 180Hz, 1ms, 1000R, HDR10)
Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite
As an affiliate, we earn on qualifying purchases.
A Simpler Test of Weight Editing
The experiment followed a report by four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—on what they call weight-space programming. In their described Qwen example, GPT-6 Astra examined another model’s weights and made a direct edit. The source article reports that the change stopped looping and raised a math score from 7.9% to 48.1%. The researchers cautioned that removing an obstacle can expose an existing ability, but cannot add an ability the model lacks.
The home trial was inspired by that prediction but did not reproduce the MIT method. Instead, it used a known approach that measures activations by running a model and then removes a selected direction. The report describes this as a simpler test of whether reducing loops would help a model that struggled on a legal benchmark.
Ornith was selected because the report says it was the local model showing the relevant looping behavior. Its publisher designed it for coding and agent work, not law. On the published Lexassay tasks, two responses hit the experiment’s 62,000-token output limit; in the final portions of those transcripts, only 2% to 13% of lines were distinct, according to the report. Other tested models either did not loop or could not be run under comparable conditions.
“The model finishes is not the answer is right.”
— Thorsten Meyer, author of the home trial
Limits of the Ten-Task Trial
The results remain preliminary. The report describes one run per setting, a small development set and a loop detector that changed during the experiment. It also says three graders came from the same model family, which may limit the independence of the grading. GPT-6 Astra reviewed the work and found faults, but the article does not describe that review as a second experiment.
The comparison across setups is another limitation: the unchanged model answered eight of 10 tasks in its published run, but four of 10 on the setup used for the edited-model comparison. The report says the model-card sampling run answered nine of 10 without an edit, but that result also used a different setup. It is therefore unclear how much of the completion difference reflects the weight change rather than sampling or other setup differences.
The published material gives results for 10 tasks; five Lexassay tasks remain hidden. The report does not establish whether the edit would generalize to those tasks, other legal benchmarks, other models or a larger set of runs. It also does not show that the edited model’s legal answers are safe or dependable for real-world use.
The report says the experiment’s runs, answers, grades and citation checks are public on Lexassay, along with a method page. Readers can examine the original run, sampling-settings run, edited model and edit control, though the source does not provide a schedule for additional testing.
The next useful evidence would include repeated runs under identical settings, a fixed loop-detection method, and evaluation by graders independent of the model family involved in the editing and blind grading. Results on the five hidden tasks could also test whether any apparent improvement carries beyond the published development set. Until such evidence is available, the reported finding is limited: the edit helped the model complete more tasks on one setup, but did not improve its legal score.
Key Questions
What changed after the weight edit?
On the same experimental setup, Ornith answered all 10 published tasks, compared with four before editing. The trial attributed the change to removing a measured loop-associated direction from part of the model’s output weights.
Did the edited model become better at law?
Not according to the reported legal score. It remained at zero, despite meeting seven of 50 checked criteria and completing more responses. The edited answers included 55 wrong or invented citations.
Was this the same method used by MIT researchers?
No. The home trial used directional ablation: it ran the model, measured an internal direction and removed it from selected weights. The MIT work described in the source involved an AI reasoning directly about another model’s weights.
How strong is the evidence?
The result is limited to one run per setting and 10 published tasks, with several methodological caveats described by the author. The article does not establish that the edit improves performance on hidden tasks or other legal tests.
Can the edited model be used for legal advice?
The source explicitly says the article is not legal advice. Its reported zero score and unreliable citations do not support relying on the model for legal guidance.
Source: ThorstenMeyerAI.com
