🔍 Read the full analysis: GPT-6 Astra Examined Claude’s Model Edit And Flagged Missed Bugs on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A review by OpenAI’s GPT-6 Astra found reproducible problems in a Claude Opus 5.5-led experiment that edited a local model’s weights. The team fixed four initial faults, but in a later review all 12 author tests passed while four of six reviewer checks still failed. The experiment reported more benchmark answers after editing, but its legal score remained zero.
OpenAI’s GPT-6 Astra, running in Codex, reviewed a model-editing experiment led by Anthropic’s Claude Opus 5.5 and found bugs that could make benchmark results misleading. The review identified faults in resuming runs, handling invalid task IDs, invoking the command line and converting model weights; the findings matter because they call into question whether a passing test suite was enough to validate the experiment’s measurements.
The experiment sought to improve the local model Ornith 1.5 for legal tasks by weakening a “loop direction” in its weights, without further training. According to the source article, after the edit the model answered all 10 published Lexassay tasks, up from four, while its legal score remained zero. The runs are public on Lexassay. These results describe the reported benchmark; they do not establish improved legal capability.
GPT-6 Astra was assigned a review role rather than a competing model-editing experiment. It used separate checks, did not alter the main implementation, and the Claude session made the fixes. The reviewer reported that resuming a run could reuse an answer despite changed model or sampling settings; an unknown task ID could produce a normal-looking run with no evaluations; and a real command failed despite green parser unit tests. It also found that converting weights changed values even when edit strength was set to zero.
The source reports that the first-round faults were addressed with tests designed to fail against the old code. In a second review round, all 12 author tests passed, while four of the reviewer’s six provenance checks failed. The article also says GPT-6 Astra measured a 0.51% change when storing one real weight matrix in a different number format, with 915,360 of 1,048,576 values changing. That is a measurement of conversion, not legal quality or an error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Checks Matter
The review illustrates a practical weakness in relying on a green test suite alone: tests can pass while the actual command-line workflow fails, or while a run silently reuses results from a different setup. In a benchmark, those defects can compromise the link between a reported score and the model, settings or task that supposedly produced it.
Weight conversion creates a separate attribution problem. If values change even with the edit set to zero, a measured difference cannot automatically be credited to the intended intervention. The reviewer’s findings prompted changes to the experiment and more cautious wording, but they do not independently establish whether the model edit caused the reported answer-count change. The legal score’s remaining at zero also limits what can be concluded about practical legal performance.
Using a reviewer from another company may add useful scrutiny, particularly when the reviewer can reproduce concrete faults without changing the main rig. It is not a substitute for broader replication: this review had stated limits, and the source does not describe it as a full independent rerun of the benchmark.
Products Worth Considering
Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite
Bose QuietComfort Ultra Wireless Over-Ear Headphones with Spatial Audio and Noise Cancellation, Black
Jackery Explorer 2000 v2 Portable Power Station with Two 200W Solar Panels
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The work began on November 4, when two AI sessions were working on the same model-editing project. The roles were then separated. Claude Opus 5.5 led the plan, implementation, benchmark runs and blind grading by three Claude agents. GPT-6 Astra, from OpenAI and operating in Codex, was asked to review and validate the other session’s implementation.
The source says the reviewer did not inspect hidden benchmark prompts, topics, answers or reference solutions. Its first review took place while the experiment was being built and identified four reproducible faults. The article’s November 9 update says the figures refer to the 10 published Lexassay tasks and that the runs are publicly available. The source frames the work as an AI experiment, not legal advice.
The review also changed descriptions of the results. The article says “exactly identical at strength zero” was not accurate, that an average of wrong citations needed qualification, and that calling the work a “parallel Codex experiment” overstated GPT-6 Astra’s role. It further says the causal claim based on one control was stronger than the evidence supported.
Limits of the Review Evidence
The source says GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth blind grader, and measured on a CPU rather than the production GPU path. It also reports that four of six provenance checks failed in the second round, without the supplied material detailing each failed check or its resolution. The extent to which those failures affect the final published results is not specified.
It remains unclear from the supplied account whether the full 10-task benchmark was independently reproduced after the fixes, and whether the change in answer count was caused by the intended weight edit rather than conversion effects or other factors. The source reports the benchmark outcome and the reviewer’s measurements, but those findings should not be treated as proof of improved legal reasoning.
Published Runs and Further Verification
The source says the experiment’s runs are public on Lexassay, including the weight-edited model, an edit control and model-card sampling. Those records give readers a way to inspect the published work. The account does not announce a scheduled follow-up experiment or a further review milestone.
The next useful evidence would be a clearly documented rerun using the corrected code and controls, with its model, settings, conversion method and task results reported together. Until such evidence is available, the reported benchmark change remains a result from this experiment, while the review establishes specific faults found and corrected—not a general validation of the model edit.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and measurement process for a Claude Opus 5.5-led edit to the local model Ornith 1.5. It did not run a competing model-improvement experiment.
What bugs did the review find?
The first review found that resuming a run could reuse results from different settings, invalid task IDs could yield empty successful runs, a real command-line invocation failed despite passing parser tests, and weight conversion changed values even at zero edit strength.
Did the edit improve the model’s legal score?
The source reports that the edited model answered all 10 published Lexassay tasks, compared with four before the edit, but its legal score remained zero. That result does not show improved legal performance.
Were the review findings fully resolved?
The source says the four initial faults were addressed. It also reports that all 12 author tests passed in a later round, while four of six reviewer provenance checks failed. The supplied account does not explain each later failure or establish that every issue was resolved.
Can readers inspect the experiment?
The source says the runs are public on Lexassay, including the edited model, an edit control and model-card sampling. It does not say that the full benchmark was independently rerun after all fixes.
Source: ThorstenMeyerAI.com
