TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A critique of the post “Astra can read and write weights to improve Qwen” says its headline impression combines distinct experiments. The Qwen math result used Python and up to 50 batches of practice problems, rather than relying only on raw weights; the critique accepts the reported benchmark change but says it does not establish that narrower capability.
A critique of the post “Astra can read and write weights to improve Qwen” says its headline Qwen result did not come from simply inspecting raw weights: the experiment used Python code and let the model run Qwen on up to 50 batches of practice problems. The distinction matters because the original post’s framing links several different demonstrations, while the critique says they use substantially different methods.
The original post’s abstract, as quoted in the critique, says GPT-6 Astra can infer what a network was trained to do from raw weights, write small game-playing networks, and improve Qwen3.5-0.8B at math by editing 0.016% of its weights. It reports a MATH-500 score increase from 7.9% to 48.1%, and an IFEval score increase described as threefold. Those are claims reported by the original post; the supplied critique does not independently verify the underlying measurements.
The critique’s experiment summary separates the raw-weight demonstrations from the Qwen test. In its account, experiments that presented raw weight values without tools or network execution involved smaller networks with 17,000 to 172,000 weights. By contrast, the Qwen experiment involved an 800-million-parameter model, Python, model execution and as many as 50 batches of practice problems. The critique says every experiment involving a real language model used Python.
The reviewer does not argue that using code or editing weights is inherently unimportant. Rather, the central objection is that the results should not be read as evidence of one method that both understands raw weights unaided and improves a large language model. The critique describes the Qwen result as showing that Astra can be prompted to perform tasks a human machine-learning engineer might carry out, including writing code to inspect or change weights and testing model behavior.
What the Qwen Test Actually Shows
The distinction affects how readers should interpret the capability being demonstrated. Reading raw weights without tools, writing a network from scratch, and iteratively modifying a model with code and test runs are different tasks. Evidence for one does not automatically establish the others.
According to the critique, the Qwen experiment may still show a useful ability: Astra devised and executed a process involving weight modification and repeated evaluation. But that is a broader, tool-assisted workflow, not a demonstration that the system understood Qwen’s weights through inspection alone. For researchers and readers comparing AI claims, identifying which inputs, tools and feedback loops were available is necessary to judge what a benchmark result supports.
The distinction also matters for comparisons with human expertise. Engineers routinely use scripts and experiments to inspect or alter model parameters. An AI system that can productively perform such work may represent a meaningful capability, but the result is different from a model independently inferring a network’s function from raw numbers. The critique argues that combining both kinds of evidence in one narrative risks making the overall capability appear more unified than the experiments establish.
Products Worth Considering
Apple Watch Series 12 GPS 46mm Smartwatch
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Separate Experiments, Different Methods
The original post brings together tests involving several kinds and sizes of networks. Its first demonstrations reportedly ask Astra to identify a game from a small trained policy and identify the output shape of a diffusion network. The critique says these tests supplied raw weights and did not run the network. It also lists an experiment in which Astra altered a small Kinetix car policy, reporting a change from 0% to 100% in one try after modifying 3.5% of the weights; the reviewer notes that this test used one recorded failure episode.
Other demonstrations used different methods. The critique says Astra wrote small MinAtar networks from scratch, but their weights began at zero, so there were no existing weights to inspect. It also describes a Go-network experiment that used Python and network execution, and tests involving Gemma that used code; one reported sycophancy test succeeded only when Astra could run the model and inspect attention patterns, while a related weight-editing attempt failed.
These details do not settle the quality of every result, but they explain the critique’s main point: the post’s experiments do not all test the same capability. The strongest headline result for Qwen belongs to the group involving code and model runs, not the smaller raw-weight-only demonstrations.
““Every experiment involving a real language model uses Python.””
— The critique’s author
What the Report Does Not Verify
The supplied material is a critique of another post, not an independent replication or full technical audit. It does not provide the original experiment logs, prompts, code, model checkpoints or statistical details needed to assess the MATH-500 and IFEval results directly. The reported score changes should therefore be understood as figures attributed to the original post, rather than independently confirmed findings here.
The critique also does not establish whether the reported Qwen improvement persists across different prompts, evaluation runs or tasks beyond those described. It says the model was tested on practice batches, but the supplied account does not explain how those examples were selected or whether they overlap with evaluation data. The scope and reproducibility of the result remain unclear from this material.
Nor does the critique establish that the original post explicitly claimed every experiment used the same method. Its argument is that the abstract and presentation can lead readers to associate the raw-weight capability with the Qwen result. The original post’s authors’ response, if any, is not included in the supplied source.
Further Tests Needed to Resolve the Dispute
The immediate next step for readers is to compare the original post’s methods and experiment descriptions with the critique’s account, especially the tools available in each test and whether the network was run. A clearer presentation would report raw-weight inspection, Python-based analysis, weight editing and model feedback as separate conditions.
To establish how broadly the Qwen finding applies, follow-up work would need to document the prompts, code, practice batches, evaluation procedure and repeated results. Independent replication could test whether the reported improvement holds under controlled conditions and whether it extends beyond the stated benchmarks. Until those details are available, the narrow conclusion supported by the critique is that the Qwen result was tool-assisted and iterative, not a raw-weights-only demonstration.
Key Questions
What does the critique say is misleading?
It says the framing can make readers think Astra improved Qwen simply by inspecting raw weights. According to the critique, the Qwen test used Python, model execution and up to 50 batches of practice problems.
Did the original post report a large improvement on MATH-500?
Yes. As quoted in the critique, the original post reported a change from 7.9% to 48.1%. The supplied material does not independently verify that result.
Does the critique say Astra cannot edit model weights?
No. It describes weight-editing experiments and argues that the Qwen result involved a broader, code-assisted workflow. Its objection is about what the experiments demonstrate, not whether weight editing occurred.
What remains uncertain about the Qwen result?
The supplied source does not include enough underlying data to assess independent reproducibility, the selection of practice batches or how well the result generalizes beyond the reported tests. The original authors’ response is also not provided.
Source: honnibal.dev
