TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Sakana AI researchers describe Multi-Agent Self-Supervision (MASS), a method that repeatedly searches for effective multi-agent workflows and trains a shared language model on selected team outputs. The report says the approach improved score per output token over the base model on research benchmarks, but the method relies on self-evaluation and the source does not provide enough detail here to establish the breadth or reliability of the gains.
Sakana AI researchers have introduced Multi-Agent Self-Supervision (MASS), a method that trains a language model on work produced by a team of agents running that same model. The report says MASS improved score per output token over the base model on research benchmarks, offering a test of whether collective agent work can help a model improve when external supervision is limited.
MASS assigns one shared model three roles: a task-solving executor, a workflow optimizer and an evaluator. The optimizer revises how agents divide tasks, share findings and check one another’s work. The evaluator compares the resulting workspaces and selects what it judges to be the stronger result. In this setup, the model being improved also supplies the feedback used to guide improvement.
The process has two loops. In the inner loop, the researchers search for a team workflow while keeping the model’s weights fixed. In the outer loop, the system runs the selected workflow, ranks the resulting workspaces using the model’s own judgments, keeps selected examples and fine-tunes the shared model on them. The updated model then takes part in the next round of workflow search and evaluation.
The report focuses on non-verifiable research tasks, defined as tasks without an available automatic rule-based signal for judging answers. The authors say they separately asked GPT-5.5 and Claude Opus 4.8 to assess completed work and withheld those judgments from MASS, using them as an outside measure. The source describes improved score per output token compared with the base model, but the supplied material does not state the size of the improvement.
How Teamwork Becomes Training Data
The proposal addresses a supervision problem that could grow as AI systems tackle work beyond the limits of easy checking. A model may generate research ideas faster than people can verify them, particularly when evaluation requires specialist knowledge, costly evidence or lengthy experiments. MASS tests whether a system can draw useful training signals from the combined work of multiple agents rather than relying only on a human-provided answer key.
If the reported results hold up, the method could offer one way to reuse the problem-solving experience created during agent collaboration. Its central proposition is that coordination can produce work or judgments that are useful to the shared model, even though each agent uses the same underlying model. That is a research direction, not evidence that the system can reliably improve itself across domains or produce independently verified discoveries.
The distinction matters because the evaluator is part of the system being optimized. A model can favor outputs that match its own preferences without those outputs being correct. Outside assessments provide a check in the reported experiments, but they do not by themselves establish that self-evaluation will remain dependable on harder tasks or in practical use.
As an affiliate, we earn on qualifying purchases.
The Self-Evaluation Research Setup
Sakana AI frames MASS as a form of homogeneous recursive self-improvement: one model serves as the optimizee, optimizer and evaluator. The report’s motivation is that, at the frontier of research, human experts may find it difficult to assess proposals or direct progress, while the model itself may be the most capable available supervisor. The researchers ask whether collective work can supply a useful signal within that constraint.
The analogy in the report is to human communities that divide work, combine ideas and pass on shared knowledge. MASS translates that idea into a computational process: a team explores a task under a written workflow, and selected executions become training examples for the model shared by the team. The report presents this as a research method and does not claim that the model has escaped human-designed tasks, data or oversight.
For evaluation, the authors distinguish the system’s internal judgments from assessments by other models. GPT-5.5 and Claude Opus 4.8 are described as external evaluators whose judgments were not used to steer MASS. This separation is intended to measure progress beyond the self-judgment signal, although the source material does not give full benchmark names, scores or experimental details.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Evidence
The supplied report material does not identify the research benchmarks or provide numerical results, sample sizes, comparison settings or uncertainty estimates. It is therefore not possible from this information to quantify the reported improvement or judge how consistent it was across tasks. The phrase score per output token describes the reported measure, but the source excerpt does not explain its calculation.
Another open issue is whether the model’s self-ranking reliably selects better work. The outside assessments by GPT-5.5 and Claude Opus 4.8 offer a separate comparison, but both are also AI systems, and the excerpt does not give their detailed findings or describe human expert validation. The accuracy of MASS’s internal evaluator remains central to whether its training loop improves task performance rather than reinforcing its own errors.
The material also does not establish how well the method transfers to other models, domains or longer training runs, or what computational costs the workflow search and fine-tuning require. No deployment results or independent replication are described in the supplied source.
As an affiliate, we earn on qualifying purchases.
Replication and Benchmark Details
The next evidence readers need is a full account of the benchmark results, including task definitions, scores, baselines, token costs and the degree of variation across runs. Those details would make it possible to assess how much MASS gains over the base model and whether the reported score-per-token measure reflects better task outcomes or a change in output length.
Further testing could compare MASS with alternative training methods and check its results against human expert judgments, especially on tasks where correctness is difficult to verify. Replication by researchers outside Sakana AI would also help establish whether the findings generalize. The supplied material does not announce a timetable for these steps, so the method’s broader performance and next milestones remain unclear.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is MASS?
Multi-Agent Self-Supervision is a method in which agents using one shared language model work under a workflow, while that model also revises the workflow and evaluates the team’s output. The system then trains the shared model on selected executions.
What does recursive self-improvement mean in this report?
Here, it means the same model serves as the task solver, the optimizer of agent coordination and the evaluator of results. The model is updated using selected team work, then used again in those roles.
Did Sakana AI report a measured improvement?
The report says MASS improved score per output token over the base model on research benchmarks. The supplied material does not include numerical scores, the size of the improvement or the benchmark names.
Were the results checked outside MASS?
The authors say they used GPT-5.5 and Claude Opus 4.8 to assess completed work and withheld those judgments from the training loop. The supplied excerpt does not provide the detailed assessments or describe human validation.
Does the report show that AI can improve itself without human oversight?
No. It describes a research method for self-supervision on selected tasks. The report material does not establish that MASS can independently verify its results, operate without human involvement or reliably improve across fields.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
