TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Memdoor workflow using five Qwen3.8-27B models averaged 9.8 correct answers out of 10 over five passes on the first 10 problems in a LiveCodeBench Hard list. Its authors report 9.6 for a re-created version of their multi-agent harness and 8.0 for Fable 5 alone, but the comparison covers a small set, uses different serving setups, and may be affected by training-data exposure.
A Memdoor workflow running five Qwen3.8-27B agents averaged 9.8 correct solutions out of 10 across five passes on the first 10 problems in a LiveCodeBench Hard list, according to results published in the project’s GitHub repository. The report compares that score with 9.6 for a re-created version of the authors’ own multi-agent harness and 8.0 for Fable 5 alone, but the limited test set and differing model-serving arrangements mean the figures do not establish a broad performance ranking.
The workflow assigns separate roles to a planner, two ideators, a solver and a manager, with a repair worker available in the loop. Each agent starts with a fresh context and exchanges information through files on disk. A shell script runs the sample tests and determines whether a solution succeeds; the report says the model itself does not decide the score. The workflow sees the public samples, while the final grading includes hidden tests.
The reported five-pass table gives the Memdoor workflow scores of 10, 10, 10, 10 and 9. The re-created harness scored 9, 10, 10, 9 and 10; Fable 5 alone scored 7, 8, 10, 8 and 7. The totals are based on 10 problems per pass, or 50 workflow runs, rather than the authors’ 100-problem evaluation. The repository says 49 of those 50 runs solved the problems.
The GitHub report says the implementation was generated from a description of the method rather than hand-written, and that its prompts and loop were rebuilt to match the authors’ published setup. It also says all runs used Qwen3.8-27B with a 128,000-token reply cap. The Memdoor runs were served through OpenRouter, while the authors’ original harness used their own vLLM deployment in FP8. Fable 5’s score comes from the authors’ result files and a single call per problem, not from running Fable 5 inside the Memdoor harness.
What the Scores Say About Agent Teams
The result offers a concrete, reproducible-looking account of how a multi-agent setup can perform on a coding benchmark, and the repository provides detailed run traces rather than only a headline score. That detail matters to developers evaluating agent workflows: the report includes requests, replies, generated programs, grader output, per-call costs and hosting information, allowing readers to inspect how the system reached its results.
However, the average difference between the Memdoor version and the re-created harness is 0.2 problems out of 10 over this particular sample. The test is too small to show whether that gap is meaningful or whether the workflow would retain its score on a larger, independent set. The figures also compare different hosting arrangements, and Fable 5 was not tested under the same harness. Readers should treat the report as a limited implementation comparison, not evidence that five agents generally outperform a single model.
The reported operational costs provide another practical data point: the 50 runs cost $35.79, or $38.77 when 16 attempts that were cut off, stopped for a fix or paused are included. The report gives a median of 40 minutes of model time per run, with the longest taking four hours. Those figures indicate that high scores in this setup involved multiple model calls and substantial elapsed time, not just answer quality.
Products Worth Considering
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Recreated
The report describes the work as a Memdoor implementation of the method associated with GVS5H (Lee, 2026), which used a five-model Qwen3.8-27B team on LiveCodeBench Hard. The implementation’s stated aim was to reproduce the authors’ prompts and loop, then compare the results on the same first 10 listed problems. The published material says the problems date from March 29 to April 6, 2025.
According to the repository, an initial version created its own task file for each problem, and the model planned less effectively from that text than from the benchmark’s exact problem statement. The authors say they made four Memdoor changes to better match the original harness: using a model’s reasoning when it returned no text, applying top-p and top-k settings from a model configuration, marking replies cut off at the token cap, and giving calls an independent four-hour clock. They report that each request in four scripted comparisons matched the original harness byte for byte.
The workflow’s code is available in the project’s GitHub repository, along with the run records in bench/results/parity-5pass. The report says the workflow can be run with a configured hosted model or a local model, but the benchmark results discussed here were served through OpenRouter, not run locally.
“49 of 50 solved.”
— Memdoor GitHub report
Limits of the Five-Pass Comparison
The main uncertainty is whether these scores generalize beyond 10 problems and five passes. The report notes that the benchmark contests were held in 2025 and says it cannot rule out the possibility that Qwen3.8-27B encountered the problems in training. It does not establish whether the results would hold on newer or independently selected problems.
The deployments were also not identical: the Memdoor run used OpenRouter, while the original harness used the authors’ own FP8 vLLM host. The report says Modal does not publish its precision and that calls shifted among providers during runs. Fable 5 was scored from the authors’ files, with one call per problem, rather than evaluated in the same workflow. These differences limit what the score gaps can show.
The report describes two cases where the workflow changed a correct program after the manager raised an incorrect counterexample or failed to indicate that work was finished. It says both problems were eventually solved within the run, but the examples show that manager decisions can damage a working solution. The authors suggest checking counterexamples against brute force on small inputs and retaining the last program that passed samples; neither safeguard was included in the reported results.
Reproduction and Broader Testing
The immediate next step for readers is to inspect or run the published workflow and traces. The repository gives setup instructions and sample files, and says users can replace the task and paired input/output tests to try the workflow on their own coding problems. Re-running the same benchmark may help check reproducibility, while a larger and newer problem set would be needed to test how well the result generalizes.
The report does not announce a follow-up evaluation or a scheduled release. Further comparisons would be more informative if they used the same serving setup, call limits and grading process for each system, and disclosed whether the test problems could have appeared in model training. Until then, the result is best understood as a documented score for one small benchmark sample, with broader performance and cost claims still open.
Key Questions
What did the Memdoor workflow score?
It averaged 9.8 correct answers out of 10 over five passes on the first 10 problems in the reported LiveCodeBench Hard list. Its pass scores were 10, 10, 10, 10 and 9.
Did the test show that the workflow beats Fable 5 in general?
No. The report gives 8.0 out of 10 for Fable 5 alone, but that score came from the authors’ files and a single call per problem, not from a same-harness comparison. The small sample also does not establish broad performance.
How many problems and runs were included?
The reported Memdoor results cover 10 problems across five passes, for 50 runs. The repository says 49 runs solved their problem. This is not the authors’ 100-problem evaluation.
Could training data have affected the result?
Yes. The report says the contests took place between March 29 and April 6, 2025 and that possible training-data exposure cannot be ruled out.
Can developers try the workflow themselves?
The report publishes the Memdoor workflow and sample files on GitHub, with instructions for running it using a hosted or local model. Its benchmark results, however, were served through OpenRouter.
Source: github.com
