📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI has achieved significant automation in engineering tasks, with benchmarks indicating near-saturation. Research, however, remains less automated, though this gap may close faster than expected. The development signals a shift in AI capabilities and strategic focus.
Recent developments confirm that AI systems are now capable of automating the core engineering tasks involved in AI research, reaching near-saturation levels on key benchmarks. Meanwhile, the automation of AI research itself remains less certain, though evidence suggests this gap may narrow quickly. This shift has significant implications for the future of AI development and research strategies.
Multiple benchmarks, including CORE-Bench and MLE-Bench, demonstrate rapid progress in AI’s ability to reproduce research and perform on Kaggle competitions, with capabilities approaching or surpassing human-level performance. For example, CORE-Bench scores have increased from 21.5% in September 2024 to over 95.5% in December 2025, with the benchmark’s author declaring it ‘solved.’ Similarly, MLE-Bench scores have risen from 16.9% to 64.4% within sixteen months, putting AI at competitive levels with mid-tier human performance.
These improvements suggest that the engineering aspect of AI research — reproducing experiments, optimizing kernels, and running complex code — is effectively automated, reducing the marginal cost and friction traditionally associated with research replication and engineering tasks. Clark’s analysis points to a structural pattern: as these benchmarks approach saturation, the capacity for automation in core engineering tasks becomes nearly complete.
In contrast, the automation of research — the creative and hypothesis-driven aspects — remains less developed. Clark notes that some research tasks may be inherently different from engineering, involving creativity and insight that are harder to automate. However, the rapid progress in engineering capabilities raises the possibility that research itself could become automated at scale, blurring the line between engineering and research processes.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
The AI Productivity Blueprint: How Software Engineers and Designers Can Save 50% of Their Time
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
AI for Scientific Discovery (AI for Everything)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Thames & Kosmos My Robotic Pet: Coding Chameleon STEM Building & Experiment Kit | Color-Sensing Coding Robot for Boys & Girls Ages 8+, Screen-Free Educational Fun, Robotic Reptile with 3 Play Modes
BUILD, CODE, PLAY & LEARN: Construct a robotic reptile pal that responds to your gestures, changes colors, and…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications of AI Achieving Near-Complete Engineering Automation
The near-saturation of AI capabilities in core engineering tasks suggests a potential paradigm shift in AI development. As the bottleneck in reproducing experiments and optimizing models diminishes, the focus may shift toward automating research discovery and hypothesis generation. This could accelerate AI progress, reduce costs, and reshape the roles of human researchers. However, it also raises strategic questions about innovation, oversight, and the future of scientific inquiry in AI.
Progress Patterns in AI Engineering Capabilities
Over the past two years, multiple independent benchmarks have tracked AI’s progress in core research skills. CORE-Bench, measuring research reproduction, improved from 21.5% to over 95.5%, with the author declaring the task ‘solved.’ MLE-Bench, assessing Kaggle competition performance, rose from 16.9% to 64.4%, indicating AI’s competitiveness with mid-tier human practitioners. Additionally, advances in kernel design and infrastructure optimization reflect ongoing research in production-grade AI engineering.
This pattern of rapid, overlapping improvements across different capabilities suggests that the engineering side of AI research is approaching automation saturation. Clark’s analysis indicates that the structural pattern of these benchmarks is a strong signal of AI’s growing mastery of engineering tasks, which are fundamental to AI research itself.
“The pattern across multiple benchmarks shows AI nearing saturation in core engineering tasks, fundamentally shifting the landscape of AI research.”
— Thorsten Meyer
Uncertainties About the Automation of AI Research
While engineering tasks are nearing full automation, it is still unclear how much of the research process — involving hypothesis generation, creative problem-solving, and novel insight — can be automated. Clark leaves open whether research tasks are fundamentally different from engineering or if they will follow the same rapid saturation pattern. Additionally, the pace at which research automation might accelerate remains uncertain, and institutional or strategic factors could influence this trajectory.
Next Steps for AI Research and Engineering Automation
In the coming months, further benchmarks and real-world implementations will clarify the extent of AI’s automation in research. Researchers and organizations will need to monitor these developments to adjust strategies accordingly. Additionally, exploring how to integrate automated research processes with human oversight will be critical. The evolution of institutional policies and funding priorities may also influence the pace and scope of automation in AI research.
Key Questions
What are the main benchmarks indicating AI automation progress?
CORE-Bench, measuring research reproduction; MLE-Bench, assessing Kaggle competition performance; and various kernel design advancements are key benchmarks showing rapid progress towards automation saturation.
Does this mean AI research will soon be fully automated?
While engineering tasks are nearing full automation, it remains uncertain whether creative and hypothesis-driven research can be fully automated. The pace of progress suggests it could happen faster than previously expected, but definitive conclusions are still pending.
What are the implications for human researchers?
Automation of engineering tasks may reduce costs and friction, allowing researchers to focus on higher-level innovation. However, it also raises questions about the future roles of human researchers and oversight in AI development.
What could accelerate or hinder the automation of research?
Factors include technological breakthroughs in AI’s creative capabilities, institutional resistance, regulatory environments, and strategic priorities within research organizations.
Source: ThorstenMeyerAI.com
