📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large-scale Italian LLM from scratch, achieving impressive benchmarks but scoring only 4.9% on Italian school exams. This reveals significant scaling challenges for sovereign-language models.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite being a technically advanced project.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, utilizing Italy’s national supercomputing infrastructure and funding through the PNRR. The project’s goal was to create a high-quality, native-language LLM that could outperform multilingual models on Italian benchmarks. The models, ranging from 350 million to 7 billion parameters, demonstrated strong performance on general Italian language tasks, outperforming comparable multilingual models.
However, when evaluated on the INVALSI Italian school exams, Minerva-3B scored only 4.9%, a result that is near chance levels. Researchers noted that while dataset composition influences model quality, the overall size of the dataset and the number of parameters are more critical for handling complex language tasks, including academic content. This finding challenges assumptions that native-language data alone suffices for deep knowledge representation in LLMs.
The results suggest that even substantial investment in native-language data and infrastructure may not be enough at current parameter scales, raising questions about the level of investment required to develop truly capable sovereign-language models. The Italian project exemplifies a long-term, large-scale approach that, despite technical achievements, exposes fundamental limitations in current scaling strategies.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Accelerate Everything with Tensor Cores: A Developer’s Guide to High-Performance AI, Efficient Training, and Scalable Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Natural Language Processing: A Textbook with Python Implementation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language AI Development
The Minerva results highlight that large-scale native-language training alone may not produce models with deep country-specific knowledge, especially at current parameter levels. This challenges the narrative that sovereign-LLMs can be developed effectively through data and infrastructure investment alone, emphasizing the need for even larger models or different approaches. The findings have broad implications for European AI policy, suggesting that national investments must consider the scale and complexity required to achieve meaningful language and knowledge depth, which could influence future funding and research strategies across Europe.
European Sovereign-LLM Strategies and Scaling Realities
The European sovereign-LLM debate has centered on two approaches: AMÁLIA’s continuation training with European data and Italy’s from-scratch training of Minerva. AMÁLIA, which layered Portuguese data onto a multilingual foundation, has not yet published weights, while Minerva made its models, data, and code openly available. Italy’s approach involved a significant investment—using 2.5 trillion tokens with half Italian content—and a dedicated institutional effort, including PNRR funding and high-performance computing resources.
Despite technical successes, Minerva’s low exam score underscores a key challenge: scaling models sufficiently to handle complex, country-specific knowledge remains difficult. Prior European projects have often assumed that data quantity and native focus are sufficient, but emerging evidence suggests that model size and training complexity are critical factors. This ongoing debate informs European AI policy, especially regarding the level of investment needed for sovereign-language models to reach practical utility.
“The Italian approach, while technically impressive, reveals the fundamental challenge of scaling models to truly capture country-specific knowledge.”
— Thorsten Meyer
Unanswered Questions About Scaling and Model Utility
It remains unclear how larger models, beyond Minerva’s current parameter scales, will perform on complex, country-specific tasks. The precise investment threshold needed to achieve meaningful country-knowledge depth is not yet established, and ongoing research aims to determine whether alternative architectures or training regimes can overcome current limitations.
Next Steps in European Sovereign-Language AI Research
The Minerva team plans to continue refining their models, including ongoing experiments with continual training and larger architectures. Future evaluations will likely assess whether increased model size can improve performance on academic and complex language tasks. Policymakers and researchers will also need to consider whether additional investments or different methodologies are required to meet national AI sovereignty goals.
Key Questions
Why did Minerva score so low on the Italian exams?
The evaluation indicates that, despite large-scale native-language training, current model sizes and training approaches may be insufficient for complex, country-specific knowledge tasks like academic exams.
Does this mean native-language training is ineffective?
Not necessarily. The findings suggest that native-language data alone is not enough at current scales; larger models or different training strategies may be needed to fully realize country-specific knowledge capabilities.
What are the implications for European AI policy?
The results imply that European investments in sovereign-LLMs must consider the scale and complexity required to produce truly effective models, potentially requiring more resources or alternative approaches.
Will larger models improve performance?
Ongoing research aims to evaluate whether increasing model size beyond Minerva’s current parameters can significantly enhance performance on complex, country-specific tasks.
How does this affect the future of European AI sovereignty?
The findings highlight that achieving deep, country-specific knowledge in LLMs is a significant challenge, influencing future strategies and investments in European AI infrastructure and research.
Source: ThorstenMeyerAI.com
