🔍 Read the full analysis: Perplexity And Intelligence on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI has published internal measurements of its models’ pretraining perplexity and argues that the metric could help track AI progress across developers. The company links lower loss to capabilities in its own models, but the figures do not establish that a particular score guarantees an ability, and no shared reporting standard has been announced.
OpenAI has published internal measurements of its models’ pretraining performance, arguing that perplexity could give AI developers a shared way to track progress and inform safety standards, as explored in the original analysis of perplexity and intelligence. The company connects falling scores with capabilities in its own models, while stressing that future abilities and the timing of further advances remain uncertain.
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values indicate better predictions. OpenAI says it has tracked the measure over time, including for models beginning with GPT-3.5, and is now publishing some internal results. The company describes the figures as a way to follow underlying predictive performance even as task-specific benchmarks change or become less useful, a question also relevant to measuring AI intelligence gains.
In the figures described in the source, OpenAI reports log-perplexity of about 0.99 for GPT-3.5 and about 0.91 for GPT-4, and says later models have reached losses below 0.8. It associates these stages with fluent conversation, productive use of chain-of-thought reasoning, and mathematics and cybersecurity capabilities. Those connections are OpenAI’s account of its own models, not evidence that a specific score reliably produces a specific capability.
OpenAI also says a model at about 0.76 log-perplexity began producing results on open mathematics problems that the post describes as groundbreaking. The company forecasts that changes in pretraining research could arrive as soon as 2027 and anticipates new capabilities around 0.68–0.7. These are projections, not confirmed milestones or a schedule for future releases, amid broader discussion of AI and abundant intelligence.
Perplexity and Intelligence A shared measure?
OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.
What the metric measures
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.
A model learns from a large dataset.
Perplexity tracks predictive performance on held-out data.
Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.
OpenAI’s reported progression
The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.
| Reported level | OpenAI’s account |
|---|---|
| ≈0.99 | GPT-3.5; fluent conversation |
| ≈0.91 | GPT-4; productive chain-of-thought reasoning |
| Below 0.8 | Later models; mathematics and cybersecurity capabilities |
| ≈0.68–0.7 | Range where OpenAI anticipates new capabilities |
Why a shared measure could matter
Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.
- Track predictive performance over time
- Compare progress if developers use shared methods
- Inform discussion of common safety standards
- What a model can reliably do
- How safely it behaves
- How much oversight development requires
What remains unresolved
The supplied material does not include the underlying chart or enough detail to independently compare the measurements.
- No standardized reporting or audit process is specified.
- No formal industry standard, timetable, or commitments from other developers are announced.
- The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
If developers report measurements using comparable methods, perplexity could help researchers track changes across model generations, including when individual benchmarks are revised or no longer distinguish between systems. OpenAI argues that a common measure could support discussion of progress and the development of shared safety standards.
But a predictive-performance score cannot, on its own, show what a model can do in practice, whether it behaves safely, or what level of human oversight it needs. The release may contribute to a wider discussion, but the measurements described do not yet establish that developers can make reliable comparisons or that particular scores should trigger specific safety measures.
Products Worth Considering

Samsung 32 Odyssey G5 QHD Gaming Monitor

Sony WH-1000XM4 Wireless Noise Cancelling Headphones (Silver)

Bose QuietComfort Ultra Headphones Gen 1, Lunar Blue
As an affiliate, we earn on qualifying purchases.
Why OpenAI Tracks Pretraining Loss
OpenAI presents pretraining as a useful stage for tracking model development because later capabilities are shaped through additional methods such as prompting, fine-tuning and reinforcement learning. The company’s argument is that a measure of predictive performance may remain informative as downstream evaluations and benchmarks shift over time.
The post draws on the company’s history with models including GPT-2, GPT-3, GPT-3.5 and GPT-4, and associates improvements in pretraining measurements with capabilities observed across those generations. OpenAI also acknowledges that the capabilities associated with particular values were not predictable in advance. Its historical examples therefore motivate tracking the metric; they do not prove the same relationship will hold for future systems.
The company says it weighed the competitive cost of disclosing more about its position against the potential value of industry-wide safety measures. Its proposal comes as a suggestion for a shared scientific basis, not as an established industry process.
Limits of the Published Measurements
The available source material does not include the underlying chart, full dataset details or enough information for independent comparisons with other developers’ models. It also does not specify a standardized reporting process, independent audit arrangements, or how a particular score should affect safety decisions.
OpenAI’s examples do not show that a given perplexity value guarantees a capability, or that the same relationship will persist as training methods change. The timing and nature of future abilities, the possibility that AI systems will accelerate pretraining research, and the forecast of advances as soon as 2027 remain uncertain. The source also reports no formal industry standard or commitments from other developers.
Whether Other Developers Adopt It
The next development to watch is whether other AI organizations publish comparable measurements. For the figures to support cross-company comparisons, researchers would need agreement on datasets, evaluation methods and reporting practices; the supplied material gives no timetable for that work.
OpenAI frames its disclosure as a first step toward shared tracking and safety discussions. Whether that proposal becomes a practical standard, and what evidence might connect scores to capabilities and risks, remains open. For now, perplexity is best treated as one measure of predictive performance, not a complete account of a model’s abilities or safety.
Key Questions
What is perplexity in an AI model?
Perplexity measures next-token prediction on a held-out dataset. Lower values indicate better predictive performance on that dataset, but do not by themselves describe every capability or safety risk.
What measurements did OpenAI publish?
The source reports OpenAI figures of about 0.99 log-perplexity for GPT-3.5, about 0.91 for GPT-4, later losses below 0.8, and a model near 0.76 associated by the post author with results on open mathematics problems.
Does a lower score prove a model has a particular capability?
No. OpenAI links lower reported loss with capabilities seen in its models, but the material does not establish that a particular score reliably produces a particular ability.
Has the AI industry adopted perplexity as a safety standard?
No formal standard or commitments from other developers are described. The company presents the release as a possible first step toward shared measures, not an agreed industry process.
Are the projected advances by 2027 confirmed?
No. The source describes possible advances as OpenAI’s forecast. Their timing, nature and connection to AI-assisted research remain uncertain.
Primary source: OpenAI · via ThorstenMeyerAI.com
