AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Perplexity And Intelligence on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

OpenAI has published internal measurements of its models’ pretraining perplexity and argues that the metric could help track AI progress across developers. The company links lower loss to capabilities in its own models, but the figures do not establish that a particular score guarantees an ability, and no shared reporting standard has been announced.

OpenAI has published internal measurements of its models’ pretraining performance, arguing that perplexity could give AI developers a shared way to track progress and inform safety standards, as explored in the original analysis of perplexity and intelligence. The company connects falling scores with capabilities in its own models, while stressing that future abilities and the timing of further advances remain uncertain.

Perplexity measures how well a model predicts the next token in a held-out dataset; lower values indicate better predictions. OpenAI says it has tracked the measure over time, including for models beginning with GPT-3.5, and is now publishing some internal results. The company describes the figures as a way to follow underlying predictive performance even as task-specific benchmarks change or become less useful, a question also relevant to measuring AI intelligence gains.

In the figures described in the source, OpenAI reports log-perplexity of about 0.99 for GPT-3.5 and about 0.91 for GPT-4, and says later models have reached losses below 0.8. It associates these stages with fluent conversation, productive use of chain-of-thought reasoning, and mathematics and cybersecurity capabilities. Those connections are OpenAI’s account of its own models, not evidence that a specific score reliably produces a specific capability.

OpenAI also says a model at about 0.76 log-perplexity began producing results on open mathematics problems that the post describes as groundbreaking. The company forecasts that changes in pretraining research could arrive as soon as 2027 and anticipates new capabilities around 0.68–0.7. These are projections, not confirmed milestones or a schedule for future releases, amid broader discussion of AI and abundant intelligence.

At a glance
reportWhen: Published recently; the source does not…
The developmentOpenAI has made some internal pretraining measurements public and proposed perplexity as a possible shared measure for tracking AI progress and informing safety discussions.
AI progress · Briefing

Perplexity and Intelligence A shared measure?

OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.

≈0.76
Log-perplexity linked to reported open math results
≈0.99
Reported for GPT-3.5
≈0.91
Reported for GPT-4
2027
Possible timing of faster research progress

What the metric measures

Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.

1
Pretrain

A model learns from a large dataset.

2
Measure

Perplexity tracks predictive performance on held-out data.

3
Shape capabilities

Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.

OpenAI’s reported progression

The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.

Reported levelOpenAI’s account
≈0.99GPT-3.5; fluent conversation
≈0.91GPT-4; productive chain-of-thought reasoning
Below 0.8Later models; mathematics and cybersecurity capabilities
≈0.68–0.7Range where OpenAI anticipates new capabilities

Why a shared measure could matter

Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.

Potential value
  • Track predictive performance over time
  • Compare progress if developers use shared methods
  • Inform discussion of common safety standards
What it cannot show alone
  • What a model can reliably do
  • How safely it behaves
  • How much oversight development requires

What remains unresolved

The supplied material does not include the underlying chart or enough detail to independently compare the measurements.

  • No standardized reporting or audit process is specified.
  • No formal industry standard, timetable, or commitments from other developers are announced.
  • The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
WatchMeaningful comparisons depend on shared datasets, reporting methods, independent scrutiny, and evidence connecting the metric to capabilities and risks.
The takeaway
Perplexity may be a useful signal of pretraining progress, but it cannot yet support reliable cross-developer comparisons or settle questions about safety and future abilities.

A Shared Yardstick for AI Progress

If developers report measurements using comparable methods, perplexity could help researchers track changes across model generations, including when individual benchmarks are revised or no longer distinguish between systems. OpenAI argues that a common measure could support discussion of progress and the development of shared safety standards.

But a predictive-performance score cannot, on its own, show what a model can do in practice, whether it behaves safely, or what level of human oversight it needs. The release may contribute to a wider discussion, but the measurements described do not yet establish that developers can make reliable comparisons or that particular scores should trigger specific safety measures.

Products Worth Considering

As an affiliate, we earn on qualifying purchases.

Why OpenAI Tracks Pretraining Loss

OpenAI presents pretraining as a useful stage for tracking model development because later capabilities are shaped through additional methods such as prompting, fine-tuning and reinforcement learning. The company’s argument is that a measure of predictive performance may remain informative as downstream evaluations and benchmarks shift over time.

The post draws on the company’s history with models including GPT-2, GPT-3, GPT-3.5 and GPT-4, and associates improvements in pretraining measurements with capabilities observed across those generations. OpenAI also acknowledges that the capabilities associated with particular values were not predictable in advance. Its historical examples therefore motivate tracking the metric; they do not prove the same relationship will hold for future systems.

The company says it weighed the competitive cost of disclosing more about its position against the potential value of industry-wide safety measures. Its proposal comes as a suggestion for a shared scientific basis, not as an established industry process.

Limits of the Published Measurements

The available source material does not include the underlying chart, full dataset details or enough information for independent comparisons with other developers’ models. It also does not specify a standardized reporting process, independent audit arrangements, or how a particular score should affect safety decisions.

OpenAI’s examples do not show that a given perplexity value guarantees a capability, or that the same relationship will persist as training methods change. The timing and nature of future abilities, the possibility that AI systems will accelerate pretraining research, and the forecast of advances as soon as 2027 remain uncertain. The source also reports no formal industry standard or commitments from other developers.

Whether Other Developers Adopt It

The next development to watch is whether other AI organizations publish comparable measurements. For the figures to support cross-company comparisons, researchers would need agreement on datasets, evaluation methods and reporting practices; the supplied material gives no timetable for that work.

OpenAI frames its disclosure as a first step toward shared tracking and safety discussions. Whether that proposal becomes a practical standard, and what evidence might connect scores to capabilities and risks, remains open. For now, perplexity is best treated as one measure of predictive performance, not a complete account of a model’s abilities or safety.

Key Questions

What is perplexity in an AI model?

Perplexity measures next-token prediction on a held-out dataset. Lower values indicate better predictive performance on that dataset, but do not by themselves describe every capability or safety risk.

What measurements did OpenAI publish?

The source reports OpenAI figures of about 0.99 log-perplexity for GPT-3.5, about 0.91 for GPT-4, later losses below 0.8, and a model near 0.76 associated by the post author with results on open mathematics problems.

Does a lower score prove a model has a particular capability?

No. OpenAI links lower reported loss with capabilities seen in its models, but the material does not establish that a particular score reliably produces a particular ability.

Has the AI industry adopted perplexity as a safety standard?

No formal standard or commitments from other developers are described. The company presents the release as a possible first step toward shared measures, not an agreed industry process.

Are the projected advances by 2027 confirmed?

No. The source describes possible advances as OpenAI’s forecast. Their timing, nature and connection to AI-assisted research remain uncertain.

Primary source: OpenAI · via ThorstenMeyerAI.com

You May Also Like

SpaceX handed lowest possible ESG rating by MSCI

SpaceX has been assigned the lowest possible ESG rating by MSCI, raising questions about its sustainability and governance practices.

Marketing First, Waste Second: How Grocery AI Finds Near‑Term Lift

AIThis post was created with the assistance of artificial intelligence (AI). TL;DR…

AI in Newsrooms: How Algorithms Are Augmenting Journalism

Navigating the evolving landscape of AI in newsrooms reveals how algorithms are transforming journalism—discover the impact and future implications.

Choosing A Postpartum Recovery Guide After Birth

An IdeaNavigator AI concept proposes tailoring postpartum recovery guidance to delivery details, with a two-hundred-mother pilot to test use through week six.