The Best AI Configuration I Tested Isn’t Running In My News Pipeline

🔍 Read the full analysis: The Best AI Configuration I Tested Isn’t Running In My News Pipeline on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a 71-configuration test of AI news-writing models found GPT-6 Astra produced the highest-scoring articles, while the much cheaper GPT-6 Luna remained his production choice. The results, published October 13 on ThorstenMeyerAI.com, also prompted him to remove two fallback models after they produced invented quotes or output his pipeline could not use.

Meyer tested configurations from September 8 to October 13, 2026, using the production prompt and real news sources. The test included API, subscription, free-tier and locally run open-weight models. Each source was turned into both an English and a German article. Two AI judges rated the work on a five-point scale, considering source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, as well as German language quality.

In the initial test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a perfect source-fidelity score. Across a further 12-article robustness run, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reported a cost of about $0.12 per article and an average generation time of around 50 seconds.

GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71 and valid output for every article in that run. Meyer reported its per-article cost at roughly $0.0012 to $0.0013 and typical generation time at 17 seconds. His report says he chose Luna for production on cost grounds, rather than selecting the highest-scoring model.

Why Cost Changed the Model Choice

The results illustrate a practical distinction between the best-scoring model and the best fit for a production workflow. Astra’s reported average was higher, but its roughly $0.12 article cost was about 100 times Luna’s. For a high-volume publisher, that difference can outweigh a modest score advantage, provided the lower-cost option meets the publisher’s quality and reliability requirements.

The test also points to a less visible operational risk: fallback models can affect accuracy when the primary model is unavailable. Meyer says one backup received the lowest possible source-fidelity score and generated quotes attributed to unnamed sources; another produced no valid articles in four attempts. Removing them could reduce the risk of fabricated material entering the pipeline, though the report does not provide a longer-term audit of the revised system.

For readers, the broader takeaway is that public model rankings may not predict performance on a specific editorial task. Meyer’s results are a single publisher’s evaluation, not evidence that the same configuration will suit every newsroom, language pair or source mix.

How the News-Writing Test Worked

Meyer said he designed the evaluation to resemble his own workflow: models received the same production prompt and were asked to turn source material into finished articles. The two test sources differed in length—a 6,340-character product filing and a 619-character political news item. He said shorter sources are common in his pipeline, making the thin-source task especially relevant to his use case.

Two AI systems, GPT-6.1 Sol and Claude Opus 5.5, served as judges and rated each article against the original source. Meyer also ran the strongest configurations on 10 additional real sources, mostly short ones. He reported 95% bootstrap intervals and said small score differences in the initial two-article test should be treated as noise; he considered the larger follow-up run a more reliable basis for ranking.

The report says increased reasoning effort did not deliver a dependable improvement for GPT-6 Luna. Across six effort settings, the high setting had a small apparent score edge, but Meyer said it fell within the robustness-run intervals. At the maximum setting, the reported cost and generation time were each about 12 times higher than at the production “none” setting, without a corresponding score gain.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Reported Results

The findings come from one publisher’s test and workflow, and the report does not establish how the models would perform on a broader, independently selected set of newsrooms, topics or languages. The main comparison began with only two sources; although Meyer added a 10-article robustness run for stronger configurations, the available description does not provide the full article set or detailed model-by-model uncertainty intervals for every result.

The ratings were made by two AI judges, not by a disclosed panel of human editors. The report describes the criteria and names the judges, but it does not establish whether human assessments would produce the same ordering. It also does not provide a longer-term comparison of error rates after the fallback models were removed. Independent replication and sustained production results remain unknown.

How the Pipeline Will Be Evaluated

Meyer says GPT-6 Luna now runs as the production model, following the October 7 removal of GPT-4.1 nano and gpt-oss-120b from the news route. The report does not set out a scheduled next benchmark or state whether the pipeline has a replacement fallback. Continued monitoring would be needed to establish whether Luna’s test performance holds across routine publishing and whether removing the backups changes reliability during failures.

The next useful evidence would include results from a larger, ongoing set of articles, with disclosed source types, validity rates and human editorial review. Until then, the published scores describe Meyer’s measured workflow, while their applicability to other publishers remains a question for testing.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the reported robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer reported that Luna cost about $0.0012 to $0.0013 per article, compared with about $0.12 for Astra, while still producing consistently strong results in his tests.

What happened to the fallback models?

Meyer says he removed GPT-4.1 nano and gpt-oss-120b from the news route on October 7. The former produced invented quotes in the test; the latter produced no valid articles in four attempts.

Did higher reasoning effort improve Luna’s articles?

Not reliably in this evaluation, according to Meyer. The highest setting took and cost about 12 times as much as the no-effort setting, while the reported scores showed no dependable gain.

Do these results show which model every newsroom should use?

No. The test reflects one publisher’s prompts, sources, languages and editorial criteria. Other newsrooms would need to evaluate models on their own tasks and standards.

Source: ThorstenMeyerAI.com