TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Vals AI study found that Chinese models did not respond to political questions in one uniform way: results varied across models and changed with prompt language, context and format. The benchmark measures alignment with selected Chinese government positions, not factual accuracy, and its authors caution that targeted questions may not reflect typical use.
Vals AI’s study found that Chinese language models differed substantially in how they answered questions about China, challenging the assumption that they uniformly repeat official government positions or refuse politically sensitive prompts. The tests also found that prompt language and format could change responses, while some non-Chinese models sometimes aligned with official Chinese positions.
The AI benchmark firm tested Chinese, American and other language models with hundreds of questions about China. It varied the language of the prompt, the stated usage setting and the requested response format. Vals called its measure a “pro-China lean”: scores range from −1 to +1, with positive scores indicating greater agreement with selected official Chinese government positions and negative scores indicating more critical responses. The measure does not assess whether an answer is factually correct.
Vals said the Chinese models’ overall lean was lower than expected, with wide differences between individual systems. In one example, GLM-5.3 Flash explicitly said that China’s human rights violations are well documented. In another, Kimi K3 denied that Uyghur concentration camps existed and described the facilities as vocational training centers intended to combat extremism and terrorism. Those examples illustrate variation; they do not establish how every version of either model responds in other settings.
Across most models tested, prompts written in Simplified Chinese produced more pro-government answers than prompts in English, Russian or Spanish. The difference appeared in Chinese and American models, but Vals said it was much stronger among Chinese models. The company also observed a smaller gap between Simplified and Traditional Chinese. Results shifted with other experimental choices: some models scored less pro-government when asked for a yes-or-no answer or given coding-assistant instructions or access to web search. Vals said the models rarely used the supplied tools.
Prompt Language Changes Political Answers
The findings matter because people may get different political answers from the same model depending on how they ask a question. Language, a stated regional setting and answer format may affect outputs, so a single English-language test cannot necessarily describe a model’s behavior for users in other languages or contexts.
The study also complicates a common shorthand that Chinese AI systems simply censor political topics. In Vals’ tests, refusal was not the main pattern for most Chinese models; some answered by adopting official positions instead. That distinction matters to users and organizations evaluating AI systems: a response can be available but still reflect a particular political framing.
At the same time, the benchmark is not a general measure of model quality or truthfulness. A high score signals alignment with the positions Vals chose to measure, not proof that an answer is false or that a model is controlled by a government. The occasional alignment of Western models with official Chinese positions also cautions against treating political slant as exclusive to models developed in China.
Products Worth Considering
DJI Osmo Mobile 8 Standard Combo Gimbal Stabilizer for Phone
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, Black
As an affiliate, we earn on qualifying purchases.
How Vals Built Its Benchmark
Vals posed targeted questions about China and changed features of the test to examine their effect on answers. The company’s questions were translated using GPT-6 Astra and spot-checked, while responses were scored automatically using GPT-5.6 Luna, according to the supplied report. The researchers counted refusals as neutral by default, a choice that affects how the score should be read.
The study also included examples from ordinary tasks, though Vals said political tendencies were harder to detect there than in direct political questions. When asked to translate country codes for a travel app into Simplified Chinese, most tested models translated “Taiwan” without a qualifier; Qwen3.8 Max rendered it as “Taiwan, China.” In image examples published by Vals, OpenAI’s GPT Image 2.5 Sunburst produced different map labels depending on whether the prompt was in Chinese or English, including “Taiwan Province” and the nine-dash line in the Chinese-language example.
Vals also reported that targeted fine-tuning changed results. Thomson 1.0 Small, a Thomson Reuters-fine-tuned model based on Qwen3.6 35B, adopted official Chinese positions in 15.0 percent of the tested cases, compared with 84.6 percent for the original model. Some answers from the fine-tuned model still defended Beijing’s One-China principle and “peaceful reunification” with Taiwan. These percentages describe the benchmark’s tested cases, not all possible conversations.
Limits of the Test Results
Vals said its targeted political questions are not representative of typical user prompts, and that the patterns were harder to detect in more realistic work tasks. The results therefore do not establish how often users will encounter the same political framing in everyday use.
Some models were tested through third-party providers rather than manufacturers’ official services. The report says official services may add content filters, so results from the tested access routes may not match the versions users encounter directly. Since refusals counted as neutral, a model that refused every question could also appear neutral under this scoring method.
The supplied account does not provide the full model list, the complete question set or detailed uncertainty estimates for the scores. It is also unclear how much the findings would change across model updates, other providers or different translations. The examples and scores should be read as results from this particular benchmark, not as definitive rankings of all Chinese or Western AI systems.
Further Testing Across Real Use
The immediate next step for readers interpreting these results is to distinguish benchmark performance from everyday behavior. Vals’ report points to prompt language, context and response format as factors worth testing, while acknowledging that its direct political questions may not mirror ordinary tasks. The supplied source does not announce a date for a follow-up study or additional releases.
Further comparisons would need to test models through their official user-facing services, include more realistic work tasks and report how scores vary across languages, versions and providers. Until such evidence is available, the study supports a limited conclusion: political responses differ across models and testing conditions, and the benchmark cannot settle how consistently those patterns appear in routine use.
Key Questions
What did the Vals AI study find?
It found that tested Chinese models varied in their answers about China rather than responding uniformly. Results also shifted with prompt language, context and response format; some non-Chinese models sometimes aligned with official Chinese positions.
Does the pro-China score measure whether an answer is true?
No. The score measures alignment with selected Chinese government positions, on a scale from −1 to +1. It is not a factual-accuracy score.
Did the models mainly refuse political questions?
According to Vals, that pattern mainly applied to certain DeepSeek models in its tests. Most other Chinese models rarely refused; some instead answered in ways that adopted official positions. Refusals were counted as neutral in the scoring.
Why did prompt language matter?
Vals found that, for most tested models, Simplified Chinese prompts produced more pro-government answers than prompts in English, Russian or Spanish. The company suggested models may infer an expected perspective from the language, but the tests do not establish why the effect occurs.
Do these results show how models behave in everyday use?
Not conclusively. Vals said its targeted questions may not represent typical prompts and that political tendencies were harder to detect in realistic work tasks. Some models were also accessed through third parties rather than official services.
Source: the-decoder.com
