TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher on coding, statistics and prediction tasks than a group using Google and reference sites. After ChatGPT was removed, the researchers found no measurable difference in technical knowledge between the groups.
A randomized trial at Boston Consulting Group (BCG) found that consultants using ChatGPT scored higher on three data science tasks than colleagues using Google and reference websites, but showed no measurable knowledge gain when later tested without the chatbot. Published October 26 in Nature Human Behaviour, the study suggests that access to generative AI can extend workers’ performance while they use it, without necessarily teaching them to do the work independently.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The trial assigned consultants to use either ChatGPT or tools including Google and Stack Overflow. The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. Participants then completed two 90-minute tasks drawn from coding, statistics and prediction. The study was conducted in late March and early April 2024 and was preregistered before data collection, according to the authors.
Of 986 consultants allocated to the study, 573 began the survey and 487 submitted work on both tasks; those 487 formed the main analysis sample. The researchers reported that the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. The scores improved relative to the control group, but the consultants still averaged below 44 BCG data scientists who completed the tasks without AI.
After the work tasks, participants answered five technical questions without ChatGPT. The researchers found no statistically significant difference between the groups: reported differences ranged from −7.7 to +4.8 percentage points, and the joint test had a p-value of 0.531. The authors also reported an exploratory finding that ChatGPT users became more confident in the system’s ability to solve problems it could not actually solve.
Performance Gains Did Not Transfer
The findings matter to employers treating access to AI as a form of worker training. In this trial, ChatGPT helped consultants complete tasks beyond their existing expertise, but the follow-up test did not show that the experience left them better able to answer related questions on their own. That distinction affects how companies evaluate productivity gains, training programs and the quality of work produced with AI assistance.
The result does not show that AI cannot support learning in other settings. It shows that these participants, tasks, brief training and study period produced measurable task improvements while the tool was available, alongside no detectable improvement on the immediate unaided test. Employers may need to distinguish assisted performance from independent skill when assigning work or assessing whether staff have learned a new capability.
The confidence finding also bears on oversight. The authors describe it as exploratory, so it should not be treated as a definitive result. Still, greater confidence in a tool’s ability to handle problems it cannot solve could affect when users verify its output. The study’s setup itself underscores that issue: the tasks were designed to make ChatGPT useful while preventing it from correctly answering every question independently.
Products Worth Considering
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
Jackery Solar Generator 1000 v2 with Two 100W Solar Panels, 1024Wh LiFePO4 Battery
DJI Osmo Mobile 8 Phone Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How BCG Tested ChatGPT Use
BCG consultants took part in the trial in late March and early April 2024. The treatment group received a short lesson on prompting ChatGPT for technical problems. The control group received instruction in using Google and reference sites such as Stack Overflow and Khan Academy. Each participant worked on two of three tasks: Python data cleaning, statistical analysis, or predicting international football match outcomes.
BCG data scientists developed the tasks with OpenAI’s Economic Impacts Research team. According to the paper, the tasks were constructed so ChatGPT could provide useful guidance but could not correctly answer every question on its own. The study also included an expert benchmark: 44 BCG data scientists completed the tasks without AI, and their average provided the reference point for comparing consultants’ results.
The study is titled “Generative AI and the temporary upskilling of knowledge workers.” Its authors characterize the observed gains as temporary and compare AI assistance to an exoskeleton: it can extend a person’s capabilities while in use, but that does not establish that the person has acquired the underlying skill.
““find no evidence””
— The study authors
Limits of the Follow-Up Test
The follow-up consisted of five technical questions administered after the work tasks. The reported result is no statistically significant difference between the groups; it does not establish that ChatGPT use can never produce learning, or that no individual participant learned anything. The study material provided here does not establish whether longer training, repeated practice or later assessments would yield different results.
The main analysis covered 487 consultants who submitted both tasks, fewer than the 986 originally allocated. The supplied account does not specify how attrition affected the results. The confidence finding was exploratory, and the study was carried out at one consulting firm using GPT-4 and tasks built for the experiment. How well the findings generalize to other jobs, tools and workplace settings remains unclear.
Testing Longer-Term Skill Retention
The study was published in Nature Human Behaviour on October 26, 2026. The available source material does not identify a scheduled follow-up study or next research milestone. Further evidence would be needed to determine whether different training methods or sustained AI-assisted practice help workers retain skills after access to the tool ends.
For employers, the immediate question is how to check whether AI-supported work is accurate and whether staff can perform the relevant tasks independently. The BCG trial does not supply a general workplace policy, but it does show why a productivity result during AI use should not by itself be treated as evidence that employees have learned the task.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than a control group using Google and reference sites. They still averaged below 44 BCG data scientists who completed the tasks without AI.
Did ChatGPT improve consultants’ technical knowledge?
The researchers found no statistically significant difference between the ChatGPT and control groups on five technical questions answered after ChatGPT was removed. The test does not establish whether longer training or later practice would produce different results.
When was the trial conducted?
The trial took place in late March and early April 2024. The paper was published in Nature Human Behaviour on October 26, 2026.
How many consultants were included in the main analysis?
Of 986 consultants allocated to the study, 573 began the survey and 487 submitted both tasks. The authors used those 487 participants for the main analysis.
Does the study show that AI cannot teach workplace skills?
No. It found no measurable advantage on an immediate, five-question test without ChatGPT under the study’s conditions. It does not settle whether other training approaches, repeated practice or longer-term use can build skills.
Source: Thorsten Meyer AI
