Tracking the Explosive World of Generative AI

Stanford/MIT Study: GPT Boosts Support Agent Productivity by up to 35%

Researchers from Stanford and MIT reveal the substantial effects of AI on customer service agents, increasing productivity and closing skill gaps, which may bring far-reaching consequences for the labor landscape.

A Stanford & MIT study has shown vast increases in customer support agent productivity from the deployment of AI software. Photo illustration: Artisana

🧠 Stay Ahead of the Curve

  • Stanford and MIT researchers reveal AI's significant impact on customer service agents' productivity in a groundbreaking study.

  • The study demonstrates AI's ability to reduce skill gaps, particularly benefiting low-skill agents and new hires in customer support.

  • As AI raises performance standards, the broader labor market may face changes in job requirements, compensation, and role availability.

By Michael Zhang

April 26, 2023

A new study by researchers from Stanford and MIT presents a first-of-its-kind analysis of the effects of large language model (LLM)-powered software on over 5,000 customer service agents at a Fortune 500 company. The software, which was powered by OpenAI’s GPT-3, delivered a significant impact that may foreshadow accelerated shifts in the labor landscape driven by generative AI's rapid influence.

A Pioneering Wide-Scale, Long-Term Analysis

The study examined more than 3 million chat sessions conducted by 5,179 agents over a year-long period encompassing both pre- and post-AI deployment. Its uniqueness stems from its focus on real-world conditions, rather than controlled lab tests, and its consideration of AI's impact across an extended timeframe on thousands of real workers.

Although the specific software remains unnamed, the researchers disclosed that it was based on OpenAI’s GPT-3 and fine-tuned on customer service interactions. The agents, primarily located overseas, worked for an undisclosed Fortune 500 company specializing in business process software.

Boosting Productivity for New and Low-Skill Agents

The researchers observed a 13.8% increase in the number of chats agents successfully resolved per hour. Specifically, the gains the researchers witnessed were:

  • A decline in the time it takes an agent to handle an individual chat.

  • An increase in the number of chats that an agent is able to handle per hour.

  • A small increase in the share of chats that are successfully resolved.

Raw productivity improvements were observed across the board after deployment of AI software. Credit: NBER

The improvement was particularly notable among lower-performing agents, who experienced a 35% increase in successful resolutions per hour. This substantial reduction in skill gaps allowed less-skilled workers to perform nearly as well as their highest-skilled counterparts, the researchers found.

Lower-skilled agents rapidly converged towards the skill level of the most talent agents after the deployment of AI software. Credit: NBER

Conversely, higher-skilled agents saw little to no increase in productivity. In some cases, they even experienced a decrease, which researchers hypothesized was due to the AI tool's distracting nature.

Notably, new agents also experienced significant performance improvements. Agents with two months of tenure, aided by AI, performed on par with agents who had been working for over six months without AI assistance.

Implications for Knowledge Worker Industries

Generative AI's rapid emergence is already transforming various industries and the work professionals do within them. We have reported on the declining job opportunities for creative artists in China as AI image generation gains widespread adoption in the gaming industry. The impact of generative AI on other roles is just beginning to surface.

For customer service agents, increased retention rates and better performance among lower-skilled agents are clear outcomes of generative AI implementation. These improvements could lead to cost savings for companies as they benefit from higher performance levels and longer employee tenures.

However, the elevation of the lowest performers to the level of the highest performers may have other consequences for the broader labor market in customer support. As AI improves the quality of the least-skilled agents, job requirements and compensation may change as companies no longer need to compete for top talent.

The researchers highlight the importance of considering compensation for workers who contribute data to AI systems. “Our findings raise questions about whether and how workers should be compensated for the data that they provide to AI systems,” the team writes, “High-skill workers, in particular, play an important role in model development but see smaller direct benefits in terms of improving their own productivity.”

As generative AI continues to make a splash in numerous roles, overall efficiency gains may result in a reduction of available support agent roles. Notably, the software examined in the study was based on GPT-3. With the rapid advancements in LLMs, even more capable software utilizing OpenAI's recently released GPT-4 could have profound consequences, including a further increase in support agent efficiency.

Given the early stage of generative AI, the researchers agree: numerous consequences “deserve further scrutiny.”

Read More: ChatGPT