ChatGPT helps, and concerns, business advisors, study findings
Last spring, when Karim Lakhani began testing how ChatGPT affected the work of specialized business consultants, he thought they would be pleased with the tool. In a pilot study of two dozen workers, the language bot helped them complete two-hour tasks in 20 minutes.
“I assumed they would think like I did, 'Great! I can do so much more!'” Harvard Business School Professor Dr. Lakhani said.
Instead, there was a sense of unease among the advisors. He appreciated that he had done better work in less time. But ChatGPT's accelerated work threatened their sense of belonging as high-skilled workers, and some feared relying on it too much. “They were really concerned and they felt like it would stigmatize them and be kind of an empty calorie for their brain,” Dr. Lakhani said.
Following these initial trials, Dr. Lakhani and his colleagues designed a larger, controlled experiment to measure how ChatGPT would affect more than 750 white-collar workers. He Study, which is under review in a scientific journal, has indicated sharply mixed results in the work product of consultants. ChatGPT significantly improved the speed and quality of work on brainstorming tasks, but it led many consultants astray when performing more analytical tasks.
The study also details workers' varied feelings about the equipment. One participant compared it to the fire that Prometheus stole from the gods to help humans. Another told Dr. Lakhani's colleague Fabrizio Dell'Acqua that ChatGPT feels like junk food – hard to resist, easy to consume but ultimately bad for the consumer.
In the near future, language bots like OpenAI's ChatGPT, Meta's Llama, and Google's Gemini are expected to perform many white-collar tasks, e.g. Copywriting, drafting legal briefs and drafting recommendation letters. This study is one of the first to show how technology can impact actual office work – and office workers.
“This is a well-designed study, especially in such an emerging field,” said Maryam Alawi, a professor in the Scheller College of Business at the Georgia Institute of Technology, who was not involved in the experiments. Dr. Allawi, who has studied the impact of new digital technologies on workers and organizations, also said the study “really shows how much more we need to learn.”
The study recruited management consultants from the Boston Consulting Group, one of the world's largest management-consulting firms. The company had barred its consultants from using AI bots in their work.
“We wanted it to involve a large group of real workers working on real tasks,” said François Candelon, managing director of the company who helped design the experiments.
The volunteers were divided into two groups, each of which worked on a different management-consulting problem. In each group, some advisors used ChatGPT after the 30-minute training, some used it without any instruction, and some did not use it.
One of the tasks was to brainstorm about a new type of shoe, create an inspiring business plan to make it, and write persuasively about it. Some researchers believed that only humans could perform such creative work.
they were wrong. Consultants using ChatGPT produced work that was rated about 40 percent better on average by independent evaluators. In fact, people who simply cut and pasted ChatGPT's output were rated more highly than colleagues who mixed its work with their own ideas. And AI-assisted advisors were more than 20 percent faster.
This year's study of ChatGPT legal analysis And white collar writing jobs It has been found that bots help low-performing people more than the most efficient ones. Dr. Lakhani and his colleagues found the same effect in their study.
However, in a task that required evidence-based reasoning, ChatGPT was not helpful at all. In this group, volunteers were asked to advise a corporation that was invented for the study. They needed to interpret data from spreadsheets and link it to fake transcripts of interviews with officers.
Here, ChatGPT provoked employees to rely on it too much. Unaided humans had the correct answer 85 percent of the time. Those who used ChatGPT without training scored over 70 percent. Those who had been trained fared even worse, getting the answer only 60 percent of the time.
In interviews conducted after the experiment, “people told us they neglected to check because it's so polished, it looks so perfect,” said Hila Lifshitz-Assaf, a management professor at Warwick Business School in Britain.
Many consultants said ChatGPT made them uneasy about how the tool would change their profession and even their understanding of themselves. Nearly three out of four participants told researchers they worried that using ChatGPT would weaken their own creative muscles, said Mr. Candelon of the Boston Consulting Group.
“If you don't have an existential crisis about this tool, you haven't used it very much yet,” said another co-author, Ethan Mollick, a management professor at the Wharton School of the University of Pennsylvania.