M.B.A. Students vs. ChatGPT: Who Comes Up With More Innovative Ideas?

How good is AI in generating new ideas?

The conventional wisdom has been not very good. Identifying opportunities for new ventures, generating a solution for an unmet need, or naming a new company are unstructured tasks that seem ill-suited for algorithms. Yet recent advances in AI, and specifically the advent of large language models like ChatGPT, are challenging these assumptions.

We have taught innovation, entrepreneurship and product design for many years. For the first assignment in our innovation courses at the Wharton School, we ask students to generate a dozen or so ideas for a new product or service. As a result, we have heard several thousand new venture ideas pitched by undergraduate students, M.B.A. students and seasoned executives. Some of these ideas are awesome, some are awful, and, as you would expect, most are somewhere in the middle.

The library of ideas, though, allowed us to set up a simple competition to judge who is better at generating innovative ideas: the human or the machine.

In this competition, which we ran together with our colleagues Lennart Meincke and Karan Girotra, humanity was represented by a pool of 200 randomly selected ideas from our Wharton students. The machines were represented by ChatGPT4, which we instructed to generate 100 ideas with otherwise identical instructions as given to the students: “generate an idea for a new product or service appealing to college students that could be made available for $50 or less.”

In addition to this vanilla prompt, we also asked ChatGPT for another 100 ideas after providing a handful of examples of successful ideas from past courses (in other words, a trained GPT group), providing us with a total sample of 400 ideas.

Collapsible laundry hamper, dorm-room chef kit, ergonomic cushion for hard classroom seats, and hundreds more ideas miraculously spewed from a laptop.

How to compare

The academic literature on ideation postulates three dimensions of creative performance: the quantity of ideas, the average quality of ideas, and the number of truly exceptional ideas.

First, on the number of ideas per unit of time: Not surprisingly, ChatGPT easily outperforms us humans on that dimension. Generating 200 ideas the old-fashioned way requires days of human work, while ChatGPT can spit out 200 ideas with about an hour of supervision.

Next, to assess the quality of the ideas, we market tested them. Specifically, we took each of the 400 ideas and put them in front of a survey panel of customers in the target market via an online purchase-intent survey. The question we asked was: “How likely would you be to purchase based on this concept if it were available to you?” The possible responses ranged from definitely wouldn’t purchase to definitely would purchase.

The responses can be translated into a purchase probability using simple market-research techniques. The average purchase probability of a human-generated idea was 40%, that of vanilla GPT-4 was 47%, and that of GPT-4 seeded with good ideas was 49%. In short, ChatGPT isn’t only faster but also on average better at idea generation.

Still, when you’re looking for great ideas, averages can be misleading. In innovation, it’s the exceptional ideas that matter: Most managers would prefer one idea that is brilliant and nine ideas that are flops over 10 decent ideas, even if the average quality of the latter option might be higher. To capture this perspective, we investigated only the subset of the best ideas in our pool—specifically the top 10%. Of these 40 ideas, five were generated by students and 35 were created by ChatGPT (15 from the vanilla ChatGPT set and 20 from the pre trained ChatGPT set). Once again, ChatGPT came out on top.

What it means

We believe that the 35-to-5 victory of the machine in generating exceptional ideas (not to mention the dramatically lower production costs) has substantial implications for how we think about creativity and innovation.

First, generative AI has brought a new source of ideas to the world. Not using this source would be a sin. It doesn’t matter if you are working on a pitch for your local business-plan competition or if you are seeking a cure for cancer—every innovator should develop the habit of complementing his or her own ideas with the ones created by technology. Ideation will always have an element of randomness to it, and so we cannot guarantee that your idea will get an A+, but there is no excuse left if you get a C.

Second, the bottleneck for the early phases of the innovation process in organisations now shifts from generating ideas to evaluating ideas. Using a large language model, an innovator can produce a spreadsheet articulating hundreds of ideas, which likely include a few blockbusters. This abundance then demands an effective selection mechanism to find the needles in the haystack.

To date, these models appear to perform no better than any single expert in their ability to predict commercial viability. Using a sample of a dozen or so independent evaluations from potential customers in the target market—a wisdom of crowds approach—remains the best strategy. Fortunately, screening ideas using a purchase intent survey of customers in the target market is relatively fast and cheap.

Finally, rather than thinking about a competition between humans and machines, we should find a way in which the two work together. This approach in which AI takes on the role of a co-pilot has already emerged in software development. For example, our human (pilot) innovator might identify an open problem. The AI (co-pilot) might then report what is known about the problem, followed by an effort in which the human and AI independently explore possible solutions, virtually guaranteeing a thorough consideration of opportunities.

The human decision maker is likely ultimately responsible for the outcome, and so will likely make the screening and selection decisions, informed by customer research and possibly by the opinion of the AI co-pilot. We predict such a human-machine collaboration will deliver better products and services to the market, and improved solutions for whatever society needs in the future.

Christian Terwiesch and Karl Ulrich are professors of operations, information and decisions at the Wharton School of the University of Pennsylvania, where Terwiesch also co-directs the Mack Institute for Innovation Management.

Berkshire began liquidating its big stake in the banking company in mid-July—and has already unloaded about 15% of its interest. The selling has been fairly aggressive and has totaled about $6 billion. (Berkshire still holds 883 million shares, an 11.3% interest worth $35 billion based on its most recent filing on Aug. 30.)

The selling has prompted speculation about when CEO Warren Buffett, who oversees Berkshire’s $300 billion equity portfolio, will stop. The sales have depressed Bank of America stock, which has underperformed peers since Berkshire began its sell program. The stock closed down 0.9% Thursday at $40.14.

It’s possible that Berkshire will stop selling when the stake drops to 700 million shares. Taxes and history would be the reasons why.

Berkshire accumulated its Bank of America stake in two stages—and at vastly different prices. Berkshire’s initial stake came in 2017 , when it swapped $5 billion of Bank of America preferred stock for 700 million shares of common stock via warrants it received as part of the original preferred investment in 2011.

Berkshire got a sweet deal in that 2011 transaction. At the time, Bank of America was looking for a Buffett imprimatur—and the bank’s stock price was weak and under $10 a share.

Berkshire paid about $7 a share for that initial stake of 700 million common shares. The rest of the Berkshire stake, more than 300 million shares, was mostly purchased in 2018 at around $30 a share.

With Bank of America stock currently trading around $40, Berkshire faces a high tax burden from selling shares from the original stake of 700 million shares, given the low cost basis, and a much lighter tax hit from unloading the rest. Berkshire is subject to corporate taxes—an estimated 25% including local taxes—on gains on any sales of stock. The tax bite is stark.

Berkshire might own $2 to $3 a share in taxes on sales of high-cost stock and $8 a share on low-cost stock purchased for $7 a share.

New York tax expert Robert Willens says corporations, like individuals, can specify the particular lots when they sell stock with multiple cost levels.

“If stock is held in the custody of a broker, an adequate identification is made if the taxpayer specifies to the broker having custody of the stock the particular stock to be sold and, within a reasonable time thereafter, confirmation of such specification is set forth in a written document from the broker,” Willens told Barron’s in an email.

He assumes that Berkshire will identify the high-cost Bank of America stock for the recent sales to minimize its tax liability.

If sellers don’t specify, they generally are subject to “first in, first out,” or FIFO, accounting, meaning that the stock bought first would be subject to any tax on gains.

Buffett tends to be tax-averse—and that may prompt him to keep the original stake of 700 million shares. He could also mull any loyalty he may feel toward Bank of America CEO Brian Moynihan , whom Buffett has praised in the past.

Another reason for Berkshire to hold Bank of America is that it’s the company’s only big equity holding among traditional banks after selling shares of U.S. Bancorp , Bank of New York Mellon , JPMorgan Chase , and Wells Fargo in recent years.

Buffett, however, often eliminates stock holdings after he begins selling them down, as he did with the other bank stocks. Berkshire does retain a smaller stake of about $3 billion in Citigroup.

There could be a new filing on sales of Bank of America stock by Berkshire on Thursday evening. It has been three business days since the last one.

Berkshire must file within two business days of any sales of Bank of America stock since it owns more than 10%. The conglomerate will need to get its stake under about 777 million shares, about 100 million below the current level, before it can avoid the two-day filing rule.

It should be said that taxes haven’t deterred Buffett from selling over half of Berkshire’s stake in Apple this year—an estimated $85 billion or more of stock. Barron’s has estimated that Berkshire may owe $15 billion on the bulk of the sales that occurred in the second quarter.

Berkshire now holds 400 million shares of Apple and Barron’s has argued that Buffett may be finished reducing the Apple stake at that round number, which is the same number of shares that Berkshire has held in Coca-Cola for more than two decades.

Buffett may like round numbers—and 700 million could be just the right figure for Bank of America.

M.B.A. Students vs. ChatGPT: Who Comes Up With More Innovative Ideas?

We put humans and AI to the test. The results weren’t even close.

How to compare

What it means

When will Berkshire Hathaway stop selling Bank of America stock?