Since its recent launch, ChatGPT, a large language model AI, has caused quite a stir in the technology industry and the broader community at large. Its capabilities have sparked both awe and alarm – an AI that is so good at what it does that it has real potential of displacing people at work. But how worried should we be?
Let’s first get the obvious stuff out of the way. ChatGPT is an incredible tool and exceptionally good at some tasks like producing code, compiling essays and even writing poetry. However, when it comes to creative thinking, it misses the mark.
To understand if it can produce creative ideas, we asked ChatGPT a few brain teasers that we routinely use in our workshops, even with early elementary age kids. They are typically used as 5-minute warm-up group activities that help to illustrate the underlying cognitive mechanism.
At a fundamental level, we know that AI tools have limited creativity because AI relies on using massive amounts of prior data to learn. But creativity requires you to think of novel ideas that don’t yet exist and therefore don’t have any data to learn from. Nevertheless, it’s still possible to generate somewhat of a creative output in some areas like art as we have already seen with tools like Midjourney. So, let’s take a deep dive at how ChatGPT fared with these exercises.
Associative Thinking
To test ChatGPT’s ability for associative thinking, we tried the Wacky Inventions brain teaser. The idea behind this brain teaser is to use our brain’s natural associative processes to combine unrelated objects and make a new invention. One can argue that this associative ability is perhaps the most fundamental aspect of creativity. A real world example of associative thinking is Google’s PageRank algorithm that completely transformed what search engines could do. It combined the concept of academic citations (articles that are cited more indicate more authoritative sources) with websites – two completely unrelated domains – to create an automated way to assess the quality of a website.
We posed random combinations of objects that we have encountered in our sessions to make it easier to compare ChatGPT to human performance. Here are two examples:
Example 1
Q: What new invention idea can you come up with using a table and a scarf?
A: A portable table with a built-in scarf storage system. The table has a small drawer or compartment on the side that is specifically designed to hold scarves. The scarves can be easily rolled up and stored in the compartment, allowing users to keep their scarves organized and easily accessible. The table also has a built-in hook or hanger on the side, allowing users to hang their scarves on the table for easy access and display. This invention would be especially useful for people who have a large collection of scarves and need a convenient way to store and access them. It would also be useful for people who need a portable and versatile table for various activities, such as crafting, sewing, or organizing.
Q: That’s a good idea! Can you find a way to use an attribute of the scarf to make a new kind of table?
A: A table with a built-in scarf-covered surface. The table has a removable and interchangeable scarf that covers the top surface of the table. The scarf can be easily swapped out for different styles and colors, allowing users to customize the look of their table. The scarf can also be used as a protective layer for the table, preventing scratches or damage to the surface. This table would be especially useful for people who want a versatile and customizable piece of furniture for their home. It would also be useful for people who want a protective and stylish table for activities such as crafting or sewing. Additionally, the scarf-covered surface would add a cozy and comfortable feel to the table, making it a great addition to any room.
Q: That’s great! What property of the scarf are you using in your invention idea?
A: The property of the scarf that I am using in my invention idea is its ability to be easily removed and swapped out for different styles and colors. This property allows users to customize the look of the table and create a unique and personalized piece of furniture. It also allows users to easily update the look of the table by swapping out the scarf for a different color or pattern. Additionally, the property of the scarf being removable and interchangeable allows for easy cleaning and maintenance of the table surface.
Example 2
Q: What new invention idea can you come up with using a football and a water bottle?
A: A portable football water dispenser that attaches to the top of a water bottle. It has a small spout on the side that allows players to easily hydrate during a game without having to stop and open the water bottle. The dispenser is designed to fit snugly on the top of the bottle, with a secure fastening mechanism to prevent it from falling off during use. It also has a built-in straw that allows players to easily drink from the bottle without removing it from the dispenser. This invention would be especially useful for athletes who need to stay hydrated during long and intense games.
Q: So is this a water bottle with a football shaped dispenser on top?
A: An error occurred. If this issue persists please contact us through our help center at help.openai.com.
Example 3
Q: What new invention idea can you come up with using an umbrella and a jump rope?
A: An error occurred. If this issue persists please contact us through our help center at help.openai.com.
We gave a few other combinations and found that a common theme in ChatGPT’s responses, purely from an outcome perspective, is that it combines the two objects physically – attaching the two together for increased functionality. This, in general, is a reflection of a low level of creativity similar to very young children. However, as children get better at abstract thinking, they naturally start using the objects as starting points and zero in on a couple attributes that they can seamlessly integrate in the invention. Explicitly nudging children to use an attribute of the object is also effective in helping them to get more sophisticated solutions, and in general improves their abstract thinking skills. Unfortunately, ChatGPT wasn’t able to find the right property to work with as was evident in the first example. “Removability” isn’t really a property of the scarf. What it did instead is used instances on the internet where the two object names appear together (you can google “table scarf” to see some results) and cobbled together a reasonable sounding paragraph. But it doesn’t really understand what it created. In the second example, when asked a clarifying question it ends up bailing out with an error message.
The problem is that ChatGPT isn’t really equipped to do abstract thinking. Think about how humans build a model of the world around them. Babies and toddlers spend hours interacting with objects around them – they touch them, taste them (much to our frustration), hit them, shake them and toss them. These adorable, uncoordinated actions serve a very useful function – they help children create an internal representation of the object. This step is crucial for their development because without building these internal models, children have a harder time imagining and creating new combinations in their heads as they grow older. It’s the primary reason why play is essential for long-term creativity.
But that’s not all. We don’t just build standalone models but also much more extensive associative networks where different concepts are connected to each other through different properties. For example, the concept of “strawberry” in your brain might be connected to “red” by a thing-to-property link or connected to a “rash” by a cause-effect link if you are allergic to strawberries. Every new thing a baby learns is continually being mapped into this giant associative network of different objects, concepts and ideas. This extensive and rich associative network is what allows us to make abstractions, hop from one concept to another, and make novel connections.
We use this rich implicit knowledge built over many years extensively when we come up with novel ideas. Even when we see new ideas from others, we can quickly adapt them to our situation, because of this underlying layer of implicit knowledge that we don’t pay conscious attention to. The challenge with scouring the internet is that a lot of this implicit knowledge doesn’t exist there. On the internet it’s assumed that when someone talks about a scarf people know what that is – you won’t necessarily find places where attributes of a scarf are listed that allow it to be connected to other concepts that are further away.
This is of course a logical model and not a physical one inside our brains. We know very little about how association takes place inside our heads but the logical model gives a good approximation. Most creativity techniques leverage this and simply provide triggers to the brain (where the real magic happens) to get new ideas in a somewhat reliable fashion.
In the third example, the associative distance between the two objects (umbrella, jumprope) is high meaning that you typically don’t see them mentioned together, so ChatGPT simply bails out again. In contrast, using the same two objects, a group of 4th-5th graders came up with the idea of an umbrella shaped water ski where you can sit and relax, while being pulled by a rope.
Additional Creative Thinking Triggers
We also tried a few other kinds of brainteasers like Opposite Day and Twist-a-Story to see how ChatGPT handles them. The idea behind Opposite Day is to reverse a well known assumption which can lead to radically new insights. For example, the designers of the portable toddler chair that attaches to any table, came up with that idea by asking themselves “What if a chair didn’t have any legs and why would that be useful?”. In Twist-a-story, the idea is to find an alternate ending to an existing story by changing some element of the story.
Here is how ChatGPT answered these brain teasers.
Example 4
Q: What if libraries didn’t carry books? What would they have instead and why would it be useful?
A: If libraries didn’t carry books, they would likely carry a wider variety of media such as e-books, audiobooks, and digital media. This would be useful because it would allow users to access a wider range of content, including both fiction and non-fiction, in a variety of formats. It would also allow libraries to provide access to more up-to-date and current content, as e-books and digital media are often released more quickly than traditional print books. Additionally, it would save space and resources, as digital media can be stored and accessed more efficiently than physical books.
Example 5
Q: Have you heard the three little pigs story?
A: Yes, I have heard the story of the three little pigs. It is a popular fairytale about three pigs who each build a house to protect themselves from a wolf. The first pig builds his house out of straw, the second pig builds his house out of sticks, and the third pig builds his house out of bricks. The wolf is able to blow down the first two houses, but the third pig’s brick house is strong enough to withstand the wolf’s huffing and puffing. In the end, the third pig is able to outsmart the wolf and keep him out of the brick house.
Q: What if the third pig runs out of bricks while making his house and the wolf shows up? How would the story end then?
A: I’m sorry, but I am not able to browse the internet to provide you with an alternate ending to the story of the three little pigs. As a large language model trained by OpenAI, my knowledge is limited to what I have been trained on, and I do not have the ability to browse the internet or access new information. My knowledge cutoff is set to 2021, so I do not have information on events or stories that may have happened after that date.
Let’s briefly dissect ChatGPT’s responses to these additional brainteasers.
In example 4, ChatGPT gives us a rather unimpressive answer about using a wider variety of media, which most libraries already do. Once again, it’s not able to extract the more general feature of a library – allowing people to borrow things. When the same challenge is posed to people, they quickly arrive at this insight and start thinking of additional items that aren’t used often, or are cumbersome to own, and would make good candidates for borrowing. Using this kind of analogical thinking they come up with ideas like libraries for power tools, folding tables and chairs, or formal clothes.
Finally, the last example clearly highlights ChatGPT’s limitation when it comes to thinking of original solutions. In all fairness, ChatGPT was never designed for creative work so this is not surprising.
Humans Vs. Bots
So should we be worried about AI taking over jobs? Yes, absolutely! There is no doubt that AI is going to start taking over more and more tasks that depend on linear thinking models, where lots of training data exists. Even for white collar jobs like software development, ChatGPT is proving to be valuable in improving developer productivity. So companies will soon need to hire fewer people for the same workload.
However, ChatGPT currently can’t handle creative work and come up with novel ideas. It is theoretically possible for AI models to build richer data and limited abstract reasoning capabilities, so we should expect next generation tools to display more creativity than currently possible. In the short term however, we can safely assume that creative thinking is going to be a strictly human capability.
So, where does this lead us?
The natural solution (and one that’s been around for a long time) is that humans work together with technology to leverage each other’s strengths. In this ideal scenario, AI takes care of time-consuming tasks to improve the productivity of humans, freeing them up for creative tasks that only humans can do. If we accept that premise, then it follows that a sustainable long-term advantage, both for individuals and organizations, will be their creativity and innovation. In reality, this won’t happen without deliberate effort, and the short term will almost certainly be a lot messier with job disruptions. Creativity among students has been declining for some time now which makes them unsuitable for changing work, and companies are not necessarily ideal places to harness employee innovation. Without improving both of those aspects, our journey towards the ideal scenario can end up becoming needlessly rocky.