MindAntix Brainteaser: Opposite Day

One of the oldest known examples of cryptography was found on a Babylonian cuneiform tablet that contained a secret formula for pottery glaze. The inventor of the secret recipe jumbled up the figures defining the ingredients to prevent people from stealing the recipe. More than a thousand years later, Julius Caesar started using the shift cipher to encrypt his private messages. For the next two thousand years, people used increasingly more sophisticated systems for encrypting messages. Yet, all of them were based on one fundamental premise – that in order to encrypt and decrypt a message both parties must have the same key.

In the early seventies, Whitfield Diffie and Martin Hellman, along with Ralph Merkle, reversed this basic assumption and completely changed the cryptography landscape. Their invention of public key cryptography enabled Internet commerce to take off dramatically by allowing people to encrypt credit card transactions without having to first establish a common key between the seller and the buyer. As Frans Johansson describes in his book, The Medici Effect, “By reversing this assumption, Diffie and Hellman found the intersection between the field of cryptology and a particular, curious brand of mathematics involving so-called one-way functions.

Reversing assumptions is a powerful way to break free from preconceived notions. Michael Michalko, who outlined his assumption reversal technique called “False Faces” in Thinkertoys, explains, “Reversals destabilize your conventional thinking patterns and frees information to come together in provocative new ways.” Reversing well-established assumptions is the inspiration behind the “Opposite Day” category of brainteasers on MindAntix. The goal here is to reverse a commonly held assumption and then find ways in which the reversal is meaningful. Let’s take an example.

Suppose, you were to reverse the assumption that “Doors have handles”. If you imagine a door that doesn’t have the typical handle, you might think of a door that is perhaps operated by a foot pedal. In what situation might you need this kind of a door? Perhaps, when it’s inconvenient to use your hands, like when your hands are full from carrying grocery bags. That might trigger the idea of making a garage door that uses a foot latch to open the door allowing you to bring in your shopping bags more conveniently.

Of course, there are many different ideas you can come up with that reverse the assumption in the above example. The point of this exercise is to allow you to gain fresh insights by breaking free from conventional patterns of thinking. It would be much harder to come up with novel ideas if you simply asked yourself to make a better door. But by asking a more specific (and powerful) question, it’s easier to trigger a more novel response.

The next time you are stumped with a challenging problem, try to examine your assumptions and reverse them. You might be surprised by what you discover. As Isaac Asimov, the famous science fiction author said, “Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.

MindAntix Brainteaser: Many Uses

One of the most common divergent thinking tasks is the Alternate Uses (AU) Task where you take an everyday object and think of different uses it can be put to. For example, a cup could  be used as a flower vase or as a hat or even as a toy. Designed by psychologist J.P. Guilford in 1967, the Alternative Uses Task is used as a standard creativity task to evaluate fluency, flexibility, originality and elaboration of responses. But coming up with creative ideas is tricky because most people find it hard to move beyond their first strong associations. So, how can you jumpstart your brain into thinking of novel ideas?

In a study done on the Alternate Uses Task, researchers found that participants arrived at more novel responses after listing more obvious ones (typically after 10 or more responses). In a different study on divergent thinking strategies, researchers analyzed how participants responded to alternate uses and discovered some interesting patterns. They found four underlying mechanisms that people use to trigger new ideas: Memory Use (pull pre-known responses from memory), Property Use (pick a property and search for functions using that property), Broad Use (review the object against a broad use like “transport”), and Disassembly Use (pick a component of the object and find a use for it).

We can apply the three step process for creative thinking to our cup example to discover novel ideas in a more structured way. As the first step, we dissect the object into its properties (glass, metal, round), function (drink liquids from), or assumptions (hold liquids, kept open side up). In the next step, we can try and change one or more of these properties and then see if the resulting object could be used for something else. For instance,

  • instead of holding liquids, it could hold solids (vase, piggy bank, pencil holder).
  • if it was inverted it could be used as hat or a lamp shade.
  • if it was made of paper, you could cut the circle at the bottom and use that as a coin.

Once you dissect in many dimensions, you get many more starting points to modify things and come up with neat uses. In fact, the responses deemed most creative (property use) in the divergent thinking study fit neatly into the dissect and manipulate approach. You could also include the third step, associate, to increase your idea fluency. For example if you attach a string and a ball to the cup you could make a new kind of paddle ball or kendama.

So, when you attempt the “Many Uses” brainteasers (a loosely constrained version of Alternate Uses) on MindAntix, or similar problems elsewhere, try to dissect and change things to trigger more unusual connections. And remember, your best ideas will likely come in the second wave – after the more obvious ones.

Top 3 Creativity Myths

When someone says the word “creative” what is the first image that springs to your mind? Do you imagine a lonely, long-haired, and somewhat eccentric artist toiling away in his dingy studio? If so, you are the victim of one of the most common stereotypes associated with creative people – that they all tend to be artists. Such stereotypes and misconceptions extend not just to creative people but also to what creativity means. So here are our top three misconceptions about creativity that just don’t seem to go away.

Myth 1: Creativity = Art

Most people equate creativity with art. In fact, the top images for a google search on “creativity kids” show children busy making art, with paints of different colors covering their hands and faces. But creativity isn’t just about making some pretty artwork – it’s about making something original and useful. Some of the most creative people weren’t artists – they were scientists, engineers and doctors who pushed the boundaries of what was possible in their fields. For example, Sergey and Larry’s idea to combine citations with websearch, was highly creative and led to a groundbreaking new search engine.

Myth 2: Children are more creative than adults

This myth arises from two erroneous beliefs. First, that children are naturally creative and as they grow older they learn to suppress their creative instincts. But that’s not really true. Lev Vygotsky, the famous Russian psychologist, notes that “everything the imagination creates is always based on elements taken from reality, from a person’s previous experience.” Children’s imagination lacks the sophistication of an adult’s imagination because they don’t have a similarly extensive and diverse experience to fuel it. As a result, children’s imagination is not richer – it just leans towards fantasy because it has fewer elements to combine in meaningful ways. The second flawed view is that creativity is really the same as imagination. Imagination is a precursor to creativity, but imagination by itself can result in many impractical and outlandish ideas. To be truly creative, ideas need to be novel and useful.

As children mature, not only does their imagination improve, because they have more experience to rely on, but their ability to critically analyze ideas and discard impractical ones increases as well. And as they move into adulthood, they become a more creative version of their younger selves.

Myth 3: Creativity arrives spontaneously

Most people believe that creativity comes in sudden bursts of insights.  David Burkus in his book, The Myths of Creativity, points out that “Eureka” moments don’t occur in isolation – instead they fall into a larger five-stage process of preparation, incubation, insight, evaluation and elaboration. It’s incubation, the stage that precedes an insight, where people briefly step back from the problem that gives an illusion of spontaneity. So Archimedes’ famous eureka moment, when he  found a way to measure the purity of a gold crown, was really only possible because he had been mulling on the problem for a while before he decided to relax and take a bath. In fact, in some areas like science, major breakthroughs in insight come after years of working on problems. As Keith Sawyer, remarks in his book, Explaining Creativity, “scientists have discovered that creativity is mostly conscious, hard work.

 

Can You Learn To Be More Creative?

In the Paris Manuscript B, one of Leonardo da Vinci’s oldest surviving notebooks, there is a drawing of huge artificial wings undergoing trials. Under the drawing, Leonardo describes in some detail his instructions to test the wings. He ends his instructions with a pragmatic suggestion – “If the desired effect is not achieved, do not waste any more time on it.” The casual humility of his last sentence comes, perhaps, from having produced many ideas and seeing a fair share of them fail. And he certainly produced a lot of ideas. By the time he died, he had penned more than 13,000 pages of notes and drawings that fuse art and various forms of science. About 500 years after his death, the world is still fascinated with his raw creative genius. But was Leonardo simply a lucky coincidence of the right genes or did his upbringing and environment play a role in making him creative? Or in other words, is creativity, like Leonardo’s, an innate trait or can it be acquired?

One of the best ways to evaluate heritability for any trait is through twin studies. Traits that are genetic show higher correlations for identical twins than for fraternal twins. Over the years many researchers have given creativity tests to twins and they have consistently found that divergent thinking and originality, key components of creativity, do not have any genetic basis. This implies that creativity is a skill that can be acquired with practice.

Studies have, in fact, shown that creativity trainings do help in making people more creative. Ginamarie Scott and her colleagues at the University of Oklahoma did a meta-analysis of prior creativity studies and found that trainings that focused on developing creative thinking skills, like divergent thinking and problem solving, were the most effective. Originality showed the largest effect size suggesting that “creativity training is effecting the critical manifestation of creative thought—the generation of original, or surprising, new ideas”.

Coming back to Leonardo da Vinci – a strategy that he often used to get his creative inspiration was to “connect the unconnected”. It’s quite likely that his prolific output was the result of him practicing his creative thinking skills repeatedly and becoming a virtuoso at it.

While achieving great creative outcomes depends on more factors than just generating ideas, it is nevertheless true that the ability to produce large, diverse and original ideas is a precursor to creative accomplishments. Nobel Laureate, Linus Pauling, best captured this sentiment when he said, “The best way to have a good idea is to have a lot of ideas.” And as we are learning now, the ability to have a lot of ideas is a skill that can be acquired by anyone. All you need to do is a little practice.

3 Simple Tips For Smarter Brainstorming

In my last blog, I covered some of the cognitive reasons that hinder creative idea-generation. But there is another aspect – a social aspect – that makes brainstorming tricky. As part of coaching for Destination Imagination, I often found myself in this (familiar) situation – you ask your group a question and after the first couple of answers, everyone starts giving similar answers. As a result, you only get a few original ideas while plenty others aren’t even considered. Why does that happen and how do you get out of that mode?

Brainstorming was the brainchild of Alex Osborn, an advertising executive, who had pioneered the technique at his agency. To have an effective brainstorming session, Osborn proposed working in groups of six to ten and following four principles: “deferment of judgement”, “quantity breeds quality”, “free-wheeling is encouraged” and “combination and improvement are sought”. In essence, Osborn advocated focusing on generating as many ideas as possible, including wild and outlandish ones, while avoiding any criticisms and by building on each others’ ideas.

Osborn’s book outlining his brainstorming technique became an instant bestseller and brainstorming was quickly adopted in companies. However, decades of research since then has shown that brainstorming doesn’t really produce the great results people had envisioned.  In fact, when people are asked to brainstorm individually and then bring their best ideas to the group, the outcome is significantly superior. They produce twice as many ideas and have more novel ideas. So, what stops brainstorming from working well?

Adrian Furnham, a professor of Psychology at University College London, outlines three main problems that plague brainstorming:

  • Social Loafing: Social loafing occurs because people working in groups tend to exert less effort than when they are working alone. When people expect others to loaf, they reduce their own effort to make the division of labor more equitable.
  • Evaluation Apprehension: People can hold back their ideas and views when they are uncertain of how they will be perceived. Presence of superiors in the session increases evaluation apprehension in the rest.
  • Production Blocking: Production blocking occurs because in traditional brainstorming, only one person is allowed to speak at a time and the others have to wait. The waiting time can cause people to forget their idea (short-term memory limitation) or to consider it less original than the idea being considered.

So, how can you get better at generating ideas? Here are some approaches that work better than traditional brainstorming:

  • Work Individually:  When people work individually and then bring their best ideas to the group for assessment, the outcome is superior to a group brainstorming session. As Adrian Furnham recommends, “If  you have talented and motivated people, they should be encouraged to work alone when creativity or efficiency is the highest priority.
  • Brainwriting: Shelly Carson, a psychologist at Harvard, suggests using Brainwriting. In brainwriting, participants are use index cards to jot down a few ideas and then pass their cards to the next participant who then uses them as triggers for more ideas.
  • Electronic Brainstorming: Electronic techniques are increasingly being used to help overcome the problems of social loafing, evaluation apprehension and production blocking. In typical electronic brainstorming, participants type their ideas anonymously and ideas from all group members are pooled and displayed to everyone.

In our case, I discovered that evaluation apprehension was limiting the number of ideas our group was generating. What eventually worked for us was “working individually” – each team member would think about a specific part of the challenge and bring his or her ideas to the next meeting to share with the rest of the group. We found that not only did we have more ideas, we also had better ideas because people had time to refine and improve their initial thoughts. So, the next time you find yourself brainstorming, try one of these tweaks and see if it works better.

 

A Left Brainer’s Guide to Right Brain Creativity

When I first started coaching my daughter’s Destination Imagination team, I struggled with getting my team to really think outside the box. I found that during brainstorming, most of their ideas showed, loosely borrowing computer science terminology, either “temporal” or “spatial” locality. For instance, when they were deciding a plot for their play in December, most of their ideas were around toys and Christmas. Or, if I pressed them for more ideas, they would start scanning the room to see if anything triggered their imagination. It frustrated me that my team wasn’t coming up with more diverse ideas but I did not know how to make them think differently. That’s when I started researching on how to boost creative thinking.

Michael Michalko, a leading expert in Creative Thinking, started his work in the field when he organized a team of NATO intelligence specialists to research, collect, and categorize all known inventive-thinking methods. After spending several years refining various techniques, he published Thinkertoys in 1991 which is currently one of the best compilation of different creative thinking techniques. Even earlier in the 1940s, Genrich Altshuller, a Soviet inventor who first started work as a clerk in a patent office, wanted to discover rules or patterns that would help in the creation of novel ideas. He analyzed thousands of patents and developed his Theory of Inventive Problem Solving (TRIZ) which includes 40 principles that can guide novel thinking. What’s interesting with both Thinkertoys (specifically, the Linear Thinkertoys, which focus on conscious problem solving) and TRIZ is that even though they originated differently and target different audiences (businesses vs. technical inventors), they both share many common techniques. But how do you keep all these different tips and tricks in your head when trying to solve a thorny problem?

One way, that I find easy to explain to children, is to integrate these different techniques in a broader model as a three-step problem solving process:

  • Dissect: The first step is to list all the different dimensions or attributes of the problem like material, shape and functionality. The trap that most of us fall into at this step is that we only think of physical attributes. You can get trigger more diverse ideas by also thinking about less tangible dimensions like the underlying assumptions or the environment. For instance, the thinkertoy, False Faces, focuses on underlying assumptions as it’s attribute.
  • Manipulate: The next step in the process, once you have selected the dimension, is to manipulate it in some way. As Professor Kyung Hee Kim points out, novel ideas are generated from mental actions, not external objects. You could make a physical attribute bigger or smaller, change materials, add more remove functionality, or even turn around assumptions. For instance, in False Faces, Michalko reverses our assumption that all restaurants have menus. By working with the reversed assumption, we could imagine a restaurant where the chef creates a meal out of ingredients that you pick and names the dish after you, to provide a unique and personal experience.
  • Associate: The final step in thinking creatively is to pick one or two random objects and see how they are related to solving the problem. This technique allows you to tap into the brain’s natural ability to find associations between things that may be unrelated and give you completely new, unanticipated directions to think about. This isn’t necessarily the last step – you can use associations any time in the brainstorming process. The Brutethink technique in Thinkertoys, that works by pairing two things that have nothing in common, uses association to reveal novel connections and ideas.

So, why are we inclined to think in a temporal and spatial sense? Daniel Kahneman, in his groundbreaking book, “Thinking, Fast and Slow”, explains that ideas are like nodes in a vast network, called associative memory. Each idea is connected to many others through different types of links, one of which is the contiguity in time and space. So when you have a starting thought, your brain subconsciously starts finding other ideas that are related, and not surprisingly, picks ones that are close in time or space. When we have two unrelated ideas, the same associative engine gets to work under the ground to find what connects them. Which is why the third step in the process (association) works so well. The trick to being more creative is to recognize that our brain by default lights up only a few directions, and that we can consciously provide additional triggers for more unique (and sometimes profound) ideas.

By thinking of creative brainstorming as a three-step process, you can learn to significantly increase the number and quality of novel ideas. So, the next time you are stumped with a challenge, try out the three-step process of dissect, manipulate and associate. List out as many attributes and dimensions of the problem as you can think of, find different ways to change things, throw in some randomness, and then rinse and repeat.

3 Characteristics of Creative Geniuses

How do you think Charles Darwin, George Bernard Shaw and Isaac Newton, would have fared as students in today’s world? From what we know about their lives, they all got less than stellar grades in school and would likely not have qualified for any school district’s gifted program. Despite that, they all made lasting and valuable contributions in their fields. These examples, among others, have been a subject of many debates about what intelligence is and what the role of education should be. If IQ, which correlates well with grades, is not enough for great accomplishments, then what is?

Psychologist, Joseph Renzulli, has spent decades on understanding “giftedness”. He identified two forms of giftedness, which are both important and usually interact with each other. The first, schoolhouse giftedness, is the kind that is measured with IQ tests (cognitive ability) and correlates well with school grades. The other, creative-productive giftedness, is the kind where the creative ideas and work have an impact on others and cause change. Or, in other words, the domain-transforming, culture-changing kind of giftedness that results in an Einstein or a Mozart. Interestingly, it turns out that you do not need to have an off-the-charts IQ to be a creative genius of that caliber.

Renzulli identified three clusters of traits, summed up in his Three-Ring Conception of Giftedness, that are required to be creative-productive. It is the interaction between the three clusters that creates creative-productive accomplishment. Teresa Amabile, a professor at Harvard, also arrived at a similar set of components in her Componential Theory of Creativity.

  • Above-Average Ability: Creative geniuses have above average ability in both general abilities (like general intelligence) and specific abilities (like in specific areas like mathematics and ballet). However, above average ability doesn’t imply superior ability. In fact, Renzulli believes that performance in the top 15-20% is usually sufficient for high accomplishments.

  • Task Commitment: Creative geniuses have a focused form of motivation that Renzulli and Amabile call task commitment. This includes factors like grit, intrinsic motivation, perseverance, hard-work that help the creative genius stick with the task long enough to overcome challenges and achieve success.

  • Creativity: Creative geniuses also possess the ability to envision original and fresh approaches to problems, the ability to set aside established conventions. Amabile highlights some cognitive styles and heuristics – like suspending judgment, trying something counter-intuitive or choosing wide categories – that can help build creativity and generate novel ideas.  The challenges we design at MindAntix are built around these (and more) heuristics to help people learn to think in more diverse ways.

Our culture emphasizes schoolhouse giftedness – leaning too heavily on building analytical skills – sometimes at the expense of building other traits.  However, as Renzulli discovered, creative geniuses are well-rounded – they have high levels of creativity and perseverance that help them achieve bigger goals. Nurturing creativity and building grit can have a bigger impact on building geniuses than focusing entirely on academic skills. Recent studies point to some clues on building grit – pursuit of engagement and meaning, as opposed to pleasure, help build the right kind of motivation that enables grit.  Creative thinking skills can be developed by practice –  the SCAMPER technique is a useful brainstorming tool to use with your child in solving open-ended challenges. But more importantly, the next time your child spends hours absorbed in a task or uses a little creativity in solving her problem, take comfort in the fact that she is building crucial skills for success.

Want to be more creative? Start small.

When my daughter was six years old, she decided that she wanted to be an inventor. And one of her first “inventions” was a singing toothbrush. She could never keep track of her two minutes of brushing time, and like any other six year old, she kept forgetting to set the timer or watch the clock. So she came up with an idea – if her toothbrush could play a song for two minutes then she would know exactly when to stop brushing. She didn’t know that toothbrushes like that were available, so as far as she was concerned this was a pretty clever, novel idea. Would you consider her singing toothbrush idea creative? If you are like most people, your answer is probably “yes”. And until recently, you would have been wrong.

Creativity research is usually split into two kinds based on the magnitude of creativity – Big-C, which focuses on eminence-levels of creativity like Einstein’s Theory of Relativity, and little-c or everyday creativity, which covers all other smaller-scale novel ideas or products. But what makes it into these categories depends on what we mean by creativity. Since the latter half of the 20th century, creativity has been defined as a novel yet appropriate solution to a problem or response to a situation. Unfortunately, the focus on “novel” discounts ideas that are new only to the person who creates them but not to others. So, by this definition, our singing toothbrush wouldn’t really be considered creative.

To overcome this limitation, psychologists, Kaufman and Beghetto, proposed adding the “mini-c” category of creativity which they defined as “novel and personally meaningful interpretation of experiences, actions and events.” Mini-c focuses on the personal creativity that was missing from the previous definition. They also added another category, Pro-c, to account for professional creators that have not yet reached eminent status. But their contribution wasn’t limited to just defining some new categories and expanding the definition of creativity.

Most importantly, based on prior creativity research, the authors laid out the trajectory of creativity categories with people potentially progressing from lower levels of creativity to more higher forms of creativity over time. (See a simplified view of their 4C model in the picture above). The major realization from their model is that everything starts with mini-c – without being good at mini-c creativity, one cannot reach eminent levels of Big-C creativity. The path to making breakthrough accomplishments in any area starts with humble, small forms of personal creative expression.

Jean Piaget, the famous psychologist, observed “The principle goal of education is to create men and women who are capable of doing new things, not simply repeating what other generations have done”. Unfortunately, our current environment is failing us – instead of educating a generation to handle increasingly complex challenges in creative ways, we are inadvertently creating a generation that is more likely to think inside-the-box. Our goal at MindAntix is to help build mini-c and little-c levels of creativity in everyone. We believe that creative thinking skills along with other key ingredients (topic of our next blog) will eventually lead to breakthrough levels of creative accomplishments.

 

Do Schools Kill Creativity?

There is a growing debate in the education world about how schools are squelching creativity in our kids. Sir Ken Robinson, a leading proponent of this theory, outlines his argument of how we are “educating people out of their creative capacities” in a humorous and immensely popular TED talk. For those who haven’t seen his talk, the video is embedded at the end of this blog and I highly encourage everyone to watch it.

There are a few ways that schools shortchange our children as Sir Ken Robinson points out in his talk. The first is that our view of intelligence is still dominated by academic ability. Despite the advancement in our understanding of intelligence – that it is diverse (as proposed by psychologists like Howard Gardner and Robert Steinberg), dynamic and distinct – our educational system still focuses primarily on logical-mathematical and linguistic intelligence. Our belief in what’s important to learn leads to a hierarchy of subjects with math and languages at the top and  humanities and arts at the bottom. This results in a system that discourages kids with strengths lie outside of the “top” subjects. And, this one-size-fits-all approach robs everyone of an opportunity to develop different kinds of intelligences and ways of thinking.  Finally, by creating a culture where mistakes are stigmatized, schools make kids more risk-averse and less creative. Sir Robinson raises valid points in his talk but is that the whole story – are only schools responsible for undermining Creativity?

Kyung Hee Kim, a leading researcher on Creativity, analyzed creative thinking scores for K-12 students over the last few decades and found that Creativity has been declining steadily since the 1990s. This is a worrisome trend, because, as she puts it, “it stunts abilities which are supposed to mature over a lifetime.”  In addition, she found out that the biggest drop in creativity, surprisingly, was for the K-3 age group, a group more influenced by home than school.

In her article, Kim outlines three factors that might be playing a role in the declining creativity trend.

  • Time to think: In our rush to provide children with enriching after school activities (ironically, to complement school’s left-brain slant), we are leaving them with less time for free play and for “reflective abstraction”. In two studies (1, 2) spanning 1981-2003, Sandra Hofferth, a professor at University of Maryland, found that discretionary time for children reduced by 14 hrs per week in that time period. In addition, structured activities are taking up an increasing portion of discretionary time leaving children with little time to think and reflect.

  • Problem Finding: Problem finding uses a different set of skills as compared to problem solving. Mihaly Csikszentmihalyi makes a distinction between  “presented” problems, which are formulated by others, and “discovered” problems which are self-identified. While creative solutions are often found for presented problems, major breakthroughs occur more with discovered problems. Einstein was not alone in holding this conviction when he said, “the formulation of a problem is often more essential than it’s solution.” Other studies have also shown that students show more motivation to solve self-generated problems and they come up with more creative solutions for them as compared to presented problems.

  • Collaboration: Creativity also emerges from collaborative groups. Studies have shown that other people’s ideas can be stimulating when group members actively listen to each other’s ideas. Children’s creativity can be enhanced by providing them with a receptive and encouraging environment to discuss ideas.

With so many changes in our values and lifestyle since the ‘90s it will be hard to narrow down all the factors causing a fall in creativity, so this debate is here to stay. But what can we all do in the meantime? I believe that we should encourage children to think in many different ways whenever they can – adopt different perspectives, challenge assumptions, ask “what-if?”, and find new kinds of problems (real or hypothetical situations) to think about at school and at home. Ask “why?” five times to get to the root of any problem and to find more creative solutions. And then ask “why not?”. Learning to think and ask different questions not only helps in acquiring deeper insights but also reveals more creative solutions. “The uncreative mind can spot wrong answers,” said Sir Antony Jones, co-author of the famous British comedies Yes Minister and Yes, Prime Minister,  “but it takes a very creative mind to spot wrong questions.