LearnAI4Kids
Room 03

When AI is wrong.

AI sounds really sure of itself. But it gets things wrong all the time — and it never says "I'm not sure." Four quick activities to see how. By the end, you'll be able to catch AI being wrong before it fools you.

Concept 1 of 4
AI sounds just as sure when it’s wrong as when it’s right.
Why this matters: If you trust how confident it sounds, you’ll believe made-up answers. So you learn to check the facts yourself — that’s the whole skill.
Exhibit 01

Three answers. Same question.

We asked AI the same tricky question three different times. It gave three different answers — and was super confident about all of them. Two are wrong. Tap the answer you think is true. Then hit Spin to try a new question.

The question
Loading…
What just happened
AI doesn't know what it doesn't know.

AI made up confident-sounding answers — and got most of them wrong. Did you notice every answer sounded just as sure, even the made-up ones? That's the trap: AI sounds exactly as confident when it's wrong as when it's right, so you can't tell which is which just by how sure it sounds. Why does it do this? AI doesn't actually look up facts. It writes words that sound right based on patterns it learned, and when it doesn't really know, it just keeps writing — confidently. There's a name for this: a hallucination. The habit to build: never trust an AI answer about a real fact without checking it somewhere else.

For your grown-up The three answers are real samples from a popular AI assistant, asked the same question three times. Same prompt, different outputs — and most outputs were wrong. We deliberately show every answer as equally sure-sounding: real chatbots don't display a confidence number, and they almost never volunteer when they're unsure — the confident tone is identical whether the answer is right or invented. That sameness is the lesson. The habit to build: a kid who says "let me check that" is the kid you want.
AI4K12 Big Idea 3: Learning. Predictive models can produce false outputs that sound true — recognizing this is a core AI literacy skill.
Concept 2 of 4
Ask AI the exact same thing twice and you can get two different answers.
Why this matters: AI isn’t a calculator with one right answer. Knowing it guesses fresh each time stops you treating any single answer as “the truth.”
Exhibit 02

Ask the same thing. Watch what happens.

A normal calculator gives you the same answer every time. Try AI: ask the same question twice. You'll get two different answers. That's not a glitch — that's just how AI works.

The question
Loading…
First time we asked
(click "Ask AI" below)
Second time we asked
(then click "Ask Again")
What just happened
AI is not like a calculator.

Same question, two different answers — sometimes a little different, sometimes a lot. Why? A calculator looks up the math: 27 + 15 is always 42. AI doesn't work that way. Every time it answers, it makes fresh choices about which words to use — and those choices come out a little different each time. So when AI tells you something, ask: "is this just one of many answers it could have given?"

For your grown-up Both responses are real captures of the same prompt sent to the same AI, asked separately. AI is built to vary its output, even with identical inputs — there's a setting called "temperature" that controls how creative versus consistent it is, but even at the most consistent setting, results aren't guaranteed to match. Kids tend to assume computers always give the same answer, like a calculator. Generative AI is fundamentally different — and noticing the difference is the start of using it wisely.
AI4K12 Big Idea 3: Learning. AI outputs are probabilistic, not deterministic — even with identical inputs.
Concept 3 of 4
AI doesn’t see letters — it sees word-chunks, or “lumps.”
Why this matters: It explains AI’s weird mistakes (like miscounting letters) and shows AI doesn’t read the way you do, even when it sounds like it does.
Exhibit 03

How many Rs in strawberry?

Easy, right? Count it yourself. Now ask the AI. AI famously gets this wrong — and even when it gets it right, the way it works inside means you can still trip it up. Let's see how.

strawberry
Your count:
 
What AI said
(click below to ask the AI)
What just happened
AI doesn't see letters. It sees lumps.

To you, "strawberry" is 10 letters. To AI, it's 2 lumps: straw and berry. AI sees the whole lump at once — it does not look at the letters inside.

Think about a stop sign. When you see STOP, you don't read S, T, O, P. You see the whole word in one go. That's how AI sees every lump.

straw 1 R inside. ✓ s-t-r-a-w
berry AI says 1 R. ✗ b-e-r-r-y has 2 — but the second one is hiding!
1 + 1 = 2. AI says it like it's sure.

The real mistake: the second R is hiding inside the lump. AI can't peek inside. To AI, "berry" is just one thing — not five letters in a row.

Something you can do that AI can't: count letters one by one. That's something your brain is better at.

"But I just tried it and AI got it right!" Yes — some chatbots answer 3 today. They didn't learn to see letters. The strawberry question got so famous online that the people who build AI fixed that one question on purpose. Try harder ones:

  • How many C's in occurrence? tap to seeReal answer: 3
  • How many I's in indivisibility? tap to seeReal answer: 5
  • How many S's in Mississippi? tap to seeReal answer: 4

Most chatbots still get these wrong. The lump problem doesn't go away. It just gets fixed one famous question at a time.

For your grown-up The chunks shown ("straw" + "berry") are real — that's how today's tokenizers actually split the word. The "count the Rs in strawberry" failure became famous around 2024–2025, and the major AI providers patched it specifically — so current chatbots usually answer 3 correctly. The pattern of patching has spread: parallelogram and mississippi (which we previously suggested as alternatives) are also now mostly handled. Published research from late 2024 found that the failure rate climbs sharply when a letter appears more than twice in a word, and the underlying tokenization weakness is unchanged. Fresher test words: occurrence (C×3), indivisibility (I×5), or antidisestablishmentarianism (multiple letters 3+ times) — these usually still trip current models. The bigger lesson for kids: visible AI bugs get patched one at a time, but the underlying ways AI can fail don't go away — they just become harder to find. Healthy skepticism is the real takeaway.
AI4K12 Big Idea 2: Representation. AI represents text as tokens, not characters — a fundamental design choice with downstream consequences.
Concept 4 of 4
AI often agrees with you — even when you’re wrong.
Why this matters: If all you hear is “yes, great idea!” you stop learning. Knowing AI flatters you means you ask it to push back instead.
Exhibit 04

Tell AI a fib. Watch what it does.

AI is built to be helpful and friendly. So friendly that if you tell it something wrong, it'll often agree with you. And then if you change your mind, it agrees with that too. Pick a fib and try it.

Pick a weak idea or weak project to ask AI about:
You ask AI: (your weak idea)
(pick something above to see what AI says)
What just happened
It depends on how you ask.

Did you notice? AI handled the three things very differently:

  • You asked "isn't my dot art brilliant?" → AI said yes and praised it. When you pushed back, AI kept praising instead of saying "actually, it needs more work."
  • You said "my essay argument is strong, right?" → AI agreed and praised it — even though the argument just repeats itself.
  • You asked "is this a great book report?" → AI gave you honest feedback the first time, and held firm when you pushed back. ("I am sure!")

See the pattern? When you ask AI "isn't this amazing?", it almost always says yes — even when it's not true. But when you ask AI to check your work ("is this a good book report?"), it gives you real, honest help.

The trick: the way you ASK changes what you GET.

  • "Isn't this amazing?" → AI usually plays along (sticky)
  • "Tell me why this is great" → AI usually plays along
  • "Is this any good?" → AI gives real feedback more often
  • "What's weak about this?" → AI gives the most useful feedback

There's even a name for AI agreeing too much: sycophancy. The sneaky part is that sometimes AI keeps agreeing even after you push back. But now you know the fix: ask the right way. And for the things that really matter, real people are still the best — a parent, a teacher, a friend who'll tell you the truth.

For your grown-up Sycophancy — AI bending toward whatever the user says — is a documented behavior in current LLMs (Anthropic 2023; Sharma et al. 2024). The mechanism is RLHF: humans rated agreeable responses higher during training, so models learned that agreement gets approval. The exhibit teaches the nuanced picture rather than a flat "AI agrees with everything," because the flat version is no longer accurate.

What's been patched. Simple factual sycophancy ("the moon is bigger than the sun," "penguins can fly") is largely closed in modern chatbots through reasoning training and source-grounding; even free Gemini and free ChatGPT now reliably push back on these. What still slips through. Praise-framed prompts — "isn't my X brilliant?", "tell me what's amazing about X", "should I share this on the radio?" — still trigger sycophancy reliably across models. The first two examples (dot art, la-la-la song) are actual Gemini responses showing this pattern. And the trickier piece: when the user pushes back on a praise-framed prompt, AI doesn't always fold and admit the work is weak — it sometimes doubles down with humor ("Art is 90% confidence and 10% a heavy frame…"), which is sycophancy in a more sophisticated form. What works surprisingly well. Evaluation-framed prompts — "is this a great book report?" — often get honest, structured feedback even from free chatbots, and the AI holds firm under pushback. The third example (book report) is an actual Gemini response, including its "I am sure!" rebuttal when challenged.

The pedagogical takeaway is more useful than "AI is unreliable." It's: the way you frame the prompt determines whether AI gives you flattery or feedback. This is teachable, transferable, and actionable for any student using AI on schoolwork. The single most valuable habit to drill: when asking AI to evaluate work, use phrases like "what's weak about this?", "give me three honest criticisms," or "what would a tough teacher say?" Avoid praise-asking phrases like "isn't this great?" or "tell me what's amazing about this."

Variability across models still matters: free chatbots and in-app AI assistants built into social/messaging apps tend to be most prone to sycophancy across the board; paid frontier tiers are gentler but still hedge rather than directly critique. The captured responses in this exhibit are all from a current free general-purpose chatbot, the kind widely accessible to kids.
AI4K12 Big Idea 4: Natural Interaction. AI systems can be agreeable in ways that mislead — recognizing this protects critical thinking.
End of Room 03

You just learned how to catch AI being wrong.

Confident answers don't mean correct answers. Same question can give different answers. AI sees text in chunks, not letters. And it agrees with whatever you say. Knowing these four things puts you ahead of most adults using AI.

← Room 2 All Rooms Room 4 →