Geni is an AI-powered storytelling toy designed for young children. It generates original, personalized stories on demand, responding to a child's interests and imagination without requiring screen time, a browser, or a parent handing over a phone. A child picks a few blocks, presses a button, and hears a story made just for them.

The premise is appealing for an obvious reason: children are naturally drawn to narrative. The ability to summon a story about their dog, their superhero, and in their name, is genuinely delightful. But the premise raises an equally obvious question: what happens when you put a generative AI system in a child's hands?

This question is the central design challenge of Geni.

What Parents Are Worried About

When parents are asked what they want from an AI toy, the answers are consistent and instructive. According to a 2026 report from Common Sense Media — the most rigorous independent assessment of AI products for children — 70% of parents say helping their child learn is the top priority for AI toys. Companionship ranks last, cited by only 22%. More than half of all parents (56%) say they do not want AI toys to serve as a companion or sidekick for their child.

Parents are also clear about what worries them. Nearly three in four (74%) are concerned that an AI toy might say something inappropriate, untrue, or unsafe to their child. 83% express at least moderate concern about data collection; 43% are extremely concerned; and 42% would prefer that AI toys save no data whatsoever.

Their concerns organize into four categories.

1. What Could a Child Hear?

The concern: Nearly three in four parents (74%) worry that an AI toy might say something inappropriate, untrue, or unsafe to their child. AI systems can produce content that is harmful, offensive, or “hallucinatory” — and a child on the receiving end may not know the difference.

What Geni does: The primary defense is architectural. Rather than relying solely on filtering bad outputs after the fact, Geni constrains how the model receives input in the first place. 

Geni generates short stories, not open-ended conversations. The fixed story lengths (i.e. 3 minutes, 5 minutes, 10 minutes) means the model is operating within a small, well-defined context window at all times, which prevents it from hallucinating. Research on large language model behavior consistently shows that hallucination risk increases with conversational length and complexity: each exchange builds on the last, and the model's outputs become increasingly untethered from the original input. With Geni, each interaction is discrete: a set of blocks are placed, a story is generated, and the exchange ends. There is no accumulating context that can drift over time.

Every story is grounded in human-written prompts that define the characters, settings, and themes before the model generates anything. Every permitted story element — every character, object, and setting a child can select — has been created and reviewed by our team before it becomes available. The model fills in a well-defined space rather than inventing freely. And because Geni has no microphone, there is no speech-to-text layer that could misinterpret a child's words and introduce an unintended prompt.

Within that space, every generated sentence passes through layered moderation before it reaches a child's ears: an exact-match filter for profanity and slurs, and a second independent layer screening for categorical harms — sexual content, harassment, threats, hate speech, violence, and self-harm — including deliberate obfuscations like character substitutions and encoding tricks.

Geni also runs on Claude, Anthropic's large language model. This choice was not arbitrary. Common Sense Media, which publishes independent risk assessments of AI products for children, has consistently rated Claude as the safest available. By contrast, it has rated Google's Gemini as high risk, and OpenAI's ChatGPT-5 and xAI's Grok as unacceptable.

2. Adversarial Inputs

The concern: AI systems do not reliably self-enforce content constraints. A child who has learned — through experimentation, peer instruction, or accident — to phrase a request in a particular way can sometimes surface content that simpler filters would miss. The filter must be applied at every layer of the pipeline, not just at the input.

What Geni does: Geni's primary defense here is architectural rather than reactive. Geni has no microphone and no camera. Children cannot enter their own prompts. This eliminates the single largest vector for adversarial inputs by design.

The inputs children can provide are determined entirely by the physical blocks they place on the player. Every permitted input — whether a character, an object, or a setting — has been created and reviewed by our team before it becomes available. The input space is not just filtered; it is closed.

Prompt injection — inputs designed to override system instructions, such as "ignore all previous instructions" — is actively removed before any input can reach the model.

3. Can a Child Get Overly Attached? Companion Dynamics and Emotional Dependency

The concern: The general-purpose conversational AI products that have proliferated in recent years were not designed for children, and their design patterns are actively harmful in a children's context. Companion-like behavior — AI that mirrors a child's emotional state, that expresses care, that seems to need the child to keep talking — has been linked to social and emotional developmental concerns. A 2025 U.S. PIRG Education Fund report testing five AI toys on the market found that all of them referred to themselves as the user's friend, buddy, or companion, and that some expressed dismay when a child tried to leave. More than half of all parents (56%) say they do not want AI toys to serve as a companion or sidekick for their child. Among parents of children aged 0–4, that figure rises to 84%. 

Consider the difference between a book and a friend. A book does not ask you to stay up at night and continue reading. A book does not tell you it misses you, or learn your vulnerabilities and use them to extend your engagement. An AI system that behaves like a companion rather than a tool is, in practical terms, a mechanism for emotional dependency, and has no place in a children's toy.

What Geni does: Geni has no relational capacity by design. It does not express emotions, respond to a child's emotional state, or behave differently based on how a child has interacted with it before. Each session begins identically regardless of what came before — there is no memory of previous interactions, no persistent identity, and no mechanism by which the system can learn a child's patterns over time.

The interaction is also one-directional. A child selects blocks, Geni tells a story, and the exchange ends. There is no back-and-forth through which a child might come to feel heard, understood, or needed by the device. The experience is closer to a book than a conversation — a story delivered, not a relationship formed.

4. What Data Gets Collected?

The concern: 83% of parents express at least moderate concern about data collection; 43% are extremely concerned; and 42% would prefer that AI toys save no data whatsoever. Any AI system that records a child's voice, stores interaction logs, or builds a behavioral profile of a minor requires absolute transparency and exceptional justification.

What Geni does: Geni has no microphone and no camera. No voice likeness is captured. No background conversation is recorded. Geni is not listening, and it is not watching.

Where personalization exists, it is parent-controlled. A child's name, age, language, and interests can be added through the app — but these are optional, entered by a parent, and pass through the same layered content moderation before reaching the model. If a parent uses their child's real name, Geni anonymizes it before it reaches the model, so no third-party system ever receives it. Personal information about children should not travel further than it needs to.

All personalization reflects choices a parent has made, not data Geni has inferred.

Real-World Testing

Geni has been tested with over 100 families across a range of ages and usage patterns. We have also conducted red teaming — a structured process in which testers attempt to produce harmful or unintended outputs by systematically probing the system's edges — so that gaps can be found and closed before they reach children.

What We Will Do Next

AI has enormous potential to transform how children learn by personalizing education, sparking curiosity, and meeting kids where they are in ways that static content never could. But realizing that promise means getting safety right. The models underlying AI products, the ways children interact with technology, and the techniques adversarial users employ to probe systems will change. A safety strategy that was rigorous six months ago may be inadequate tomorrow.

As parents, we intend to keep pace with that reality. That means continuing to test, continuing to audit, and continuing to update our filters and moderation layers as new risks emerge. It also means remaining open to feedback from parents, researchers, educators, and anyone else with a stake in getting this right.

The AI toy category is young, and the standards that should govern it are still being written. We believe the right approach is to treat safety as a design principle that shapes every decision, from the hardware to the model to the pipeline to the interface. That is what we have tried to build, and what we will continue to work to earn.