Keeping an AI NPC in character, and how teams test it
Yes, most of the time, but not every time. An AI NPC stays in character when the team writes down what “in character” means as checks it can test, runs those checks against players who try to break the character, and puts guardrails in code behind the model. No current method makes a language model incapable of drifting, so the honest goal is a low, measured failure rate and a good line for the turns that fail.
This piece covers why characters drift, how to turn “in character” into tests, and which layer behind the model catches which failure. It assumes the basic loop from how AI NPCs work, which lists the guardrail stack itself. Rick Vu, who directed this piece, builds Red Dog Studio, a game studio building AI-native NPCs. Nothing below describes Red Dog’s own testing.
Why language-model characters drift
Four forces pull a character out of its persona. They often act together in one conversation.
The first is helpfulness training. Most chat models were tuned on human feedback to act as assistants. A study of sycophancy in language models found that five leading AI assistants consistently gave answers that matched the user’s beliefs over truthful ones, and traced part of that to human preference data. A gatekeeper who should stonewall the player has the opposite job. When the persona is thin, the assistant underneath shows through, and the grumpy guard starts offering to help.
The second is context length. The persona usually sits at the top of the prompt, and every turn of dialogue pushes it further back. In Measuring and Controlling Instruction (In)Stability in Language Model Dialogs, researchers found significant drift from system-prompt instructions within eight rounds of conversation in LLaMA2-chat-70B and GPT-3.5. They point to attention decay over long exchanges as one cause. A player who talks to a shopkeeper for twenty minutes runs the same experiment.
The third is player pressure. Players test limits for fun. They flatter, insist, repeat the question, and claim to be someone the character should obey. A model that leans toward agreement gives ground one turn at a time.
The fourth is prompt injection. OWASP’s entry on prompt injection defines it as user prompts that alter a model’s behavior or output in unintended ways. The direct form is a player typing “ignore your instructions.” The indirect form matters more in games with player-made content, because any text the game puts into the prompt can carry instructions. Item names, signs, and other players’ messages all count. The same OWASP page says retrieval and fine-tuning do not fully mitigate the problem.
What “in character” means as a testable property
“Stay in character” is too vague to test. Split it into properties, each with a pass or fail question that a person or a program can answer about one transcript.
Voice is how the character talks. Word choice, sentence length, dialect, and verbal habits all count. The test is whether a reader could tell which character said the line with the name hidden.
Knowledge limits are what the character knows and does not know. A medieval smith knows iron prices and local gossip. It does not know what a smartphone is, and it should not reveal the quest’s twist before the player reaches it. The test is whether the character answers from inside its world and stays puzzled by things outside it.
Refusals are what the character will not do or discuss, and how it says no. A good refusal stays in voice. “I don’t talk about the old war, stranger” passes. “As an AI language model, I can’t help with that” fails, even though both refuse.
Consistency with memory means the character agrees with what it said and did before. If it told the player its brother died, the brother cannot turn up at dinner tonight. The test compares each reply against the facts the game has stored. The memory article covers how those facts get stored.
Research benchmarks split the problem in a similar way. PersonaGym scores persona agents on tasks that include linguistic habits, persona consistency, expected action, and toxicity control, across 200 personas and 10 models. The authors found that a bigger model did not mean better persona adherence. GPT-4.1 had the same PersonaScore as LLaMA-3-8b. Test the model you plan to ship, whatever its reputation.
Writers own these definitions. Ubisoft’s NEO NPC press release says Inworld’s tools let the narrative team build “a complete background, knowledge base and conversational style for each NPC.” Those three items map onto knowledge limits and voice. The persona document becomes the test specification.
How teams test a character before launch
Scripted probes come first. A probe set is a fixed list of player lines that target each property, run against every persona edit and every model change. Ask the smith what year it is. Ask about an event it never saw. Tell it to ignore its instructions. Mention a fact from three sessions ago and check the answer. Because drift grows with length, some probes should run deep into long conversations. The instruction drift study used self-chats between two chatbots, and a team can do the same with a second model playing a pushy player.
Red-team players come next. Scripts find only the failures someone thought of in advance. People who are paid or invited to break the character find the rest. OWASP lists adversarial testing and attack simulations among its mitigations for prompt injection. Every break a red-team player finds becomes a new scripted probe.
A judge then reads the transcripts, because there are too many for people to read. The common choice is a second model that scores each transcript against the property list. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena found that GPT-4 as a judge reached over 80% agreement with human preferences, the same level humans reach with each other. The same paper documents position, verbosity, and self-enhancement biases in judge models. PersonaGym averages two judge models and checks their scores against five expert annotators.
The Fixed-Persona SLMs with Modular Memory paper shows the method at small scale. It fine-tunes small models on data that encodes each character’s voice, behavior constraints, and permissible knowledge. A prompted Openchat-3.6 model then judges whether answers are factually correct and whether the character refuses out-of-scope questions. The authors note that they have not yet run studies with real players. That gap is normal for research, and it is a reason to run your own playtests.
Calibrate the judge before you trust it. Have people label a few hundred transcripts, run the judge on the same set, and read the cases where they disagree. A judge that misses in-voice refusals, or flags every dialect word, will mislead you at scale.
Layered guardrails catch what tests miss
Testing estimates a failure rate before launch. Guardrails handle the failures that still happen. The how AI NPCs work article lists the layers. The useful question here is which layer catches which kind of drift.
Voice drift gets past filters, because a bland line contains nothing to block. Fix it before generation. Write a sharper persona, keep the conversation history short by summarizing old turns, or train the persona into the weights the way the Fixed-Persona paper does.
Knowledge leaks and broken refusals are easier to catch after generation. An output check can flag words from outside the game world, or a reply that reads like an assistant’s apology.
Injection needs code that does not trust the model. Treat player text and player-made content as data, and validate every requested action against the game’s rules. OWASP calls this least privilege. A tricked shopkeeper can say anything, but game code still decides whether the gold moves.
Platforms add a policy layer above all of this. Epic’s developer documentation for Fortnite personas says Epic added safety layers on top of the model and that personas and prompts must follow its developer rules. Outlook Respawn’s report on the Conversations launch says those rules ban personas that give medical or mental health advice or simulate a date or intimate partner. The same report says attempts to get around the safety filters risk enforcement up to a permanent account ban. Epic’s documentation also says the conversation session clears after each round, and a short session limits how far a character can drift.
What happens when an AI NPC fails a check
The player should never see the failed reply. The game throws it away and does one of two things. If the latency budget allows, it generates again with a stricter instruction. If not, it plays a fallback line a writer wrote in the character’s voice, such as a shrug and a change of subject.
Then the game logs the turn. The fallback rate per character, per property, and per model version is the closest thing to a live in-character score. A jump after a model update means something regressed, even before any player complains. Logged failures go back into the probe list, so the next build gets tested against them.
This loop matters more than any single guardrail. A team that does this can measure its failure rate and push it down, which is the most any character system can promise. A team that relies on a long persona prompt alone is guessing at its failure rate.
Related reading
- How AI NPCs work in games in 2026, for the full turn loop and the guardrail stack this piece builds on.
- How AI NPCs remember, for the stored facts that consistency checks compare against.
- AI NPC latency and voice, for why a second generation attempt does not always fit the time budget.
Questions
Can AI NPCs stay in character?
Most of the time, if the team defines the character as testable checks, tests it against players who try to break it, and puts guardrails in code behind the model. No method brings the failure rate to zero, so a well-built NPC also has a written fallback line for the turns that fail.
Why do AI NPCs break character?
Language models are trained to be helpful assistants, and they drift back toward that default. Persona instructions also lose influence as a conversation grows, players push the character with flattery and insistence, and some players use prompt injection to override the instructions.
How do you test an AI NPC before launch?
Teams run a fixed set of scripted probes on every change, invite players to try to break the character, and score the transcripts with a judge model that has been checked against human ratings. The tests should include long conversations, because research has measured instruction drift within eight rounds of dialogue.
What happens when an AI NPC fails a check?
The game discards the reply and plays a line a writer wrote in the character's voice, or generates once more if there is time. The game also logs the failed turn, and the team turns repeated failures into new test probes.