Skip to content
  • There are no suggestions because the search field is empty.

Building and Iterating on Roleplay Environments

Learn how to take an environment from a first draft to something you trust: set a baseline, test it with realistic learner approaches, tune it, and roll it out in stages.

You can generate a usable environment in one step. To trust it with your learners, though, you need to test it. The first draft gets the scenario, goals, and AI character roughly right. Testing it against a few realistic learner approaches and tuning what you find is how you get a character that pushes back in the right places and scores that separate strong performances from weak ones. It is also how you build confidence before the environment reaches your learners.

Choose Your Starting Point

Every route produces the same kind of environment, which you can test and refine with the same steps below.

  • Start from a Template - the fastest route when a ready-made scenario is close to what you need. See Starting a Roleplay Environment from a Template.
  • AI-Assisted Authoring - describe the scenario and upload your source material (product briefs, playbooks, call scripts, rubrics). Best when you have real content the AI character should know. See the Environment Editor Walkthrough.
  • Your own AI assistant - draft the environment outside the platform in Claude, ChatGPT, or a similar tool, then paste it into the editor. Ask Customer Solutions for the environment template and prompt.
  • Blank Canvas - write every field yourself. Best for highly specific scenarios.

Source material matters more than the route. If the AI character needs to answer technical questions or quote your pricing, upload the documents that contain those facts. The character can only be as expert as what you give it.

Understand the Tradeoffs

A few choices shape how the environment behaves and how useful its scores are.

  • Difficulty and score spread. If every learner meets every goal, the scores can't tell your learners apart. Aim for a scenario where an average learner meets some goals, a strong learner meets most of them, and a perfect score takes real skill. This matters most if you plan to compare or rank learners.
  • Character realism and length. Real people answer in short turns and don't volunteer everything they know. A character that gives up information too easily makes the conversation easier to complete but less useful as practice.
  • Number of goals. Three to five specific, observable goals score more consistently than a long list or broad goals like "build rapport".
  • Voice or text. Text sessions are faster to run, so they're good for testing what the character says and how sessions score. Voice adds speech recognition and pacing, so if your learners will practice by voice, run at least a few voice sessions before you roll out. See Text Mode in Roleplay.

Build and Iterate

1. Establish a baseline

Before you change anything, decide what "working" looks like and record how the first draft performs.

  • Write down three to five test approaches that reflect real learners: a strong performance, an average one, a weak one, and at least one that goes off script (an unexpected question, a pushy or rude learner, an attempt to get the character to give in).
  • Run a session for each approach and note the score, which goals were Met, Partially Met, or Not Met, and anything the character did that a real customer wouldn't.

Use the same approaches every round so you're comparing like with like.

2. Evaluate the draft

Compare the results with what you expected. Check the behaviors your scenario depends on:

  • Does the character stay in role and push back where a real customer would?
  • Does it reveal key information only when the learner earns it?
  • Does it use the facts from your source material accurately?
  • Do the scores rank your test approaches in the right order? The weak approach should score clearly lower than the strong one.
  • Does the feedback for each goal match what actually happened in the conversation?

3. Tune and retest

Change one thing at a time so you know what made the difference.

  • Use Refine with AI on the Description or AI Character to make targeted changes, for example "Make the character more skeptical about cost" or "Don't reveal the budget until the learner asks about business impact."
  • Make goals more specific if they score inconsistently.
  • Add or replace source material if the character gets facts wrong.

Then rerun the same test approaches and compare against your baseline. Stop when the strong, average, and weak approaches land where you expect and the character behaves the way it would in a real conversation.

4. Roll out gradually

Release to a small group first, such as a few experienced team members or a pilot cohort.

  • Review their sessions in Reporting: read a few transcripts, check how scores are spread, and look for goals that almost everyone meets or misses.
  • Tune again based on what real learners do, which is rarely what you predicted.
  • Then publish to your full audience.

Get Support

Customer Solutions can review your environment, help you calibrate difficulty, or build the first draft with you from your source materials.

Need help? Submit a ticket to Customer Solutions for help building, testing, or tuning Roleplay environments.