Using Live LLMs as Experimental Stimuli: A Literature Review on the “Clean Manipulation” Problem

Published

August 2, 2026

Using Live LLMs as Experimental Stimuli: A Literature Review on the “Clean Manipulation” Problem

August 2, 2026

I’ve been designing a few experiments where participants interact with an AI companion driven by an LLM. Using an LLM as a stimulus creates a distinctive problem: the manipulation is almost never “clean”.

So how do we design experimental research involving interaction with AI? What is the current best practice? I did a comprehensive review of the literature on exactly this question, and I wanted to share what I found in case anyone is interested in this topic.

It’s a bit long and dense. If you only have five minutes, read the TL;DR, the current best practices, and my recommendations.

TL;DR

  • The core problem is real and now formally named: when you deploy a live LLM as a treatment, you control the generating policy (model + system prompt + decoding parameters), not the realized conversation each participant experiences; the estimand shifts from “the effect of a fixed message” to “the effect of assignment to a conversational policy,” and the two are not the same (Fresh & Shin, 2026, arXiv 2607.03597).

  • Best practice has converged on a reproducible-policy discipline: hold the model constant across conditions and manipulate only the focal system-prompt variable; pin exact model version strings, decoding parameters, and access dates; log full transcripts linked to participant ID and condition; run manipulation checks AND post-hoc treatment-fidelity audits of transcripts; preregister prompts; open-source deployment code; and report using GUIDE-LLM (Feuerriegel et al., 2026, Nature Human Behaviour) and the Abdurahman et al. (2025, AMPPS) primer.

  • Irreducible limitations remain: proprietary APIs are not deterministic even at temperature 0 (silent updates, expert routing, numerical rounding; Coqueret et al., 2026, arXiv 2607.24372); model deprecation threatens replicability; findings can be highly prompt- and model-specific (Hackenburg et al., 2025, Science); and a flagship exemplar (Costello et al., 2024, Science) now carries an Editorial Expression of Concern (Thorp, 2026). Treat the live LLM as a stimulus distribution, not a fixed text, and model stimulus variability rather than pretending it away.

What I Find in Literature (Key Findings)

  • The historical arc runs from tight control with poor realism (vignettes, mock transcripts) toward high realism with poor control (live LLMs). Each earlier method traded away one of the two. Live LLMs collapse the labor and believability problems of Wizard-of-Oz and rule-based bots but reintroduce a control problem at a new level of abstraction — you now control the policy, not the utterances.

  • “Generate-then-freeze” is the crucial intermediate. LLMs generate candidate stimuli that are then validated and presented statically, preserving classic experimental control. The four-preregistered-experiment communication paper (SocArXiv 10.31235/osf.io/xb2n6, v2 posted May 10, 2025) compared ChatGPT (GPT-4)-generated versus human-generated message stimuli and found, verbatim, that “GPT-generated message stimuli performed equivalently to or even surpassed human-generated stimuli in manipulating target variables, while the performance in controlling unintended variables was mixed.”

  • Best practice = control the policy, audit the realization. You cannot fix the words, so you fix the distribution that generates them (system prompt, model, temperature) and then measure treatment fidelity in the transcripts post hoc, borrowing the clinical fidelity-instrument tradition.

  • A mature toolkit ecosystem now exists (DiSCoKit, Simple Chat, the Oh et al. Qualtrics toolkit, the Yale/Kalla AWS Bedrock recipe, SMARTRIQS-derived chatboxes), removing the engineering barrier and standardizing logging/security.

  • The estimand has been formalized. Conversational experiments identify the effect of assignment to a policy, not the effect of any fixed message; more granular estimands (message-level, feature-level, dosage) require extra identifying assumptions.

More Details from the Lit Review

1. Historical approaches and their trade-offs

  1. Static pre-tested stimuli, vignettes, mock chat transcripts/screenshots. The classic experimental-control tradition (Shadish, Cook & Campbell, 2002, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Houghton Mifflin) prizes a fixed, pre-tested manipulation so that any between-condition difference is attributable to the manipulated construct and internal validity is protected. In communication and media psychology this became the workhorse: vignettes, constructed news articles, and mock social-media/chat screenshots, each pretested for the target construct and confounds. The enduring critique is ecological/external validity: static stimuli capture stated responses to a frozen artifact rather than actual behavior in an interactive medium. The information-systems HCI literature made this explicit: Williams, Jenkins, Valacich & Byrd (2017), “Measuring Actual Behaviors in HCI Research - A Call to Action and an Example,” AIS Transactions on Human-Computer Interaction, 9(4) — arguing the field over-relies on self-report/intention proxies rather than measured behavior. For interactive technology, a screenshot of a chatbot is not a chatbot.

  2. Human confederates following scripts. To recover interactivity, lab studies used trained confederates delivering scripted conversational turns. This buys real-time contingency and social presence but is labor-intensive, hard to scale, and vulnerable to operator inconsistency (confederates drift from script, leak demand characteristics, and vary by rapport/appearance). Standardization across sessions and confederates is the persistent threat to internal validity.

  3. Wizard of Oz (WoZ). A human “wizard” simulates an intelligent system, usually covertly, so participants believe they are interacting with a machine. The method originates with Kelley (1984), “An Iterative Design Methodology for User-Friendly Natural Language Office Information Applications,” ACM Transactions on Office Information Systems, 2(1), 26–41, and was codified by Dahlbäck, Jönsson & Ahrenberg (1993), “Wizard of Oz Studies — Why and How,” Knowledge-Based Systems, 6(4), 258–266 (also IUI ’93, pp. 193–200). WoZ delivers realism and design flexibility before a system exists, but its problems are: (i) deception and the ethics/debriefing burden; (ii) heavy labor; and (iii) operator inconsistency — wizards vary in production and recognition. Riek (2012), “Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines,” Journal of Human-Robot Interaction, 1(1), 119–136, reviewed 54 WoZ experiments published in the primary HRI venues from 2001–2011. Researchers “mainly used WoZ for verbal (72.2%) and non-verbal (48.1%) processing”; most constrained wizard production (90.7%) but few constrained wizard recognition (11%), few measured wizard error (3.7%), and few reported pre-experiment wizard training (5.4%) — quantifying the inconsistency threat. Modern LLM-era work (Fang et al., 2024, “On LLM Wizards: Identifying Large Language Models’ Behaviors for Wizard of Oz Experiments,” IVA ’24) uses LLMs to standardize wizard behavior.

  4. Rule-based / scripted / decision-tree chatbots (ELIZA lineage). Weizenbaum’s ELIZA (1966, “ELIZA — A Computer Program for the Study of Natural Language Communication Between Man and Machine,” Communications of the ACM, 9(1), 36–45) established pattern-matching, keyword-and-template conversational agents; the lineage runs through PARRY (Colby, 1975), ALICE/AIML (Wallace, 2009), and modern decision-tree platforms (Landbot-style). The “ELIZA effect” — users over-attributing understanding — made even primitive bots socially potent. But scripted bots have brittle, declining believability: they cannot handle out-of-script input and break the illusion of a genuine interlocutor, capping realism and constraining the constructs one can manipulate. A relevant control-vs-realism datapoint from the virtual-human tradition: Lucas, Gratch, King & Morency (2014), “It’s only a computer: Virtual humans increase willingness to disclose,” Computers in Human Behavior, 37, 94–100.

  5. The “generate-then-freeze” intermediate. LLMs generate candidate stimuli that are then validated and presented statically — keeping classic control while cutting stimulus-production cost. The key communication-methods paper is “Leveraging Large Language Models in Message Stimuli Generation and Validation for Experimental Research” (2025; SocArXiv/OSF preprint 10.31235/osf.io/xb2n6, v2 posted May 10, 2025). Across four preregistered experiments it compared ChatGPT (GPT-4)-generated versus human-generated message stimuli on (1) manipulating target variables (discrete emotions and moral intuitions) and (2) controlling unintended variables. Verbatim: “We found GPT-generated message stimuli performed equivalently to or even surpassed human-generated stimuli in manipulating target variables, while the performance in controlling unintended variables was mixed.” This is the bridge: it uses the LLM’s generativity while retaining a frozen, pre-tested treatment - the very “clean manipulation” that live deployment sacrifices.

2. Current best practices for deploying live LLMs as stimuli

Control at the level of the generating policy. Because you cannot fix the realized text, fix what generates it. Hold the model constant across conditions and manipulate only the focal variable through condition-specific system prompts; this is exactly the Costello et al. (2024) design logic: same model in every condition, only the system-prompt goal differs (persuade vs. control). Hackenburg et al. (2025, Science) is the empirical warning that the policy matters enormously: prompting and post-training move persuasiveness far more than scale or personalization.

Parameter and version control. Fix and report decoding parameters (temperature, top-p), pin the exact model version string, and record access dates. Abdurahman et al. (2025, AMPPS) recommend prioritizing models that are “stable over time and permanently available” and note open-weight models (e.g., Llama 4, Gemma, Mistral, Phi-4, Qwen 3.5, DeepSeek V4 Pro, GLM-5.2, Kimi K2.6, Nemotron) allow weight archiving impossible with proprietary APIs. Hold temperature constant across conditions or vary it as an explicit factor.

Tight task scoping. Constrain the interaction (turn caps, scoped goals, bounded topic) to reduce behavioral variance across participants and keep the realized treatment closer to the intended construct. (Costello et al.’s DebunkBot, for example, structured the exchange as a describe-belief → rate → rebuttal loop.)

Extensive piloting before launch to characterize the output distribution and catch guardrail refusals or off-target behavior.

Logging and fidelity auditing. Log full transcripts linked to participant ID and condition (all toolkits below do this). Then audit treatment fidelity post hoc, borrowing the clinical concept: human coders or validated fidelity instruments score whether the realized conversation actually delivered the intended treatment. The template is Kuhlmeier et al. (2025), an LLM behavioral-activation chatbot evaluated with the Quality of Behavioral Activation Scale (Q-BAS), a validated 14-item fidelity instrument, by ten psychotherapists across 48 sessions (JMIR preprint #94781; arXiv 2503.21540); see also TherapyGym (2026, arXiv 2603.18008) using the Cognitive Therapy Rating Scale (CTRS) plus a validation set of expert-rated dialogues. Pair this with standard manipulation checks.

Preregistration, open code, security, privacy/IRB. Preregister prompts and the fidelity-coding plan. Open-source deployment code. Keep API keys and system prompts server-side and inaccessible to participants. The Yale/Kalla recipe routes calls through AWS Lambda precisely so “users (survey participants) can only access our chat through the Lambda URL … They have no access to our API keys and lack access to our system prompts.” Use enterprise API agreements that do not train on submitted data, and pseudonymous participant IDs.

Reporting standards / checklists.

  • GUIDE-LLM: Feuerriegel, Barrie, Crockett, et al. (2026), “A reporting checklist for large language models in behavioural science,” Nature Human Behaviour, 10, 1182–1186, doi 10.1038/s41562-026-02492-7 (llm-checklist.com; sfeuerriegel.github.io/llm-checklist). A consensus-based reporting checklist for transparency, reproducibility, and ethical accountability across all stages of LLM-based research.

  • Abdurahman, Salkhordeh Ziabari, Moore, Bartels & Dehghani (2025), “A Primer for Evaluating Large Language Models in Social-Science Research,” Advances in Methods and Practices in Psychological Science (AMPPS), 8(2), 25152459251325174, doi 10.1177/25152459251325174. Good for methodological rigor, replicability, validity; explicit recommendations for both authors and reviewers. Also check: Abdurahman et al., 2024, “Perils and opportunities in using large language models in psychological research,” PNAS Nexus, 3(7), p. 245.

  • Randomness/reproducibility: Coqueret et al. (2026), “Randomness in large language models: What researchers need to know (and report),” arXiv 2607.24372: setting temperature to zero “removes deliberate sampling when that option is available, but it does not eliminate the other sources of randomness” (silent model updates, numerical rounding, expert routing); “exact reproduction is therefore generally not possible when using proprietary application programming interfaces.” See also the “background temperature” formalization (arXiv 2604.22411) on batch-size variation, kernel non-invariance, and floating-point non-associativity at nominal T=0.

3. Exemplar studies and deployment toolkits

Exemplar studies.

  • Boissin, Costello, Spinoza-Martín, Rand & Pennycook (2025), “Dialogues with large language models reduce conspiracy beliefs even when the AI is perceived as human,” PNAS Nexus, 4(11), pgaf325, doi 10.1093/pnasnexus/pgaf325 (preregistered, N = 955)

  • Hackenburg, Tappin, Hewitt, Saunders, Black, Lin, Fist, Margetts, Rand & Summerfield (2025), “The levers of political persuasion with conversational artificial intelligence,” Science, 390(6777), eaea3884, doi 10.1126/science.aea3884 (three experiments, N = 76,977 responses from 42,357 people; 19 LLMs; 707 political issues; 466,769 LLM claims fact-checked). Post-training boosted persuasiveness by as much as 51% and prompting by as much as 27%, dwarfing personalization and increasing model scale; strikingly, “where they increased AI persuasiveness, they also systematically decreased factual accuracy.” Direct implication for stimulus design: findings are highly specific to the prompt and model, so “the LLM” is not a stable stimulus.

Deployment toolkits.

  • DiSCoKit — Banks, Stromer-Galley, Singh & Capano (2026), “DiSCoKit: An Open-Source Toolkit for Deploying Live LLM Experiences in Survey Research,” preprint (arXiv 2602.11230; github.com/SHASTAlink/DiSCoKit). A daemon-like middleware app that brokers data among the survey platform, a database, and an LLM service, providing persistent transcript storage plus experimental control over AI behavior while remaining transparent to participants.

  • Simple Chat — Bermudez Schettino, Dasmeh & Brinkmann (2025), “Facilitating the Integration of LLMs Into Online Experiments With Simple Chat,” arXiv 2511.19123. Open-source chat interface for Qualtrics, oTree, and LimeSurvey; connects to commercial and open-weight models; streaming responses to preserve conversational flow; an administrative interface for fine-grained prompt/interface control; unified participant experience across conditions.

  • Oh, Calabrese, Zhang, Rasul & Xue (2026), “Embedding an AI Chatbot in Qualtrics: Chatbot Toolkit for Communication and Social Science Researchers,” SSRN 6780043 (doi 10.2139/ssrn.6780043; May 16, 2026). A self-contained chatbot inside a single Qualtrics question, no external server, configured by editing five clearly marked blocks in one JavaScript file; connects to OpenAI’s Chat Completions API, captures the full transcript as embedded data, and gates survey progression by keyword trigger, turn-count cap, or both.

  • Chen, Kalla, Le, Nakamura-Sakai, Sekhon & Wang (2026), “Embedding LLM Chatbots into Qualtrics Surveys” (joshuakalla.com/llm_persuasion); integrates Qualtrics with Anthropic’s Claude via AWS Bedrock (inference) and AWS Lambda (request handling), keeping keys/prompts server-side.

  • Other platforms worth noting: SMARTRIQS (Molnar, 2019, “A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys”); oTree-GPT (McKenna, 2023); G4R / GPT for Researchers (Kim, 2025); CLPC (Customizable LLM-Powered Chatbot, Lamprou & Moshfeghi, 2025, arXiv 2501.05541, which lets participants switch models mid-conversation); Talk2X (arXiv 2504.03343); OASIS (Open Agentic Survey Interview System); and Cloze (arXiv 2606.15033) for mental-health contexts with a non-removable safety floor. Note that oTree 6.0 Beta has begun adding native “web API” support for LLM calls, though it is not yet stable.

4. Remaining drawbacks and open methodological questions

Irreducible stochasticity with proprietary APIs. Even at temperature 0, frontier APIs are not deterministic: silent model updates, mixture-of-experts routing, numerical rounding, and batch/hardware effects inject variance (Coqueret et al., 2026, arXiv 2607.24372; “background temperature,” arXiv 2604.22411). Coqueret et al. document that at T=0 the output distribution “becomes much more concentrated but does not collapse to a single value.” Exact replication is generally impossible on proprietary APIs; open-weight, self-hosted models with fixed weights are the only route to true reproducibility (Abdurahman et al., 2025).

Construct-validity gap between prompt and realized conversation. The manipulation is the prompt; the treatment is the conversation. These are linked only stochastically, and each participant co-produces a different realization. The “From Prompts to Constructs: A Dual-Validity Framework for LLM Research in Psychology” (arXiv 2506.16697) and the analytic-flexibility critique (arXiv 2509.13397) both formalize how prompt-level manipulations may not map cleanly onto the intended construct and how the near-infinite dimensionality of prompt space creates researcher degrees of freedom.

Treatment heterogeneity and the estimand reframe. Fresh & Shin (2026), “What Is the Causal Effect of a Conversation? Estimands and Inference in AI-Mediated Conversations,” arXiv 2607.03597. Their central point, verbatim: the conversation “is generated jointly by the respondent and the conversational agent and is thus endogenous to who the respondent is,” so “random assignment to a conversational condition identifies the causal effect of assignment to that condition. However, it does not, without additional assumptions, identify the effect of the realized conversation or of the conversational content that the assignment is intended to induce.”

Model deprecation and silent updates threaten longitudinal replicability: the model you ran may not exist next year, and “the same” endpoint may behave differently.

Guardrail/safety-training conflicts with desired experimental behaviors: safety training can cause refusals or blunt a manipulation (e.g., an incivility, deception, or aggressive-persuasion condition), biasing the realized treatment toward the model’s aligned defaults and truncating the construct range you can study.

Generalizability across models and prompts is weak: Hackenburg et al. (2025) show effects swing widely by model and prompt, so a single-model, single-prompt finding may not transport. This is the empirical face of the estimand problem: you have estimated the effect of one policy, not of “AI conversation” in general.

Ethics. Deception about AI identity (WoZ-descended), persuasion/manipulation risk (Hackenburg et al.; the Yale democratic-risk framework), and the ELIZA-effect vulnerability of participants to anthropomorphic disclosure (Lucas et al., 2014) all require IRB attention, honest consent design, and careful debriefing.

The conceptual reframe: stimulus-as-distribution/policy, not stimulus-as-fixed-text. The oldest relevant statistics literature is directly applicable. Clark (1973), “The Language-as-Fixed-Effect Fallacy,” Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359, and Judd, Westfall & Kenny (2012), “Treating Stimuli as a Random Factor in Social Psychology: A New and Comprehensive Solution to a Pervasive but Largely Ignored Problem,” Journal of Personality and Social Psychology, 103(1), 54–69, argue that when stimuli are sampled, treating them as fixed inflates Type I error; the fix is crossed random effects for participants and stimuli. Applied to live LLMs, each realized conversation is a draw from the policy’s distribution; the statistically honest move is to model that variability - treat the realized stimulus as a random factor and use mixed-effects models - rather than pretend the treatment is a single fixed text. A subtle but important point: Judd et al. note that in the limiting case where every participant receives a unique stimulus set, the standard model is the statistically appropriate one, which is close to the live-LLM situation and forces you to state explicitly what population of stimuli/policies you intend to generalize over.

My Recommendations

Stage 1 — Decide whether you even need a live LLM. If your construct can be delivered as a frozen, pre-tested message, use “generate-then-freeze” (e.g., LLM-generated, validated, static stimuli per the 2025 SocArXiv message-stimuli paper). You keep clean manipulation and lose nothing except interactivity. Threshold to escalate to live deployment: your hypothesis concerns adaptive, multi-turn, participant-contingent behavior that a fixed text cannot instantiate (e.g., real-time rebuttal, personalized correction, relational dynamics).

Stage 2 — If live, engineer the policy for control. Hold the model constant across conditions; manipulate only the focal variable via condition-specific system prompts; pin the exact model version and decoding parameters; scope the task tightly (turn caps, bounded goal); pilot to characterize the output distribution; route through a server (Lambda-style) so keys/prompts are inaccessible; use a no-training enterprise API or an open-weight self-hosted model if replicability is paramount. Pick a toolkit: DiSCoKit or Simple Chat for multi-platform/open-weight flexibility; the Oh et al. single-question Qualtrics tool for the lightest lift; the Yale Bedrock recipe if you want managed cloud inference.

Stage 3 — Measure what you could not control. Log every transcript with participant ID and condition. Run manipulation checks AND a preregistered treatment-fidelity coding of transcripts (human coders and/or machine coding). Report the realized distribution of the manipulated feature, not just its intended level, and report refusal/guardrail rates.

Stage 4 — Analyze stimuli as random, not fixed. Fit mixed-effects models with random effects for the realized conversation/stimulus where you intend to generalize beyond the specific realizations (Judd, Westfall & Kenny, 2012). Frame your estimand explicitly as the effect of assignment to a conversational policy (Fresh & Shin, 2026); if you want message- or feature-level effects, state the additional identifying assumptions and design for them (e.g., randomize opening messages, or use message-level intervention regimes).

Stage 5 — Report to standard. Complete GUIDE-LLM (Feuerriegel et al., 2026) and follow the Abdurahman et al. (2025) primer; preregister prompts and the fidelity plan; open-source deployment code and (privacy permitting) transcripts; report access dates and acknowledge irreducible API stochasticity per Coqueret et al. (2026). Benchmark that would change the recommendation: if you can self-host an open-weight model with fixed seeds and batch-invariant kernels, you can credibly claim reproducibility and de-emphasize the stochasticity caveat; if you must use a proprietary API, treat every reported result as conditional on a model snapshot that may vanish.