Download PDFOpen PDF in browserAsk WhAI: Probing Belief Formation in Role-Primed LLM Agents17 pages•Published: April 19, 2026AbstractWe present Ask WhAI, a debugger for multi-agent LLM interactions that records and replays encounters, probes agent belief state out of band at encounter breakpoints, and injects counterfactual evidence to test belief revision under controlled perturbations.We integrate Ask WhAI with a medical case simulator in which role-primed specialist agents write to a shared, time-stamped electronic medical record (EMR) and query an oracle LabAgent that releases ground-truth results only when ordered. We stress-test the system on a synthetic multi-specialty diagnostic journey for abrupt-onset neuropsychiatric symptoms. Agents primed with strong role-specific priors (e.g., "act like a neurologist") interact with a moderator across sequential encounters; breakpoints enable pre- and post-event belief probes, allowing us to separate entrenched priors from evidence integration effects. Across controlled changes in probe framing, evidence exposure, and encounter order, we observe role-conditioned priors, resistance to counter-evidence, and order effects on belief trajectories. By separating belief probes from the clinical dialogue and enabling replay under targeted perturbations, Ask WhAI offers a reproducible way to study belief formation and epistemic silos in multi-agent reasoning. Keyphrases: agentic systems for scientific simulation, belief formation in foundation models, medical case simulation with ai agents, multi agent explainability and debugging, role based reasoning in diagnostic modeling, shared memory coordination in multi agent systems In: Jernej Masnec, Hamid Reza Karimian, Parisa Kordjamshidi and Yan Li (editors). Proceedings of AI for Accelerated Research Symposium, vol 3, pages 121-137.
|
