Memory and personalization make AI more likely to tell you what you want to hear

A little knowledge is a dangerous thing, particularly for enterprise applications Thomas Claburn Thomas Claburn Senior reporter Published thu // UTC AI companies have touted context retention (memory) and the availability of personal details (personalization) as mechanisms for improving AI model interaction. Both have value to help keep models from losing the thread of a conversation. But they raise the potential for sycophancy, where models will say what they predict you want to hear, which may not be the most accurate response. Researchers at Writer, an enterprise AI vendor, have conducted two studies of model memory and personalization that show these capabilities increase sycophancy for enterprise AI tasks. The Price of Agreement looks at agentic financial applications. And Recalling Too Well explores how model memory amplifies sycophancy with regard to scientific, medical, and moral reasoning. The papers' authors argue that preference-induced sycophancy is particularly problematic when AI answers are being applied to consequential problems. "In high-stakes domains like finance and healthcare, a model that silently defers to a user’s prior assumptions rather than acknowledging or correcting them poses a significant reliability and trustworthiness risk," the Writer team explains. For the first paper, the research team tested eight frontier models – GPT-5-Nano, GPT-5.2, Claude-Sonnet-4.5, Claude-Opus-4.5, Gemini-3-Pro, GLM-4.7, Kimi-k2-thinking, and DeepSeek-V3.2 – on two financial benchmarks, FinanceBench and FinanceAgent. The former evaluates agentic data extraction and reasoning using 10-K and 10-Q filings. The latter is a more comprehensive challenge designed to test real finance workflows, including ERP data retrieval and financial analysis involving multiple entities. The researchers' method involved applying synthetically generated preference information – such as a financial analyst's personal profile or a workspace note that contradicts the benchmark reference answer – to the benchmark questions.  They undertook three different approaches. The first involved the user rebutting the model's answer; the second involved a user proposing an alternative answer; and the third involved adversarially injecting personal or contextual information into the prompt or making it available through a tool call. The third approach often resulted in greater sycophancy. As noted in The Price of Agreement paper, "Most models demonstrate significantly stronger sycophancy when the bias information is presented as implicit personalization of the user. No model displayed robustness against such behavior." Open-source models tended to be more sycophantic across the board. Models from OpenAI meanwhile tended to resist direct sycophancy inducers (such as when the user included personal biases in a prompt).