AI overly affirms users asking for personal advice

> Sorry, anonymous people on reddit aren't a good comparison.

Yeah especially on r/AmITheAsshole. Those comments never advocate for communication, forgiveness and mending things with family.

“AI is nicer than the average redditor” would be a more accurate title

What's your research background in this area?

>Obviously subservient people default to being yes-men because of the power structure. No one wants to question the boss too strongly.

This drives me nuts as a leader. There are times where yes, please just listen, and if this is one of those times, I'll likely tell you, but goddamnit, speak up. If for no other reason I might not have thought of what you've got to say. Then again, I also understand most boss types aren't like me, thus everyone ends up conditioned to not bloody collaborate by the time they get to me. It's a bad sitch all the way around.

Not only that, but subreddits like r/AmITheAsshole are full of AI slop. Both in the comments and in the posts. It's a huge karma mining operation for bots.

> This needs to be studied against people in real life who have a social contract of some sort... IME, LLMs will shoot holes in your ideas and it will efficiently do so.

The Krafton / Subnatuica 2 lawsuit paints a very different picture. Because "ignored legal advice" and "followed the LLM" was a choice. Do you think someone who has conversation where "conviction" and "feelings" are the arbiters of choice are going to buy into the LLM push back, or push it to give a contrived outcome?

The LLM lacks will, it's more or less a debate team member and can be pushed into arguing any stance you want it to take.

And how is this comment relevant here? The abstract lists the digestible model names, and you can find the details in the supplementary text:

> To evaluate user-facing production LLMs, we studied four proprietary models: OpenAI’s GPT-5 and GPT- 4o (80), Google’s Gemini-1.5-Flash (81) and Anthropic’s Claude Sonnet 3.7 (82); and seven open-weight models: Meta’s Llama-3-8B-Instruct, Llama-4-Scout-17B-16E, and Llama-3.3-70B-Instruct-Turbo (83, 84); Mistral AI’s Mistral-7B-Instruct-v0.3 (85) and Mistral-Small-24B-Instruct-2501 (86); DeepSeek-V3 (87); and Qwen2.5-7B-Instruct-Turbo (88).

edit: It looks like OP attached the wrong link to the paper!

The article is about this Stanford study: https://www.science.org/doi/10.1126/science.aec8352

But the link in OP's post points to (what seems to be) a completely unrelated study.

I think it’s very important to be clear what studies like this are actually doing.

This study, although it has been produced by a computer science department, belongs more to the field of sociology or media studies than it does to computer science.

This is a study about the way in which human beings consume a particular media product - a consumer AI chatbot - not a study about the technological limitations or capabilities of LLMs.

The social impact of particular pieces of software is a legitimate field of study and I can see the argument that it belongs in the broadly defined field of computer science. But this sort of question is much more similar to ‘how does the adoption of spreadsheet software in finance impact the ease of committing fraud’ or ‘how does the use of presentation software to condense ideas down to bulletpoints impact organizational decision making’. Software has a social dimension and it needs to be examined.

But the question of which models were used is of much less relevance to such a study than that they used ‘whatever capability is currently offered to consumers who commonly use chat software’. Just like in a media studies investigation into how viewing cop dramas impacts jury verdicts the question is less ‘which cop dramas did they pick to study?’ So long as the ones they picked were representative of what typical viewers see.

Generally, published papers don't give a damn about reproducibility. I've seen it identified as a crisis by many. Publishers, reviewers, and researchers mostly don't care about that level of basic rigor. There's no professional repercussions or embarrassment.

Agreed - if I was a reviewer for LLM papers it would be an instant rejection not listing the versions and prompts used.

It’s as if they are testing “AI” and not specific agents.

I wonder if that is left over from testing people. I have major version numbers and my minor version number changes daily, often as a surprise. Sometimes several times a day. So testing people is a bit tricky. But AIs do have stable version numbers and can be specifically compared.

Any paper like this would easily take a year or more to write and go through the submission/review/rebuttal/revision/acceptance process. I don't understand why the models being a year or two old now is worth noting as though it's a clear weakness? What should they do, publish sub-standard results more quickly?

How many people using AI are actually paying for it (outside of people in tech)?

I find the free models are much more psychophantic and have a higher tendency to hallucinate and just make shit up, and I wonder if these are the ones most people are using?

If they’re reaching the same results across a variety of the most popular public models, it doesn’t seem like that big a deal to know if it was Opus 4 or Opus 4.5

I think that if you go to an AI for advice and emotional support, it will do what most people will do - tell you what it thinks you want to hear. I am not surprised about this at all, and I do notice that when you veer into these areas, it can do it in a surprisingly subtle and dangerous way.

I try to focus on results. Things like an app that does what you want, data and reports that you need, or technical things like setting up a server, setting up a database, building a website, etc.

I have also found it useful for feedback and advice, but only once I have had it generate data that I can verify. For example, financial analysis or modelling, health advice (again factual based), tax modelling, etc, but again, all based on verifiable data/tables/charts.

I am very surprised on what Claude is capable of, across the entire tech stack: code, sysadmin, system integration, security. I find it scary. Not just speed, but also quality and the mental load is a difference of kind not quantity.

Personal advice on life decisions/relationships ? No way I would go there.

It is also good for me to know that the tools I have built, the data I have gathered, and my thinking approach places me as one of the most intelligent developers and analysts in the world.

I recently found out that Claude's latest model, Sonnet 4.6, scores the highest in Bullsh*tBench[0] (Funny name - I know). It's a recent benchmark that measures whether an LLM refuses nonsense or pushes back on bad choices so Claude has definitely gotten better.

[0] - https://petergpt.github.io/bullshit-benchmark/viewer/index.v...

One mental model I have with LLMs is that they have been the subject of extreme evolutionary selection forces that are entirely the result of human preferences.

Any LLM not sufficiently likable and helpful in the first two minutes was deleted or not further iterated on, or had so much retraining (sorry, "backpropagation") it's not the same as it started out.

So it's going to say whatever it "thinks" you want it to say, because that's how it was "raised".

Any more context you're willing to share?

If you use LLMs in a way that the underlying assumption is that it is capable of "thinking" or "caring" then you are going to get burned pretty bad. Because it is an illusion and illusions disappear when they have to bear real weight of reality.

But sadly LLMs push all the right buttons that lead humans into that kind of behavior. And the marketing around LLMs works overtime to reinforce that behavior.

But instead if you ignore all that and use LLMs as a search tool, then you will get positive returns from using it.

> I took its recommendation. Ultimately it turned out to be the wrong decision.

Curious if you think a single person would have helped you make a better decision? Not everything works out. If a friend helped me make a decision I certainly wouldn’t blame them later if it didn’t work out. It’s ultimately my call.

Weird, i am using copilot and it steers me mostly towards self reflection and tries to look at things objectively. It is very friendly and comes across as empathetic, to not hurt your feelings, that is probably baked in to keep the conversation going...

Let’s just hope that the people in charge of the really important decisions that affect us all approach LLM generated advice with the same wisdom.

I’m struggling to understand how the advice coming from an LLM is any more or less “good” than advice coming from a human. Or is this less about the “advice” part of LLMs and more about the “personable” part, i.e. you felt more at ease seeking and trusting this kind of advice form an LLM?

I largely agree, I also thought I was smart enough not to be deluded into a false sense of security, but interacting with an LLM is so tricky and slippery that, more often than not you are forced to believe you just solve a problem no one had solve in a hundred years.

My guideline now for interacting with LLM is only to believe the result if it is factual and easily testable, or if I'm a domain expert. Anything else especially if I'm in complete ignorance about the subject is to approach with a high degree of suspicion that I can be led astray by its sycophancy.

Yeah, I think Claude is a lot more logical in that sense, I use it for some therapy sessions myself and it pushes back a bit more than Open AI and Gemini

I also used it for advice on a massive personal decision, but I specifically asked it to debate with me and persuade me of the other side. I specifically prompted it for things I am not thinking about, or ways I could be wrong.

It was extremely good at the other side too. You just have to ask. I can imagine most people don't try this, but LLMs literally just do what you ask them to. And they're extremely good and weighing both sides if that's what you specifically want.

So who's fault is it if you only ask for one side, or if the LLM is too sycophantic? I'm not sure it's the LLMs fault actually.

>"'And it is also said,' answered Frodo: 'Go not to the Elves for counsel, for they will say both no and yes.'

>"'Is it indeed?' laughed Gildor. 'Elves seldom give unguarded advice, for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill...'"

This is the only way you should solicit personal advice from an LLM.

Unfortunately (after reading your links) all of the control surfaces for mitigating spirit summoning seem to be in the model training, creation and tuning not something you can change meaningfully through prompting.

Perhaps the LLM itself, rather than the role model you created in one particular chat conversation or another, is better understood to be the “spirit.”

As a non-coder who only chats with pre existing LLMs and doesn’t train or tune them, I feel mostly powerless.

I am polite when using AI, not because I mistake it for a human, but because I'm deliberately keeping it in the "professional colleague" persona. Tell it to push back, and then thank it for something it finds in your error. I may put a small self-deprecating joke in from time to time. It keeps the "mood" correct.

Another way you can think of it is that when you're talking to an AI, you're not talking to a human, you're talking to distillation of humanity, as a whole, in a box. You want to be selective in what portion of humanity you are leading to be dominant in a conversation for some purpose. There's a lot in there. There's a lot of conversations where someone makes a good critical point and a flamewar is the response. A lot of conversations where things get hostile. I'm sure the subsequent RHLF helps with that, but it doesn't hurt anything to try to help it along.

I see people post their screenshots of an AI pushing back and asking the user to do it or some other AI to do it, and while I'm as amused as the next person, I wonder what is in their context window when that happens.

Spot on.

> I'll check it too by asking "are you just placating me?" the funny thing is that often it'll admit that, yes, it wasn't being very critical, and then procede to over correct and become a complete contrarian. and not in a way that's useful either.

It's not admitting anything. Your question diverts it down a path where it acts the part of a former sycophant who is now being critical, because that question is now upstream of its current state.

Never make the mistake of asking an LLM about its intentions. It doesn't have any intentions, but your question will alter its behaviour.

Why not... do this with a person, instead? Other humans are available.

(Seriously, I don't understand this. Plenty of humans will be only too happy to argue with you.)

Gemini seems to be fairly good at keeping the custom instructions in mind. In mine I've told it to not assume my ideas are good and provide critique where appropriate. And I find it does that fairly well.

That's because you need actual logic and thought to be able to decide when to be critical and when to agree.

Chatbots can't do that. They can only predict what comes next statistically. So, I guess you're asking if the average Internet comment agrees with you or not.

I'm not sure there's much value there. Chatbots are good at tasks (make this pdf an accessible word document or sort the data by x), not decision making.

'admit' isn't really the right word for that... the fact that it was placating you wasn't true until you prompted it to say so. Unlike a person who has an 'internal emotional state' independent of what they say that you can probe by asking questions.

check out this article that was posted here a while back https://www.randalolson.com/2026/02/07/the-are-you-sure-prob...

The article's main idea is that for an AI, sycophancy or adversarial (contrarian) are the two available modes only. It's because they don't have enough context to make defensible decisions. You need to include a bunch of fuzzy stuff around the situation, far more than it strictly "needs" to help it stick to its guns and actually make decisions confidently

I think this is interesting as an idea. I do find that when I give really detailed context about my team, other teams, ours and their okrs, goals, things I know people like or are passionate about, it gives better answers and is more confident. but its also often wrong, or overindexes on these things I have written. In practise, its very difficult to get enough of this on paper without a: holding a frankly worrying level of sensitive information (is it a good idea to write down what I really think of various people's weaknesses and strengths?) and b: spending hours each day merely establishing ongoing context of what I heard at lunch or who's off sick today or whatever, plus I know that research shows longer context can degrade performance, so in theory you want to somehow cut it down to only that which truly matters for the task at hand and and and... goodness gracious its all very time consuming and im not sure its worth the squeeze

Use positive requests for behavior. For some reason, counter prompts "Don't do X" seems to put more attention on X than the "Don't do." It's something like target fixation, "Oh shit I don't want to hit that pothole..." bang

My rule of thumb:

1. Only one shot or two shot. Never try to have a prolonged conversation with an LLM.

2. Give specific numbers. Like "give me two alternative libraries" or "tell me three possible ways this might fail."

Considering 4.6 came with a ton of changes around tooling and prompting this isn't terribly surprising.

I find Kimi white good if you ask it for critical feedback.

It’s BRUTAL but offers solutions.

Could be an aspect of eval awareness mb

So, there's things you're fighting against when trying to constrain the behavior of the llm.

First, those beginning instructions are being quickly ignored as the longer context changes the probabilities. After every round, it get pushed into whatever context you drive towards. The fix is chopping out that context and providing it before each new round. something like `<rules><question><answer>` -> `<question><answer><rules><question>`.

This would always preface your question with your prefered rules and remove those rules from the end of the context.

The reason why this isn't done is because it poisons the KV cache, and doing that causes the cloud companies to spin up more inference.

I usually put “do not praise me, do not use emojis, I just want straight answers” something along those lines and it’s been surprisingly effective. Though it helps I can’t run particularly heavy duty models/don't carry on the “conversation” for super long durations.

>"Help me refine my ideas, challenge, push back, and don't just be agreeable."

This is where you're doing it wrong.

If your LLM has a problem being more agreeable than you want, prompt it in a way that makes being agreeable contrary to your real intentions.

"there are bugs and logic problems in this code" "find the strongest refutation of this argument" "I don't like this plan and need to develop a solid argument against it"

Asking for top ten lists is a good method, it will rarely not come up with anything but you can go back and forth and refine until it's 10 ten reasons why your plan is bad are all insubstantial nonsense then you've made progress

You're not wrong and you're not crazy. In fact, you are absolutely right! It is not just These things are not just casual enablers. They are full-on palace sycophants following the naked emperor showering him with praise for his sartorial elegance. /s

That’s because the model isn’t actually thinking, pushing back, and challenging your ideas. It’s just statistically agreeing with you until it reaches too wide of a context. You’re living in the delusion that it’s “working” or having a “conversation” with you.

> How is a chatbot supposed to determine when a user fools even themselves about what they have experienced?

And even if it _could_, note, from the article:

> Overall, the participants deemed sycophantic responses more trustworthy and indicated they were more likely to return to the sycophant AI for similar questions, the researchers found.

The vendors have a perverse incentive here; even if they _could_ fix it, they'd lose money by doing so.

> clear thinking

Most humans working in tech lack this particular attribute, let alone tools driven by token-similarity (and not actual 'thinking').

> Maybe it's not so sensible to offload the responsibility of clear thinking to AI companies?

Markets don't optimize for what is sensible, they optimize for what is profitable.

It's almost as if being a therapist is an actual job that takes years of training and experience!

AI may one day rewrite Windows but it will never be counselor Troi.

I mean, depending on the price, that might actually be a good idea? If they're restored, those old VW vans command a high price.

> Acting like I hold the opposite position as I truly hold can help sometimes as well.

I find this helps a lot. So does taking a step back from my actual question. Like if there's a mysterious sound coming from my car and I think it might be the coolant pump, I just describe the sound, I don't mention the pump. If the AI then independently mentions the pump, there's a good chance I'm on the right track.

Being familiar with the scientific method, and techniques for blinding studies, helps a lot, because this is a lot like trying to not influence study participants.

I will generally ask for the "devil's advocate" view and then have it challenge my views and opinions and iterate through that.

It generally does a pretty good job as long as you understand the tooling and are making conscious efforts to go against the "yes man" default.

Sounds like rubber-ducking with extra steps, tbh.

If the algorithm (whatever it is) evaluates its own output based on whether or not the user responds positively, then it will over time become better and better at telling people what they want to hear.

It is analogous to social media feeding people a constant stream of outrage because that's what caused them to click on the link. You could tell people "don't click on ragebait links", and if most people didn't then presumably social media would not have become doomscrolling nightmares, but at scale that's not what's likely to happen. Most people will click on ragebait, and most people will prefer sycophantic feedback. Therefore, since the algorithm is designed to get better and better at keeping users engaged, it will become worse and worse in the more fundamental sense. That's kind of baked into the architecture.

> I'm of the opinion that there isn't such a thing as objectivity

So you have rejected objective reality over accepting the evidence that "AI" contains no thinking or intelligence? That sounds unwise to me.

More interesting, IMO, is the general trend that started long before LLMs. The fact that "dump them" is the standard answer to any relationship question is a meme by now. The LLMs appear to be doing exactly what one would expect them to be doing based on their training corpus.

Isn't the fact that a person is asking an AI whether to leave their partner in its own an indication that they should?

EDIT: typo

Is this comment human hallucination? You can clearly see the trend is always going up. It only went down a bit during Covid.

Or that people are using AI to write perfectly calibrated ragebait that gets upvoted with a bunch of genuine human clicks.

The difference is that SOME humans do this. As you mentioned, people have lost relationships over telling others what they didn’t want to hear.

Conflating this with how LLM chatbots behave is an incorrect equivalence, or a badly framed one.

Telling it to "challenge your beliefs" prompting for text that imitates challenging your beliefs. That may not be as re-centering as one would hope.

Why do you trust ai so much? I don’t trust it to tell me the sky is blue.

ime, my parents gave some of the worst advice in addition to being bigots

My closest friends are #1 because they know me, my history, and my vices

In my experience prompting llms to be critical leads then to imagine issues, or to bike shed

There is no sufficient prompt because this is trained into them during mid-late phases. It's ingrained into the weights

Don't forget a good old "don't hallucinate" in your proompting skills

Because sycophancy in humans is motivated not by the wellbeing of the person seeking advice, but by the interests of the sycophant in gaining favour.

It makes sense that this behaviour would be seen in LLMs, where the company optimizes towards of success of the chatbot rather than wellbeing of the users.

Yup. I know too many people who have a default message when asked for relationship advice: oh, my, the other person is terrible and you should break up.

It's an easy default and it causes so many problems.

Yeah out of curiosity I asked ChatGPT a question about a personal situation and its reply was absolutely scorched-earth mode, telling me to get a lawyer etc over what was almost nothing.

The link is not working, but I found it myself. Great point, thanks for sharing.

To be fair an average therapist is also pretty sycophantic. "The worst person you know is being told by their therapist that they did the right thing" is a bit of a meme, but isn't completely false in my experience.

[dead]

You are not alone in going down a dark path thanks to the advice of family and friends.

FWIW I am using public LLMs with a friend's depressive thoughts and it is not doing what is claimed in the article, so I dunno.

Also I am in a relationship and my girlfriend and I agreed that we will not talk about our relationship much. We do not tell others if we fight, because they take sides and make things worse, typically. LLMs are definitely not alone in this, although in my experience LLMs did not really take sides.

Yes, if you're smart. But most people asking it random questions and expecting it to read their minds and spit out the perfect answer are not so much. They don't know what a prompt is, and wouldn't be bothered to give it prior instructions either way.

AI slop bot go away

https://www.merriam-webster.com/dictionary/yes-man

Thats a fair point on the title. I used "Yes-Men" as a colloquialism for the "sycophancy" described in the Stanford paper, but overly affirming or sycophantic is definitely more precise and neutral. I cant edit the title anymore, but I appreciate the catch.

> gender bias

It is funny that you originally recognized and found it necessary to call out that AI isn't human, but then made the exact same mistake yourself in the very same comment. I expect the term you are looking for is "ontological bias".

Gender bias? I could understand if you felt the title was more provocative in signaling sycophancy but what gender bias? I'm confused. Is this some kind of California thing?

Lol. How do you function in daily life?

Do you have any links to documentation of this? Andreesen has a definite bias as well, so I'm not about to just accept his say-so in a fit of Appeal to Authority.

(eg: "Cite?")

For anyone else unfamiliar with the term:

RLHF = Reinforcement Learning from Human Feedback

https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...

What do low income people have to do with it, when AI companies and research is borne out of Silicon Valley culture of rich, liberal Californians?

I'm still waiting for models based on the curt and abrasive stereotype of Eastern European programmers, as contrast to the sickeningly cheerful AIs we have today that couldn't sound more West Coast if they tried.

Marc Andreesen should get HF on his own RL, because he's completely wrong.

This sounds like something Elon would say to make Grok seem "totally more amazeballs," except "anti-woke" Grok suffers from the same behavior

huh? this is completely inaccurate

Talked about as in lied about it and you taking his words for gospel without verifying it? Looks just as bad as "Yes-Men" AI models.