One of my kids recently had a no-contact knee injury while playing basketball. He immediately started limping and crying and I had to carry him from the court to the car.
I did some searching with Grok and I found out:
- no contact injuries are troubling b/c it generally means they pulled something
- kids don't generally tear an ACL (or other ligament)
- it's actually way more common for the ligament to pull the anchor point off of the bigger bone b/c kid bones are soft
I asked it to differentially diagnose the issue with the details of: can't hold weight, little to no swelling and some pain.
It was adamant, ADAMANT, that this was a classic case of bone being pulled off by the ligament and that it would require surgery. It even pointed out the no swelling could be due to a very small tear etc. It gave me a 90% chance of surgery too.
I followed up by asking what test would definitely prove it one way or the other and it mentioned getting an X-Ray.
We go off to the urgent care, son is already kind of hobbling around. Doctor says he seems fine, I push for an X-Ray and turns out no issue: he probably just pulled something. He was fully healed in 2-3 days.
As someone who has done a lot of differential diagnosing/troubleshooting of big systems (FinTech SRE) I find it interesting that it was basically correct in what could have happened but couldn't go the "final mile" to establish it correctly. Once we start hooking up X-Rays to Claude/Grok 4.2 etc equivalent LLMs, will be even more interesting to see where this goes.
I and many of my friends have used ChatGPT extremely effectively to diagnose medical issues. In fact, I would say that ChatGPT is better than most doctors because most doctors don't actually listen to you. ChatGPT took the time to ask me questions and based on my answers, narrowed down a particularly scary diagnosis and gave excellent instructions on how to get to a local hospital in a foreign country, what to ask for, and that I didn't have to worry very much because it sounded very typical for what I had. The level of reassurance that I was doing everything right actually made me feel less scared, because it was a pretty serious problem. Everything it told me was 100% correct and it guided me perfectly.
I was taking one high blood pressure medication but then noticed my blood sugar jumped. I did some research with ChatGPT and it found a paper that did indicate that it could raise blood sugar levels and gave me a recommendation for an alternative I asked my doctor about it and she said I was wrong, but I gently pushed her to switch and gave the recommended medication. She obliged, which is why I have kept her for almost 30 years now, and lo and behold, my blood sugar did drop.
Most people have a hard time pushing back against doctors and doctors mostly work with blinders on and don't listen. ChatGPT gives you the ability to keep asking questions without thinking you are bothering them.
I think ChatGPT is a great advance in terms of medical help in my opinion and I recommend it to everyone. Yes, it might make mistakes and I caution everyone to be careful and don't trust it 100%, but I say that about human doctors as well.
I agree that absolute deference to doctors is a mistake and that individuals should be encouraged to advocate for themselves (and doctors should be receptive to it) but I'm not so convinced in this specific case. Why do high blood sugar levels matter? Are there side effects associated with the alternative treatment? Has ChatGPT actually helped you in a meaningful way, or has the doctor's eventual relenting made you feel like progress has been made, even if that change is not meaningful?
In this context, I think of ChatGPT as a many-headed Redditor (after all, reddit is what ChatGPT is trained on) and think about the information as if it was a well upvoted comment on Reddit. If you had come across a thread on Reddit with the same information, would you have made the same push for a change?
There are quite a few subreddits for specific medical conditions that provide really good advice, and there are others where the users are losing their minds egging each other on in weird and whacky beliefs. Doctors are far from perfect, doctors are often wrong, but ChatGPT's sycophancy and a desperate patient's willingness to treat cancer with fruit feel like a bad mix. How do we avoid being egged on by ChatGPT into forcing doctors to provide bad care? That's not a rhetorical question, curious about your thoughts as an advocate for ChatGPT.
> Why do high blood sugar levels matter?
Are you asking why a side effect that is actually an entire health problem on its own, is a problem? Especially when there is a replacement that doesn’t cause it?
> most doctors don't actually listen to you.
> doctors mostly work with blinders on and don't listen
This has unfortunately been my experience as well. My childhood PCP was great but every interaction I've had with the healthcare system since has been some variation of this. Reading blood work incorrectly, ignoring explanations of symptoms, misremembering medications you've been taking, prescribing inappropriate medications, etc. The worst part is that there are a lot of people that reflexively dismiss you as a contrarian asshole or, even worse, a member of a reviled political group that you have nothing to do with just because you dare to disagree with what The Person With A Degree In Medicine said about your own health problems.
Doctors aren't immune to doing a bad job. I don't think it's a secret that the system overworks them and causes many of them to treat patients like JIRA tickets - I'd just like to know what it would take for people to realize that pointing this out doesn't make you a crackpot.
As an aside I use Claude primarily for research when investigating medical issues, not to diagnose. It is equally likely to hallucinate or mischaracterize in the medical domain as it is others.
ChatGPT for health questions is the best use case I have found (Claude wins for code). Having a scratch pad where I can ask about any symptom I might feel, using project memory to cross reference things and having someone actually listen is very helpful. I asked about Crohn's disease since my grandfather suffered from it and I got a few tests I could do, stats on likelihood based on genetics, diet ideas to try and questions to ask the doctor. Much better than the current doc experience which is get the quickest review of my bloods, told to exercise and eat healthy and a see you in six months.
I’ve heard many people say the same (specifically about ppl being better than doctors because they listen) and I find it odd and wonder if this is a specific country thing?
I’ve been lucky enough to not need much beyond relative minor medical help but in the places I’ve lived always found that when I do see a GP they’re generally helpful.
There’s also something here about medical stuff making people feel vulnerable as a default so feeling heard can overcompensate the relationship? Not sure I’m articulating this last point well but it comes up so frequent (it listened, guided me through it step by step etc.) that I wonder if that has an effect. Feeling more in control than a doctor who has other patients and time constraint just say it’s x or do this
+100 to this from my personal experience
No, no, no. You can change your doctor, and get one that listens to you - you can't change the fact that ChatGPT has no skin in the game - no reputation, no hippocratic oath, no fiscal/legal responsibility. Some people have had miracles with Facebook groups, or WebMD, but that doesn't change where the role of a doctor is or mean that you should be using those things for medical advice as opposed to something that allows you to have an informed conversation with a doctor.
Neither do most doctors. No gp will get disbarred for giving the wrong diagnosis on a first consult.
They have 15 minutes and you have very finite money.
Medical agents should be a pre consult tool that the patient talks to in the lobby while waiting for the doctor so the doctor doesn't waste an hour to hear the most important data point and the patient doesn't sit for an hour in the lobby doing nothing.
Doctors have no skin in the game too. Our society is built on the illusion of 'skin in the game' of professionals like doctors and lawyers (and to a lesser extent, engineers), but it's still an illusion.
I don't know if I could trust AI for big things, but I had nagging wrist pain for like a year, any time I extended my wrist (like while doing a pushup). It wasn't excruciating but it certainly wasn't pleasant, and it stopped me from doing certain activities (like pushups)
I visited my GP, 2 wrist specialists, and physical therapist to help deal with it. I had multiple x rays and an MRI done. Steroid injection done. All without relief. My last wrist specialist even recommended I just learn to accept it and don't try to extend my wrist too much.
I decided to ask Gemini, and literally the first thing it suggested was maybe the way I was using the mouse was inflaming an extensor muscle, and it suggested changing my mouse and a stretch/massage.
And you know what, the next day I had no wrist pain for the first day in a year. And it's been that way for about 3 weeks now, so I'm pretty hopeful it isn't short term
And why should anyone trust you?
> she said she was aware that DeepSeek had given her contradictory advice. She understood that chatbots were trained on data from across the internet, she told me, and did not represent an absolute truth or superhuman authority
With highly lucid people like the author's mom I'm not too worried about Dr. Deepseek. I'm actually incredibly bullish on the fact that AI models are, as the article describes, superhumanly empathetic. They are infinitely patient, infinitely available, and unbelievably knowledgeable, it really is miraculous.
We don't want to throw the baby out with the bathwater, but there are obviously a lot of people who really cannot handle the seductivity of things that agree with them like this.
I do think there is pretty good potential in making good progress on this front in though. Especially given the level of care and effort being put into making chatbots better for medical uses and the sheer number of smart people working on the problem.
If the computer is the bicycle of the mind, GenAI is a motor vehicle. Very powerful and transformative, but it's also possible to get into trouble.
Well yes, but as an extremely patient person I can tell you that infinite patience doesn't come without its own problems. In certain social situations the ethically better thing to do is to actually to lose your patience, may it be to shake the person talking to you up, may it be to indicate they are going down a wrong path or whatnot.
I have experience with building systems to remove that infinite patience from chatbots and it does make interactions much more realistic.
I'm reminded of the monologue from Terminator 2:
> Watching John with the machine, it was suddenly so clear. The Terminator would never stop, it would never leave him... it would always be there. And it would never hurt him, never shout at him or get drunk and hit him, or say it couldn't spend time with him because it was too busy. And it would die to protect him. Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice.
The AI doctor will always have enough time for you, and always be at the top of their game with you. It becomes useful when it works better than an overworked midlevel, not when it competes with the best doctor on their best day. If we're not there already, we're darn close.
If you get the "You're absolutely right!" response from an LLM that screwed up on a field you're familiar with and still let them play with your health, you're...courageous to say the least.
I prepared for a new patient appointment with a medical specialist last week by role playing the conversation with a chatbot. The bot turned out to be much more responsive, inquisitive and helpful. The doctor was passive, making no suggestions, just answering questions. I had to prompt him explicitly to get to therapy recommendations, unlike the AI. I was glad that I had learned enough from the bot to ask useful questions. It would have been redundant at best if the doctor was active and interested, but that can't be depended on. This is standard procedure for me now.
This was not what I was expecting. The doctors I know are mostly miserable; stuck between the independence but also the burden of running their own practice, or or else working for a giant health system and having no control over their own days. You can see how an LLM might be preferable, especially when managing a chronic, degenerative condition. I have a family member with stage 3 kidney disease who sees a nephrologist, and there's nothing you can actually do. No one in their right mind would recommend a kidney transplant, let alone dialysis for someone with moderately impaired kidneys. All you can do is treat the symptoms as they come up and monitor for significant drops in function.
For major medical issues it may well be best practice to use the four eyes principle like we do for all safety related systems. Access is key and at this time getting a second pair of eyes in close timely proximity is a luxury few have and even fewer will have looking at the demographics in the developed world. Human doctors are failable as is AI. For the time being having a multitude of perspectives may well be the best in most cases.
The problem is not reliance on AI but that the AI is not ready yet and using general-purpose models.
There isn't simply enough doctors to go around and the average one isn't as knowledgeable as you would want. Everything suggests that when it comes to diagnosis ML systems should be better in the long run on average.
Especially with a quickly aging population there is no alternative if we want people to have healthcare on a sensible level.
They really have arrived just in time, I think.
Considering how difficult it is to get patients to talk to doctors, using AI can be a great way to get some suggestions and insight _and then present that to your actual doctor_
The dangers are obvious (and also there are some fascinating insights into how healthcare works practically in China). I wonder if some kind of "second opinion" antagonistic approach might reduce the risks.
Medical Advice Generative Adversarial Networks would be a good idea.
I see some of this adversarial second-guessing introspection from Claude sometimes. ("But wait. I just said x y and z, but that's inconsistent with this other thing. Let me rethink that.")
Sometimes when I get the sense that an LLM is too sycophantic, I'll instruct it to steelman the counter l-argument, then assess the persuasiveness of that counter-argument. It helps.
Reminds me of an excellent paper I just read by a former Google DeepMind Ethics Research Team member
https://www.mdpi.com/2504-3900/114/1/4 - Reinecke, Madeline G., et al. "The double-edged sword of anthropomorphism in llms." Proceedings. Vol. 114. No. 1. MDPI, 2025 Author: https://www.mgreinecke.com/
Worriesome for sure.
However I would say that the cited studies are somewhat outdated already compared e.g. with GPT-5-Thinking doing 2mins of reasoning/search about a medical question. As far as I know Deepseeks search capabilities are not comparable and non of the models in the study spend a comparable amount of compute answering your specific question.
How much nvidia u holding bro
This almost certainly isn't only a China problem. I've observed UK users asking questions about diabetes and other health advice. We also have an inexpensive (free-at-point of use for most stuff) but stretched healthcare system. Doubtless there are US users looking at the cost of their healthcare and resorting to ChatGPT instead too.
In companies people talk about Shadow-IT happening when IT doesn't cover the user needs. We should probably label this stuff Shadow-Health.
To some extent, the deployment of a publicly funded AI health chat bot, where the responses can be analysed by healthcare professionals to at least prevent future harm is probably significantly less bad than telling people not to ask AI questions and consult the existing stretched infrastructure. Because people will ask the questions regardless.
The joke of looking symptoms up on WebMD and determining you have cancer has been around for... geez over 20 years now. Anti-vaccine sentiment mostly derived from Facebook. Google any symptom today and there are about 10 million Quora-esque websites of "doctors" answering questions. I'm not sure that funneling all of this into the singular UI of an AI interface is really better or worse or even all that different.
But I do agree that some focused and well funded public health bot would be ideal, although we'll need the WHO to do it, it's certainly not coming from the US any time soon.
Access trumps everything else. A doctor is fine with you dying while you wait on his backlog. The machine will give you some wrong answers. The mother in the story seems to be balancing the concerns. She has become the agent of her own life empowered by a supernatural machine.
> She understood that chatbots were trained on data from across the internet, she told me, and did not represent an absolute truth or superhuman authority. She had stopped eating the lotus seed starch it had recommended.
The “there’s wrong stuff there” fear has existed for the Internet, Google, StackOverflow. Each time people adapted. They will adapt again. Human beings have remarkable ability to use tools.
> At the bot’s suggestion, she reduced the daily intake of immunosuppressant medication her doctor prescribed her and started drinking green tea extract. She was enthusiastic about the chatbot
I don't know enough about medicine to say whether or not this is correct, but it sounds suspect. I wouldn't be surprised if chatbots, in an effort to make people happy, start recommending more and more nonsense natural remedies as time goes on. AI is great for injuries and illnesses, but I wonder if this is just the answer she wants, and not the best answer.
med student here: Reducing immunosuppressant not something to be taken lightly for kidney transplant patients. I was shocked when I read that sentence.
As soon as the model detects user pleasure at not needing a scary surgery (especially if you already confided you're scared) then it'll double down on that line of thinking to please the user.
A sick family member told me something along the lines of, "I know how to work with AI to get the answer." I interpret that to mean he asks it questions until it tells him what he wants to hear.
Indeed, real doctors have the advantage of understanding how to treat humans that are incapacitated. =3
Just seeing that guy’s face and hearing his voice makes me uneasy. That channel is total body-horror. Glad that guy ended up okay, unlike this poor soul:
Indeed, the instinctual pucker factor for some things in this world are warranted. That case is famous, and still controversial to those with inflated hubris. =3
I think the article can basically be summed up as "GenAI sychophancy should have a health warning similar to social media". It's a helluva drug to be constantly rewarded and flattered by an algorithm.