Demand for healthcare AI is real, but the questions people actually search reveal what concerns them: Are AI chatbots HIPAA compliant? Can you trust ChatGPT for medical advice? Is it safe to put patient information into a chatbot? These are the due-diligence questions asked by teams scoping a genuine deployment. Below are the three risks that matter most, and how to design around them.
Risk 1: Hallucination - when the chatbot invents an answer
Why shouldn't you use ChatGPT for medical advice? Because a general-purpose LLM generates plausible-sounding text without knowing whether it is true. Ask it about a drug interaction or a symptom and it will answer confidently regardless of whether it has any basis for doing so. Two error types are especially dangerous in a clinical setting: false reassurance, where the system tells a patient not to worry when they should, and fabricated instruction, where it cites a dose, guideline or eligibility rule that does not exist.
Addressing this requires an architectural solution rather than a refined prompt. The most reliable medical chatbots do not allow the model to generate clinical answers freely. They match each query against a verified, curated knowledge base, and where no trusted answer exists they say so and escalate rather than guess. A system that responds with "I don't have a verified answer; let me connect you to a clinician" is behaving correctly.
Risk 2: Privacy - PHI, HIPAA and what the model does with your data
Are AI chatbots HIPAA compliant? Not by default. The consumer tiers of tools such as ChatGPT (Free, Plus, Pro, Team and Business) are not HIPAA-eligible, and entering protected health information (PHI) into them can constitute a breach. Only specific tiers support compliant use: ChatGPT Enterprise, Edu and the dedicated ChatGPT for Healthcare product, or OpenAI's API, each requiring a signed Business Associate Agreement (BAA). The same pattern holds across the major providers, with Anthropic, Microsoft, Google and AWS all offering BAAs on their enterprise or API products but not on consumer tiers. A BAA is a contract making the vendor legally accountable for safeguarding PHI, but it covers only the vendor's obligations. The organisation remains the covered entity, responsible for how the tool is configured, who can access it, and what staff are permitted to enter.
This is a common point of failure in healthcare deployments. Does ChatGPT train on your data? Depending on the tier and settings, inputs may be retained or used for training, which is unacceptable for patient records. Is it safe to put patient information into a public chatbot? No. The safe pattern is an architecture in which data never leaves a controlled, encrypted environment and is never routed to third-party APIs. Retrofitting privacy onto a system built around external APIs is both costly and fragile, so data sovereignty should be designed in from the outset.
Risk 3: No human oversight. Guardrails and the human-in-the-loop
What are AI guardrails, and what is human-in-the-loop AI? Guardrails are the rules and limits that prevent a chatbot from acting outside its safe scope. Human-in-the-loop means a person reviews or takes over before any high-stakes decision is finalised. In healthcare, the most damaging failures occur when a chatbot handles a case it should have referred onward — a distressed patient, an ambiguous symptom, or a question that requires clinical judgement.
A well-designed system monitors the entire conversation and detects when a human is needed, regardless of whether it technically knows the answer. It should also be monitored after launch. Tracking accuracy, escalation rates and edge cases is essential, because a model that performs well at launch can degrade over time.
How we solve this - and why the same principles apply across regulated sectors
At PQ Impact, we build systems to address exactly these problems. For TaxNav, an HMRC-recognised tax platform, we designed and built AI4U — an AI support system that automates the majority of customer queries with zero hallucinations, because it answers from a verified knowledge base rather than a free-generating model. User data stays within a self-contained, encrypted environment and is never sent to external APIs. When the system cannot answer, it escalates: raising a ticket, notifying the team and providing a full conversation summary, so that the agent begins with full context.
Tax and medicine differ in their detail, but the underlying constraints are the same. The cost of an incorrect answer is high, the data is sensitive, and the subject matter is both nuanced and regulated. The same architecture applies equally to finance, insurance, legal and clinical software. We have built knowledge-base-driven assistants that resolve routine queries instantly, intelligent escalation that supports human experts rather than replacing them, and private deployments for organisations that cannot allow data to leave their control. In most cases the knowledge base is generated automatically from existing documentation, so organisations with reasonable help content are often closer to deployment than they expect.
How to choose a healthcare AI vendor
The search data is clear that buyers are right to lead with risk. When evaluating a medical AI chatbot, four questions are worth asking the vendor. Where does our data live, and will you sign a BAA? Does the system generate answers or retrieve verified ones? What happens when it does not know the answer? And how will it be monitored after deployment? A vendor unable to answer these questions clearly should be treated with caution.
Adding AI without a clear plan introduces unnecessary risk. The safer approach is to define the failure mode first, design for privacy and accuracy from the outset, and treat AI as a tool that supports clinicians and support teams rather than replacing their judgement. Handled this way, a chatbot can become a reliable asset in a sector where safety is non-negotiable.
Considering AI for your healthcare or regulated product? Contact PQ Impact to discuss support automation, onboarding or internal tools built for accuracy, privacy and compliance from day one.
