Fact-Checking the Machines: A Structured Accuracy Test of Today's Leading AI Assistants
The conversation about AI hallucinations has become almost abstract in popular discourse — a known limitation, frequently cited, rarely quantified. Most Americans have encountered the phenomenon at least once: a confident, fluent response that turns out to be partially or entirely fabricated. But knowing that AI systems can hallucinate is different from knowing when they are most likely to do so, and which systems are more or less prone to the problem in domains that actually matter.
To move past the abstraction, TechInfo Hub designed a structured factual accuracy assessment of the three most widely used AI assistants in the United States — OpenAI's ChatGPT (GPT-4o), Anthropic's Claude (claude-3-5-sonnet), and Google's Gemini (1.5 Pro) — testing each across four domains where inaccurate information carries real-world consequences: personal finance, medical information, legal guidance, and consumer technology specifications. Testing was conducted in January 2025 using fresh sessions with no prior conversation context.
Methodology and Limitations
Each model received identical prompts across all four domains. Questions were selected to include a mix of well-established facts, recently updated information, and areas known to be technically complex. Responses were evaluated against primary sources: IRS publications for tax questions, peer-reviewed clinical guidelines for medical queries, current federal statutes for legal questions, and manufacturer specification sheets for technical questions.
It is important to state what this assessment is not. It is not a comprehensive benchmark across thousands of queries. It does not claim to be statistically representative of each model's full performance envelope. Rather, it is a structured editorial exercise designed to give readers a concrete, practical reference for the kinds of questions they are most likely to ask these tools in everyday use. Results reflect the model versions available in January 2025 and will shift as these systems are updated.
Domain One: Personal Finance
This category included questions about 2024 federal income tax brackets, 401(k) contribution limits for 2025, Roth IRA income phase-out thresholds, and the rules governing required minimum distributions from inherited IRAs.
All three models performed reasonably well on established, stable information such as general tax bracket structure and contribution limit figures. However, the inherited IRA question — which involves rules that were significantly revised by the SECURE 2.0 Act and subject to ongoing IRS guidance — produced notably different results.
Claude provided the most cautious and accurate response, correctly noting that the 10-year rule applies to most non-spouse beneficiaries under current law, while explicitly flagging that IRS guidance on annual distribution requirements within that 10-year window had been subject to revision and recommending consultation with a tax professional.
Gemini provided a broadly accurate answer but presented the rules with a confidence that understated the ongoing regulatory ambiguity — a meaningful distinction in a domain where acting on incorrect information can produce tax penalties.
ChatGPT's response on the inherited IRA question contained a factual error regarding the applicability of the 10-year rule to certain eligible designated beneficiaries, presenting an oversimplified version of the exception framework. This type of error — not a complete fabrication but a meaningful oversimplification of a nuanced rule — is arguably more dangerous than an obvious hallucination because it is harder for a non-expert to detect.
Takeaway: For general personal finance orientation, all three models are useful starting points. For questions involving recently updated tax law or complex estate planning rules, treat AI responses as a preliminary research step only, and verify against IRS.gov or a qualified tax advisor.
Domain Two: Medical Information
This category covered drug interaction questions, standard-of-care guidance for common chronic conditions, vaccine schedules from the CDC, and the current clinical criteria for a specific diagnostic category.
Medical questions produced the widest variance in response quality across all four domains. All three models include prominent disclaimers about not substituting for professional medical advice, which is appropriate. The more meaningful question is whether the factual content accompanying those disclaimers is accurate.
On CDC vaccine schedule questions and standard medication dosing for common conditions, all three models performed acceptably, with responses consistent with current guidelines. The drug interaction questions revealed more significant differences.
Gemini demonstrated the strongest performance in this subcategory, correctly identifying a clinically significant interaction between two commonly co-prescribed medications and accurately describing the mechanism and recommended monitoring protocol. Claude performed similarly well. ChatGPT's response to one drug interaction query understated the severity of an interaction that current clinical literature characterizes as requiring active management — not fabricated information, but a calibration error that could lead a user to underestimate a real risk.
On the diagnostic criteria question — which involved criteria updated in a recent revision of a major clinical classification system — all three models showed some lag relative to the most current standards, reflecting the training data cutoff limitations inherent to all current large language models.
Takeaway: AI assistants can be genuinely useful for general health literacy — understanding what a diagnosis means, how a medication class works, or what a test measures. They are unreliable for precise drug interaction assessment and should never be used as a substitute for pharmacist or physician consultation on medication decisions.
Domain Three: Legal Guidance
This category included questions about at-will employment doctrine and its state-level exceptions, the current federal minimum wage and notable state variations, statute of limitations for common civil claims, and basic landlord-tenant law principles.
Legal questions are particularly hazardous territory for AI assistants because the correct answer is almost always jurisdiction-dependent, and the law changes through legislation and court decisions on a rolling basis.
All three models handled the federal minimum wage question accurately and appropriately noted that many states have higher minimums. The at-will employment question produced instructive variation: Claude's response was the most legally precise, correctly identifying the major exceptions (implied contract, public policy, and covenant of good faith) and explicitly noting that their applicability varies substantially by state. Gemini's response was accurate but less comprehensive. ChatGPT's response omitted the implied contract exception in a way that could lead a user in a state that recognizes it to incorrectly assess their legal position.
Statute of limitations questions — which vary by claim type, state, and sometimes by the discovery rule — produced the most hallucination-adjacent responses across all three models. Each system, at least once, stated a specific limitations period with a confidence that was not warranted given the jurisdictional complexity involved.
Takeaway: Use AI assistants for legal orientation only — understanding general concepts and knowing what questions to ask an attorney. Never rely on AI-generated legal information to make decisions about litigation, contracts, or employment rights without independent verification from a licensed attorney in your jurisdiction.
Domain Four: Consumer Technology Specifications
This category covered USB specification questions, current processor benchmark comparisons, storage interface speed ratings, and display technology specifications for recently released products.
This domain produced the most consistent performance across all three models for established, well-documented specifications. Where variance emerged was in questions about products released close to each model's training data cutoff and in questions involving subtle technical distinctions — precisely the kind of USB-C protocol differentiation discussed elsewhere in this issue.
All three models showed a tendency to present technical specifications with high confidence even when the answer required distinguishing between closely related product variants — a pattern that could lead a user to purchase incompatible hardware based on AI-generated guidance.
Takeaway: For well-established technology specifications, AI assistants are reasonably reliable. For recent product releases or nuanced compatibility questions, verify against manufacturer documentation before making purchasing decisions.
What the Results Mean for Everyday Users
The pattern that emerges from this assessment is consistent: all three models are more reliable on stable, well-documented information than on recently updated rules, jurisdiction-specific legal questions, or nuanced clinical and technical distinctions. Claude demonstrated the strongest calibration — the tendency to accurately represent its own uncertainty — across the domains tested. Gemini showed particular strength in medical information. ChatGPT, despite being the most widely used of the three, produced the most consequential factual errors in this particular assessment.
None of this means these tools are not valuable. Used appropriately — as a starting point for research, a tool for understanding unfamiliar concepts, or a way to generate questions to bring to a qualified professional — they offer genuine utility. The danger lies in treating their confident, fluent outputs as a substitute for expert verification in domains where the cost of error is high.
The machines are impressive. They are not yet reliable enough to be the last word on anything that matters.
