I recently pasted only the abstract of a paper into ChatGPT.

I assumed the tables and figures copied too.

But they were actually images.

Since only the legends (“Table 1, Table 2…”) were present, ChatGPT did what it does best: predict.

It invented entire datasets that never existed.

That was the day I realized: hallucinations aren’t random—they’re predictable failure modes.

So why do hallucinations happen?

Think back to a childhood game. You whisper a word to a friend, they pass it to another, and so on down the line. By the time it reaches the last person, the word is often distorted beyond recognition.

ChatGPT works in a similar way. It’s a word prediction machine. Each token builds on the last. If an error slips in, the next word compounds it, and the chain quickly spirals away from the truth.

This is, of course, a simplification. But the core idea holds: hallucinations are baked into how large language models operate. Eliminating them completely isn’t realistic—but managing them is.

A new research paper from OpenAI examines why this happens and whether hallucinations can be reduced.

The paper describes that a major reason for hallucinations is structural: models are rewarded for confident guessing rather than honest uncertainty. Evaluations make this worse. Current benchmarks treat outputs like multiple-choice exams where guessing is rewarded and saying “I don’t know” gets you zero.

The paper’s proposed fix: Penalize confident errors more heavily than uncertainty, and reward appropriate expressions of doubt.

That’s useful guidance for AI engineers. But for us, the researchers using ChatGPT in day-to-day work, the more important question is: What can we do right now to mitigate hallucinations?

Here are 5 simple strategies I use to keep ChatGPT honest in research 👇

1️⃣ Tell the AI to validate itself

The first guardrail is to remind the model to double-check its own output.

AI models are designed to sound confident even when they’re wrong. One wrong number in a results section snowballs into a cascade of errors.

So I now explicitly ask it to validate.

Example:

When drafting a Results section, I now append:

“Validate all numbers against the input. If any are missing, say so instead of making them up.”

Prompt (Persona(l) GOAL):

• P: Act as a meticulous research editor.
• G: Draft the Results section using ONLY the data inside <…>. Validate all numbers, and if any are missing, flag as [MISSING]. End with a mismatch-check list.
• O: Results section text followed by a validation table.
• A: Do not invent numbers, percentages, or statistics not in the input.
• L: Context is clinical research results.

This small nudge forces the model to pause and compare, rather than guess.

Does it catch every issue? No. Maybe 999 times it works, but once in a while it will still sneak in a hallucination. Which is why you must also validate numbers yourself. But the self-check step eliminates the majority of slip-ups.

2️⃣ Double-check high-risk areas yourself

Some areas are just more prone to hallucinations. Citations are at the very top of that list.

Even with search enabled, ChatGPT tends to get citation details wrong.

Titles are often correct because they appear directly in the text or search snippets. But the rest—authors, journal names, years—often get predicted instead of retrieved.

Here’s how it happens:

If ChatGPT sees an arthritis machine learning paper, it may insert the name of a well-known arthritis AI researcher (I being one of them) into the citation, even if that person had nothing to do with the study. It’s not making up names out of thin air—it’s substituting “likely suspects.”

This is pattern-based substitution, and it makes references look convincing. That’s what makes them dangerous.

Prompt (Persona(l) GOAL):

• P: Act as a reference manager.
• G: Extract citation leads ONLY from <<>> and output them as a table: [Title] | [Authors] | [Journal] | [Year] | [DOI]. If any field is missing, write [NOT IN TEXT]. End with a reminder to verify manually.
• O: A structured citation table with explicit [NOT IN TEXT] flags.
• A: Do not guess or invent citation details.
• L: Context is scientific manuscripts.

While this prompt mitigates this issue, some slip ups still happen. Which is why you must still verify all citations manually before they go into a draft.

This was a big issue while building research boost, and we had to build a pipeline for verifying each reference one by one programmatically. You can check out the output yourself HERE: https://researchboost.com/

3️⃣ Control the input scope

Here’s something I learned the hard way.

Hallucinations are inversely related to the amount of information you feed in.

Too little context? The model fills in the blanks. Too much? It drops chunks silently.

One day, I pasted in the abstract of a paper. I thought the tables and figures had copied over, but they were actually images. The only thing present were table legends like “Table 1” and “Figure 2.”

ChatGPT didn’t stop. It invented entire datasets to match the missing tables. Confidently.

That’s what happens when you give it too little.

But the opposite problem is just as bad. If you paste in 100 pages, the model only keeps what fits in its “context window.” Anything beyond that is ignored—and once again, it starts guessing.

[A word of caution: although context windows have grown significantly (e.g., ChatGPT has a context window of 128k tokens, roughly 300-400 pages of text), the model still may lose some information from anything you put that is more than 5 pages or so. ]

The trick is to chunk your inputs. Process manuscripts step by step: summarize section by section, then draft based only on what’s inside each chunk. Never mix. Never assume.

Prompt (Persona(l) GOAL):

• P: Act as a structured summarizer.
• G: Process manuscripts in steps. Step 1: Summarize <<>> into bullet points. Step 2: Summarize <<>>. Step 3: Draft Results ONLY from <<>>. Do not mix content across chunks.
• O: Bulleted summaries for each section, followed by a chunk-specific draft.
• A: Do not add facts from outside the current chunk.
• L: Context is long academic manuscripts.

4️⃣ Watch for “pattern-based” substitutions

This one is subtle.

ChatGPT is great at recognizing patterns—but sometimes, it cares more about the pattern than the truth.

For instance, if you feed it a dataset summary that says:

“40% of patients improved, 30% had no change, 20% worsened.”

The model might “balance the pattern” by making the percentages add to 100%, even if the input was incomplete. If you had only given two numbers (say 40% and 30%), it may confidently invent the missing third category so that the pattern feels “complete.”

That’s not random—it’s substitution based on what should be there.

To counter this, I often run a second pass asking the AI.

Prompt (Persona(l) GOAL):

• P: Act as a cautious fact-checker.
• G: Review <<>> and flag any elements that appear inferred rather than explicitly present (e.g., numbers balanced to 100%, missing categories filled in, assumed details). Mark these as [LIKELY SUBSTITUTION] and explain why.
• O: A list of flagged outputs with short explanations of what seems inferred.
• A: Do not smooth over uncertainty and do not invent replacements. Only flag.
• L: Context is clinical research datasets and manuscripts.

When the model highlights what it filled in, it becomes easier to spot these false patterns before they sneak into a draft.

5️⃣ Use structured prompting

Finally, vague prompts are an open invitation for hallucinations.

If you just say “Write the Results,” the model has too much freedom. It will smooth over gaps with invented content.

Good prompt structure is the fix here.

I use my Persona(l) GOAL framework to nail down exactly what role the model should play, what output format I need, and what constraints matter most.

Prompt (Persona(l) GOAL):

• P: Act as a peer-review-ready research editor.
• G: Draft a manuscript section ONLY from <…>. After drafting, create a table comparing each number in the draft to <…> with [Match/Mismatch].
• O: A structured draft plus a verification table.
• A: Don’t create new numbers or citations.
• L: This is a PsA prevalence study from the UK THIN database.

Rule of thumb:

More structured, relevant input = fewer hallucinations.

But no matter how careful you are, once you push past the context window, contradictions creep back in.

👉 What’s the worst hallucination you’ve caught in your own work—and which of these 5 guardrails would’ve prevented it?

PROMPT OF THE WEEK

Research Meeting Preparation Framework

Persona(l): Act as my executive assistant/chief of staff. Be concise, anticipatory, and protective of my time. Challenge assumptions, surface risks, and prepare me so I never walk in blindsided.

Goal: Using <<EMAIL>> and <<PRIOR_CONTEXT>> (summaries/notes from past manager/team discussions), prep me for the next meeting about <<MEETING_CONTEXT>> by delivering:

 1. A 5-line Executive Summary of the selected email.

 2. Meeting Prep Notes: key points, open threads, and 5–7 questions I should raise.

 3. Action Items with owner, what/why, due date, and status.

 4. Team/Manager Alignment: prior agreements, decisions, and dynamics to remember.

 5. Red Flags: risks, tensions, gaps, plus proposed mitigations.
Also propose a tight agenda (30–60 min) and list pre-reads (from the thread/context only).

Output format: Output should be decision-ready, 300–450 words, and written like a chief of staff equipping an NIH-funded PI for a high-stakes discussion. Fromat is as one-page brief with these sections (use bullets, bold labels, and a table):

 - Executive Summary

 - Meeting Prep Notes

 - Action Items (table: [Item] | [Owner] | [Due] | [Status] | [Dependency])

 - Team/Manager Alignment

 - Red Flags & Mitigations

 - Proposed Agenda (timeboxed)

 - Pre-reads & Docs (links/titles found in the materials)

Avoid: Do not invent facts. If a detail isn’t present in <<EMAIL>> or <<PRIOR_CONTEXT>>, write [NOT IN MATERIALS]. Quote dates/times exactly and include day-of-week. Flag any contradictions across sources. Keep tone direct and practical. No outreach or scheduling—analysis only.

Lens of Context: Academic medicine/clinical research lab or leadership meetings. 

P.S. Research Boost AI is live. We have done everything to make sure that all citations are real and high-quality and verify all claims before providing the output.

Sign up here to get 5,000 words FREE today:

https://researchboost.com/