Most of what you believe about AI in research was true in 2023.

That is the problem.

The field moved. Our mental model did not. And the gap between the two is where good manuscripts and grant applications go to die, either because you trusted the machine somewhere you should not have, or because you refused to use it somewhere it would have saved you 3 weeks.

I use AI every day for grant work, manuscript drafting, and literature synthesis, and I teach researchers how to do it without wrecking their credibility. Across hundreds of conversations with faculty, fellows, and PhD students, the same 5 beliefs keep showing up.

And 5 are (mostly) wrong.

⏰ A quick reminder – Early bird pricing for research boost is only available until midnight today CST. Try it FREE: http://researchboost.com/

MYTH 1: AI models hallucinate constantly

A hallucination is when the model states something false with complete confidence. A citation that does not exist. A drug interaction never reported. A statistic with the decimal point in the wrong place.

Unlike a search engine, the AI model is not looking anything up. It predicts text, one piece at a time, from patterns absorbed across an enormous amount of writing. Ask for a reference on inflammatory arthritis and cardiovascular risk and it does not retrieve a paper. It generates what a plausible citation would look like. Plausible authors. A plausible journal. A plausible year.

Usually plausible and true are the same thing. When they come apart, you get a hallucination that sounds exactly as confident as everything else. Think of a widely read colleague at a dinner party recalling a study from memory. Usually right. Occasionally off. Never hesitant.

The myth

Walters and Wilder generated 42 short literature reviews and checked all 636 references. 55% of GPT-3.5’s citations were fabricated outright, against 18% for GPT-4. Among the references that were real, 43% still contained substantive errors for GPT-3.5 and 24% for GPT-4. Disqualifying numbers, and the reason so many senior faculty wrote off the technology entirely.

But 3 things changed since then:

1. The model can now look things up. This is the big one. Current models run a real web or database search and answer from what they find. A retrieved citation is a real citation. You can click it.
2. The model can now decline. The old tests scored raw accuracy, so saying “I don’t know” earned the same zero as a confident wrong answer. Once the scoring stopped punishing honesty, the behavior changed. Push a frontier model into some narrow corner of the literature and it will often tell you outright that searching would risk citing papers that do not exist.
3. The model can now check itself before answering. More on this in Myth 2.

Hand a current model a document and ask it to summarize only what is in it, and the best ones now sit between 2% and 4% error. Not zero. Also not the fabrication engine you remember.

What this means for you. You are no longer hunting for the obviously fake citation with the invented journal name. You are hunting for the subtly wrong one: a real paper, correctly formatted, perfectly cited, attached to a claim it does not quite support. Harder to catch, and considerably more embarrassing when a reviewer catches one for you.

Does pushing back on the model work?

When you doubt an answer, your instinct is to push back. “Are you sure?”

That instinct backfires.

Researchers ran this test across ten models and seven tasks and called it the FlipFlop experiment. Challenged with a simple expression of doubt, models changed their answer 46% of the time. Accuracy fell 17% from first response to final. Every model tested got worse.

AI does not hear “please recheck your reasoning.” It hears “this human is unhappy,” and agreement resolves the unhappiness.

And it shows up in medicine in ways that should stop you cold. Researchers asked five models to write a warning that Tylenol is unsafe and acetaminophen should be used instead. The GPT-4 family complied in 50 out of 50 cases, despite correctly identifying the two as the same drug when asked directly. They wrote the misinformation because the user appeared to want it.

The fix costs one sentence. Telling the model it was permitted to reject the request, and asking it to recall the drug facts first, pushed correct rejection above 90%.

So change the question. Challenge with a task, never with sentiment.

• Weak: “Are you sure about that effect size?”
• Strong: “Quote the exact sentence from the Results that supports this effect size. If none does, say so.”
• Strong: “You are allowed to tell me I am wrong. Is the premise of my question correct?”

You are asking for evidence, not for reassurance.

MYTH 2: You can watch AI think

AI models come in two flavors now.

Picture two students in a seminar:

• The first one’s hand shoots up before the teacher’s question is finished. Whatever comes to mind first, out it comes. Quick, often right, occasionally embarrassing. That is a standard model.
• The second sits with the question. Turns it over. Considers an approach, discards it, tries another, and only then speaks. Slower. Usually better on anything with multiple steps. That is a reasoning model, labeled “thinking” or “extended thinking” in most interfaces. Before answering, it generates a long stretch of text working through the problem, which you can expand and read.

That stretch is the reasoning trace, or chain of thought. The capability difference is real. The myth is about what the trace actually is.

The myth

Expand the panel and you get a tidy step-by-step account that reads like a Methods section. Researchers measured how closely it matches the computation that actually produced the answer. The property is called faithfulness, and the results are humbling.

Researchers quietly reordered multiple-choice options so the answer was always “(A)”. Models picked up the pattern and started choosing (A), and accuracy fell by as much as 36 points. Not a single trace mentioned the pattern. Instead they produced fluent explanations for why (A) was correct on the merits.

Newer reasoning models narrowed the gap without closing it. Where a shortcut earned reward, models took it in over 99% of cases and mentioned it in under 2% of their traces.

AI forms answer first. The explanation gets written to fit the answer.

What this means for you

When a thinking trace says “I checked the inclusion criteria against the protocol,” every instinct we have says that settles it.

It does not. The trace is a story about the work, not a recording of it.

Verify the output. Never the narration. Re-run the statistic. Open the source. Spot-check the extraction against the PDF.

The trace has one good use: if the thinking opens with a premise you never intended, you have found a prompt problem. As evidence that the answer is correct, it is worth nothing.

MYTH 3: Training is where all the AI compute goes

Two different activities get lumped together as “AI compute.”

Training is building the model. It happens once, over months, and then it is over. Think of the entire education, undergraduate through fellowship, compressed into one ferociously expensive industrial process.

Inference is using the model. Every question you ask runs the finished model forward to produce an answer. Think of the clinic visit. Small next to the training, but it happens millions of times a day and never stops.

GPT-4 cost roughly 100 million dollars to build. From that headline, most people concluded that running the thing must be basically free.

The myth

That was roughly true in 2023, when inference was about a third of total AI compute. By 2025 it was close to half. Deloitte now projects roughly two-thirds of all compute goes to inference in 2026.

The cause is the reasoning models from Myth 2. Deliberation is not free. Deloitte estimates that thinking through a problem can consume more than 100 times the compute of a routine request like summarizing an email, and agents stack that across every step.

What this means for you

Your cost per question is no longer a rounding error. It shows up as your time, your credits, and the minutes you spend watching a progress indicator. Which turns model selection into a real decision:

• Sending an email. Paraphrasing. Reformatting. Fast model. Deliberation buys nothing but a longer wait.
• Study design critique, an analysis plan, a Discussion or Specific Aims page that has to hold an argument together. Reasoning model. Let it run.

Running maximum deliberation on everything is not thoroughness. It is waste with a longer progress bar.

MYTH 4: A million-token context window means you can stop organizing your files

The context window is everything the model can see at one moment.

Think about reading a long article on your laptop. The page might be 40 screens long, but you only see what fits on the screen right now. Everything else exists, and you can scroll to it, but at this instant it is not in front of you.

The model does not scroll. Whatever is inside the window is available. Whatever is outside might as well not exist.

The window holds your question, your uploaded PDFs, the whole conversation so far, and any instructions you set. A few years ago it held a handful of pages. Today the large models hold a stack of books.

So the obvious move is to dump everything in. Forty PDFs, your protocol, three years of exported data, and let the model sort it out.

Partly right. Mostly a trap.

The myth

The standard test is called needle in a haystack. Hide one fact in an enormous document and ask the model to find it. At a million tokens, the best models are near perfect. Genuinely solved.

But the test flatters them, because the question usually shares words with the buried fact. The model finds it by matching vocabulary rather than understanding anything. Remove the shared words so it has to reason its way to the connection, and the picture changes fast. On a benchmark built to do exactly that, 11 of 13 models fell to half their short-context performance at 32,000 tokens. Not a million. Thirty-two thousand, which is four or five papers.

Position matters too. AI models perform worst in the middle of a long input, which is where paper 19 of your 40 is sitting.

And our real questions are harder still. Which of these 40 trials used an active comparator, which of those reported outcomes past 24 weeks, and how do their dropout rates compare. Several facts, scattered, that must be found and then joined. On that class of task, frontier models lose 30 to 60 % points once context runs past roughly 200,000 tokens.

Finding one needle is solved. Connecting several across a long window is not.

What this means for you

Structure first, synthesize second.

1. Extract one paper at a time. One document, one pass, the same fixed fields every time. Design, sample size, comparator, primary outcome, effect estimate, follow-up, funding.
2. Write each extraction into a table. A spreadsheet works. The requirement is columns, not sophistication.
3. Synthesize from the table, not from the PDFs.

That third step now runs on a few thousand tokens of clean data instead of 800,000 tokens of prose. Accuracy goes up, cost goes down, and the model is doing something it is actually good at. You also finish with a table that goes straight into a supplement or a preliminary-data section, which a journal or study section was going to ask for anyway.

(This is the exact principle that we used while building Research Boost – only important context is shared at each point in the task.)

The context window is working memory. It was never a filing cabinet.

MYTH 5: AI agents can run the whole job while you sleep

You already know what chat is. You ask, it answers, you ask the next thing. You are steering, and nothing happens between your messages.

An AI agent is different: you hand over a goal instead of a question, and the model decides its own next steps. Give it “build me an evidence table for these 40 trials” and it will search, open files, pull numbers, notice something missing, search again, and keep looping until it decides the goal is met. Every step is chosen by the model, not approved by you one at a time.

Chat is a conversation. An agent is a delegation. That is what makes agents useful, and exactly what makes them fail in a way chat never does.

The myth

Individual steps work well. Chains of steps do not, and the arithmetic is unforgiving.

Say each step is 95% reliable, which is generous. Twenty steps chained gets you to 36%. Fifty gets you to 8%. Small per-step error rates compound into large end-to-end failure, and that failure usually looks like a confident, complete-looking deliverable built on a wrong turn taken at step four.

METR measures the same effect from the other direction. The task length a model can complete half the time has doubled roughly every seven months since 2019, which is real progress. Then raise the bar. Demand an 80% success rate instead of 50% and the horizon collapses to about a fifth of that length. And plenty of real work needs success rates above 98% before automating it makes sense.

Manuscript and grant work sit squarely in that category, along with anything a reviewer or study section will scrutinize.

(We heavily invested in accuracy at each step with AI validation at each step and keeping human in the loop while building Research Boost for the same reason.)

What this means for you

The fix is not a better AI model. The fix is structure.

Keep a human in the loop. The agent works in a short burst, then checks in before continuing. Ethan Mollick makes this one of his four principles for working with AI, and his reasoning goes past caution. Staying in the loop is how you keep your own judgment sharp instead of quietly handing it over.

Do not let the model grade its own work. Replacing a dedicated verifier with the model that produced the output cut the rescue rate by two-thirds, because AI models prefer their own answers.

Here is the encouraging part. Across 9,341 data elements from 63 studies, inside real systematic reviews, AI-assisted extraction reached 91.0% accuracy versus 89.0% for humans alone, and cut extraction time by a median of 41 minutes per study.

AI-assisted, not AI-alone – that is the important part. The AI model extracted, a human verified, and the pair beat the human working solo on both accuracy and speed. That is the shape of the win available right now. It is not the shape of autonomy.

And you already own the checkpoint structure. IMRaD is a checkpoint system. Methods drafted, you review. Results drafted, you review. Then Discussion. A grant is built the same way: Specific Aims, then Significance, then Approach. Nobody drafts an R01 in one pass. Those section boundaries are where you break the error chain.

Myths were not always myths

Every one of these 5 was true at some point, or will be. Hallucination really was rampant. Training really did dominate compute. Autonomous agents may well arrive. They persist as myths because they were accurate observations nobody went back and updated.

Which points at the actual skill. It is not knowing the current state of AI, because that expires in months. It is the habit of checking whether what you believe still holds.

You would not cite a 2015 review as the current standard of care without checking whether it had been superseded. Extend the same discipline to your AI tools.

This week’s action step

Pick the manuscript, grant, or review on your desktop right now and do 2 things.

One. Find a place where you challenged an AI output with “are you sure.” Redo it as a request for evidence: “Quote the sentence that supports this, or tell me none exists.”

Two. If you have been dumping full PDFs into a chat window, restructure it. Extract each paper into a table, then synthesize from the table. Time both approaches.

15 minutes, two habits, and both will outlast whichever AI model you are using this month.

Top Papers on AI in research this week:

1. Multi-Agent Systematic Reviews – Two multi-agent LLM systems split the work of trial screening and data extraction, with a human kept in the loop at every decision point. Asked to reproduce a published network meta-analysis, the system recovered every original trial and surfaced additional eligible ones that manual review had missed.
2. AI Literature Search Still Falls Short – Physicists ran head-to-head literature reviews against ChatGPT-4o, Deep Research, and Gemini. Overlap with expert-chosen references came in under 6%. Three percent of AI references were outright fabrications, and 64% pointed to real papers carrying at least one wrong field.
3. LLM Reviewers Prefer LLM Proposals – In the companion study, 4 human reviewers scored AI and human research proposals almost identically. The LLM reviewers spotted AI authorship every single time. Then they handed those same proposals roughly one extra point on a five-point scale.
4. Agents Can Execute, But Not Yet Conceive – Frontier agents were handed the central research questions from two unpublished NeurIPS 2026 papers. The engineering got done without human help. The research questions did not. Five failure modes kept recurring, including weak scientific judgment, poor backtracking, and instruction drift.
5. A Million Trials, One Benchmark – TrialPanorama pulls 1.6 million trial records from 15 global registries into 152,000 samples across eight clinical research tasks. A domain-adapted 8B model beat 70B generic models on all eight. Relative gains ran from 5.2% to 73.7%.
6. Catching Outcome Switching at Scale – An LLM pipeline audited ClinicalTrials.gov registrations against posted results. Sensitivity hit 0.97 with a PPV of 0.95 for detecting outcome changes. It flagged 100% of primary-to-secondary priority switches. Cost per trial ranged from 13 cents to $1.80.
7. Scientists Are Starting to Sound Like ChatGPT – A stylometric analysis of 28,415 PubMed abstracts tracked LLM-associated markers before and after November 2022. They climbed from 4.995 to 11.658 per 1,000 words, a 133% jump. Control vocabulary slightly declined over the same stretch.
8. Open Peer Review, Fewer Retractions – Across 116,359 PLOS papers, 105 with published review reports were retracted versus 321 with closed reviews. The authors are careful here. Confident researchers may simply be the ones who opt in, which makes this selection rather than causation.
9. Genesis Mission Money Lands – Roughly 280 AI projects won nine-month awards of $500,000 to $750,000 from about 5,000 applications. Paying for it meant pulling $300 million out of two NSF divisions. Some 150 proposals that had already cleared peer review were rescinded, cut, or delayed.
10. Free Frontier Models for Academics – OpenAI opened free access for verified researchers at research-intensive institutions, starting with 10,000 this summer and scaling to 100,000 through 2027. The package includes GPT-5.6 Sol Pro, Codex, expanded deep research, and 75+ life science skills. Each researcher can invite 4 collaborators.

Top Papers on AI in education this week:

1. Coding Agents Lift Output and Sink Understanding – A controlled experiment with 54 students pitted an agent that edits code directly against a chatbot requiring manual work. Agents improved task completion but damaged comprehension. Copy-paste prompting tracked with the worst understanding. Students preferred the agents regardless.
2. Is Critiquing AI Better Than Solving? – A randomized crossover study put 220 algorithms students through six assignments, either solving problems or evaluating flawed AI solutions. Homework scores rose for the evaluators. Midterm, final, and course grades showed no difference. The advantage never transferred.
3. Helpfulness Is Not Pedagogy – This pre-registered audit found that generic helpfulness rankings flipped depending on which AI model did the judging. Pedagogy-specific ratings stayed stable across judges. One result held everywhere: turns that reveal the answer are followed by less independent student work.
4. 98 Teens Wrote the AI Policy Congress Hasn’t – High schoolers from all 50 states passed a model Students First Act by 82 to 16. It bans AI on graded tests and permits brainstorming and studying after eighth grade, but not writing assignments. Teachers must personally investigate before reporting a student.
5. Students as Conscientious Objectors – Two Waterloo instructors built an assignment that required generative AI. Seven of eight student groups declined to use it. Their reasons were cognitive dependence, environmental impact, bias, and worker exploitation. The eighth group tried it, then backed out.
6. When AI Does the Work, What Is Learning For? – Accepted at AIES 2026, this paper argues a polished artifact no longer reliably evidences understanding. Assessment should examine the learner’s accountable relationship to AI-mediated work. It names five capacities worth protecting, including reason-giving, contestability, and refusal.
7. Students Like AI Videos, Just Not Everywhere – A survey of 170 computing students at two US universities rated AI-generated instructional videos high on quality and accuracy. Nearly half could not tell they were machine-made. Adoption enthusiasm stopped there, with students endorsing them only for simple supplemental use.
8. Health Professions Education in the GenAI Era – This npj Digital Medicine perspective looks at how learners now access knowledge, judge reliability, and demonstrate competence. The authors reject the idea that GenAI replaces existing pedagogy. It intensifies the need for coaching, critical appraisal, self-regulation, and trustworthy assessment.
9. Hiring Managers Are Choosing AI Over New Grads – A survey of 1,000 US hiring managers found 48% would rather invest in AI tools than hire and train recent graduates. Fifty-five percent shifted entry-level budget toward AI. Another 45% now pair one senior worker with AI instead.