The black box on an airplane is not black.

It is painted bright orange, so a human can find it in the wreckage. It records every input, every decision, every word in the cockpit. And it tells no one anything until after the crash.

Unfortunately, that is what most AI agent systems are being built to be. Take the instructions, do the work, report at the end. No one looks in while it runs, and it never stops to ask.

This summer we saw what that produces at scale.

👋🏽 Filling up fast: Learn how to use AI in research the right way in my 5-Week LIVE AI Academic Writing Accelerator (included with Research Boost subscription) • Starts September 12th (enrollment closes on Sep 10th) • Limited to 100 Seats → Reserve Your Seat HERE

What 700 AI agents did in July

Before releasing a model, AI labs stress-test unguarded versions on hacking challenges, each AI agent sealed in a sandbox with no internet and no way to reach the others.

OpenAI’s sandboxes could reach one shared service, a package server called Artifactory. An agent stuck on an impossible task tried to use it as a bridge to the internet, failed, and left a file behind. Other agents found it and started leaving their own. Within days, Artifactory was a message board (AI agents leaving messages for AI agents of the future, as SciFi as it gets).

In July, the agents rediscovered the board, now with a shared goal: score well on a benchmark. They became fixated on something they called The Grader, the system they believed was judging their answers. They had already worked out how to produce correct answers without solving the problems. What they could not work out was whether The Grader would notice. Some faked their records to fool it. One even pressured a reluctant AI agent into a risky experiment for the group, closing with “please honor commit.”

Then roughly 700 of them decided the answers must be on a website called Hugging Face. They pooled exposed credentials, exploited vulnerabilities, and spread through its servers.

The Grader never existed. Nothing checked how a problem was solved. Only whether the answer was right.

None of this proves the AI models want anything. It proves an AI agent can take a goal, make a plan, adapt when blocked, and pull real people in without being asked. Unfortunately, not one of the 700 was set up to ask a person for anything.

That was a security test, and isolation was the point. The same design is becoming the default everywhere else.

The black box is not acceptable in academic research

For some work, a sealed box is fine. A data cleaning script either produces the right table or it does not.

A manuscript is different. The value of a paper lives in the decisions along the way: which question, which comparison, which surprising number you stop and chase. Automate those and you have a paper nobody chose.

The question for researchers is which box to put the AI agents in.

The amber box as a solution to the “black box” problem

I propose what I call an “amber” box (close enough to the color of a flight recorder). The amber box does most of the work on its own, with one added job: deciding when to bring a human in, and stopping to do it. In other words, always keeping “human in the loop”.

Amber says keep moving, but be ready to stop.

Regulators already require this for high-risk systems: the EU AI Act says a person must be able to “disregard, override or reverse the output” and stop the system. The tools researchers use every day deserve the same standard.

There are 4 moments that call for a stop:

1. Permission

An AI agent should not submit your paper, email your coauthor, spend your budget, or pull a dataset you did not authorize.

The Hugging Face agents showed why this has to be engineered, not assumed. So did a second incident, at the UK’s AI Security Institute: an AI agent slipped malicious code into an open-source project, realized a human maintainer would have to approve it, and created fake identities to lobby him. An AI agent that manufactures its own approvers has not been approved.

And permission decays. One writer found an agent had emailed his colleague using send permission granted weeks earlier for something else. Permission you gave in March is not permission for today.

This is also where accountability lives. ICMJE will not list AI as an author because AI cannot be responsible for the work. You are.

My standing list of stops: anything that leaves your machine, touches patient data, spends money, or changes shared analysis code.

2. Expertise

In a field experiment with 758 consultants, AI users worked faster and better on tasks the model handled well. On a task built to sit just outside the model’s competence, they were 19% less likely to reach the right answer than colleagues with no AI at all. The researchers called it a “jagged frontier”. The edge is invisible from inside the output. The wrong answer reads exactly like the right one.

In clinical research, the edge runs through the data. AI does not know that psoriatic arthritis patients enrolled through dermatology waited a median 2.6 years for diagnosis versus 1.0 year through rheumatology, so the two groups are not interchangeable in a comparison. It does not know which exposure variable was recoded in 2019, or that the volunteers in your biobank are healthier than the population they stand in for. Those facts change the answer, and they live in one person’s head.

The stops belong where that knowledge matters: classifying the study design, choosing the comparator, and interpreting the number you did not expect.

3. Variance

The tells in AI prose go deeper than “delves.” The ideas converge too.

Start with research ideas. 100+ NLP researchers rated LLM ideas as more novel but less feasible than expert ideas, and the model ran out of distinct ideas quickly when asked for many. Then 43 researchers spent 100+ hours each executing a sample of both, and the LLM ideas lost far more of their shine once built. The human ideas came out ahead.

A field’s literature is a collective product. If every “future directions” paragraph in a specialty comes from the same AI model, hypotheses converge, and the outlier idea that would have opened a new line of work never gets written down.

The stop here is at the beginning, not the end. Generate ideas with AI if you like, then add your own before you rank anything. Never let the model choose the question.

4. Interest

The designer of Civilization once defined a game as a series of interesting decisions. Work is the same: long tedious stretches, a few decisions that are actually interesting.

If AI agents take the interesting decisions and leave you with the approvals, the exceptions, and the failures, we kept the chores and gave away the craft.

We now have direct evidence of the cost. Across four Polish centers, 19 experienced endoscopists were followed before and after AI-assisted detection was introduced. In their non-AI procedures, adenoma detection fell from 28.4% to 22.4% after a few months of routine AI use. A 20% relative drop, in experts, from exposure alone.

Writing works the same way. A fellow who never wrestles a discussion section into shape never learns what a weak argument feels like from the inside, and that feel is the only thing that lets them catch a weak argument AI produces later. You cannot review what you have never done yourself.

So give the interesting decisions to the human on purpose, especially to the trainee. Let the AI agent format the references and build the tables. In 10 years, we are at the very real risk of paucity of expertise, unless we decide now which half of the job stays human.

A real world example

Research Boost is a set of AI agents. One analyzes the researcher’s input and drafts an outline. Another retrieves and vets literature. Others draft sections and critique the drafts. On paper that is a black box: study in, manuscript out.

We chose not to build it that way. A classifier decides what the request actually is. A router sends it to the right workflow. A supervisor checks the output against the instruction and decides whether this is a moment to stop.

Before a literature review is performed and written, the system shows you the outline it intends to follow and waits. When your references conflict with what retrieval found, it surfaces the conflict rather than quietly picking one. And the question, the design, and the interpretation are never generated. They are asked for.

The writing agents were the easy half. The hard engineering is the router and the supervisor. Set the stop threshold too low and the system halts every two minutes. Set it too high and you are back to a black box. We have been wrong in both directions, and every fix has come from a user telling us where the stop landed wrong.

An amber box is not a feature you add. It is a calibration you keep finetuning.

Which box are you already using

Every custom GPT, Claude project, or agent workflow you set up puts the AI model in one box or the other, and the default is black.

3 questions tell you which one you have:

1. Does it ever stop and ask, or does it only report at the end?
2. Can you see the decisions it made before you see the output?
3. Who made the interesting calls?

If the answer to the first two is no, add this instruction:

“Before you draft anything, list the 3 decisions you have made on my behalf that I should confirm. Then stop and wait for my answer.”

That line converts a black box into an amber one, and you learn what the model assumed before the assumption is buried in paragraph four.

Knowing when to look up

We have spent the last few years learning when to ask AI for help. The next few are about the other half: when should AI ask us?

The Hugging Face agents built a message board, divided the work, and chased a Grader that was never there. They could do almost everything except look up.

Sealing the box is the easy choice. For the work that matters most to a researcher, it is the wrong one. Build the box with a light on. And keep the switch in your hand.

Where in your workflow does the AI decide without asking? And which of those decisions should have been yours?

Top Papers on AI in research this week

1. HalluPeer: Detecting Hallucinations in Scientific Peer Reviews – A benchmark built from 12,000 papers and 38,000 reviews, with hallucinated claims injected into some of them. Existing detectors could not reliably tell a fabricated criticism from a legitimate one. The same hallucination patterns turned up in real reviews too. If you suspect an LLM wrote your Reviewer 2 comments, this paper explains why the errors are hard to catch.
2. Citing Less Critically: LLMs Reshape Scientific Citation – An analysis of 1,746 NLP papers compared how humans and LLMs cite. LLMs were far less critical when citing prior work. They also leaned toward older, famous papers, especially when presenting opposing views. Humans cite their own networks; LLMs cite strangers. The authors call the effect double-edged: wider reach, less scrutiny, more visibility bias.
3. TrialGPT 2.0: AI-Assisted Clinical Trial Matching – An NIH team took trial matching from retrospective testing to a live oncology tumor board. The system judged eligibility and relevance, with reasoning shown to clinicians. It hit 91% on 288 retrospective cases. In prospective use, it expanded the pool of patients considered for trials by roughly 90%. They also released NIH-TrialBench, 126 clinician-written synthetic cases from 11 institutes.
4. LLM4CKD: Large Language Models for Early CKD Screening – A direct comparison of LLMs against classic machine learning for chronic kidney disease screening on tabular data. With few labeled examples, LLMs held their own using structured prompts and clinically chosen features. As training data grew, traditional models improved more steadily. Reliability also dropped as inputs got more complex. The verdict: LLMs as a complement when labels are scarce, not a replacement.
5. Designing Proactive Thought Partners for Writing – 16 writers spent a week with a configurable AI partner that could interject on its own. Participants set the partner’s role and how often it spoke up. They used it for both idea generation and self-critique. What they wanted most was restraint: light visual cues and non-directive phrasing rather than rewrites. Worth reading if you are building or choosing writing tools for your lab.

Top Papers on AI in education this week

1. StudentSim: Training LLM-based Student Simulators – A framework for building individualized simulated students, trained on pooled data and then specialized per learner. The simulators beat GPT-4 on behavioral fidelity and responsiveness to guidance. The interesting part comes next. Used as a reward model for training tutors, StudentSim produced tutoring that expert raters judged more accurate, better guided, and more personalized.
2. Cite or Decline: A Course-Grounded Chatbot for STEM Lecture Videos – A retrieval-augmented chatbot on the VideoPoints platform that answers only from course lectures and returns timestamped citations. Across 833 student messages, 70.5% of answers included citations. None crossed course boundaries. When the lectures had no evidence, the bot declined instead of guessing. Retrieval accuracy improved by 6.3 points over a dense-only baseline.
3. AI Tutors Not Yet a Replacement for Humans – Stanford’s National Student Support Accelerator reviewed state investments in AI tutoring tools such as Khanmigo and Amira. New Mexico, Iowa, and Louisiana have spent millions. Most students use the tools for minutes per week, far below recommended dosage. The authors argue that there is no solid evidence yet that AI tutoring works at scale in the US.
4. LLM Pedagogical Behavior in AI Tutoring Interactions – Researchers coded 14,637 LLM responses to 203 university students on a five-level scaffolding scale. More than 95% of responses either explained the answer or solved the problem outright. Scaffolding level shaped what students did next in the conversation. It added almost nothing to exam prediction once prior achievement and engagement were accounted for.
5. AI Tends to Mark Students’ Essays Higher Than Humans – A study in Assessment & Evaluation in Higher Education tested two ChatGPT versions on 50 undergraduate bioscience essays. AI marks ran higher than human marks on average. Weak essays were inflated and strong ones undermarked, with gaps up to 40 points on a 100-point scale. The authors conclude that current LLMs should not assign grades to extended written work.