Evidence Review: Human Oversight of Generative AI at Work

Research question

Why is it unsafe to assume that a tool which helps with one work task will also help with a different one?

Review method

This is an RDE targeted narrative desk review of published evidence, not an original study or a systematic review. Sources were searched and checked on 8 October 2026. Search terms: Dell Acqua jagged technological frontier 2026 field experiment NIST AI 600-1 human oversight. Sources were selected for direct relevance, identifiable authorship, accessible methods or an official technical basis. Commercial marketing, unsourced statistics and advice presented as research were excluded. The small, purposive selection may miss relevant or conflicting studies. No pooled effect estimate was calculated and no participants or original data were collected.

Evidence and interpretation

Dell’Acqua and colleagues report a preregistered experiment involving 758 consultants, assigned to no AI, GPT-4 or GPT-4 with a prompting overview. On tasks within the tested model's capability, AI-assisted participants completed work faster and produced higher-quality output. On a complex task outside that capability, AI-assisted participants were less likely to reach the correct answer. [1] Published field experiment.
NIST's 2024 Generative AI Profile proposes risk management, testing, monitoring and human review appropriate to context. It is a cross-sectoral guidance document informed by consultation, not a trial proving a specific review process effective. [2] NIST profile.

What this evidence cannot establish

The experiment concerns particular consultants, tasks and a particular model. Its findings are not a benchmark for every current product. Having a person involved did not guarantee correctness; the study does not prove that simply adding a final approval checkbox solves the problem. NIST provides an organising framework rather than a guarantee. Effective oversight must itself be examined: can the reviewer detect the errors that matter?

RDE practical interpretation

The following activity is an RDE learning exercise, not a tested intervention or a finding reported by the cited authors. Use it to make a decision explicit and identify the evidence you would still need. Record uncertainty rather than filling gaps with an attractive explanation.

• Select a low-consequence practice task with a known source of truth. Keep confidential or personal information out of the exercise.
• Define what counts as a satisfactory answer before looking at the AI output. Identify claims requiring verification.
• Ask a reviewer to check the draft against the original records, not merely its fluent wording. Record missed errors as well as corrected ones.
• Compare the reviewed output with the reference answer. Decide which task boundaries, evidence or review skills need changing.
• Document when the workflow should pause and refer the decision to a suitably qualified person. This exercise is not an approval for high-consequence deployment.

Check your learning

Which statement above is a published finding, which is guidance, and which is an RDE interpretation? What additional evidence would change your decision? Write a short answer before applying the exercise to a real situation.

References

• [1] Dell’Acqua, F. et al. (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science. Published online 11 March 2026. DOI: 10.1287/orsc.2025.21838.
• [2] NIST (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, July 2024. DOI: 10.6028/NIST.AI.600-1.

Review version 1.0. Sources checked 8 October 2026. Follow the links for full methods and context. The cover is illustrative and does not depict study participants or results.

Cover image: AI-generated illustrative scene created specifically for this resource.