Ironies of Automation

Automation is introduced because humans are slow or fallible. The human then remains the supervisor for exceptional situations that automation cannot handle — precisely the situations requiring the strongest knowledge and fastest diagnosis. Yet routine automation has meanwhile taken the operator out of the loop.

Bainbridge describes how automation can reduce practice and current process knowledge while people remain expected to intervene. In a randomised mathematics study at one high school, Bastani and colleagues (2025) found better practice performance with GPT-4, but worse unassisted performance shortly afterwards in the version without learning safeguards. That disadvantage was not detected in the version with learning safeguards. This does not establish long-term retention loss or recovery capability in organisations.

Automation may reduce active practice and feedback while responsibility for exceptions remains with the human. This creates a mismatch between responsibility and capability, especially when exceptions are rare and difficult.

AI transfer to investigate: Modern AI extends this classical mechanism into professional and cognitive work. The new elements are the application area and much larger scale; the underlying mechanism already existed.

Proposed test, not yet conducted: Compare human-only work, direct-answer AI, analytical-support AI and high-offloading AI with periodic takeover or rehearsal. Measure normal performance and, later, unaided exception detection, diagnosis and adaptation.

“Human in the loop” is not evidence of capability. If human intervention is part of the safety or knowledge design, that skill must be maintained and remain demonstrable.

AI may amplify an existing mechanism here. Reduced practice can impair recovery, but suitable training or system design may prevent this. The proposed test has not been run.

Bainbridge (1983), Ironies of automation

Bastani et al.(2025), Generative AI without guardrails can harm learning — author manuscript

Bainbridge (1983), full article via IFAC

Bainbridge discusses skill, monitoring and practice under automation. Bastani and colleagues examine short-term mathematics performance at one school. That additional study does not establish long-term skill loss or organisational recovery effects.