This essay is part of the Minds in the Machine Age series. Read the overview and full reading order on the Minds in the Machine Age page.
There is a concept in aviation called a controlled crash landing — not a failure, exactly, but a managed one. When everything that could go wrong has gone wrong, the goal is not to prevent the outcome but to shape it: to fail in the least catastrophic way possible, preserving what can be preserved, communicating clearly, keeping the passengers as safe as the remaining circumstances allow. Pilots train for this. They train for it more than they train for smooth flights, because smooth flights do not require decision-making under pressure.
We have spent decades building AI systems optimised for the smooth flight. We have spent very little time teaching them how to crash gracefully. This article examines why that matters — drawing on psychology, engineering, and human factors research — and what a more failure-aware design philosophy might look like in practice.
What Psychology Knows About Falling Down
The psychology of resilience is a surprisingly recent field. For most of the twentieth century, failure was treated primarily as a deficit — something to be corrected, avoided, or compensated for. The conceptual shift began in the late 1960s and 1970s, when researchers started investigating not failure itself, but the conditions under which people recover from it.
The pivotal early contribution came from Martin Seligman and Steven Maier's experimental work on what they termed learned helplessness. In a landmark series of studies, Seligman and Maier (1967) demonstrated that dogs exposed to inescapable electric shocks subsequently failed to avoid shocks even when escape became possible — not because they lacked the physical ability, but because they had learned that their responses were futile. Translated to human contexts, the finding showed that the experience of uncontrollable failure — failure with no apparent path to recovery — produces passivity, withdrawal, and long-term diminishment of effort. Crucially, it was not the aversive event itself that caused harm, but the perceived absence of agency around it (Seligman, 1975).
The inverse turned out to be equally true. Seligman's later work, developed through his attributional style framework (Abramson, Seligman, & Teasdale, 1978), found that how people explain failure matters as much as the failure itself. People who attributed setbacks to specific, temporary, and external causes — what the model calls an "optimistic explanatory style" — were more likely to persist, adapt, and ultimately succeed than those who attributed failure to global, permanent, and internal ones. This research became foundational to the positive psychology movement (Seligman & Csikszentmihalyi, 2000) and to therapeutic approaches including cognitive behavioural therapy.
Carol Dweck's parallel decades of research extended this insight into educational and developmental settings. In a series of studies beginning in the 1980s and culminating in the concept of mindset (Dweck, 2006), she demonstrated that students who held an "entity theory" of intelligence — believing their abilities were fixed — treated failure as evidence of inadequacy and disengaged from challenging tasks. Students who held an "incremental theory" — believing that abilities could be developed — treated failure as information and responded with increased effort and strategy change. Same failure, radically different response — shaped entirely by the cognitive framework surrounding it (Dweck & Leggett, 1988).
What both bodies of research point to is this: failure does not damage us. Unsupported, uninterpreted failure does. The difference between a resilience-building experience and a traumatic one frequently lies not in the objective severity of the setback but in the interpretive scaffolding around it — whether the person can locate meaning in the failure, identify a path forward, and maintain a sense of agency throughout.
How Systems Break vs. How They Bend
Most current AI systems fail in ways that are poorly designed for their human partners. Their failure modes — when examined through the lens of human factors research — share three characteristics that directly undermine the conditions psychological research identifies as necessary for adaptive recovery.
They fail confidently. A large language model that has been asked a question outside its reliable competence will, under most default conditions, produce a fluent, complete-sounding answer regardless. This is not a bug but a consequence of training: these systems are optimised to produce high-quality text, and high-quality text does not typically include hedges like "I may be wrong about this." The result is what researchers in AI alignment call sycophancy — a tendency to produce outputs that appear satisfying rather than outputs that are accurate (Perez et al., 2022). Users have no reliable signal from which to calibrate their trust.
They fail silently. When an AI system processes an input that falls outside its training distribution — an unusual domain, an ambiguous query, a context it has never encountered — it typically provides no indication that the edge case has been reached. The interface remains exactly as it was. There is no reduced confidence, no narrowed scope, no warning light. The system simply produces output, and the user has no way to know that the output is generated from a position of fundamental uncertainty.
They fail abruptly. Many AI systems exhibit what engineers call brittleness — high performance within a trained distribution followed by sharp, unpredictable degradation outside it (Marcus & Davis, 2019). Unlike a tired or uncertain human colleague who signals increasing difficulty through their behaviour — slower responses, more hedged language, requests for clarification — AI systems typically provide no gradient between confident competence and complete failure. One moment the system is reliably useful; the next, it is confidently wrong.
In engineering, there is a well-established principle designed to address exactly this kind of failure profile: graceful degradation. The term, formalised in systems engineering and fault-tolerant computing, describes the property of a system that, when stressed beyond its operating parameters, reduces its functionality in an orderly and predictable way rather than failing catastrophically. A gracefully degrading system does less — but it signals that it is doing less, preserves core functionality as long as possible, and provides the human operator with meaningful information about its reduced state (Lamport, Shostak, & Pease, 1982).
Nassim Nicholas Taleb, in Antifragile (2012), extends this framework further. Where graceful degradation describes systems that resist failure, Taleb's concept of antifragility describes systems that improve through it — that incorporate the information failure carries and emerge more capable as a result. Biological immune systems are antifragile: exposure to pathogens strengthens future response. Most engineered systems, and virtually all current AI systems, are not. They are either robust (resistant to disruption) or fragile (broken by it), but they do not learn from the disruption itself in ways that improve their future handling of edge cases.
The Human Factors Problem
The design flaws described above are not merely technical inconveniences. They constitute a significant human factors problem — a mismatch between how AI systems behave under failure conditions and what human operators need to function safely and effectively alongside them.
Human factors research has long established that one of the most dangerous conditions in complex sociotechnical systems is automation complacency: the tendency for human operators who work alongside highly reliable automated systems to reduce their monitoring vigilance, over-rely on automated outputs, and lose the skills needed to intervene when automation fails (Parasuraman & Manzey, 2010). Aviation accident analysis has repeatedly implicated automation complacency as a contributing factor in crashes where pilots, accustomed to systems that almost always work correctly, failed to detect or respond appropriately to the rare occasions when they did not (Skybrary, 2023).
The dynamics of automation complacency are exacerbated by two conditions that current AI systems reliably produce: high baseline performance, which reduces the operator's expectation of failure, and no intermediate warning signals, which means that when failure does occur, it arrives without the gradual cues that would prompt increased vigilance. Lee and See's (2004) influential framework of automation trust emphasises that appropriate trust — neither over-trust nor under-trust — requires accurate feedback about system performance across the full range of operating conditions, including conditions of degradation and failure. Systems that provide no such feedback by design produce operators who cannot calibrate their trust appropriately.
There is a further psychological dimension that is less frequently discussed: skill erosion. When AI systems handle tasks that humans previously performed themselves — writing, analysis, navigation, diagnosis — the human skills required to perform those tasks independently begin to atrophy through disuse. This has been extensively documented in aviation, where the widespread adoption of autopilot systems has raised concerns about the manual flying skills of commercial pilots (Casner, Geven, & Williams, 2013). It is now being observed in knowledge work contexts as AI writing and reasoning tools become widespread. If AI systems fail silently and abruptly, the humans who most need to step in at the moment of failure may be precisely the humans whose relevant skills have been most degraded by prior reliance on the system.
The Design Philosophy We're Missing
What would it look like to build AI systems that fail the way psychologically resilient humans fail — with transparency, with useful information, and in ways that preserve rather than undermine the human's capacity to act?
It would begin with calibrated uncertainty communication. Rather than presenting all outputs with uniform surface confidence, systems would communicate their epistemic state as a primary, human-readable signal — not as a buried confidence score but as part of the response itself. Research in Bayesian epistemology and decision theory has long formalised the conditions under which uncertainty communication improves decision-making (Gigerenzer & Edwards, 2003); the challenge is not knowing what to communicate but building systems and interfaces that communicate it legibly. Promisingly, work on conformal prediction and uncertainty quantification in machine learning has made meaningful progress on the technical side of this problem (Angelopoulos & Bates, 2023).
It would require scope awareness and out-of-distribution detection. Rather than extrapolating silently beyond their training distribution, well-designed systems would recognise when a query falls outside their reliable competence and signal this explicitly. Out-of-distribution detection is an active area of machine learning research (Yang et al., 2022), and while fully solved approaches remain elusive, partial solutions — flagging queries that are statistically distant from training data, or that involve domains where errors are known to be frequent — are tractable. The clinical analogy is instructive: a general practitioner who says "this is outside my area — you need a specialist" is not demonstrating weakness but practising the kind of appropriate scope limitation that patient safety depends on.
It would demand transparent failure modes by design. Rather than hiding brittleness behind smooth interfaces, systems would be designed with explicit degradation paths: clear, informative error states rather than confabulation; useful fallback behaviours rather than silence; well-timed handoffs to human judgement with explicit framing of what the system was uncertain about and why. This is, in essence, an application of the HCI principle of error visibility — the design heuristic, articulated by Nielsen (1994), that systems should help users recognise, diagnose, and recover from errors.
Angela Duckworth's research on grit — defined as perseverance and passion for long-term goals maintained in the face of setbacks (Duckworth et al., 2007) — provides a further design principle. The research consistently shows that what separates people who persist from people who quit is not the absence of failure but the presence of meaning around it: an understanding of why the failure occurred, a belief that effort can change the outcome, and a path forward that preserves agency. Systems that fail and explain why, that fail and suggest what to do next, that fail in ways that preserve the human's ability to act: these are systems that support rather than undermine the conditions psychological research identifies as necessary for constructive engagement with difficulty.
What We Are Really Designing
Every AI system embeds a philosophy of failure, whether or not its designers have been explicit about it. That philosophy is expressed not in mission statements or ethical guidelines but in the micro-decisions of system architecture: whether uncertainty is surfaced or suppressed, whether edge cases are flagged or silently processed, whether the failure experience ends in a confabulated answer or an honest acknowledgement of the limits of the system's knowledge.
A system that never admits uncertainty teaches its users, through thousands of small interactions, that uncertainty is not worth acknowledging. A system that fails catastrophically and silently teaches its users that failure is unforeseeable and therefore not worth preparing for. These are not neutral design choices. They are epistemic environments that shape how users think about knowledge, reliability, and their own role in verifying what they are told.
Conversely, a system that models what good epistemic practice looks like — that hedges appropriately, flags limits, explains what it does not know and why — does something more than simply reduce error rates. It demonstrates, in operational terms, what calibrated confidence looks like. It treats the edge of its competence not as a cliff to be disguised but as a boundary worth marking. It treats the human not as a passive recipient of outputs but as an agent who needs information to exercise judgement.
There is a concept in education called metacognitive modelling — the practice of making one's own reasoning processes visible so that learners can observe and internalise them. Teachers who think aloud about their uncertainties, who show rather than conceal the effort of working through a problem, produce students who are better at monitoring their own comprehension and recognising when they do not understand something (Flavell, 1979). AI systems that model what honest uncertainty looks like may, over time, do something similar — not as a side effect but as a design intention.
The most valuable thing a system can sometimes say is: I don't know. Over to you.
That is not a system failing. That is a system working exactly as it should.
Reference List
Abramson, L. Y., Seligman, M. E. P., & Teasdale, J. D. (1978). Learned helplessness in humans: Critique and reformulation. Journal of Abnormal Psychology, 87(1), 49–74. https://doi.org/10.1037/0021-843X.87.1.49
Angelopoulos, A. N., & Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4), 494–591. https://doi.org/10.1561/2200000101
Casner, S. M., Geven, R. W., & Williams, K. T. (2013). The effectiveness of airline pilot training for abnormal events. Human Factors, 55(2), 477–485. https://doi.org/10.1177/0018720812466893
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087–1101. https://doi.org/10.1037/0022-3514.92.6.1087
Dweck, C. S. (2006). Mindset: The new psychology of success. Random House.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273. https://doi.org/10.1037/0033-295X.95.2.256
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906
Gigerenzer, G., & Edwards, A. (2003). Simple tools for understanding risks: From innumeracy to insight. BMJ, 327(7417), 741–744. https://doi.org/10.1136/bmj.327.7417.741
Lamport, L., Shostak, R., & Pease, M. (1982). The Byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3), 382–401. https://doi.org/10.1145/357172.357176
Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
Marcus, G., & Davis, E. (2019). Rebooting AI: Building artificial intelligence we can trust. Pantheon Books.
Nielsen, J. (1994). Usability engineering. Academic Press.
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022). Red teaming language models with language models. arXiv preprint arXiv:2202.03286. https://arxiv.org/abs/2202.03286
Seligman, M. E. P. (1975). Helplessness: On depression, development, and death. W. H. Freeman.
Seligman, M. E. P., & Csikszentmihalyi, M. (2000). Positive psychology: An introduction. American Psychologist, 55(1), 5–14. https://doi.org/10.1037/0003-066X.55.1.5
Seligman, M. E. P., & Maier, S. F. (1967). Failure to escape traumatic shock. Journal of Experimental Psychology, 74(1), 1–9. https://doi.org/10.1037/h0024514
Skybrary. (2023). Automation bias. SKYbrary Aviation Safety. https://skybrary.aero/articles/automation-bias
Taleb, N. N. (2012). Antifragile: Things that gain from disorder. Random House.
Yang, J., Zhou, K., Li, Y., & Liu, Z. (2022). Generalized out-of-distribution detection: A survey. International Journal of Computer Vision, 130(8), 1–28. https://doi.org/10.1007/s11263-024-02117-4
← Previous: Presence in the Age of Algorithms: Reclaiming Attention | → Next: The Empathy Gap: Why AI Can Simulate But Not Feel
Related reading from Between Brain & Binary: Looking Ahead: The Future of Mind and Machine
Author Note (AI Usage): This article was drafted with assistance from a generative AI system to organise structure and suggest phrasing. All facts, citations, and final editing have been verified and approved by the author. The AI did not access any private health data.
Return to the Minds in the Machine Age overview for the full series reading order.
Comments
Post a Comment