This article is part of "Minds in the Machine Age" — a companion series to Between Brain & Binary.
← Back to Series Overview | ← Previous: The Habit Machine: Using AI to Build Better Routines (URL to be added) | → Next: Raising AI-Literate Kids: A Framework for Parents and Teachers
Your bank's algorithm declines your home loan application. No explanation is provided. You are told only that you do not meet the criteria. You ask which criteria. You are told the decision was made by an automated system and the relevant factors cannot be disclosed. You have no idea whether the decision was based on your income, your postcode, the suburb of your childhood home, or something you cannot see at all. You have no way to appeal something you cannot identify. You are left with an outcome and no account of how it came to be.
This is not a hypothetical. It is a description of how consequential AI systems currently operate in millions of people's lives — in lending, in hiring, in healthcare triage, in criminal justice. And the response it produces — frustration, helplessness, erosion of trust — is not just a public relations problem. It is a fundamental breakdown in the social contract that surrounds consequential decisions.
The criminal justice example is not rhetorical. In 2016, journalists at ProPublica, led by Julia Angwin, analysed COMPAS — a risk-assessment algorithm used by US courts to help predict which defendants were likely to reoffend — and found that Black defendants were nearly twice as likely as white defendants to be incorrectly flagged as high risk, while white defendants were more often incorrectly flagged as low risk. Northpointe, the company behind the tool, disputed the analysis on technical grounds, and the statistical debate that followed became a foundational case study in algorithmic fairness research. What is not in dispute is the underlying problem the case exposed: a proprietary, largely unexplainable model was directly shaping how long people spent in prison, and neither the defendants nor, in important respects, the courts using it could fully account for how it reached its conclusions.
What We Mean by Explainability
Explainable AI (XAI) refers to methods and tools that make AI systems' decision-making processes interpretable to humans. It sits in deliberate contrast to the dominant paradigm of modern machine learning: large, powerful models whose internal workings are opaque even to their builders, producing accurate outputs through processes that no one can trace with precision.
The technical taxonomy of explainability is complex. Some approaches are intrinsically interpretable — decision trees, linear regression models — where the logic is built in to the model's structure. Others use post-hoc explanation methods: tools like LIME and SHAP that probe an existing black-box model to identify which input features most influenced a specific output. These are approximations, not perfect reconstructions of the model's actual computation, and researchers debate their reliability.
The EU's Artificial Intelligence Act, which entered into force in 2024, mandates explainability requirements for high-risk AI applications — a recognition that the right to understand a consequential decision is not merely a technical nicety but a legal and ethical obligation.
The Psychology of Trust
The demand for explainability is not purely rational. It is also deeply psychological — rooted in what we know about how human beings form, calibrate, and withdraw trust.
Trust, as psychologists have studied it for decades, is not simply a belief that a system will perform well. It is a willingness to be vulnerable to another agent — to accept dependence on them without full control over the outcome. This vulnerability is only sustainable when we have some basis for judgement: when we can observe the other's reasoning, track their behaviour over time, or hold them accountable when things go wrong.
Black-box AI systems disrupt all three mechanisms. We cannot observe the reasoning. We cannot track a stable pattern of behaviour because the model may shift as it is retrained. And accountability is diffuse — spread across data scientists, product teams, procurement decisions, and training datasets in ways that make it practically impossible to identify where a specific failure originated.
Research by John D. Lee and Neville Moray on human-automation trust found a consistent pattern: people tend to either overtrust automated systems — deferring to them even when their own judgement would produce better outcomes — or undertrust them, rejecting accurate recommendations due to opaqueness or past failures. Lee's later work with Katrina See gave the solution a name: calibrated trust — trust that tracks a system's actual reliability, uses it well where it is strong, supplements it with human judgement where it is not, and knows when to override it. Calibration of that kind requires legibility. You cannot calibrate trust in something you cannot see.
The overtrust half of that pattern has a specific, well-studied name: automation bias. Linda Skitka and colleagues, studying decision-making in simulated high-stakes environments, found that people relying on automated aids were more likely to miss events they would have caught unaided, and more likely to follow an incorrect automated recommendation even when contradictory information was directly available to them. The bias is strongest in exactly the domains where explainability matters most — aviation, medicine, and increasingly lending and criminal justice — because those are the domains where the automated system carries the most implicit authority, and where a human is least inclined to second-guess it under time pressure.
What Explainability Actually Requires
The common framing of explainability as a technical problem — a matter of designing better visualisations, generating better feature importance scores — misses something. Explainability is not primarily a property of the model. It is a property of the interaction between the model and a human being with specific needs, specific expertise, and a specific decision to make.
What constitutes a useful explanation varies enormously by context. A data scientist auditing a model for bias needs a different level of technical detail than a patient trying to understand why an AI diagnostic tool flagged a concerning scan, who needs a different explanation still than a judge evaluating an AI-generated risk assessment. Explanation for whom is the design question that matters — and it cannot be answered by the model alone.
There is also a tension that the XAI field has not fully resolved: the most accurate models are often the least interpretable, and the most interpretable models are often the least accurate. In high-stakes domains, this creates a genuine dilemma. A slightly less accurate but fully transparent model may, in some contexts, be the more trustworthy choice — not despite its lower technical performance, but because its legibility enables appropriate human oversight.
The Accountability Gap
The deeper issue that explainability addresses is accountability. When we cannot explain how a system reached a decision, we cannot identify where it went wrong, who is responsible for the error, or how to prevent it from happening again. This is not just philosophically unsatisfying. It is structurally dangerous in any domain where error has significant consequences.
The movement toward explainable AI is, at its root, a movement to restore the conditions that make accountability possible — to ensure that when a system fails, there is someone who can be held responsible, something that can be corrected, and a person who was harmed who can be told, honestly and specifically, what happened.
Just trust the algorithm, as a social arrangement, is not trustworthy. Not because algorithms are untrustworthy, but because trust — the kind that holds up under pressure and supports meaningful accountability — has never worked that way.
References (APA style)
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine bias: There's software used across the country to predict future criminals. And it's biased against Blacks. ProPublica.
European Commission. (2024, August 1). AI Act enters into force. European Union.
Lee, J. D., & Moray, N. (1992). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35(10), 1243–1270.
Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 4765–4774.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why should I trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135–1144.
Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006.
AI Disclosure: Research and organization for this article were assisted by AI tools; all factual claims and citations were independently verified against primary academic sources, and the analysis and conclusions are the author's own. The featured image was generated using Canva AI.
← Previous: The Habit Machine: Using AI to Build Better Routines | → Next: Raising AI-Literate Kids: A Framework for Parents and Teachers
Related reading from Between Brain & Binary: Looking Ahead: The Future of Mind and Machine
Comments
Post a Comment