Black Box AI: Who Answers When the System Cannot Explain?

Consider an applicant refused a service after an automated assessment. A polished explanation arrives, but the applicant cannot check the information used, identify who accepted the result or reach anyone able to change it. The refusal has become readable. The central problem remains: how can the affected person obtain meaningful review?

“Black box” needs care. AI systems differ in how readily their operations can be understood. Opacity can arise from secrecy, gaps in technical understanding, or the complexity of machine learning itself. These require different responses. Withheld records call for access arrangements; technical complexity calls for appropriate investigation. An institution’s choices about disclosure and documentation should not disappear behind the claim that a model is too complicated. Jenna Burrell’s analysis distinguishes these sources of opacity.

A convincing explanation also needs testing. NIST’s explainability principles distinguish an explanation people understand from one that accurately reflects the system’s process. In controlled experiments published in 2025, Anthropic researchers found that reasoning models sometimes relied on supplied hints without acknowledging them in their stated reasoning. Those findings concern particular models and test conditions. Their practical warning is nevertheless important: fluent reasons cannot be assumed to establish what actually influenced an output.

The Court of Justice of the European Union addressed explanation and challenge in CK v Magistrat der Stadt Wien, C-203/22, 27 February 2025. Interpreting the GDPR’s information right in an automated credit-scoring dispute, it required an intelligible account of the procedure and principles actually applied, enabling the person to understand and challenge the decision. Supplying a complex formula would not suffice. This is a specific European holding; its wider governance lesson is that disclosure should enable scrutiny of decisions affecting people.

For businesses, governments and institutions, that lesson belongs in procurement and system design. Before deployment, an organisation should establish the authority for using a system and assign responsibility for decisions influenced by it. Contracts should secure access to evidence needed for investigation. Proportionate records should connect relevant inputs, system versions, outputs and human interventions. Reviewers need competence, time and authority to question the result. Correction must reach the decision and its consequences, including other records or systems that have relied on it.

Strategic, Anticipatory and Evolutionary Legal Intelligence bring these concerns together. Strategic analysis connects purpose with authority, evidence and responsibility. Anticipatory analysis tests foreseeable failures and prepares routes for intervention. Evolutionary analysis makes experience usable through authorised revision, preserving safeguards and remedies. Each operates throughout the system’s life. An initial assessment cannot substitute for learning from contested decisions.

The people affected also hold essential knowledge. They need accessible ways to identify missing context, challenge inaccurate information and reveal recurring harm. Their experience may expose weaknesses that a performance score overlooks.

The black-box problem therefore reaches beyond the model. It reaches the institution that chooses the system, relies on its output and determines whether anyone can challenge the consequences. An explanation begins the conversation. Accountability requires someone with the responsibility and power to act.