The Part of the Picture Your AI Cannot See

Read the written version →

Subscribe: Spotify · RSS feed

Show notes

This episode is based on Coinmedia's insight piece 'What Your AI System Can See — and What It Cannot', published in September twenty twenty-six. The piece examines the practical consequences of information boundaries in deployed AI systems — not as a technical problem but as a governance and design problem that sits with the leadership teams running those systems.

The statistic about two thirds of senior technology executives being accountable for AI systems they cannot realistically supervise, and the figure of fifty-four AI agent incidents per organisation per year, comes from a global study of senior technology executives published by IBM. The episode cites these figures as the article presents them; the original research is referenced in the article itself.

The finding that large language models attend more reliably to information at the beginning and end of long inputs — sometimes called the 'lost in the middle' effect — has been documented in published academic research on model behaviour with extended context windows. The episode does not name specific papers; the article from which this episode is drawn does not cite them by name either, and the episode does not go beyond what the source supports.

The three visibility gaps named in the episode — the recency gap, the tacit knowledge gap, and the organisational boundary gap — are framing developed in the Coinmedia article from observed patterns across operational and advisory contexts, not from a single external study. The five questions at the end of the episode are drawn directly from the article's closing section and are intended as prompts for internal conversation, not as a technical audit framework.

Chapters

• 0:08 Cold open
• 0:51 Why the boundary that matters most goes undrawn
• 1:50 Why making the system bigger does not fix this
• 3:16 Three gaps that appear again and again
• 5:02 The real danger is misplaced confidence
• 6:17 What deliberate visibility design actually looks like
• 8:25 Close

Transcript

Cold open

Imagine you have a capable analyst on your team. You give them a file and ask for their assessment. They read everything carefully, structure their thinking well, and come back with a confident, well-written recommendation. What you did not realise — and what they had no way of knowing — is that you gave them only part of the file. The rest was sitting in a different folder. They did not ask for it. They did not know it existed. And so their recommendation does not include it.

That is the situation most organisations are in with their AI systems right now. Not because the systems are unintelligent. Because no one has asked the right question about them. Not what can this system do — but what can this system actually see?

Why the boundary that matters most goes undrawn

When a business deploys an AI system — for research, for drafting, for analysis, for routing customer queries — someone decides what information it can access. But that decision is almost never made deliberately. It happens as a byproduct of which tools were connected and which data sources were in scope when the system was set up. It is rarely documented. And it is almost never revisited.

Yet that boundary is one of the most consequential design choices in the whole system. Everything outside it is invisible to the AI. The system cannot flag what it has not seen. It cannot add a caveat about information it does not know exists. It will reason from whatever is in front of it — and it will produce the most coherent output it can from that material.

That output will not say: I should mention that I have not seen anything about X. It will simply produce an answer that does not include X. And the reader, unless they already know about X, will have no reason to doubt what they are reading. The output looks complete. It reads as authoritative. And it may be missing something that changes the picture entirely.

Why making the system bigger does not fix this

The instinctive response to this problem is to assume it solves itself as AI systems improve. And in one narrow sense, capacity has grown significantly. The amount of information a model can process in a single operation has expanded substantially in recent years. So the obvious thought is: connect it to more data, give it a bigger window, and the visibility problem goes away.

But capacity and completeness are not the same thing. A system can have a very large window onto a very incomplete picture. The question is not how much information it can hold — it is what has been placed inside it, and by whom, using what criteria, on what schedule.

There is also a subtler issue. Research into how these systems perform on long inputs has found a consistent pattern: models tend to use information more reliably when it appears at the beginning or the end of what they are given. Material buried in the middle gets less attention. So even when a large volume of information is provided, the ordering and distribution of that information shapes what the system actually uses when it forms its response. More data fed in does not automatically produce better reasoning out. It depends on which data, structured how, with the most decision-relevant facts prominent rather than buried.

Three gaps that appear again and again

Across a range of operational and advisory contexts, three kinds of visibility gap appear with enough regularity to be worth naming.

The first is the recency gap. AI systems are trained up to a point in time, and the data sources they connect to are updated on some schedule — hourly, daily, weekly, or ad hoc. Any business condition that has changed since the last update is invisible to the system. Pricing shifts, personnel changes, a competitor's announcement, a regulatory development — in fast-moving commercial situations, that lag can be the difference between a useful output and a misleading one.

The second is the tacit knowledge gap. Much of what makes a business function well lives in people's heads, not in any document. The informal understanding between a founder and a long-standing client. The reason a particular process has an exception that was agreed eighteen months ago. The commercial nuance behind a contract clause that is technically standard but practically significant. None of this exists in a system unless someone has deliberately put it there. And it rarely has been.

The third is the organisational boundary gap. In most businesses, information is held across different systems, different departments, and different relationships. An AI system with access to the C R M but not to the finance system is reasoning about a customer relationship without the revenue context. One with access to completed project documentation but not to the current pipeline is advising on capacity without knowing what is actually committed. The system does not know it is working in a silo. It simply works.

The real danger is misplaced confidence

What makes these gaps genuinely dangerous is not ignorance — it is the appearance of certainty. If an AI system labelled its outputs clearly: based on partial information, significant caveats apply — most readers would treat them accordingly. They would probe. They would ask what else might be relevant. But that is not how these systems present themselves.

They produce fluent, well-structured, apparently complete responses. And fluency is, in most circumstances, a feature — it is one of the reasons these systems are useful. The problem is that fluency can mask uncertainty. A reader who is not already expert in the subject has no reliable way to distinguish a fully-informed output from a well-expressed partial one.

A global study of senior technology executives published by I B M found that two thirds are accountable for AI systems they cannot realistically supervise. The same research recorded an average of fifty-four AI agent incidents per organisation in the preceding year that required human correction — with a significant proportion involving data exposure, cascading failures, or compliance issues. These are not primarily failures of AI capability. They are failures of visibility — on both sides. The AI could not see what it needed to. And the humans running it could not see that the AI was operating blind.

What deliberate visibility design actually looks like

The response to this is not to stop using AI systems in consequential situations. It is to be deliberate about what you place in front of them — and to build that deliberateness into the process, not leave it to chance.

The most important shift is treating information design as a first-order decision. Before deploying a system for a particular use, the question to ask is not what can this system do — it is what does this system need to see in order to reason about this situation well. That question, asked carefully, tends to surface the gaps immediately. It makes the invisible boundary visible.

It also changes what governance looks like. Effective oversight is not just reviewing the outputs of an AI system. It includes reviewing what went in — checking that the information provided was current, representative, and complete enough for the decision at hand. There is a real distinction between humans reviewing AI outputs and humans supervising AI behaviour. The first catches errors after the fact. The second catches the conditions that produce errors in the first place.

There are five questions worth bringing to your own team. For each AI system you are running, can you describe — in plain terms — what it can and cannot access? When was the information it works from last updated, and is that frequency appropriate for the decisions it supports? What tacit knowledge is critical to the areas it operates in, and has any of it been deliberately encoded? Who is responsible for reviewing not just what the system produces, but what it was given to work with? And when it produces a confident-sounding output, what is the process for assessing whether that confidence is warranted? These are not technical questions. They are governance questions. Most leadership teams have not yet had that conversation.

Close

The one thing to take away from this episode is simple. The boundary of what an AI system can see is a design choice. It is not a default. It is not something that resolves itself as the technology improves. It is something someone decides — deliberately or not — and it shapes every output the system produces from that point forward.

Left unexamined, that boundary tends to be drawn too narrowly. The gaps it creates do not announce themselves. The outputs look complete. The confidence sounds warranted. And the decisions made on the back of them carry a risk that no one has named.

The question to take back to your team this week is this: for the AI systems you are currently running, who owns the decision about what those systems can see — and when did they last review it?

If that conversation surfaces something worth thinking through further, you are welcome to get in touch. The contact details are on the episode page.

Each episode is written from that week's Coinmedia insight and voiced with a synthetic model of Zsófia's own voice. The thinking, the editorial line and the approval to publish are human.