What Your AI System Can See — and What It Cannot

There is a question most leadership teams do not think to ask about their AI systems: what, exactly, can this system see? Not in a technical sense — not how many pages of text it can process or which databases it is connected to — but in a practical, commercial sense. What portion of the situation it is reasoning about has actually been placed in front of it? And what has not?

This matters because an AI system, however sophisticated, does not go looking for context. It works with what it is given. If the information it receives is incomplete, dated, siloed or subtly misleading, the output it produces will reflect that — often without any visible warning that something is missing. The system will reason coherently and confidently over a partial picture of reality, and deliver a result that looks authoritative.

That is the problem. Not that AI systems are unintelligent. Rather, that they are very good at appearing complete when they are not.

The boundary no one draws

When a business deploys an AI system — whether for research, drafting, analysis, customer communication or internal routing — someone decides what information it can access. Usually this decision is made implicitly, as a byproduct of which tools are connected and which data sources are in scope. It is rarely made deliberately, and it is almost never documented as a boundary.

Yet that boundary is one of the most consequential design choices in the system. Everything outside it is invisible to the AI. It cannot flag what it has not seen. It cannot qualify its answer with a caveat about information it does not know exists. It will simply reason from what is in front of it and produce the most coherent output it can.

In practice, this means an AI system analysing a client situation may not have access to the most recent conversation that account manager had with the client last week. A system summarising competitive positioning may be working from documentation that was accurate six months ago but has since shifted. A system helping to qualify a sales opportunity may have no view of internal capacity constraints that are currently being negotiated at a senior level.

None of these gaps will appear in the output. The AI will not write: 'I should note that I have not seen any information about X.' It will simply produce an answer that does not include X — and the reader, unless they already know about X, will have no reason to doubt what they are reading.

Why bigger does not mean more complete

It is tempting to assume that as AI systems become more capable — larger, faster, better connected — the visibility problem resolves itself. In one narrow sense, capacity has grown significantly. The amount of information a model can process in a single operation has expanded substantially in recent years.

But capacity and completeness are different things. A system can have a very large window onto a very incomplete picture. The question is not how much it can hold, but what has been placed inside it — and by whom, using what criteria, on what schedule.

Research into how these systems actually perform on long inputs has found a consistent pattern: models tend to attend more reliably to information at the beginning and end of what they are given, and less reliably to material buried in the middle. So even when a large volume of information is provided, its distribution and ordering within that input shapes what the system actually uses when forming its response.

For a leadership team, the practical implication is this: increasing the volume of information fed to an AI system does not automatically improve the quality of its reasoning. It depends on which information, how it is structured, and whether the most decision-relevant facts are prominent or buried.

The three visibility gaps that matter most in practice

Having worked across a range of operational and advisory contexts, certain visibility gaps appear with enough regularity to be worth naming directly.

• The recency gap. AI systems are typically trained up to a point in time and connected to data sources that are updated on some schedule — hourly, daily, weekly, or ad hoc. Any business condition that has changed since the last update is invisible to the system. In fast-moving commercial situations — pricing shifts, personnel changes, regulatory developments, a competitor's announcement — this lag can be the difference between a useful output and a misleading one.
• The tacit knowledge gap. Much of what makes a business function well lives in people's heads rather than in any document. The informal understanding between a founder and a long-standing client. The reason a particular process has an exception that was agreed eighteen months ago. The commercial nuance behind a contractual clause that is technically standard but practically significant. None of this exists in a system unless someone has deliberately encoded it — and it rarely has been.
• The organisational boundary gap. In most businesses, information is held in different systems, departments and relationships, and not all of it flows to a central point. An AI system that has access to the CRM but not to the finance system is reasoning about the customer relationship without the revenue context. One that has access to completed project documentation but not to the current pipeline is giving advice about capacity without knowing what is actually committed.

The confidence problem

What makes these gaps dangerous is not ignorance — it is misplaced confidence. If an AI system produced outputs clearly labelled 'based on partial information, significant caveats apply,' most readers would treat them accordingly. But that is not how these systems present themselves. They produce fluent, well-structured, apparently complete responses.

This is not a design flaw in a simple sense. Fluency is, in most circumstances, a feature. The problem is that fluency can mask uncertainty, and a reader who is not already expert in the subject matter being discussed has no reliable way to distinguish a fully-informed output from a well-expressed partial one.

A global IBM study of senior technology executives found that two-thirds are accountable for AI systems they cannot realistically supervise. The same research recorded an average of 54 AI agent incidents per organisation in the preceding year requiring human correction — with a significant proportion involving data exposure, cascading failures or compliance issues. These are not primarily failures of AI capability. They are failures of visibility — on both sides. The AI could not see what it needed to. And the human could not see that the AI was operating blind.

What useful visibility design looks like

The response to this problem is not to avoid using AI systems in consequential situations. It is to be deliberate about what you put in front of them — and to build that deliberateness into the process rather than leaving it to chance.

The most effective approach treats information design as a first-order decision, not an afterthought. Before deploying a system for a particular use, the question to ask is not 'what can this system do?' but 'what does this system need to see in order to reason about this situation well?' That question, asked carefully, usually surfaces the gaps immediately.

It also changes what governance looks like. Rather than reviewing only the outputs of an AI system, effective oversight includes reviewing what went in — checking that the information provided was current, representative and complete enough for the decision at hand. This is the distinction between humans reviewing AI outputs and humans supervising AI behaviour, and it is a meaningful one.

Practical questions for leadership teams

The following questions are not a technical checklist. They are prompts for a conversation that most leadership teams have not yet had about the AI systems they are currently running or considering.

• For each AI system in use, can you describe — in plain terms — what information it can and cannot access? If not, who in the organisation knows?
• When was the information your AI system works from last updated? Is that update frequency appropriate for the decisions it is supporting?
• What tacit knowledge is critical to the business areas this system operates in? Has any of it been deliberately encoded, or does it exist only in people?
• Who is responsible for reviewing not just what the AI produces, but what it was given to work with?
• When the system produces a confident-sounding output, what is the process for assessing whether the confidence is warranted?

The practical discipline

There is an analogy worth holding onto. A very capable analyst who has been given access to only part of the file will produce analysis that reflects only that part. If they are diligent, they will flag the limitation. But if they do not know the rest of the file exists, they will not know to flag anything — and their analysis will be presented, and likely received, as complete.

AI systems are, in this respect, like very capable analysts who never ask for the rest of the file. They work with what they have been given. The discipline of deciding what that is — and reviewing it regularly — sits entirely with the people running them.

That discipline is not complex to establish. But it requires acknowledging that the boundary of what an AI system can see is a design choice, not a default. And that left unexamined, it tends to be drawn too narrowly.

If there is a recurring situation in your business where you are relying on an AI system's output without being certain it has the full picture, that is worth a conversation. The gap between what the system sees and what is actually true is usually smaller than it first appears — and almost always addressable.