Why your AI keeps starting from scratch at work

One of the first things most of us learn when using ChatGPT, Claude or another general-purpose AI is that better context usually produces a better answer. Give it the background, explain what you are trying to achieve, add the relevant documents and the output improves considerably.

That works well for a task that starts and ends in one session. Business work rarely does.

A sales deal can run for six months. A customer relationship can last for years. A project may involve hundreds of emails, meetings, decisions and changes along the way. Information gets distributed across CRM records, inboxes, meeting transcripts, Slack conversations, documents and people’s own notes. Some of it is still relevant, some has become outdated and some contradicts what was true three months ago.

The problem is therefore not simply giving AI access to more information. It is maintaining an understanding of what matters as the work changes.

This distinction is becoming important enough that the language around AI is changing too. Anthropic describes context engineering as a progression from prompt engineering: instead of concentrating mainly on the instructions given to the model, you think about the entire information state available to it. Their guidance makes another important point: context is finite, and adding more of it can eventually make it harder rather than easier for a model to focus on the information that matters.

That is very different from the early idea that better business AI would mainly come from writing better prompts.

Access to your systems does not mean understanding your work

Connecting AI to business systems is already becoming much easier. Through APIs, connectors and MCP, an AI can increasingly search a CRM, read documents, inspect email and access other tools without someone copying the information into a chat manually.

That is a significant improvement, but I think there is an important difference between access to information and understanding the state of something.

Sales provides a good example because the problem is easy to see.

Imagine asking an AI:

Where do things stand with the Acme deal?

If the AI is connected directly to your CRM, email and meeting recorder, technically it has everything it needs. But it may still need to find the right opportunity, inspect the CRM record, identify the relevant contacts, search the email history, locate recent meetings and read enough of the transcripts to understand what has happened.

Then it has to resolve the meaning behind all of that. Perhaps the CRM says the deal is in negotiation, but the latest customer conversation says budget approval has been delayed. A note from three weeks ago says the champion is taking the proposal to the CFO, but a more recent email shows that the CFO has now delegated the decision to procurement. The next step in the CRM is already in the past.

The raw information is all there. The current state of the deal is not.

The AI has to construct it.

Ask another question tomorrow and much of that work may need to happen again. The model does not automatically have a durable business memory simply because it can reach the systems where the information lives.

This is not unique to sales. The same thing happens when AI works with a long-running customer account, a hiring process, a strategic project or a product initiative. The information may be accessible, while the current understanding still has to be reconstructed.

OpenAI’s own guidance for long-running agents describes the same underlying problem from a technical angle: carrying too much history forward can create distraction and inefficiency, while retaining too little causes the agent to lose coherence. Techniques such as trimming, compaction and persistent memory exist because continuously replaying everything that happened before is not a good long-term architecture. OpenAI: Session memory

Reconstructing context has a real cost

I have become particularly interested in this problem through my current work at Optivian, where it appears in a very concrete form.

A salesperson might connect Claude directly to HubSpot, their email and a meeting recorder and ask it to analyse an opportunity. That works. But the model has to spend part of its effort locating and reconstructing the deal before it can start reasoning about the actual question.

Optivian published a test of this approach using a representative customer run with Claude Cowork. The same deal question was asked with CRM, email and Gong connected directly to the AI and then through Optivian’s pre-processed deal context.

The direct approach consumed roughly 919,000 cumulative input tokens, compared with about 154,000 through the structured context layer. Total billable tokens were 933,294 versus 158,691, around 5.9 times the amount. The direct route also required 30 inference calls compared with 11. These are figures from one representative customer run, not a universal benchmark, but they illustrate how expensive repeated reconstruction can become.

The token cost itself is interesting, but I think the more important issue is what the model is spending those tokens doing.

If a frontier model has to repeatedly search raw transcripts, scan long email threads and piece together what happened before it can answer a question, a meaningful part of the work is information retrieval and reconstruction rather than reasoning.

It is similar to hiring a very capable analyst and giving them access to your entire company every morning, but deleting their notes every evening.

They can work out what is happening. You are simply making them rediscover it over and over again.

Persistent context is not the same as a bigger context window

An obvious answer is bigger context windows. Models can already process far more information than they could a few years ago, and that capability will continue to improve.

I don’t think that removes the underlying problem.

If you have six months of emails, meetings and CRM history, the ideal solution is probably not to put all six months into the model every time you ask a question.

More information is not automatically better context.

Some information is outdated. Some is duplicated. Some is irrelevant to the question. Some matters only because it changes the meaning of something else.

Anthropic makes essentially this argument in its context-engineering guidance. Even as context windows become larger, context pollution and information relevance remain problems. Its recommended principle is to provide the smallest set of high-signal information that gives the model what it needs.

This suggests a different architecture for business AI.

Instead of thinking of context as a pile of source material that gets loaded whenever the AI needs to do something, it can be treated as something that is continuously maintained.

When a new meeting happens, the system does not simply store another transcript. It works out what changed.

When a customer sends an email, it does not just add another document to the archive. It updates what is currently known about the relationship.

Older information does not disappear, but the system maintains a current representation of the important state and knows where the underlying evidence came from.

Then, when AI needs to reason about the situation, it starts from that current understanding and retrieves the detailed source material only when it needs it.

This is roughly the idea behind the deal intelligence layer I see through my work at Optivian. Each customer interaction updates an always-current view of the deal: where it stands, who the stakeholders are, what has changed, what is blocking progress and what may be at risk. The original emails, meetings and CRM records still matter, but the AI does not need to reread the whole archive before every question.

Sales is just a particularly clear example of a more general problem.

Business context is state, not just information

I think this is the part that is easy to miss when talking about AI at work.

Most business systems were designed primarily to store information. CRMs store records. Email stores messages. Meeting tools store recordings and transcripts. Project-management tools store tasks. Documents store decisions and plans.

Humans build the state in their heads.

Someone who has worked on an account for six months knows that the original project scope changed, one stakeholder left, another became more influential, procurement is now the main blocker and the deadline mentioned in the CRM is probably unrealistic. They do not reread the entire history every morning to reach that understanding. They continuously update their mental model as new information arrives.

For AI to become genuinely useful in longer-running work, it needs some version of the same capability.

That does not mean storing everything forever or giving an AI unlimited memory. In fact, that would probably create a different version of the same problem. Useful memory requires deciding what should persist, what has been superseded, what is uncertain and which original source should be retrieved when more detail is needed.

This is why I think context engineering will matter much more than prompt engineering for business AI.

The prompt still matters. The model still matters. Access to the right tools matters.

But if an AI joins a piece of work halfway through and has to reconstruct months of history before it can contribute, there is a limit to how useful it can become.

The interesting shift happens when the AI no longer starts from scratch.

It knows what has happened before, what has changed since, what is currently believed to be true and where to look when it needs the details.

At that point it stops behaving like a very smart stranger with access to your files and starts behaving more like something that actually understands the work.

That, to me, is where business AI gets considerably more interesting.