You've done it. You dropped a 20-page contract into an AI tool, asked it to flag the risky clauses, and walked away feeling covered. Then someone read it carefully and found the clause that mattered. You pointed it out to the AI. It said, "Oh, you're right."
That's the problem. Not the miss. The silence.
At All Things AI 2026, I walked through why chat-style AI fails in high-stakes document review and what it actually takes to build a system you can trust. The demo used contract review as the example, but the patterns apply anywhere documents, policies, or compliance checklists need to be reviewed reliably.
When you hand a large document to an LLM naively, a few things happen that you don't see:
.doc files extract as hundreds of lines of SharePoint XML garbage. The model doesn't flag it. It just processes whatever came through.This is what I call the tragedy of the context. The LLM is confident. The output looks complete. But you have no idea what it didn't read.
The mental model most people use for AI context windows is wrong. They treat it like storage: add more, get more.
But context degrades as it fills. LLMs pay more attention to the beginning and the end. The middle gets fuzzy. The longer the conversation runs without a reset, the more noise accumulates and the less reliable the outputs become.
The right mental model: context is a workspace. You bring in what you need, do the work, clear it out, and bring in the next thing. That shift in thinking is the foundation for everything that follows.
Instead of feeding an entire document to the model, parse it first using deterministic code. Split on structural headings. Turn a 20-page contract into 97 discrete clauses. Store those in a vector database like ChromaDB.
When the model needs to review a specific issue, it retrieves the relevant clauses by semantic similarity, not by reading the whole document. The result: a full document review session that uses fewer than 9,000 tokens in context. If you are running a smaller model with a 50,000 token window, you are still well within budget.
The key shift here is moving from "feed the LLM everything" to "give the LLM exactly what it needs."
"Find the troubling items in this document" is not a useful prompt for high-stakes review. It's open-ended, hard to verify, and produces inconsistent results.
What works better is defining specific acceptance criteria for each review task. Evaluate IP and ownership issues. Evaluate indemnification clauses. Evaluate submission requirements. Each task has its own target, its own search pass through the vector database, and its own output format.
This is eval-lite thinking borrowed from ML evaluation frameworks. You define the goals first. You measure against them. You can audit what was checked and what was found.
A single agent reviewing a complex document is the same problem at a different scale. It still fills a single context window. It still misses things as the window degrades.
The better approach is orchestration: multiple agents, each with its own clean context window, each assigned a specific job. One agent handles IP review. Another handles red flag identification. Another handles adversarial review, taking a second pass to challenge the findings of the first.
Each sub-agent gets 200,000 tokens of fresh context. None of them carry the noise of the full conversation history. The orchestrator assembles the results into a final report.
In the demo (video below), one slash command triggered the full pipeline: document intake, clause extraction, multi-agent review, adversarial validation, and a formatted report with critical and high severity findings. The pipeline reviewed all 97 clauses and produced an audit log of every action taken, stored locally in the MCP server.
The patterns above handle how the review runs. This one handles how you prove it did.
Every action the agent took during the review session was logged to a local database via a post-chat hook. It captured who reviewed what, when, and what it found.
The audit log is what makes the whole system trustworthy. It's what lets you say with confidence that the system actually reviewed clause 47, not just that it produced output that looks like it did.
Human-on-the-loop is the right framing here. Not human-in-the-loop, where you approve every step. Not fully autonomous, where you trust blindly. You design the process. You define the criteria. You review the output. The agent does the work at scale.
If your team is using AI for document review today, ask these questions:
The tools to build this exist today. The standards for defining agents, skills, and hooks are mostly platform-agnostic. The patterns scale beyond contract review to policies, compliance audits, code review, and test planning.
The barrier isn't the technology. It's building the system intentionally instead of assuming a chat interface is enough.
Watch my All Things AI talk “Orchestrate Agentic AI: Context, Checklists, and No-Miss Reviews.”
All demo assets, agent definitions, and the Claude Cowork plugin from this talk are available in this GitHub repository. The slides are there too if you want to follow along.
Ready to build AI your team can trust? Explore Six Feet Up’s AI capabilities.
Six Feet Up, Inc.
11208 Windermere Blvd.
Fishers, IN 46037 - USA
Phone: +1 (317) 861-5948
Email: info@sixfeetup.com
