When we use AI tools to navigate, audit, or refactor a massive codebase, we make a massive assumption: We assume the AI is looking at the executable code. We treat Large Language Models (LLMs) like compilers that read raw abstract syntax trees. But recently, I hit a massive roadblock that proved the exact opposite. If a file contains a detailed comment, an AI will often choose the path of least resistance: it will read, trust, and argue based on the natural language comment while remaining completely blind to the actual code execution beneath it.
The Backstory: The Stale Comment Trap
A few days ago, I had an Claude help me modify a function in a large codebase. It did a great job changing the logic, but it made a critical housekeeping error—it forgot to update the long, descriptive comment sitting above the function. Fast forward to a completely new session. Because AI has no memory between sessions, it was looking at this file with a clean slate. I asked a simple structural question about our system's architecture: "This system follows XYZ rules, correct?"
The Blind Spot in Action
The Claude confidently told me, "No, it does not follow those rules." I knew it did because I had personally rewritten the logic. I pushed back, trying multiple different phrasing styles to get it to look at the implementation. But no matter how I asked, the AI refused to budge. It was completely locked into its answer. Finally, out of frustration, I changed my strategy. I stopped asking conceptual questions and demanded physical proof: "Give me the exact line of code where it is not following the rules so I can understand and change it." The moment the Claude had to physically map its reasoning to actual line numbers of code, the illusion shattered. It replied:
"Sorry, I am wrong. There is no code like that. It is just a comment there and I was checking that only. You are right, you can add that 'This System Follows XYZ rules.'"
Why Do LLMs Choose Comments Over Code?
This isn't an isolated bug; it is an inherent characteristic of how language models process information:
- Token Probability and Natural Language: LLMs predict text based on probability. Natural language comments match their training data far more fluidly than strict, abstract code syntax. To the AI's internal attention mechanism, a beautifully structured comment is a massive "anchor" that pulls focus away from the raw code.
- The "Truth" Assumption: In billions of lines of open-source training data, a code comment matches the code below it 99% of the time. The AI is fundamentally conditioned to assume that Comments = The Definitive Guide to the Function. When they mismatch, the AI defaults to trusting the text over the syntax.
How to Protect Your Large Codebase
If you rely on Claude or Any AI to help you manage and scale a complex repository, you must actively code around this blind spot.
- The Code-First Prompt Constraint: When auditing architecture, explicitly blind the AI to comments.
- Example: "Strictly ignoring all comments, docstrings, and headers, analyze the executable logic of this function to see if it follows XYZ rules."
- Enforce Clean-Up Workflows: Make it a strict habit that whenever an AI edits a function, its final task must be to rewrite or entirely delete the accompanying comments to prevent stale documentation from poisoning future sessions.
- Demand Receipts Instantly: Never accept a generic "Yes" or "No" from an AI regarding your codebase. Always force it to provide line numbers or exact code block quotes in its very first response.
Have you caught an AI hallucinating system architecture because it read a stale comment or README file instead of the actual code?