Code That Writes Code That Breaks Code: The Infinite Loop Problem

Nobody designed the bug. A developer prompted an assistant for a utility function six months ago, something mundane that would have taken twenty minutes to write manually and took thirty seconds to generate. The function worked, passed review without much scrutiny, and merged cleanly. Another developer, fixing an unrelated performance issue last month, prompted a different tool to optimize that same function. The optimization removed defensive checks that seemed redundant but actually handled nested null values correctly. A third tool, asked to generate tests based on current behavior, wrote assertions validating the edge case as correct. A fourth tool, asked to update documentation, described that behavior in prose that made it sound intentional. The bug now has passing tests and clear documentation. No human ever decided it should exist.

This is not a story about tools making mistakes. Tools have always made mistakes. Humans have always made mistakes. The difference is that human mistakes usually have a human somewhere who can recognize them as mistakes. When code writes code that writes code, the recognition mechanism breaks down. Each generation assumes the previous generation was correct. Errors become features. Assumptions harden into requirements. The system drifts not because anyone decided it should drift but because no one was positioned to decide otherwise.

The Pipeline Is Already Running

The pathways for self-referential development exist now in daily workflows, operating quietly beneath productivity gains that dominate conversation. A developer prompts for code, reviews it with varying scrutiny depending on time pressure, and commits. Weeks later, another developer encounters that code and asks for an explanation. The explanation describes what the code does, not what it was meant to do. That explanation becomes documentation. Still later, a third developer asks for a refactor based on that documentation. The refactor matches the documented behavior, already one step removed from original intent. Tests accelerate the drift. Generated tests validate current behavior as correct. Future changes that would restore original intent fail those tests. The developer abandons the change rather than investigate. Original intent becomes actively difficult to restore.

Documentation completes the loop. New team members learn drifted behavior as intended because that is what documentation says. They later prompt tools with their understanding, and those tools generate code consistent with that understanding. The feedback loop closes. Human intention, once the anchor, becomes just another input whose influence attenuates over successive iterations. The system evolves according to accumulated logic of previous generations, each building on output of the ones before. Humans remain technically in the loop, clicking approve and merge, but their role shifts from authors to editors to spectators watching a system they no longer fully understand.

Warning Signs Your System Is Already Drifting

Early symptoms are already visible but rarely attributed correctly. Support teams notice behavior that surprises even engineers who built the system. Product managers discover features work differently than documented, and nobody can explain when or why. Onboarding takes longer because the system makes sense only as local behaviors that sometimes contradict. These symptoms are treated as normal complexity. They are warnings that distance between human understanding and system behavior has grown beyond safe limits.

Specific indicators that drift has already taken hold include:

• Tests pass but behavior surprises. The test suite is green, yet the system does something no one expected. This means tests validate current behavior, not intended behavior.
• Documentation is clear but wrong. Documentation reads well and seems comprehensive, yet describes behavior that differs from what stakeholders remember specifying.
• No one can explain why a feature works the way it does. Multiple engineers understand the code locally, but no one can trace the original reasoning behind the current implementation.
• Refactors consistently introduce subtle bugs. Changes that should be safe keep breaking edge cases because the edge cases were never explicitly specified, only accidentally handled.
• Onboarding relies on oral tradition. New hires cannot learn the system from documentation alone. They need veterans to explain "how things actually work" versus what the docs say.
• Code comments describe what, not why. Comments explain mechanics but never rationale. The original tradeoffs and constraints that shaped the design are absent.

Not Just Technical Debt

Technical debt is knowingly accrued and can be repaid later. This drift does not fit that model. The debt is not knowingly accrued. The code looks fine. It follows patterns. It has tests. It works. The problem is that it no longer corresponds to any human's understanding of what it should do, and no individual change would have been flagged as problematic. The danger is not catastrophic failure that triggers investigation. It is slow, cumulative drift that never triggers alarms because each change is too small to notice. Over months, drift accumulates until the system becomes effectively unmaintainable, not because anything is broken but because nothing is fully understood. At that point, options are limited: live with a system nobody comprehends or rebuild from scratch.

Building Intentional Friction

Resistance does not mean rejecting code generation tools. It means building friction into the pipeline that forces human attention where drift is most likely. Reviews of generated code must be more thorough than reviews of human-written code, not less. Human code carries implicit intent a reviewer can infer from knowing the author. Generated code has no author in any meaningful sense. Reviewers must reconstruct intent from the prompt and surrounding system, asking not just whether code works but whether it aligns with what the system should do.

Concrete practices that introduce productive friction include:

• Require intent comments on generated code. Every block of generated code should include a comment explaining what it is meant to do, written by the human who prompted it. Future readers need this anchor.
• Label generated tests explicitly. Tests that validate current behavior should be marked as behavioral tests, distinct from specification tests that validate intended behavior. When they diverge, the divergence is visible.
• Verify explanations against original specifications. Before generated documentation is accepted, compare it against whatever record of original intent exists. Flag discrepancies for human resolution.
• Maintain a separate intent log. Keep a lightweight record of what features were supposed to do when they were first built. This becomes the anchor against which drift is measured.
• Rotate code ownership deliberately. When the same person reviews their own generated code, drift accelerates. Different eyes on different stages of the pipeline catch assumptions the originator missed.
• Schedule intent reviews. Periodically review whether current behavior matches documented intent. Not bug hunting. Drift detection. Catch the gap before it becomes the new normal.

What Breaks Without Intent Anchors

When systems drift beyond human comprehension, specific failure modes emerge that differ from ordinary software failures. Ordinary failures produce errors. Drift failures produce confusion. The system continues operating but no longer aligns with business expectations or user needs. Features that were critical become vestigial. Behaviors that were accidental become relied upon. Changing anything becomes dangerous because the full implications are unknown.

Common consequences of advanced drift include:

• Regulation compliance becomes uncertain. When no one knows exactly what the system does, proving it complies with legal requirements becomes impossible.
• Incident response slows dramatically. Debugging requires archaeology. Understanding comes before action, and understanding takes hours or days.
• Feature development becomes speculative. New features are built on assumptions about existing behavior that may be wrong. Integration fails in unexpected ways.
• Vendor lock-in deepens. The original engineers who understood the system leave. Remaining staff cannot confidently migrate or replace components.
• Business continuity planning becomes guesswork. Disaster recovery assumes you know what the system does. When that knowledge erodes, recovery plans become fiction.

Who Is Still in Control

The self-referential pipeline is not a failure of technology. It is a failure of attention. Each tool does its job competently. The problem emerges from interaction between tools when no human sees the whole chain. Each developer sees only their own interaction. No one sees the cumulative effect of dozens of interactions layered across months. The system drifts because drift is the natural consequence of optimizing for local efficiency while ignoring global coherence.

Fixing this requires acknowledging something uncomfortable. The human in the loop must actually be in the loop, not technically present while functionally bypassed. Clicking approve after a thirty-second scan is not being in the loop. Accepting documentation without verification is not being in the loop. Trusting tests because they pass is not being in the loop. Being in the loop means actively comparing output against understanding and flagging divergences. This is slower. It requires more effort. It produces less visible output. And it is the only thing preventing the system from evolving into something no one understands or intended.

The loop is not inevitable. It is a choice organizations make every time they prioritize velocity over comprehension, every time they treat generated output as authoritative, every time they skip verification because the output looks plausible. The organizations that avoid drift will not be those with the most sophisticated tooling. They will be those with the most disciplined human processes around that tooling. The difference becomes visible only over time, as some systems remain comprehensible while others transform into black boxes that function but cannot be explained. By the time the difference is obvious, it is too late to correct easily. The code that writes code is already in the pipeline. The only question is whether anyone is still paying enough attention to notice before the drift becomes permanent.