There's a mass migration happening right now where developers are offloading more and more of their code writing to LLMs. Cursor, Copilot, Claude Code, aider, etc. — the tooling has gotten good enough that the default workflow for a lot of people is now "describe what you want, get code back, review it, ship it." And it works surprisingly well. Until it doesn't.
I've been paying attention to the ways it doesn't work, and the interesting thing is that the failures are not random. They cluster into a small number of recurring patterns, and once you see them, you can predict when they'll happen. More importantly, you can write rules — in your system prompt, your CLAUDE.md, your .cursorrules, whatever — that prevent most of them. But to write good rules, you need to understand why the failures happen. A rule without a mental model is just cargo cult.
This is the big one. When you ask an LLM to "add authentication to this Express app," it's not looking at your app. It's pattern-matching against the distribution of Express auth implementations in its training data and sampling from it. The output is the modal implementation — the one that shows up most often in GitHub repos, tutorials, and Stack Overflow answers. It's a perfectly reasonable auth implementation. It's just not your auth implementation.
This shows up everywhere: wrong libraries (it uses axios when your project uses fetch), wrong patterns (it writes a class when your codebase is functional), wrong conventions (it uses camelCase when your project uses snake_case). The code works. It just looks like it was written by someone who has never seen the rest of the codebase. Because it was.
The mechanism: The model has a strong prior from training, and the context window (your prompt + the files you've shown it) is the only evidence it has to update that prior. If you don't show it enough of the codebase, the prior dominates. It's literally Bayesian inference with a weak likelihood — the posterior just collapses back to the prior.
What this means for your rules: Any effective CLAUDE.md will have "read the codebase before writing" as rule #1, and it's not enough to just say "read it." You need to specify what to read: the files being modified, the import conventions, the test files, how similar features are implemented elsewhere. The more specific you are, the stronger the likelihood term, and the less the prior leaks through.
Every implementation involves choices that aren't specified in the prompt. JWT vs sessions. Postgres vs SQLite. REST vs GraphQL. Optimistic vs pessimistic locking. I call these "hyperparameters" of the implementation because they're not learned from the prompt — they're set before generation begins, usually implicitly.
The problem is that LLMs pick these silently. They don't say "I'm choosing JWT because X." They just... use JWT. And you don't notice until you're three files deep and the auth architecture doesn't match what you had in mind.
The mechanism: The model has to commit to a token sequence, and the first few tokens of an implementation effectively lock in the approach. There's no built-in mechanism for "pause here and ask whether this is the right direction." The autoregressive nature of generation means decisions get baked in incrementally, one token at a time, and by the time you see the output, the architectural decision was made somewhere around token 50 and is now load-bearing for the other 500 tokens.
What this means for your rules: You want a rule that forces the model to state assumptions before writing code. But you also want to identify which assumptions matter most — specifically the irreversible ones. Changing a variable name is cheap. Changing the database schema is expensive. Changing the auth architecture three sprints in is a rewrite. Your rules should be weighted by reversibility cost.
This extends to destructive operations too. Deleting code, dropping columns, running migrations — these are irreversible state transitions. The model should be required to flag these with the same urgency as architectural decisions, because the cost of getting them wrong is the same: you can't undo them.
This is one of my favorites because the analogy to ML is almost exact. A model with too many parameters relative to its training data will overfit — it'll memorize the noise instead of learning the signal. LLM-generated code does the same thing: it produces abstractions, configuration layers, and extension points that vastly exceed what the actual requirements justify.
You ask for a function that sends a welcome email. You get an EmailService class with a strategy pattern, a template engine, retry policies, and a provider abstraction. The model has seen thousands of email-sending codebases in its training data, and it's generating the average of all of them. That average includes every abstraction anyone has ever needed. But you don't need the average. You need sendWelcomeEmail(user).
The mechanism: The model is optimizing for plausibility, not minimality. A 200-line implementation with proper abstractions looks more like "production code" in the training distribution than a 15-line function. So the model generates the 200-line version because it assigns it higher likelihood. It's doing maximum likelihood estimation, and the MLE for "production code" is over-engineered code.
What this means for your rules: The hard part is that "don't over-engineer" is not a verifiable rule. The model can't easily evaluate it on its own output. You need to reformulate it as something checkable. The best version I've seen: "don't abstract until the same code has been duplicated at least twice." This converts a subjective judgment into a countable predicate. Another good one: "can you remove any abstraction layer and still pass all tests? If yes, it's dead weight." These are the kinds of rules that actually change generation behavior because the model can apply them as a filter on its own output.
LLMs have a "style" in the same way that a generative model has a mode. Left to its own devices, a model will generate code in the style that's most probable in its training distribution: modern ES6+, functional-leaning, well-commented, with descriptive variable names. This is fine if your codebase looks like that. It's not fine if your codebase uses var, has terse variable names, and was written in 2015.
The mechanism: The model's style prior is very strong because style features are incredibly consistent across tokens. Once it starts generating in one style, the autoregressive conditioning keeps it there. Overriding this requires explicit, repeated evidence in the context that the style should be different.
What this means for your rules: "Match existing style" is necessary but insufficient. You might need to be specific: "use the same quote style, naming convention, indentation, and semicolon usage as the existing file." The more concrete the instruction, the more likely it overrides the prior. This is also why showing the model the actual file it's editing — not just describing what it should do — is so important. The file itself is the strongest style signal.
There are certain classes of bugs that LLMs produce where the code looks correct on casual review but is fundamentally broken. These cluster around two areas: security and concurrency. And the reason is the same for both: the model's training distribution is dominated by code that runs in a single-user, trusted-input, sequential-execution environment.
Security: The vast majority of code on GitHub does not properly handle adversarial input. Tutorials routinely show db.query('SELECT * FROM users WHERE name = ' + name) because it's shorter and the tutorial is about something else. The model has seen this pattern millions of times and will reproduce it confidently. SQL injection, XSS, hardcoded secrets — these are all cases where the modal training example is the wrong example.
Concurrency: Most code in the training set runs sequentially. The model has limited exposure to concurrent execution patterns and almost no ability to reason about interleaved execution. Race conditions, missing awaits, unhandled promise rejections — these bugs don't manifest in the single-threaded test environment that the model implicitly assumes.
What this means for your rules: You need explicit rules for these because the model won't get them from its prior — its prior is actively wrong here. "Validate all user input," "never hardcode secrets," "consider race conditions when mutating shared state" are not things the model will do by default. They need to be in the system prompt, and they need to be specific. "Think about security" is too vague to shift behavior. "Never concatenate user input into a SQL query; always use parameterized queries" is specific enough to actually work.
When an LLM edits existing code, it has a strong tendency to "improve" things it wasn't asked to improve. Rename a variable that's fine. Reorder imports. Add comments. Reformat whitespace. Convert a for loop to a .map(). Each individual change is arguably an improvement, but the aggregate effect is catastrophic for code review: the actual change is buried in noise, the git blame is polluted, and there's a nonzero chance one of the "improvements" introduced a subtle bug.
The mechanism: The model generates tokens that are high-probability given the context. When it sees code that doesn't match its style prior, "fixing" it is the high-probability continuation. It takes an explicit constraint to override this. Without that constraint, every code edit becomes a mini-refactor.
What this means for your rules: "Minimize the diff" is good. "Every changed line must be directly traceable to the task" is better because it's mechanically verifiable — you can look at a diff and check each line. This is also one of the strongest arguments for showing the model the existing file before asking for changes: the style signal from the file competes with the style prior and reduces the urge to "fix" things.
If you're writing a CLAUDE.md or similar ruleset, the above gives you a framework for why each rule needs to exist. The most common mistake I see in these rulesets is listing rules without understanding the failure mode they're preventing. The result is rules that are either too vague to change behavior or too specific to generalize.
A few principles for writing good rules:
Priority order matters. Rules will conflict. "Write minimal code" vs "handle all errors" is a real tension. If your rules don't have explicit priority, the model resolves conflicts using its prior, which defeats the purpose. Put "read the codebase first" at P0 because nothing else matters if the model is generating from the wrong distribution.
Verifiable beats aspirational. "Write clean code" changes nothing. "Every abstraction must be justified by at least two concrete use sites" changes behavior. Whenever you write a rule, ask: can the model check whether its output satisfies this before returning? If not, rewrite it.
Token efficiency is not optional. Your rules compete with the actual codebase for context window space. A 3000-token essay explaining why minimal diffs are important is 3000 tokens that could have been the actual source files the model needs to read. Compress your rules into the minimum token count that preserves their meaning. Write the essay for yourself; ship the directives to the model.
Eval your rules. This is the step almost nobody does. Take your CLAUDE.md, run it on 20 representative coding tasks with and without the rules, and check if the rules actually improve the output. If a rule doesn't measurably reduce the failure mode it targets, it's wasting tokens. This is just normal ML practice — you wouldn't ship a model without evaluating it, so don't ship a system prompt without evaluating it either.
The deeper pattern here: LLMs fail at code generation in exactly the same ways they fail at everything else. Strong priors override weak evidence. Mode collapse produces generic outputs. The training distribution doesn't match the deployment distribution. The solutions are also the same: provide more evidence (read the codebase), constrain the output space (specific rules), and evaluate on the actual target distribution (test against your real tasks). If you've trained neural nets, you already know all of this. You just need to recognize it in a different context.