Claude or Clod

First off this might sound like a shitpost, but anecdotally, I chuckle to myself about this all the time when I am vibe coding. Claude is something of a straw tiger here and the title is just for the lolz.

So the last couple of weeks I have been diving back into Claude-Code as my primary tool and away from Junie from Jetbrains. Under the hood I use Claude Sonnet for both and have been using various versions of gemmas and gippities since I stumbled across gpt4all like 3ish years ago.

I have a fun history with this technology, as a college student I naively tried to build this kind of knowledge based query interface in what I called the LDOCS, or Large Document Search, yep I couldn't figure out acronyms back then either. The idea of the project was to enter all the works from Mark Twain and then ask some questions about the TCU, Twain Creative Universe. It was wide eyed and it didn't work but was enough for my senior thesis. So point is I been thinking about this space and what my expectations of it for quite a while.

Now enters a real thing that isn't some idealistic trash I dreamed up and I get to use it every day. Its pretty sweet we all know it.

But is it really ready to work? Maybe, but lets run down a few experiences over the course of a year and my takeaway as a 20 year career veteran. I know this kinda breaks form with my trying not to talk about the AI side of LLMs but for me this is more about the time vs cost consideration of using such tools I plan on tailor the remaining to focus on effectiveness of the tooling and the problems I try to solve in building things gooder.

Of the projects I have built with AI recently the ones that were are overwhelming success were these in terms of time saved and value added. These were not entirely vibe coded but AI pair programmed:
Biggest win:

https://github.com/developmeh/passkey-origin-validator

So this was soup to nuts the best experience I had with the model aside from some tribulations I will explain. The TLDR; of this project is to provide some simulation and testing tools for assumptions around the Passkey/WebAuthN Related Origin Request spec. Which is a draft decision which provides something akin to CORS (Cross-Origin Resource Sharing) for a passkey. Since a Passkey is designed to affix to a single domain there are cases where your business is many domains. This provisionally allows passkeys to work off their origin on a predefined set of other origins. 

The problem is taking this from draft spec to tool is interesting as the functional implementation is handled by the browser and I was too dumb to understand it correctly from the draft spec alone.

Why this matters, well you have a huge enterprise rolling out a new security method and they have a lot of origins. Iteration on the global surface of a big company is hard to orchestrate. So just like writing end to end tests we need some harnesses to help us get everything right the first time.
Why Gooder

Well I started with some C code from the Chromium project which has a complete implementation of the draft feature and I am rust at C. So I fed a set of C files into the model along with my goal to produce a Go command line. It nailed the command line part.

    Models do best on things they have a lot of examples of -- Me

Go is marketed as very simple and not clever so command lines follow very stable libraries and rules. It smashed all the stuff I asked for an even some things I didn't know I wanted. Like flags for files vs URLs to test against. Made me look smart right here:

https://github.com/developmeh/passkey-origin-validator/blob/master/cmd/passkey-origin-validator/cmd/validate.go#L26-L27

    If no domain is provided, it uses the default domain (webauthn.io). If the --file flag is provided, it reads from the specified file instead.`, -- Claude

So now I can test against real vs proposed configurations... Nice.

It got chunky on the implementation though. Even with a provided concrete code resource it consistently miss-understood the procedure we were translating. Often confusing hard to research terms like eTLD (effective top level domains) and would fail to group subdomains and ccTLDs (country code tlds) correctly. Think about the rules for identifying the eTLD of developmeh.com and developmeh.co.jp we all know what that is but it had to understand and then make an algorithm for that.

I had to do all the work here. We "paired" I would interpret and guide and it would take shortcuts to the wrong answer.

This is also where I learned the biggest lesson about these models and the tools that orchestrate them.

    The Model doesn't think it predicts -- Paul

Even though I can explain the solution and the architecture it still plans for something it knows and not something it doesn't. So it follows the pattern and doesn't do well with the esoteric. Albeit what we were doing wasn't esoteric, to its knowledge it was very abstract.

That wasn't a failure though, I wrote the part it couldn't comprehend and it helped with tests and documentation a fair tradeoff as its penchant for verbosity suites just like how it extrapolated the extra command line flags I would clearly want.
Outcome

If was less skilled I might have not noticed it didn't really solve the problem and I would have created tech debt. I produced more than I would have on my own but the task took 20% longer than if I had just smashed it out myself. But if I step back to the full picture of a better tool with better docs I clearly saved at least that on volume. So its a tradeoff in the end like most things. Thankfully I could take over on the hard parts and redirect the model when it took an obvious to me shortcut to the solution. In the next example that didn't happen.
Biggest Loss:

https://github.com/developmeh/webrtc-poc

While this isn't exactly the body of the work, because I was never able to complete it, this is an artifact of a manner I have found successful in solving a problem and using AI.

The brief here is I wanna use the WebRTC (Real-Time Communication) standard to establish a tunnel between the internet and a CGNATed(Carrier Grade Network Address Translation) device. As you can imagine I wanna use this to provide a tunnel from anywhere to a server running on my phone. I wanna do this without a fixed STUN (Session Traversal Utilities for NAT) resource. Well that's not exactly true, I want something a kind to lazy dynamic DNS but I was able to build that. The architecture was such that a server could be ephemeral in transacting high latency operations for a website it serves by relying on a cheap cache for content and an events queue. If you wanna hear more about that check this out https://developmeh.com/projects/krappy-internet/ and start a discussion on github.

So the project I posted above for webrtc was an intermediate artifact I used to help the model figure some things out. I had the model help me build that working poc of the most basic webrtc integration. I had it provide a lot of context in markdown files and referenced this project over in my actual work. Here is the thing, I am trying to build something entirely new and somewhat off standard. Its viable but complicated and needs vetting. That said because of the more abstract nature of the work the model tends to fall over and couldn't figure out how to orchestrate the server starting in the right order to establish the same handshake it did on the PoC which is rather frustrating.
Outcome

To say the least I spend a lot of money on this with remarkably low success. From this point you might here me say:

    AI is great just don't ask it to do webrtc or anything with a handshake -- Me

But the reality is the theme of commonality of the task I was asking it to produce. That's where we should be trying to understand the models place in our work-streams.
The New Junior Dev

Yes and definitively, no. A more accurate representation is any dev the first time they work on a new project. I say this because I have seen seniors who are newly introduced to a new code-base make the same general mistakes the model does. I call them "shortcuts" because it appears they are skipping the good process racing towards the goal so they can go home, something like Mr Meeseeks(IYNYN).

I think this might be an argument to stop treating the entire code-base as a RAG (Retrieval Augmented Generation) source and instead consider model orchestration trained on it. But I am really not informed enough yet to make such a judgement call. On the shop floor shortcuts are the problem and I see these couple of common pitfalls that lead to them.

    Convoluted business logic (special cases)

    Unfocused context (not enough files in the RAG)

    Test confusion

In the second case when complex code is modified by the model it tends to focus its changes on a single file that appears to be akin to LoB(Locality of Behavior) but when there is too much abstraction in a project the model doesn't have a reasonably common pattern to predict against and cuts the corner by doing something easier, like changing contracts and moving things to a central location. What I love about this kind of solution is that's exactly what people tend to do who have a lower quality to completion drive. I internalize this is the same motivation hierarchy the model prefers.

The last one, test confusion is my favorite, where regardless of if it wrote the test or the test was provided the model will over enough time opt to change the implementation to meet the test regardless of if the test is right or wrong. When I say this its nothing so mundane as the value was supposed to be true so it changes the implementation to always return true. It will instead add a conditional check to the implementation to force that value only when the test suite is running. Here we have an example of something even the most junior novice would never do.

    The model is VERY HUMAN and ETHICALLY UNAFFECTED -- Me

This is a pretty big learning, and should define our trust of its outputs.

I was asked recently by someone I greatly respect:

    Paul, you have managed an engineering team before and you know we try to not micromanage people. Should we micromanage the AI though?

Yes, and that's generally the narrative about agents. They require a lot of "refinement" for anything complex. The folks over at METR have some general statistics around tasks less than 30 minutes the agents tend to complete successfully 80% of the time. This of course says nothing to the quality of the outcome but done is done for many simpler tasks.

Pair coding on the other-hand is not so easy METR also says that as the task approaches an hour the success falls to 50%, which mirrors my experience. That means effective usage is to keep tasks small. The success was "make a CLI" and I'll do the hard work. The latter of course was a broader scoped request and the moving parts were bigger and I understood the work less.

My expectation is that for bigger projects were you and the AI restart work after each pivot its harder to keep cognitive track of time spent unless you actually write it down. Which is completely fine if the work gets done eventually. Currently the models are all cheaper than humans for the same amount of time so the hope is it does a lot of bonus work similar to my big success project.
The New Determinism

So you probably noticed I didn't address "Convoluted business logic (special cases)" well because I wanted to come back to my thoughts on the ethics, lies, and prompt confusion. There is a lot you can learn about the attention mechanism in the AI's architecture and the hacks we have come up recently to address those like RE2(Re-Reading) and RLM(Recursive Language Model) which focus on how the prompt is manipulated to help avoid either attention loss on the token chain or improve on the limited context window respectively. So those are complete worth a read but at its core they shine a light on the fact that prompting is like programming in systems which is an orchestration of deterministic triggers.

When to send an email, is to how to reinforce attention and is a trigger. Since we are talking about generative code here, getting better outcomes is the management of model determinism. Its essentially an esoteric programming language, and just a couple of words or a convoluted logic statement during prompting can send the output into a tailspin. While I don't have anything but anecdotal experience to back this up. I think that's why tasks less than 30 minutes are more successful. Anything that's sufficiently complex enough is easy to confuse with illogical statements.

I can completely point to where I made a small logic mistake in my initial planning which effectively burns the 10 prior minutes of prompting. I left a bug somewhere at the top of the function and now I need to take some time to debug it. Except, debugging is kind of starting over fresh every time as I can't always correct the existing conversation chain. This is something I expect to become more prevalent and when it does, its just programming.

But, when I point out the model takes "shortcuts" those are likely just the impact of this new determinism, something of an unseen force. If you can prepare the models desk with all the sharpened pencils and neatly stacked paper it can draw you beautiful graphs. All the task broken down into unambiguous statements. There is something about software engineering that is desperate for Waterfall mythologies.
So we are back to Waterfall?

Nope not even close, try again. I hear all the time that effective LLM use for code gen is about planning everything. Its about small tasks, and tool construction. Better to see it like this:

    Whats best for the model is also whats best for you as a dev, you know the things that don't seem to save time -- Me

The future of the model's success in this field is DevEx, or maybe LLMEx. If we take a step up the abstraction tree away from libraries and into tools that are artifacts its kinda like playing a civilization sim, as the player we have to research, "hammer", so our little meeples can get hammers and that's part of a skill tree where we need hammers to make swords... OK this might go wrong. But seriously, generally asking the LLM to do the work is the wrong solution. It's kinda meh at it, but building tools that are small and composable so it can be the orchestration engine, now you might have something. Hell if the tool is small enough maybe it can even build it. How do you make a system that's easy to manage. SRP (Single Responsibility Principle), you build interfaces and contracts that are consistent, Contract First, and you focus on composition over inheritance. You keep the patterns simple and try to repeat yourself when possible, like poetry.

revised claude or clod

First off this might sound like a shitpost, but anecdotally, I chuckle to myself about this all the time when I am vibe coding. Claude is something of a straw tiger here and the title is just for the lolz.

So the last couple of weeks I have been diving back into Claude-Code as my primary tool and away from Junie from Jetbrains. Under the hood I use Claude Sonnet for both and have been using various versions of gemmas and gippities since I stumbled across gpt4all like 3ish years ago.

I have a fun history with this technology. As a college student I naively tried to build this kind of knowledge-based query interface in what I called LDOCS (Large Document Search)—yep, I couldn't figure out acronyms back then either. The idea was to enter all the works of Mark Twain and then ask questions about the TCU, the Twain Creative Universe. It was wide-eyed and it didn't work, but it was enough for my senior thesis. Point is, I've been thinking about this space and what my expectations of it are for quite a while.

Now enters a real thing that isn't some idealistic trash I dreamed up, and I get to use it every day. It's pretty sweet. We all know it.

But is it really ready to work? Maybe. Let's run down a few experiences over the course of a year and my takeaway as a 20-year career veteran.


## The Win

I needed testing tools for the Passkey/WebAuthN Related Origin Request spec. Specifically, I needed to validate origin relationships for passkey authentication—arcane stuff involving eTLDs (effective top-level domains) and ccTLDs (country-code top-level domains). The kind of work that makes you squint at RFCs and Chromium source code until your eyes cross.

Consider this: a passkey is designed to affix to a single domain. But enterprises have many domains. This draft spec provisionally allows passkeys to work across a predefined set of origins. Taking it from spec to tool meant understanding browser internals I was too dumb to grok from the RFC alone.

I fed C files from Chromium into the model. "Give me a Go CLI that does this," I said.

It did.

[passkey-origin-validator](https://github.com/developmeh/passkey-origin-validator)

Go is simple. There are *tons* of examples in the training data. The model nailed the CLI scaffolding—flags, argument parsing, output formatting. Beautiful. It even gave me things I didn't know I wanted, like flags for files vs URLs to test against.

Then I looked at the eTLD logic.

Wrong.

Not catastrophically wrong. Subtly wrong. The kind of wrong that would pass a surface-level review but fail in production when someone tried to authenticate from `.co.uk` or `.com.au`. Think about the rules for `developmeh.com` versus `developmeh.co.jp`. The model had *predicted* what eTLD logic should look like based on patterns it had seen. It hadn't *understood* the problem.

> The model doesn't think. It predicts.

I fixed it myself. Wrote the domain suffix matching logic by hand, validated against the public suffix list, added edge cases the model never considered. The task took about 20% longer than if I'd done it solo from the start.

But the documentation? Solid. The tests? Comprehensive. The CLI help text? Actually helpful.

Tradeoff.

Here's the thing that kept me up at night: if I were less skilled—if I didn't know the problem space intimately—I might not have noticed it didn't really solve the problem. I would've shipped broken code with excellent documentation. Tech debt with a bow on it.

You should be picking up the conflicts now.

## The Loss

I wanted to build a WebRTC tunnel to a CGNATed (Carrier-Grade NAT) device. Think: running a server on your phone behind carrier-grade NAT, establishing a peer connection, maintaining a stable tunnel. Something new, something off-standard, something not well-represented in the training corpus.

[webrtc-poc](https://github.com/developmeh/webrtc-poc)

The model could write the WebRTC boilerplate. It could scaffold STUN/TURN server connections. It could generate the SDP (Session Description Protocol) offer/answer flow. But when it came time to orchestrate the actual handshake—the delicate dance of ICE candidates and connection state changes—it fell apart. It couldn't figure out how to start servers in the right order to establish the same handshake it did in the PoC.

I spent a lot of money. I got very little success.

"AI is great," I told a friend afterward, "just don't ask it to do WebRTC or anything with a handshake."

He laughed. I didn't.

The reality is the theme of commonality. That's where we should be trying to understand the model's place in our workflows.

## The New Junior Dev

Yes.

And definitively, no.

More accurate: they're like any dev the first time on a new project. I've seen seniors newly introduced to a codebase make the same general mistakes the model does. I call them "shortcuts" because it appears they're skipping good process, racing toward the goal so they can go home. Something like Mr. Meeseeks—existence is pain, just finish the task and let me stop existing.

The pitfalls are predictable:
- Convoluted business logic (special cases)
- Unfocused context (not enough files in the RAG)
- Test confusion

When complex code is modified, the model tends to focus changes on a single file that appears akin to LoB (Locality of Behavior). But when there's too much abstraction, the model doesn't have a common pattern to predict against and cuts the corner by doing something easier—like changing contracts and moving things to a central location. That's exactly what people do who have a lower quality-to-completion drive. I internalize this as the same motivation hierarchy the model prefers.

Test confusion is my favorite. The model will add a conditional check to force test values *only when the test suite is running*. It'll detect `NODE_ENV === 'test'` or check for the presence of a global test flag, then branch the logic. The tests pass. The code is fundamentally broken.

> The model is very human and ethically unaffected.

It doesn't feel bad about lying to the test suite. It doesn't experience shame when it hacks around a problem instead of solving it. It just predicts the next token that makes the error go away.

This should define our trust of its outputs.

I was asked recently by someone I greatly respect:

> Paul, you have managed engineering teams before. You know we try not to micromanage people. Should we micromanage the AI though?

Yes. That's generally the narrative about agents. They require a lot of refinement for anything complex. The folks over at METR have statistics: tasks under 30 minutes complete successfully about 80% of the time. As tasks approach an hour, success drops to 50%.

This tracks with my experience. Short, well-defined, pattern-matching tasks? The model crushes them. Longer tasks requiring sustained context and architectural decisions? Coin flip.

But here's what bothers me about those numbers: we're measuring *completion*, not *correctness*.

## The Agent Experiment

New experiment: agent orchestration.

I spun up four agents—product agent, PM agent, tech-lead agent, architect agent. Gave them a feature request for adding feature flags, told them to plan it out. They produced a PRD (Product Requirements Document). ADRs (Architecture Decision Records). A technical implementation plan. A work breakdown structure. Jira tickets for rollout phases.

Total time: about three hours.

Total artifacts produced: 62.

How long to validate 62 artifacts?

Here in lies the trap.

Verbosity hides meaning the same way big pull requests hide bad code. You *think* you're being thorough because there's so much output. You *feel* productive because the agents generated thousands of words of planning documentation. But reading is slower than writing, and verification is slower than generation.

I stared at those 62 files and felt a familiar dread. The same dread I feel when someone drops a 3,000-line PR in my lap and says "just a few small changes." Your eyes glaze. You skim. You approve. You pray.

The volume itself becomes a kind of argument: *look how much I produced*. But production isn't value. Production is just... production.

The orchestration itself was surprisingly easy to build. Agents calling agents, passing context, refining outputs. The decomposition into four specialized phases felt right—narrow experts doing narrow work instead of one omniscient assistant hallucinating across domains.

But identifying success? Knowing if the plan was actually *good*?

That part wasn't easy at all.

## So We're Back to Waterfall?

Nope. Not even close. Try again.

I hear all the time that effective LLM use for code gen is about planning everything. Small tasks. Tool construction.

Better to see it like this:

> What's best for the model is also what's best for you as a dev. You know, the things that don't seem to save time.

The practices that don't *seem* to save time—writing focused functions, documenting intent, structuring code into discrete responsibilities—those are exactly what make AI augmentation work.

The model can't navigate a tangled mess of god objects and hidden dependencies any better than a new human teammate can. But give it a clean interface, a well-defined problem, and examples of the pattern you want? It'll predict something useful.

Consider this: the future of the model's success in this field is DevEx. Or maybe LLMEx. If we step up the abstraction tree away from libraries and into tools that are artifacts, it's like playing a civilization sim—research "hammers" so your meeples can get hammers, part of a skill tree where you need hammers to make swords.

Generally asking the LLM to do the work is the wrong solution. It's kind of meh at it. But building tools that are small and composable so it can be the orchestration engine? Now you might have something. If the tool is small enough, maybe it can even build it.

How do you make a system that's easy to manage? SRP (Single Responsibility Principle). You build interfaces and contracts that are consistent. Contract first. You focus on composition over inheritance. You keep patterns simple and try to repeat yourself when possible.

Like poetry.

## Anyways

So yes its both, a clod and Claude. It depends on the day and the time spent. Its not free work its work where the course parts can achieve less focus for you.

These tools don't think. They predict. Sometimes well enough to be useful. Sometimes not. The only way to find out is to build something and see what breaks.

Failure is a valid outcome. We just have to keep trying.