TDAD

TDAD.md

Conclusion

https://arxiv.org/pdf/2603.17973

The main contribution of TDAD is not that it proves Test-Driven Development is ineffective for coding agents, but that repository-aware impact analysis is highly effective. The paper demonstrates that supplying an agent with precomputed code-to-test relationships dramatically reduces regressions compared to both a baseline agent and a TDD-prompted agent.

However, the comparison between TDAD and TDD has an important confounding factor. A TDD workflow requires the agent to generate tests, execute them, analyze failures, revise tests, and iterate before implementation. This process creates large amounts of self-generated context, including reasoning traces, test code, stack traces, and intermediate hypotheses. As a result, the TDD agent may suffer from context accumulation, premature commitment to an incorrect theory of the bug, and reduced implementation quality due to longer trajectories.

By contrast, TDAD primarily provides compact, high-information-density context such as impacted tests, dependency mappings, and graph-derived relationships. Rather than asking the model to discover what should be tested, TDAD tells the model which existing tests are likely relevant. This shifts the problem from test generation to test retrieval.

Therefore, the strongest conclusion supported by the paper is that graph-based impact analysis and targeted test retrieval improve coding-agent reliability. The evidence is less conclusive on whether TDD itself is detrimental. To establish that claim, future work would need ablation studies that isolate the effects of context growth, trajectory length, context resets between phases, and test-generation overhead.

More broadly, TDAD suggests that deterministic repository analysis may provide greater gains than additional prompting complexity. Instead of spending tokens rediscovering code dependencies, agent systems can precompute that information and provide it as structured context, reducing regressions while improving engineering efficiency.

How TDAD Creates Test-Mapping Context

Overview

The central idea behind TDAD (Test-Driven Agentic Development) is that coding agents should not have to rediscover repository dependencies every time they modify code. Instead, repository structure is analyzed offline and converted into a test-mapping artifact that agents can use during development.

Rather than asking the LLM:

Which tests should I run after changing this code?

TDAD attempts to answer that question ahead of time and provide the answer as context.

---

Step 1: Extract Repository Structure

TDAD parses the repository and identifies:

• Files
• Classes
• Functions
• Methods
• Test files
• Test functions

It also extracts relationships such as:

• CALLS
• IMPORTS
• CONTAINS
• INHERITS

For example:

def create_order():
    validate_order()
    save_order()

becomes:

create_order
 ├─CALLS─> validate_order
 └─CALLS─> save_order

This forms the foundation of a repository dependency graph.

---

Step 2: Connect Tests to Production Code

The system establishes links between tests and the code they exercise.

Example:

test_create_order
        ↓
   create_order

Multiple signals can be used:

Direct References

A test directly invokes a function:

def test_create_order():
    create_order(...)

Coverage Relationships

Coverage data reveals which functions are executed when a test runs:

test_create_order
    covers:
       create_order
       validate_order
       save_order

These relationships are considered high-confidence mappings.

Transitive Dependencies

The graph can identify indirect relationships.

Example:

test_create_order
        ↓
   create_order
        ↓
    save_order

Even if the test never references save_order directly, changes to save_order may still require running test_create_order.

---

Step 3: Perform Impact Analysis

Once code-to-test relationships exist, TDAD computes likely affected tests for every code element.

Example:

Function:
    save_order

Likely affected tests:
    test_save_order
    test_create_order
    test_bulk_orders

Relationships can be ranked using confidence scores derived from:

• Direct test links
• Coverage data
• Call chains
• Import dependencies

The result is a prioritized list of tests that should be checked when a particular function changes.

---

Step 4: Generate Agent-Consumable Artifacts

Instead of exposing the graph directly to the LLM, TDAD serializes the information into simple text artifacts.

Example:

Function: save_order

Related tests:
  tests/test_save_order.py
  tests/test_create_order.py
  tests/test_bulk_orders.py

Or repository-wide mappings:

save_order
  -> tests/test_save_order.py

create_order
  -> tests/test_create_order.py

validate_order
  -> tests/test_validation.py

The paper references artifacts such as:

test_map.txt
SKILL.md

These files are placed in the agent's workspace and can be searched using normal tools.

---

Agent Workflow

With the test map available, the coding agent follows a simpler workflow:

Modify code
    ↓
Look up impacted tests
    ↓
Run relevant tests
    ↓
Fix failures
    ↓
Submit patch

Instead of:

Modify code
    ↓
Infer architecture
    ↓
Infer dependencies
    ↓
Infer test ownership
    ↓
Guess which tests to run

---

Key Insight

The most important contribution of TDAD is not the graph itself but the externalization of repository knowledge.

The system performs dependency analysis algorithmically and converts the result into compact context that the LLM can consume efficiently. This reduces the need for the model to spend tokens rediscovering code-test relationships and allows it to focus on implementing and validating changes.

In practical terms, TDAD transforms repository structure into a searchable test map, enabling targeted test execution and significantly reducing regressions caused by incomplete impact analysis.

Architectural Quality, Complexity, and Maintainability Were Not Evaluated

Another limitation of the evaluation is that it focuses primarily on behavioral correctness as measured by test outcomes. The paper does not evaluate whether the generated patches preserve or improve the architectural quality, complexity, or maintainability of the codebase.

A patch can pass all tests while still degrading the design and long-term health of the system.

Abstraction Leakage

An agent may solve a bug by bypassing existing abstractions and reaching directly into lower-level implementation details.

For example:

service.repository.connection.execute(...)

instead of using an existing repository interface:

service.repository.save(...)

Both implementations may pass all tests, but the former increases coupling and exposes implementation details across module boundaries.

---

Modularity Degradation

A patch may introduce dependencies between components that were previously independent.

Examples include:

• Cross-module imports that violate architectural boundaries
• Business logic copied into presentation layers
• Database logic embedded into service layers
• Direct access to internal state that was previously encapsulated

These changes may not affect test outcomes but can reduce maintainability over time.

---

Increased Coupling

An implementation may become more tightly coupled to:

• Specific classes
• Concrete implementations
• Particular data structures
• Global state

while remaining functionally correct.

Such coupling often makes future changes more difficult and increases the risk of regressions later.

---

Reduced Cohesion

An agent may place new functionality in a convenient location rather than the most appropriate one.

Examples include:

• Utility methods added to unrelated classes
• Domain logic placed in controllers
• Infrastructure concerns mixed with business logic

Tests may continue to pass even though the overall structure of the system becomes less coherent.

---

Duplication and Technical Debt

A coding agent may duplicate existing logic rather than discovering and reusing an existing abstraction.

For example:

# Existing validation logic
validate_user(user)

# New duplicated implementation
if not user.email:
    ...

The patch may be correct today while increasing maintenance costs and future inconsistency risks.

---

API and Boundary Violations

A patch may introduce dependencies on internal implementation details that were never intended to be part of a module's public contract.

Such violations often remain invisible to functional tests because current behavior remains unchanged.

However, they reduce the freedom to refactor and evolve the system.

---

Cyclomatic Complexity Was Not Measured

The paper does not report any measurements related to cyclomatic complexity.

A patch can preserve behavior while significantly increasing decision complexity.

For example:

if condition_a:
    ...
elif condition_b:
    ...
elif condition_c:
    ...
elif condition_d:
    ...
elif condition_e:
    ...

may satisfy all existing tests while making the code substantially harder to understand, maintain, and extend.

The study does not evaluate whether TDAD-generated patches increase or decrease complexity relative to baseline agents.

---

Cognitive Complexity Was Not Measured

Beyond cyclomatic complexity, modern code-quality tools often track cognitive complexity, which attempts to measure how difficult code is for humans to reason about.

Examples include:

• Deep nesting
• Complex branching
• Multiple levels of indirection
• Long chains of conditional logic
• Complicated control flow

A patch may remain functionally correct while becoming significantly harder for future developers to understand.

---

Method and Class Growth

The evaluation does not report metrics such as:

• Function length
• Class size
• Number of responsibilities per component
• File growth

An agent may solve a problem by appending additional logic to an existing method rather than extracting a cleaner abstraction.

This would likely pass all tests while increasing maintenance costs.

---

Maintainability Metrics Were Not Evaluated

The study does not evaluate broader maintainability indicators such as:

• Maintainability Index
• Dependency complexity
• Coupling metrics
• Cohesion metrics
• Duplication metrics
• Technical debt indicators

These measures are commonly used to assess long-term software quality.

---

The "Correct but Worse" Problem

One possible outcome that would not be captured by the paper's methodology is:

A patch fixes the issue, passes all tests, avoids regressions, but leaves the codebase in a worse state than before.

Examples include:

• More branching logic
• Increased complexity
• Duplicated code
• Reduced modularity
• Greater coupling
• Abstraction leakage
• Larger methods
• Architectural boundary violations
• Additional technical debt

Under the paper's evaluation framework, such a patch would still be counted as a successful outcome.

---

Implications

The reported results support the claim that TDAD reduces test-detectable regressions. However, they do not establish whether TDAD improves overall software quality.

The paper demonstrates that graph-based impact analysis helps agents preserve test-covered behavior, but it does not measure whether agents are producing software that remains maintainable, modular, and architecturally sound.

A more comprehensive evaluation could compare pre- and post-patch values for:

• Cyclomatic complexity
• Cognitive complexity
• Coupling
• Cohesion
• Duplication
• Maintainability indices
• Architectural rule compliance
• Dependency boundary violations
• Technical debt metrics

Such measurements would help determine whether graph-based impact analysis improves not only correctness, but also the long-term quality of generated software.

Key Takeaway

The paper demonstrates improvements in behavioral correctness as measured by tests. It does not evaluate whether those improvements come at the cost of increased complexity, abstraction leakage, reduced modularity, higher coupling, or additional technical debt.

Consequently, the results should be interpreted as improvements in regression prevention and preservation of test-covered behavior rather than comprehensive improvements in software engineering quality.

How TDAD creates test-mapping context is likely the most valuable takeaway from this study because of its simplicity and effectiveness

rephrased: "The most valuable takeaway from this study is not the specific GraphRAG implementation, but the idea of generating and exposing test-mapping context to the agent. Its simplicity, low implementation cost, and strong empirical results suggest that precomputing repository knowledge may be more impactful than increasingly sophisticated prompting strategies."