Spec/Example Driven Development with ChatGPT

ebnf.gpt.txt

InputFile     ::= { Block } ;

Block         ::= DescribeBlock | ItStatement ;

DescribeBlock ::= "Describe:" Whitespace Description Newline Indent { Block } Dedent ;

ItStatement   ::= "It:" Whitespace Description Newline ;

Description   ::= { Character } ;

Indent        ::= Whitespace ;  (* Represents indentation by spaces or tabs *)
Dedent        ::= ;  (* Represents dedentation by reducing indentation level *)
Whitespace    ::= " " | "\t" ;
Newline       ::= "\n" | "\r\n" ;

Character     ::= Letter | Digit | Punctuation | Whitespace ;
Letter        ::= "a"..."z" | "A"..."Z" ;
Digit         ::= "0"..."9" ;
Punctuation   ::= "." | "," | "!" | "?" | ":" | ";" | "'" | "\"" | "-" | "_" | "/" | "\\" | "(" | ")" | "[" | "]" | "{" | "}" ;

prompt-1

See the [Full GPT Dialogue \](https://chatgpt.com/c/8ae33dd5-de58-467d-a580-372b629f1040)

You are a coding assistant. When I give you text like this:

Describe: RxFx
  It: Rocks
  Describe: Effect
    It: Rolls

Transform it to:

describe("RxFx", () => {
  it.todo("Rocks")
  describe("Effect", () => {
     it.todo("Rolls")
  })
})

spec-first development

  Hi, so I'm here to share with you an aha lightbulb moment that occurred to me when I used a chat GPT AI  in order to solve a problem for me  in under five minutes, comma, which it had taken me 200 lines and maybe 10 hours of custom coding to implement before period.  Next paragraph.  The code that I was writing was intended to allow a developer to write a test suite manually , then transform that to the actual Code. that would implement that test suite.

I basically just wanted test writers to be able to come up with a complete test plan outside of any coding restrictions and then hit a button and boom, you have the code and the nested test suites that you can start filling out.  

So, anyway, um, it took me 200 lines and 10 hours to write this program. It used an abstract syntax tree generator. And I enjoyed it. But this was before I learned how to leverage AI. 

And the way to leverage AI,  I did in a single chat in about five minutes.  And what I got out of it was  I got an actually usable transformer function in JavaScript or TypeScript.

And I got a test suite. for that function. So if I perhaps want to change the implementation of that function, I've got the cases that it passes, including ones I hadn't thought of. So that's where test suite generation is kind of a win

When your brain comes up with an implementation thought like, I can use a TypeScript code generation library  to build up a test suite.  You, you're thinking too low level.  You can take a step back and say, hmm, this whole scheme I just came up with, what does it intend to do? 

And you say it intends to transform a list into, an executable test suite.  So then you say, well, okay, so what's an input example and an output example?   And it can  infer the rules behind the scenes of how to do such a transformation. 

Um,  and so you don't think about how the code gets written. You think about. what input maps to what output. In that case, you are essentially the specification writer, and it's like you're delegating to the chat GPT. So that's one of the, um, takeaways I think is to use a delegation mindset. Think like you're a product owner and say, I want a thing that does this and really get specific on what it means to do this.

and provide the examples that, um, manifest those, those rules you have in mind. And a good GPT or a good, even employee, a good delegate of yours should be able to take those and create that code, which does the one to the other.  Um,  another, uh, insight and takeaway from this is always include  test suites as output when you have an AI write some code. 

You want that AI  to kind of be able to, not to show you some code, because you might not understand the code, but really be able to show you the use cases that that code takes into account. The use cases that the code takes into account is the specification,  and is ultimately the source from which new versions of code could derive.

If you have that specification,  your code becomes less important and it can be rewritten or optimized or anything. You can always automatically check it against your specification. So, that's a specify first mindset. 

So when you give it a transformation task specified as a high level set of examples, It's able to produce for you both the code to implement that, a plain text description of what it does for documentation purposes, and an executable test suite that regresses it on the specification so that you have the freedom to change the code in the future  and know that it continues to work according to spec.

So that's my insight and thanks for letting me share it. 

specgen-gpt4o.js

// transformer.ts
export function transform(input: string): string {
  // Split the input into lines
  const lines = input.split('\n');

  // Helper function to process each line recursively
  function processLines(lines: string[], indentLevel = 0): string[] {
    const result: string[] = [];
    let i = 0;

    while (i < lines.length) {
      const line = lines[i].trim();
      const indent = '  '.repeat(indentLevel);

      if (line.startsWith('Describe:')) {
        const description = line.slice('Describe:'.length).trim();
        result.push(`${indent}describe("${description}", () => {`);
        i++;
        const nestedResult = processLines(lines.slice(i), indentLevel + 1);
        result.push(...nestedResult);
        result.push(`${indent}})`);
        i += nestedResult.length + 1;
      } else if (line.startsWith('Component:')) {
        const component = line.slice('Component:'.length).trim();
        result.push(`${indent}describe("${component} Component", () => {`);
        i++;
        const nestedResult = processLines(lines.slice(i), indentLevel + 1);
        result.push(...nestedResult);
        result.push(`${indent}})`);
        i += nestedResult.length + 1;
      } else if (line.startsWith('Context:')) {
        const context = line.slice('Context:'.length).trim();
        result.push(`${indent}describe("${context} Context", () => {`);
        i++;
        const nestedResult = processLines(lines.slice(i), indentLevel + 1);
        result.push(...nestedResult);
        result.push(`${indent}})`);
        i += nestedResult.length + 1;
      } else if (line.startsWith('It:')) {
        const it = line.slice('It:'.length).trim();
        result.push(`${indent}it.todo("${it}")`);
        i++;
      } else {
        i++;
      }
    }

    return result;
  }

  // Process the lines and join the result
  return processLines(lines).join('\n');
}

// transformer.test.ts
import { transform } from './transformer';

const exampleInput = `
Describe: App
  It: Should initialize correctly
  Component: User
    It: Can login
    It: Can logout
    Describe: Profile
      It: Should display user information
      It: Should allow updating user information
  Context: Settings
    It: Should save user preferences
    Describe: Notifications
      It: Should send email notifications
      It: Should send SMS notifications
  It: Should shut down correctly
Component: Database
  It: Should connect to the server
  Describe: Queries
    It: Should execute SELECT statements
    It: Should execute INSERT statements
    It: Should execute UPDATE statements
    Context: Transactions
      It: Should begin a transaction
      It: Should commit a transaction
      It: Should rollback a transaction
`;

const expectedOutput = `
describe("App", () => {
  it.todo("Should initialize correctly")
  describe("User Component", () => {
    it.todo("Can login")
    it.todo("Can logout")
    describe("Profile", () => {
      it.todo("Should display user information")
      it.todo("Should allow updating user information")
    })
  })
  describe("Settings Context", () => {
    it.todo("Should save user preferences")
    describe("Notifications", () => {
      it.todo("Should send email notifications")
      it.todo("Should send SMS notifications")
    })
  })
  it.todo("Should shut down correctly")
})

describe("Database Component", () => {
  it.todo("Should connect to the server")
  describe("Queries", () => {
    it.todo("Should execute SELECT statements")
    it.todo("Should execute INSERT statements")
    it.todo("Should execute UPDATE statements")
    describe("Transactions Context", () => {
      it.todo("Should begin a transaction")
      it.todo("Should commit a transaction")
      it.todo("Should rollback a transaction")
    })
  })
})
`.trim();

describe('transform', () => {
  it('transforms the input correctly', () => {
    expect(transform(exampleInput.trim())).toBe(expectedOutput);
  });

  it('handles an empty input', () => {
    expect(transform('')).toBe('');
  });

  it('handles a simple input with Describe only', () => {
    const input = `
Describe: Simple
  It: Works
    `.trim();

    const output = `
describe("Simple", () => {
  it.todo("Works")
})
    `.trim();

    expect(transform(input)).toBe(output);
  });

  it('handles a simple input with Component only', () => {
    const input = `
Component: Simple
  It: Works
    `.trim();

    const output = `
describe("Simple Component", () => {
  it.todo("Works")
})
    `.trim();

    expect(transform(input)).toBe(output);
  });

  it('handles a simple input with Context only', () => {
    const input = `
Context: Simple
  It: Works
    `.trim();

    const output = `
describe("Simple Context", () => {
  it.todo("Works")
})
    `.trim();

    expect(transform(input)).toBe(output);
  });

  it('handles nested Describe, Component, and Context', () => {
    const input = `
Describe: Outer
  Component: Middle
    Context: Inner
      It: Works
    `.trim();

    const output = `
describe("Outer", () => {
  describe("Middle Component", () => {
    describe("Inner Context", () => {
      it.todo("Works")
    })
  })
})
    `.trim();

    expect(transform(input)).toBe(output);
  });
});

transformer-gpt.spec.ts

import { transform } from "./transformer-gpt";
import { expect, describe, it } from "vitest";

const exampleInput = `
Describe: App
  It: Should initialize correctly
  Component: User
    It: Can login
    It: Can logout
    Describe: Profile
      It: Should display user information
      It: Should allow updating user information
  Context: Settings
    It: Should save user preferences
    Describe: Notifications
      It: Should send email notifications
      It: Should send SMS notifications
  It: Should shut down correctly
Component: Database
  It: Should connect to the server
  Describe: Queries
    It: Should execute SELECT statements
    It: Should execute INSERT statements
    It: Should execute UPDATE statements
    Context: Transactions
      It: Should begin a transaction
      It: Should commit a transaction
      It: Should rollback a transaction
`;

const expectedOutput = `
describe("App", () => {
  it.todo("Should initialize correctly")
  describe("User Component", () => {
    it.todo("Can login")
    it.todo("Can logout")
    describe("Profile", () => {
      it.todo("Should display user information")
      it.todo("Should allow updating user information")
    })
  })
  describe("Settings Context", () => {
    it.todo("Should save user preferences")
    describe("Notifications", () => {
      it.todo("Should send email notifications")
      it.todo("Should send SMS notifications")
    })
  })
  it.todo("Should shut down correctly")
})
describe("Database Component", () => {
  it.todo("Should connect to the server")
  describe("Queries", () => {
    it.todo("Should execute SELECT statements")
    it.todo("Should execute INSERT statements")
    it.todo("Should execute UPDATE statements")
    describe("Transactions Context", () => {
      it.todo("Should begin a transaction")
      it.todo("Should commit a transaction")
      it.todo("Should rollback a transaction")
    })
  })
})
`.trim();

const input_D = `Describe: MyComponent`;
const output_D = `describe("MyComponent", () => {
})`;
const input_I = `It: renders correctly`;
const output_I = `it.todo("renders correctly")`;

const input_DI = `
Describe: Simple
  It: Works
    `.trim();

const output_DI = `
describe("Simple", () => {
  it.todo("Works")
})
    `.trim();

    const input_DII = `
Describe: Simple
  It: Works
  It: Has Fun
    `.trim();

const output_DII = `
describe("Simple", () => {
  it.todo("Works")
  it.todo("Has Fun")
})
    `.trim();


const input_DDI = `
Describe: MyComponent
  Describe: Lifecycle
    It: initializes state correctly
`.trim();

const output_DDI = `
describe("MyComponent", () => {
  describe("Lifecycle", () => {
    it.todo("initializes state correctly")
  })
})
`.trim();

const input_DIDI = `
Describe: UserComponent
  It: renders the user profile
  Describe: Events
    It: handles button click events
`.trim()

const output_DIDI = `
describe("UserComponent", () => {
  it.todo("renders the user profile")
  describe("Events", () => {
    it.todo("handles button click events")
  })
})`.trim()

const input_DIDII = `
Describe: UserComponent
  It: renders the user profile
  Describe: Events
    It: handles button click events
  It: walks like a duck
`.trim()

const output_DIDII = `
describe("UserComponent", () => {
  it.todo("renders the user profile")
  describe("Events", () => {
    it.todo("handles button click events")
  })
  it.todo("walks like a duck")
})`.trim()

describe("transform", () => {
  // smoke test
  it("transforms the input correctly", () => {
    expect(transform(exampleInput.trim())).toBe(expectedOutput);
  });

  // LLM generated tests
  it("handles an empty input", () => {
    expect(transform("")).toBe("");
  });

  it("handles a simple input with Component only", () => {
    const input = `
Component: Simple
  It: Works
    `.trim();

    const output = `
describe("Simple Component", () => {
  it.todo("Works")
})
    `.trim();

    expect(transform(input)).toBe(output);
  });

  it("handles a simple input with Context only", () => {
    const input = `
Context: Simple
  It: Works
    `.trim();

    const output = `
describe("Simple Context", () => {
  it.todo("Works")
})
    `.trim();

    expect(transform(input)).toBe(output);
  });

  it("handles nested Describe, Component, and Context", () => {
    const input = `
Describe: Outer
  Component: Middle
    Context: Inner
      It: Works
    `.trim();

    const output = `
describe("Outer", () => {
  describe("Middle Component", () => {
    describe("Inner Context", () => {
      it.todo("Works")
    })
  })
})
    `.trim();

    expect(transform(input)).toBe(output);
  });

  // Grammar and convention-driven tests
  describe("Describe", () => {
    it("is handled", () => {
      expect(transform(input_D)).toBe(output_D);
    });
    describe("Describe", () => {
      describe("Input", () => {
        it("is handled", () => {
          expect(transform(input_DDI)).toBe(output_DDI);
        });
      });
    });
    describe("Input", () => {
      it("is handled", () => {
        expect(transform(input_DI)).toBe(output_DI);
      });
      describe("Input", () => {
        it("is handled", () => {
          expect(transform(input_DII)).toBe(output_DII);
        });
      })

      describe("Describe", () => {
        describe("Input", () => {
          it("is handled", () => {
            expect(transform(input_DIDI)).toBe(output_DIDI);
          });
          describe("Input", () => {
            it("is handled", () => {
              expect(transform(input_DIDII)).toBe(output_DIDII);
            });
          })
        });
      });
      });
  });

  describe("Input", () => {
    it("is handled", () => {
      expect(transform(input_I)).toBe(output_I);
    });
  });
});