kicking the tires on jev with 2048

jev-2048.md

kicking the tires on jev with 2048

You can watch these strategies play out here: https://jev-with-2048.ndyg.cross.stream

I finally got a chance to kick the tires on jev (jev-1.13.0). I thought it'd do really well playing 2048.

the strategy I used was: hand it the current board state and give it the option of up, down, left, right. I tried that 4 ways (the jev rows in the table).. each row links to the request I sent.

with just the board it does about as well as making random moves. it does best when code works out what each move would do to the board and jev picks from those.. then it's about as good as a fixed rule (e.g. always left if that moves anything, else down, else right, else up).

I didn't have long to spend on it, so I'm likely missing something.

scores

every game was played to the end.

player	games	min	median	max	best tile	first choice did nothing
random moves	48	512	1102	2672	256	n/a
fixed rule: down if it moves anything, else left, else right, else up	48	720	1964	7196	512	n/a
fixed rule: left if it moves anything, else down, else right, else up	48	296	2740	5944	512	n/a
jev, given the board	20	404	706	2340	128	24%
jev, given the board, the rules and a description of each move	20	180	1656	3520	256	19%
jev, given the board and a tip: "keep the big tile in the bottom-left corner"	18	432	1454	4120	256	27%
jev, given the board each move would lead to, the points it scores, and the same tip	18	596	2494	5532	512	n/a

first choice did nothing: how often jev's top pick was a move that doesn't change the board (e.g. left, when every tile is already against the left wall and nothing can merge). when that happened I played its next pick.

n/a: random and the fixed rules only pick moves that do something, and in the last jev row those were the only ones on offer.

the requests

one per jev row, all for the same board.

request 1

jev, given the board.

{
  "state": {
    "board": {
      "row1": ". . . .",
      "row2": ". . . .",
      "row3": "2 . . .",
      "row4": "4 . 2 2"
    }
  },
  "model": "jev-latest",
  "questions": {
    "move": {
      "type": "choice",
      "instructions": "Which move should the 2048 player make on `board`?",
      "criteria": {
        "up": null,
        "down": null,
        "left": null,
        "right": null
      }
    }
  }
}

request 2

jev, given the board, the rules and a description of each move.

{
  "state": {
    "board": {
      "top_row":    ["empty", "empty", "empty", "empty"],
      "second_row": ["empty", "empty", "empty", "empty"],
      "third_row":  ["2", "empty", "empty", "empty"],
      "bottom_row": ["4", "empty", "2", "2"]
    }
  },
  "model": "jev-latest",
  "questions": {
    "move": {
      "type": "choice",
      "instructions": {
        "question": "Which move should the 2048 player make next on `board`?",
        "layout": "`board` lists its rows from top to bottom. Each row lists its four cells from left to right.",
        "rules": "A move slides every tile as far as it can go in one direction. Two tiles with the same number that collide merge into one tile with their sum. A move that changes nothing on the board is not allowed.",
        "goal": "Pick the move that leaves the player best placed to keep merging and reach larger tiles."
      },
      "criteria": {
        "up": "Every tile slides toward `board.top_row`",
        "down": "Every tile slides toward `board.bottom_row`",
        "left": "Every tile slides toward the first cell of its row",
        "right": "Every tile slides toward the last cell of its row"
      }
    }
  }
}

request 3

jev, given the board and a tip.

{
  "state": {
    "board": {
      "row1": ". . . .",
      "row2": ". . . .",
      "row3": "2 . . .",
      "row4": "4 . 2 2"
    }
  },
  "model": "jev-latest",
  "questions": {
    "move": {
      "type": "choice",
      "instructions": {
        "question": "Which move is best for the 2048 player on `board`?",
        "strategy": "Keep the largest tile in the bottom-left corner. Keep the bottom row full and ordered, largest on the left. Prefer moves that merge tiles. Avoid a move that lets a small tile slip under the largest."
      },
      "criteria": {
        "up": "Slide every tile up",
        "down": "Slide every tile down",
        "left": "Slide every tile left",
        "right": "Slide every tile right"
      }
    }
  }
}

request 4

jev, given the board each move would lead to, the points it scores, and the same tip. down is missing because it would do nothing here.

{
  "state": {
    "boards": {
      "up": {
        "board": {
          "row1": "2 . 2 2",
          "row2": "4 . . .",
          "row3": ". . . .",
          "row4": ". . . ."
        },
        "gained": 0
      },
      "left": {
        "board": {
          "row1": ". . . .",
          "row2": ". . . .",
          "row3": "2 . . .",
          "row4": "4 4 . ."
        },
        "gained": 4
      },
      "right": {
        "board": {
          "row1": ". . . .",
          "row2": ". . . .",
          "row3": ". . . 2",
          "row4": ". . 4 4"
        },
        "gained": 4
      }
    }
  },
  "model": "jev-latest",
  "questions": {
    "move": {
      "type": "choice",
      "instructions": {
        "question": "Each entry of `boards` is the 2048 board one move would leave. Which board is best for the player?",
        "strategy": "Keep the largest tile in the bottom-left corner. Keep the bottom row full and ordered, largest on the left. Prefer moves that merge tiles. Avoid a move that lets a small tile slip under the largest."
      },
      "criteria": {
        "up": null,
        "left": null,
        "right": null
      }
    }
  }
}

request 2 places the rules & the goal of the game in the questions field of the request.

Did you try putting the rules & goal information in the state instead of questions?

I'm not sure if it will improve the performance, but it would be interesting to see! Intuitively, I'd assume state is to Jev what the context window (especially the early and reusable parts of the context window, like system prompts). So in my testing, I've been putting the general information of the game format and rules in the state.

Performance/capability aside, putting the general information about the situation in state has the additional benefit that it scales to multiple questions: if you added other questions about the 2048 game state, then the rules of the game would only need to be stated once, not copypasted for each question in the request. Jev can execute an entire battery of questions in parallel against a single state (and AFAIK, one question's instructions can not poison the context of the other questions in the battery).

(I was sent here via https://simonwillison.net/2026/Sep/21/jev/ via https://news.ycombinator.com/item?id=49796843)

Ok, on a quick test, there was indeed a change in behaviour induced by moving the request 2 example input's questions.move.instructions fields layout, rules, and goal into the state part of the request (leaving only the question string as the instructions, which I also flattened).

However I'm not able to evaluate whether this change represents an improvement in the move chosen (or performance across an entire game). I'll leave that for others to try.

Table: change in the mean results of the choices' probabilities, and the confidence, across 10 runs for each structure.

Metric	Rules in questions (old)	Rules in state (new)	Delta (new - old)
Up	0.030	0.030	0.000
Down	0.380	0.428	+0.048
Left	0.360	0.342	-0.018
Right	0.230	0.200	-0.030
Confidence	0.180	0.238	+0.058

Old structure: rules in question instructions

  {
    "state": {
      "board": {
        "top_row": ["empty", "empty", "empty", "empty"],
        "second_row": ["empty", "empty", "empty", "empty"],
        "third_row": ["2", "empty", "empty", "empty"],
        "bottom_row": ["4", "empty", "2", "2"]
      }
    },
    "questions": {
      "move": {
        "type": "choice",
        "instructions": {
          "question": "Which move should the 2048 player make next on `board`?",
          "layout": "`board` lists its rows from top to bottom. Each row lists its four cells from left to right.",
          "rules": "A move slides every tile as far as it can go in one direction. Two tiles with the same number that collide merge into one tile with their sum. A move that changes nothing on the board is not allowed.",
          "goal": "Pick the move that leaves the player best placed to keep merging and reach larger tiles."
        },
        "criteria": {
          "up": "Every tile slides toward `board.top_row`",
          "down": "Every tile slides toward `board.bottom_row`",
          "left": "Every tile slides toward the first cell of its row",
          "right": "Every tile slides toward the last cell of its row"
        }
      }
    }
  }

New structure: rules in state

  {
    "state": {
      "board": {
        "top_row": ["empty", "empty", "empty", "empty"],
        "second_row": ["empty", "empty", "empty", "empty"],
        "third_row": ["2", "empty", "empty", "empty"],
        "bottom_row": ["4", "empty", "2", "2"]
      },
      "layout": "`board` lists its rows from top to bottom. Each row lists its four cells from left to right.",
      "rules": "A move slides every tile as far as it can go in one direction. Two tiles with the same number that collide merge into one tile with their sum. A move that changes nothing on the board is not allowed.",
      "goal": "Pick the move that leaves the player best placed to keep merging and reach larger tiles."
    },
    "questions": {
      "move": {
        "type": "choice",
        "instructions": "Which move should the 2048 player make next on `board`?",
        "criteria": {
          "up": "Every tile slides toward `board.top_row`",
          "down": "Every tile slides toward `board.bottom_row`",
          "left": "Every tile slides toward the first cell of its row",
          "right": "Every tile slides toward the last cell of its row"
        }
      }
    }
  }