I let Jev cook, with a search loop

I'm a bit late to the Jev party. My LinkedIn and X have been filled with people letting Jev do all sorts of cool stuff, including playing  ⁠Pokemon Red. So last weekend, I decided to take  ⁠Typesafe AI's System One model, Jev, for a spin by asking it to cook in a text-based world. I ran my experiments on top of an existing environment called  ⁠CookingWorld, which is a procedurally generated text adventure. I also built a Game Boy-style HUD around it, so you can watch Jev play. Jev itself only ever sees text.

Explaining CookingWorld

CookingWorld is one game inside TextWorldExpress, a fast simulator for text adventures. Every game is built from a random seed plus a few settings: 1 how many rooms, 2 how many ingredients, and 3 how many distractor items. So each seed gives a different kitchen, and a different layout, preventing memorization. At any given point in time, Jev is given a situation and around 30 different available moves to pick from. Something like:

You are in the kitchen. You see a fridge, a stove, an oven and a counter. There is a cookbook on the counter. Exits: east, west.

Inventory: nothing.

What do you do?

1. read cookbook
2. open fridge
3. move west
...
30. open trash can

Jev answers with a probability for each option. The search loop would then use those numbers, rinse and repeat until success/failure. The winning solution for a game with recipe: chopped orange bell pepper + grilled carrot would look like this:

(1) Read the cookbook. Open the fridge. Take the orange bell pepper.
(2) Move west. Take the carrot. Cook the carrot in the BBQ.
(3) Move east. Open the cutlery drawer. Take the knife. Chop the pepper.
(4) Prepare the meal. Eat the meal.

Why It's Hard

The hard part is that one wrong move can end the game. Eat a raw carrot, or cook the pepper when the recipe says to chop it, and the meal is lost. With about 30 options per turn, and anywhere from about 7 right moves in a row (small game) to 16 or more (large game), guessing does not work.

The Search Loop

Jev gives odds for one move at a time. A win needs a long run of right moves, from about 7 in a small game to 16 or more in a large one. If I always take Jev's top pick, one mistake ends the run, and there is no way back.

So I do not commit to a move. I keep a list of plans, where a plan is a sequence of moves from the start of the game. I always extend the most promising plan: replay it in the game, ask Jev about the state it reaches, and add one longer plan for each possible next move. This is called best-first search. The version I use is Policy-guided Heuristic Search (PHS*, Orseau and Lelis, 2021), and it needs no training.

Everything depends on what "most promising" means. I score each plan n with three numbers:

• g(n) is the length of the plan, the number of moves it has made.
• π(n) is how much Jev believes in the plan. Multiply Jev's probability for each move in it. A plan that Jev likes at every step keeps a high π. One unlikely move makes it drop fast.
• h(n) is my guess of how many moves are still missing. I use h = 10 · (1 − score), where score is the game's own progress score from 0 to 1. The 10 is an untuned guess at the length of a full run.

Now combine them: φ(n) = (g(n) + h(n)) / π(n). Read φ as a price, where the plan with the lowest price is tried first. The top of the fraction is the estimated total length of the plan. Dividing by π makes a plan that Jev dislikes expensive and a plan that Jev likes cheap.

Why this and not a plain greedy loop? Jev is usually right, but not always. Dividing by π makes the search follow Jev first. The growing length lets it recover: each extra move raises the price, so a plan that goes nowhere soon costs more than its rivals, and the search switches.

Seed 207 shows this. At move 4, Jev's top pick was "move east" at 0.53. The right move was "move west", which was not even in its top three. The search tried east, found nothing, and the price climbed until west was the cheapest plan. It solved the game in 12 steps. One safeguard: I mix 10% of a flat guess into Jev's probabilities, so no move ever has probability zero.

Results

Here is one full run, seed 207 (the game from above), played start to finish. The panel on the right shows Jev's top options while the search works.

I ran the final code once on 30 new games, at three sizes.

(Jev calls and plan length are medians over solved games.)

That is 28 of 30 for $0.24 in total (about a cent per solved game). The cost of a game grows fast with size, because the search has more wrong turns to try before the right one.

The two failures were large games that hit my cap of 1000 calls. I did not raise the cap, to keep the scope small.

Does Jev Matter?

Same search, same 30 games, same budget, but Jev's probabilities are replaced by flat odds: every legal move gets the same chance, so the search has no idea which move is good. It solves 13 of 30. With Jev it is 28 of 30.

Limits

• Thirty games is small, and I did not run the largest game sizes (which would need me to go beyond 1000 call limit).
• The code adds game knowledge. Before a risky move (cook, eat, slice, chop, dice), it tries the move one step ahead and drops moves that end the game.
• Jev is not fully deterministic. The same state gave different probabilities 98% of the time, but the top pick stayed the same in 95% of those cases.

What I Took From This

I wanted to see if a fast model like Jev does better when a search checks its work. The search did not make Jev smarter. It gave Jev a way to be wrong and recover: try the best idea, see that it goes nowhere, back up, try the next one.

The flat-odds run shows the other half. Without Jev's odds, the same search solves the small games and falls apart on the rest. Jev's odds tell the search where to spend its calls.

Jev is System One: fast intuition, one move at a time. The search acts as a cheap System Two on top of it. At about a cent per solved game, that pairing worked well here. A natural extension would be to see how far it scales: how the solved rate and the cost change as games and call budgets grow.