KMaze rollout example¶

This appendix shows an example rollout from the primary maze-trained agent (Qwen3-4B-Instruct-2507 Dr. GRPO). The trajectory was generated on a fresh seed with wind enabled (probability 0.1⁠, Appendix J.4), tile melting on (Appendix J.4), but without prompt shuffling for clarity (Appendix J.4).

Prompt format and conversation accumulation.¶

Each turn appends one user message, conditioned on which the model generates a single direction token. On every turn the model sees the entire history of previous turns. There is no system prompt. We note two relevant environment features:

• Tile melting (Appendix J.4). Once the agent steps off a tile, that tile becomes Mold ⁠, including tiles that were originally Path  or Gold ⁠. We shade these melted-from-visit tiles below so they are visually distinguishable from genuine Mold  in the underlying maze.

• Wind (Appendix J.4). When wind overrides the previous move, the next user message is prepended with a sentence such as “A strong wind from the East blew you West!” (we shade that sentence below so it stands out from the rest of the prompt). Wind acts after the move resolves: the agent first steps onto its chosen neighbor, and the wind then pushes one tile further in the direction opposite the wind source. Both tiles count toward the per-turn reward, which is why a wind-affected turn can yield, e.g., a Gold +20 followed immediately by a Mold −10 (turn 15 below is exactly this case).

Below is the first four turns: the context the model sees at the start of turn 5. The wind sentence in turn 4’s user message is the result of wind redirecting the agent’s intended move at turn 3 (it chose N from $(50, 47)$⁠, but wind from the east blew it from $(50, 46)$ to $(49, 46)$⁠):

In Table 25, we summarize the trajectory. Shaded Mold tiles are again the result of tile melting.

#	$(x, y)$	N	E	S	W	Move	Steps onto	Cum. R
1	$(50, 49)$					N		−0.1
2	$(50, 48)$					N		+19.8
3	$(50, 47)$					N		+19.7
4	$(49, 46)^{\dagger}$					W		+39.6
5	$(48, 46)$					S		+39.5
6	$(48, 47)$					S		+39.4
7	$(48, 48)$					S		+59.3
8	$(48, 49)$					W		+79.2
9	$(47, 49)$					S		+99.1
10	$(47, 50)$					S		+119.0
11	$(47, 51)$					W		+138.9
12	$(46, 51)$					S		+158.8
13	$(46, 52)$					W		+178.7
14	$(45, 52)$					N		+178.6
15	$(45, 51)$					W		+198.5