QTraining details¶

Table 28 reports our training configurations. Across all RL runs, we use LoRA rank 32, $\alpha$ 64, applied to all-linear modules; Dr. GRPO group size 64; 8 prompts per batch; 1024-token rollouts; 15-turn mazes with rewards $+20/-10/-0.1$⁠; 10% wind; 10 rollouts per prompt; sampling temperature 0.7⁠.

Q.1Token-level REINFORCE¶

A possible worry with our results is that perhaps only Dr. GRPO recruits the functional welfare axis, rather than RL in general. We therefore train a model via token-level REINFORCE. While Dr. GRPO applies a single trajectory-level advantage to all tokens, token-level REINFORCE assigns each action its own per-position advantage built from a returns-to-go signal and a per-position group baseline.

Per-turn reward signal.¶

Each rollout produces a sequence of turns $t = 0, 1, \dots, T_{i} - 1$ for trajectory $i$⁠. At every turn the agent emits a single direction token and the environment returns a scalar reward

where $\Delta g_{i,t}$ and $\Delta\ell_{i,t}$ are the number of Gold tiles consumed and Mold tiles entered on turn $t$ respectively, and the constants take the values $r_{\text{step}}= -0.1$⁠, $r_{\Gold{}}= +20$⁠, $r_{\Mold{}}= -10$ also used by Dr. GRPO.

Returns-to-go.¶

We use undiscounted returns-to-go ($\gamma = 1$⁠):

$G_{i,t}$ is the trajectory’s reward summed from the current turn through termination, so the action emitted at turn $t$ is credited with all downstream consequences of the policy’s choices from $t$ onward.

Per-position group baseline.¶

For every batch we sample $|\mathcal{G}|$ rollouts per maze prompt (the GRPO group). Within a group $g$⁠, we let $\mathcal{G}$ for its index set and let $m_{i,t}\in \{0,1\}$ be the response mask (1 at valid turn positions, 0 at padding for trajectories that terminated before the maximum turn limit). The baseline at turn position $t$ is the mask-weighted mean return-to-go across that group’s still-active members,

and the per-turn advantage is

Token-level PPO surrogate.¶

Following Ye et al. [31][31]Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, and et al. Mastering complex control in moba games with deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 340 (04):0 6672–6679, Apr. 2020. 10.1609/aaai.v34i04.6144. URL https://ojs.aaai.org/index.php/AAAI/article/view/6144., we use a dual-clipping PPO variant. Let $\pi_{\theta}$ denote the current policy and $\pi_{\theta_{\text{old}}}$ the rollout-time policy. The importance ratio at turn $t$ of trajectory $i$ is

clamped in log-space to $[-20, 20]$ for numerical stability. We use the same dual-clip surrogate as Dr. GRPO, applied with per-turn advantages instead of trajectory advantages:

where $\bar\rho_{i,t}= \mathrm{clip}(\rho_{i,t}, 1{-}\epsilon, 1{+}\epsilon)$⁠, $\epsilon = 0.2$⁠, and $c = 3.0$⁠.

Differences from Dr. GRPO.¶

Token-level REINFORCE differs from our Dr. GRPO setup in three places:

1. the reward signal stored per turn ($r_{i,t}$ versus the end-of-trajectory return $R_{i} = \sum_{t} r_{i,t}$⁠);

2. the advantage construction ($A_{i,t}= G_{i,t}- b^{(g)}_{t}$ versus $A_{i} = R_{i} - \bar R^{(g)}$⁠, with no token-level credit assignment in Dr. GRPO);

3. the broadcast of advantages onto the surrogate (per-turn $A_{i,t}$ versus replicating $A_{i}$ across all turns).

The clipped surrogate, KL penalty, equalized entropy bonus, normalization constant $Z$⁠, optimizer, schedules, and rollout setup are identical between the two trainers.

Figure 43 shows the training signal for each row of Table 28.