A verifier-driven continual learning recipe that combines self-generated traces, replay, and a failure-driven curriculum to learn new task families while limiting forgetting.
Dogukan Tuna · · 5 min read
Most fixes for catastrophic forgetting arrive with a cost attached: a teacher model that has to exist somewhere, golden demonstrations that someone has to write, a reward model that has to be trained and then kept honest. Each is reasonable alone; together they add up to a lot of infrastructure before a single task family is learned. I am releasing a narrower recipe, Verified Replay Distillation, or VRD, that trades all of that for one weaker requirement: a programmatic verifier for the domain. It lives inside continual-learning-recipes.
Continual learning, in the form I care about here, is the problem of training a single model on a sequence of task families that arrive one after the other. A new domain appears, or a new difficulty level, and the model is asked to become competent at it without losing what it already knew. The failure mode is old and familiar: fine-tune on the second family, and the first family quietly degrades.
The only requirement is a programmatic verifier for the domain of interest: a function that can look at a task and a candidate solution and return a judgement. Verifiable domains are more common than they may appear: arithmetic is verifiable, code with unit tests is verifiable, and a great many structured science questions, phrased as multiple choice or short exact-match answers, are verifiable too. Where such a verifier exists, it plays the role of teacher, critic, and reward model at once, cheap and directly auditable, though its specification can still be incomplete or exploitable.
Within each stage, training runs through a small number of cycles that form the core loop. The model samples a handful of candidate solutions for each task, drawing from its own current policy. The verifier looks at each candidate and produces a scalar reward, which the recipe turns into a per-trace weight through a softmax at a chosen temperature. The weighted traces are then folded back into the model via a single cross-entropy loss on a LoRA adapter. At the end of the cycle, a refreshed sampler is created from the updated weights, and the next cycle begins from there.
Around this core loop sit two small mechanisms that do most of the work in keeping training stable across a long sequence of stages.
The replay buffer keeps every verified trace, and at every subsequent cycle, some fraction of the batch is drawn from the pool of traces produced in previous cycles and previous stages. When the model trains on the second stage, the first stage is still gently present in every batch; when it trains on the third, the first two are present as well. The replay multiplier controls how loud this background voice is, but the essential idea is that the model is never asked to learn the new thing in the absence of the old things.
Between cycles, a failure-driven curriculum looks at the per-task pass rates from the most recent round of sampling. Tasks that were solved easily have their search budget reduced; tasks that were failed have theirs increased. The total budget across tasks is held approximately constant, so compute flows from the parts of the distribution that are already covered to the parts that still need attention. Tasks the model has never seen are treated as maximally difficult.
There's no reset between stages, either. The LoRA weights and the optimizer state are carried forward instead: the checkpoint from stage N becomes the initialization for stage N plus one, so the second stage begins in a place already shaped by the first, rather than one scrubbed back to the base model.
The current recipe is not yet a complete continual-learning comparison. It still needs matched-compute ablations to isolate how much of the gain comes from replay, the failure-driven curriculum, carrying optimizer state forward, or simply training longer on self-generated traces.
The next evaluation should compare sequential fine-tuning, VRD without replay, VRD without curriculum reallocation, and the full recipe under the same generation and update budget. Repeating the task-family order and random seeds would also show whether the retention pattern is stable or specific to one sequence.
There's no policy gradient term in the loss, no KL penalty against a frozen reference, and no soft target from a teacher distribution. The loss is cross-entropy on hard targets, with the rewards entering only through the relative weighting of candidates within each task. With fewer terms in play, it's easier to reason about why a given update moved the model in a given direction, and easier to diagnose the recipe when it misbehaves.
There's no off-policy correction, either. Fresh candidates in each cycle are sampled from the current policy, and every training trace was produced by some version of the model and then scored by the verifier. Once replay is enabled, however, the update is not strictly on-policy: each batch also contains verified traces from earlier policy snapshots. VRD deliberately uses those older traces without off-policy correction because the update is reward-weighted supervised cross-entropy rather than a policy-gradient estimator. The useful distinction is that the data is self-generated and verifier-filtered, while replay provides the retention signal.
The verifier is the contract: substituting a new verifier for a new domain is the main extension point, and the rest of the recipe is designed to sit around it without getting in the way. A code verifier, a trajectory-level tool-use checker, a theorem-proof checker: each of these would slot into the same outer loop without asking the recipe to be rewritten.
The code is in continual-learning-recipes, and the VRD recipe lives under recipes/vrd/. The result I would trust next is not a larger headline gain; it is an ablation showing which part of the loop preserves old domains and which part actually learns the new one.