Dogukan TunaResearchMail

Contents

  • The problem with starting cold
  • Three timescales, running concurrently
  • What it looks like in practice
  • Does it help
  • What's under the hood
  • What the benchmark does not establish
  • The transfer question

Tags

LLM inference, memory augmentation, recursive language models, reinforcement learning, AI agents, open source, test-time compute, strategy learning

Share

XLinkedIn

Mem-RLM — Memory-Augmented Inference for Recursive Language Models

A memory layer for Recursive Language Models that records trajectories, extracts reusable strategies, and lifts a weaker model's benchmark score by 26% over three rounds.

Dogukan Tuna · February 23, 2026 · 7 min read

Mem-RLM — Memory-Augmented Inference for Recursive Language Models
Share:XLinkedIn
← Back

The problem with starting cold

Every inference-time system without memory pays the same tax: run the same category of problem ten times and it makes the same mistake on the tenth run that it made on the first. All the execution experience from the first nine runs just disappears.

But RLM inherits the gap above along with everything it does well. Every run starts completely fresh. The model has no memory of what worked before, what failed, or which strategies suit which problem types.


Three timescales, running concurrently

Mem-RLM operates on three timescales that run concurrently during inference:

Fast — standard RLM execution. Every iteration, the model writes code, executes it in the REPL, and reads the output. This is unchanged from base RLM, with one addition: if a strategy was selected, it's already been injected into the system prompt before the first iteration starts.

Medium — trajectory recording and evaluation. After each completion, the full run gets recorded: the prompt, the model's response, every iteration of code and output, token counts, timing, and whether errors occurred. An evaluator (a separate LLM call) scores the trajectory on a 0-1 scale based on correctness and execution quality. Strategy ratings update in real-time with a weighted running average.

Slow — strategy extraction. After enough trajectories accumulate for a given problem type, the system analyzes patterns across successful and failed runs, then generates new reusable strategies. These aren't generic advice like "plan before coding" — they're concrete tactics extracted from what actually worked, like "compute each intermediate result in a separate variable, print it to verify correctness, then use it in the next step."

Selection is epsilon-greedy. 90% of the time the system picks the highest-scoring strategy for the current problem type; 10% of the time it explores a random one. Strategies that consistently underperform get deactivated automatically.


What it looks like in practice

python
from memrlm import MemRLM

mem = MemRLM(
    backend="openai",
    backend_kwargs={"model_name": "gpt-4.1-mini"},
    environment="local",
    environment_tag="math",
    auto_evaluate=True,
)

result = mem.completion("What is the sum of the first 100 prime numbers?")
print(result.response)
# The sum of the first 100 prime numbers is 24133.

First run starts cold — no strategies available, pure RLM. But the trajectory gets recorded and scored. By the third or fourth run on similar problems, the system has extracted patterns from what worked and starts injecting them. The model stops repeating the same mistakes.

You can also seed strategies manually if you already know what works:

python
mem.register_strategy(
    environment_tag="math",
    name="direct_compute",
    env_tip=(
        "When solving math problems:\n"
        "1. Parse the problem into variables\n"
        "2. Compute the answer directly in Python\n"
        "3. Assign the result and use FINAL_VAR()"
    ),
)

Does it help

I ran a 10-problem benchmark across math, combinatorics, algorithms, linear algebra, and graph theory — then ran it multiple rounds to see if strategy accumulation actually helps.

GPT-4.1-nano (weaker model — benefits most from guidance):

RunScoreAvg IterationsErrors
Raw RLM (baseline)0.45011.06
Mem-RLM Round 10.41011.46
Mem-RLM Round 20.47011.93
Mem-RLM Round 30.56513.84

A 26% relative improvement in benchmark score by Round 3, from 0.450 to 0.565. Round 1 actually underperforms the raw baseline, which is consistent with starting without a useful strategy library; by Round 3, the accumulated guidance produces the strongest result in this small run.

GPT-4.1-mini (stronger model):

RunScoreAvg IterationsErrors
Raw RLM (baseline)0.8556.01
Mem-RLM Round 10.7205.22
Mem-RLM Round 20.9254.91
Mem-RLM Round 30.8605.42

An 8% relative improvement in benchmark score at peak. Mini is already strong, so the baseline is high, but Round 2 still improves from 0.855 to 0.925 while using fewer average iterations. The score falls back to 0.860 in Round 3, so the result also shows that strategy accumulation is not monotonically beneficial.

The takeaway: Mem-RLM helps most when the base model struggles. Weaker models have more to gain from accumulated strategy guidance, which makes sense — if you already get 85% right, there's less room to improve than if you're at 45%.


What's under the hood

Everything persists in a database — SQLite by default, but it supports PostgreSQL and MySQL for production use. The data model is simple: trajectories (raw execution records), strategies (learned patterns), and run records (links between them).

Scoring decays toward recent performance. It's a weighted running average with exponential decay toward recent scores, so a strategy that worked well historically but started failing recently drops faster than a simple average would show. Strategies that fall below 0.25 average score after 5+ uses get automatically deactivated, but they're still eligible during exploration to see if conditions have changed.

The evaluator sees everything a human reviewer would. It's a separate LLM call given the full picture — the original prompt, the model's final response, the actual stdout/stderr from the REPL, and execution stats. No regex heuristics, no hardcoded rules.


What the benchmark does not establish

This is a small, repeated 10-problem benchmark, not evidence that Mem-RLM improves every model or task distribution. The evaluator is itself an LLM, the same problems recur across rounds, and the experiment does not yet isolate whether gains come from transferable strategies, problem-specific leakage, evaluator variance, or ordinary sampling noise.

A stronger evaluation would use held-out problems from the same families, multiple random seeds, a verifier independent of the model judge where possible, and ablations for strategy injection, extraction, and selection. The current result is a useful proof of mechanism: stored trajectories can change later behavior, and sometimes improve it. It is not yet a general learning curve.


The transfer question

RLMs are one of the more interesting inference paradigms to come out recently — giving models a real execution environment changes what they can do. But stateless inference means every run is independent, and that's a waste. Models encounter the same problem types repeatedly and there's no mechanism to carry forward what they've learned.

Mem-RLM makes RLM inference stateful. Over time, the system builds a repertoire of strategies associated with problem domains. The important research question is whether those strategies transfer to unseen problems rather than merely helping on tasks that resemble the extraction set. That is the next benchmark the project needs.

→ github.com/dtunai/mem-rlm