In March 2026, Andrej Karpathy dropped autoresearch, a minimalist loop-based framework that allows AI agents to perform autonomous AI research. Instead of configuring hyperparameters manually, you let an LLM directly hack the codebase based on a high-level goal in program.md. The loop is a git-based ratchet: if a change reduces validation loss, the harness commits it; if it breaks or performance degrades, the system runs git restore and tries a different path.
But a lot of people are like: ok, I'm sold on the loop. But how do we build one that actually scales past simple leaderboard toys? More importantly, how do we make it solve real scaling bottlenecks?
Because wrapping a model in a while True loop and letting it edit Python code is the easy 5%. The real work is building a robust agentic harness that can handle open-ended R&D, and pushing it down to bare-metal GPU optimization where it actually counts.
That experience led me to build Ultraresearch. I'm setting out to develop the ultimate research engine, a custom system engineered to discover scaled search spaces that push far beyond today's standard scaling recipes. I designed the system from the ground up to support a broad feature set from the very beginning. The client is fully operational, ready to start running and executing experiments. You can sign up and download the client for Windows, Linux, and macOS at tryultraresearch.com/login. Before running your first experiment, you will need to generate an API key from the dashboard to authenticate your requests. Note that Ultraresearch itself is completely free; you can configure it with your own OpenAI API key, or let the system automatically detect an active Claude Code subscription on your machine to run out-of-the-box.
This first worklog details my architecture, early results, and the roadmap ahead. I will be sharing all updates and future progress directly here on my blog.
The Scaffolding of Intelligence: Harness Engineering & Runtime Systems
Look at Lilian Weng's Lil'Log post from July 2026, "Harness Engineering for Self-Improvement". Her point is simple: recursive self-improvement (RSI) is not going to start with models hacking their own weights directly. It is all about Harness Engineering: building the sandboxes, loops, and file-based state systems that surround the model. The harness is the software scaffolding. It decides how the model plans, runs tools, delegates to sub-agents, and manages context.
We're climbing an abstraction ladder here:
- Static Prompts: Basic instruction templates (the bottom of the ladder).
- Context Engineering: Structured memory systems like Agentic Context Engineering (ACE).
- Workflow Graphs: Dynamic execution graphs search using MCTS (Aflow, ADAS).
- Self-Modifying Code: Letting the agent optimize its own harness code directly (STOP, Self-Harness, Darwin Gödel Machines).
When you let the agent inspect its own run logs and propose code fixes to its own system, the harness itself becomes part of the search space. And by using persistent files instead of dumping massive contexts into the prompt window, you prevent context collapse and work with the model's natural strength: reading and writing code.
The Cognitive Bottleneck: Modeling and Systematizing "Research Taste"
But here's where these loops hit a wall: Research Taste. Demis Hassabis has talked about this a lot: taste is the judgment to select the high-impact question and frame a testable hypothesis. That’s what separates a pioneer scientist from a code executor. Current LLMs can generate syntax all day, but they lack the taste to propose new paradigms instead of just tuning hyperparameters on existing baselines.
How do you model taste? Albert Ying (Kejun Ying) uses a graph metaphor: if ideas are nodes in a network, taste is the ability to locate the high-centrality nodes, the exact leverage points that will impact the maximum number of connected ideas once solved.
And the gap between human ideas and LLM ideas is massive. A 2026 study by Yale and Chicago researchers ("Measuring the Gap Between Human and LLM Research Ideas") analyzed 11,000 papers and showed that LLMs rely on "bridging/synthesis" (simply merging concept A with concept B) nearly 64% of the time, compared to only 12% for humans. Humans focus on failure analysis, mechanism discovery, and system building; LLMs mostly combine existing things.
But taste can be learned. If you train a model on community citation feedback and peer reviews (the core thesis of "AI Can Learn Scientific Taste" and LessWrong's TastyBench), you can teach the orchestrator how to prioritize and filter hypotheses.
The Bare-Metal Frontier: Autonomous Kernel Synthesis & Hardware Co-Design
If you want to maximize training and inference speeds, you have to go below PyTorch eager mode down to raw hardware execution: GPU Kernel Engineering. In modern AI scaling, memory bandwidth is the bottleneck. Writing high-performance kernels requires hardware-algorithm co-design: fusing mathematical reformulations with Hopper and Blackwell execution paths.
This bare-metal layer is undergoing rapid automation:
- Algorithm-Hardware Co-Design: Tri Dao (creator of FlashAttention) has shown that maximum hardware efficiency requires co-design. While FA-1 utilized IO-awareness, FA-4 exploits Hopper and Blackwell architectures via asynchronous memory pipelines, warp-specialization, TMA, and FP8 precision.
- Triton as a Target: Due to its high-level Python syntax, OpenAI's Triton has become the preferred compilation target for kernel-writing agents. Research from Sakana, Stanford, NVIDIA, and Meta actively leverages Triton for automated optimization.
- The Performance Gap: Stanford's KernelBench project shows that LLMs only beat PyTorch eager mode about 20% of the time. Compiling isn't the hard part; writing hardware-aligned code is.
Evaluating Agents Against Physical Limits
To prevent reward-hacking, Elliot Arledge's kernelbench.com measures agents against actual physical rooflines, not baseline PyTorch quirks:
- The Hard Suite: Tests agents on H100 and B200 nodes. GPT-5.6 Sol currently tops the chart, capturing up to 30% of the hardware's theoretical roofline.
- The Mega Suite: Tasks agents with building complete GPU megakernels (like a Kimi-Linear W4A16 decode megakernel). Top models like Claude Fable 5 achieve up to 18.7x speedups over reference implementations.
- The Multi Suite: An upcoming benchmark designed to test NVLink bus-bandwidth efficiency (
busbw) across 8xH100 nodes.
Synthesizing the Stack: The Vision for Ultraresearch
Connecting these dots, the trajectory is clear. Karpathy's autoresearch proved we can automate the training loop on a single GPU. But scaling past easy wins means solving two massive challenges at opposite ends of the stack:
- At the top: We need Research Taste so agents select high-leverage research directions.
- At the bottom: We need Hardware Co-Design so agents write high-performance kernels aligned with physical hardware limits.
This is exactly why I'm building Ultraresearch. My goal is to design this agentic harness around an exceptional developer experience, directing it fully toward solving next-generation scaling bottlenecks in model architectures and high-performance computing capabilities. Specifically, I want to optimize the system to handle large-scale, open-ended tasks in AI research.
I believe that the future of AI research lies in unifying these layers into a single, closed-loop orchestrator. Ultraresearch operates as a co-evolving harness that maps high-impact research directions, orchestrates model training, and dynamically generates custom, hardware-optimized CUDA kernels to accelerate execution. By closing the loop between cognitive hypothesis generation and low-level hardware co-design, the goal is to accelerate the timeline to achieving superhuman scientific discovery.