January 7, 2025 in Features, Advanced by Xiao Constantine8 minutes
Learn how RLM enables LLMs to programmatically explore large contexts through iterative REPL-based execution, achieving significant token efficiency gains.
Large Language Models have revolutionized how we process text, but they face a fundamental constraint: context window limits. Even with models supporting 100K+ tokens, processing massive documents, codebases, or datasets often requires either:
What if there was a way to give LLMs the ability to programmatically explore context of any size, querying exactly what they need, when they need it?
Enter Recursive Language Models (RLM).
RLM is a novel inference paradigm developed by researchers at MIT’s OASYS Lab ( arXiv:2512.24601). Instead of stuffing everything into a single prompt, RLM provides the LLM with a REPL (Read-Eval-Print Loop) environment where:
Think of it as giving the LLM a programming environment where it can methodically work through large contexts instead of trying to process everything at once.
RLM revolves around two key primitives:
// The LLM can query sub-sections or make sub-LLM calls
result := Query("Summarize the key findings in section 3")
// When done, the LLM signals completion
FINAL(finalAnswer)This simple pattern enables sophisticated exploration strategies:
We’ve implemented RLM as a first-class module in DSPy-Go, fully integrated with our module system, interceptors, and tracing infrastructure.
package main
import (
"context"
"fmt"
"time"
"github.com/XiaoConstantine/dspy-go/pkg/core"
"github.com/XiaoConstantine/dspy-go/pkg/llms"
"github.com/XiaoConstantine/dspy-go/pkg/modules/rlm"
)
func main() {
ctx := context.Background()
// Initialize your LLM
llm, _ := llms.NewLLM("your-api-key", core.ModelAnthropicSonnet)
// Create RLM module with configuration
rlmModule := rlm.NewFromLLM(llm,
rlm.WithMaxIterations(30),
rlm.WithTimeout(5*time.Minute),
rlm.WithVerbose(true),
rlm.WithTraceDir("./rlm_logs"),
)
// Process a large document
largeDocument := loadDocument("research_paper.txt") // 50K+ tokens
result, err := rlmModule.Process(ctx, map[string]any{
"context": largeDocument,
"query": "What are the three main contributions of this paper?",
})
if err != nil {
panic(err)
}
fmt.Printf("Answer: %s\n", result["response"])
fmt.Printf("Iterations used: %d\n", result["iterations"])
fmt.Printf("Total tokens: %d\n", result["total_tokens"])
}The RLM module orchestrates an iterative loop:
┌─────────────────────────────────────────────────────────────────┐
│ RLM Execution Flow │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. Initialize │
│ └── Load context into REPL environment │
│ └── Set up Query() and FINAL() functions │
│ │
│ 2. Iteration Loop (max 30 iterations) │
│ ┌──────────────────────────────────────────────────┐ │
│ │ a) LLM receives: │ │
│ │ • System prompt with RLM instructions │ │
│ │ • Current query/task │ │
│ │ • Iteration history │ │
│ │ • Available variables in REPL │ │
│ │ │ │
│ │ b) LLM responds with: │ │
│ │ • Reasoning (exploration strategy) │ │
│ │ • Code block (Go code to execute) │ │
│ │ • Action (explore/analyze/final) │ │
│ │ │ │
│ │ c) System executes code in sandboxed REPL │ │
│ │ • Captures stdout/stderr │ │
│ │ • Processes any Query() sub-calls │ │
│ │ • Updates variable state │ │
│ │ │ │
│ │ d) Check completion: │ │
│ │ • FINAL() called? → Return answer │ │
│ │ • Otherwise → Continue loop │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ 3. Return Result │
│ └── Final answer + metadata (tokens, iterations, time) │
│ │
└─────────────────────────────────────────────────────────────────┘DSPy-Go’s RLM uses Yaegi, a Go interpreter, to execute code within a sandboxed environment:
// Available in the REPL environment:
// The full context as a variable
context string // Your large document/data
// Query function for sub-LLM calls
Query(prompt string) string
// Batch queries for efficiency
QueryBatched(prompts []string) []string
// Signal completion
FINAL(answer interface{})
FINAL_VAR(variableName string) // Return a variable's valueSecurity considerations:
os, sys, subprocess, etc.)RLM achieves approximately 40% token savings compared to naive approaches. Here’s how:
Prompt = Full Document (50K tokens) + Question
───────────────────────────────────────
Every query sends the entire documentIteration 1: Question + "What sections exist?" (~500 tokens)
Iteration 2: Question + "Read section 3" (~2K tokens)
Iteration 3: Question + "Analyze finding X" (~1K tokens)
Iteration 4: FINAL(synthesized answer) (~500 tokens)
───────────────────────────────────────
Total: ~4K tokens vs 50K+ traditionalThe savings come from:
DSPy-Go tracks token usage across the entire RLM execution:
result, _ := rlmModule.Process(ctx, inputs)
usage := result["usage"].(rlm.TokenUsage)
fmt.Printf("Root LLM tokens: %d\n", usage.RootTokens)
fmt.Printf("Sub-call tokens: %d\n", usage.SubCallTokens)
fmt.Printf("Total tokens: %d\n", usage.TotalTokens)
fmt.Printf("Iterations: %d\n", result["iterations"])// For complex analysis tasks
rlm.NewFromLLM(llm,
rlm.WithMaxIterations(50), // Allow more exploration
rlm.WithTimeout(10*time.Minute),
)
// For quick extraction tasks
rlm.NewFromLLM(llm,
rlm.WithMaxIterations(5), // Faster completion
rlm.WithTimeout(1*time.Minute),
)Recent RLM work added explicit context-policy presets instead of relying on one fixed history strategy:
// Replay everything verbatim
rlm.NewFromLLM(llm,
rlm.WithContextPolicyPreset(rlm.ContextPolicyFull),
)
// Summarize older entries, keep recent ones verbatim
rlm.NewFromLLM(llm,
rlm.WithContextPolicyPreset(rlm.ContextPolicyCheckpointed),
rlm.WithHistoryCompression(2, 300),
)
// Start full, then checkpoint once the history gets large enough
rlm.NewFromLLM(llm,
rlm.WithContextPolicyPreset(rlm.ContextPolicyAdaptive),
rlm.WithAdaptiveCheckpointThreshold(5),
)This makes long-running RLM traces much cheaper to replay while still preserving recent local context.
Nested sub-RLMs are now explicitly budgeted:
rlm.NewFromLLM(llm,
rlm.WithSubRLMConfig(rlm.SubRLMConfig{
MaxDepth: 3,
MaxIterationsPerSubRLM: 6,
MaxDirectSubRLMCalls: 2,
MaxTotalSubRLMCalls: 8,
}),
)That gives you direct-child and whole-tree delegation guardrails instead of unbounded recursive fan-out.
Enable detailed tracing for debugging and analysis:
rlm.NewFromLLM(llm,
rlm.WithVerbose(true),
rlm.WithTraceDir("./rlm_traces"),
)This produces JSONL trace files containing:
RLM works seamlessly with other DSPy-Go components:
// Use with interceptors
rlmModule := rlm.NewFromLLM(llm)
rlmModule.SetInterceptors([]core.ModuleInterceptor{
interceptors.LoggingModuleInterceptor(),
interceptors.TracingModuleInterceptor(),
})
// Compose with other modules
program := core.NewProgram(
map[string]core.Module{
"extract": rlmModule,
"summarize": modules.NewPredict(summarySignature),
},
func(ctx context.Context, inputs map[string]any) (map[string]any, error) {
// RLM extracts key points
extracted, _ := program.Modules["extract"].Process(ctx, inputs)
// Then summarize
return program.Modules["summarize"].Process(ctx, map[string]any{
"content": extracted["response"],
})
},
)result, _ := rlmModule.Process(ctx, map[string]any{
"context": entireCodebase, // Thousands of files
"query": "Find all security vulnerabilities related to SQL injection",
})The LLM can:
result, _ := rlmModule.Process(ctx, map[string]any{
"context": researchPaper,
"query": "Compare the methodology with Smith et al. (2023) and identify gaps",
})result, _ := rlmModule.Process(ctx, map[string]any{
"context": serverLogs, // Gigabytes of logs
"query": "Find the root cause of the outage on January 5th",
})result, _ := rlmModule.Process(ctx, map[string]any{
"context": legalContract,
"query": "What are all the termination clauses and their conditions?",
})// Good: Specific and actionable
"query": "List all API endpoints that don't have authentication middleware"
// Less effective: Vague
"query": "Tell me about the API"| Task Type | Recommended Max Iterations |
|---|---|
| Simple extraction | 5-10 |
| Analysis | 15-25 |
| Complex synthesis | 30-50 |
if usage.TotalTokens > expectedBudget {
log.Warn("Token usage exceeded expectations",
"expected", expectedBudget,
"actual", usage.TotalTokens)
}When results are unexpected, enable tracing to understand the LLM’s exploration path:
rlm.NewFromLLM(llm,
rlm.WithVerbose(true),
rlm.WithTraceDir("./debug_traces"),
)pkg/modules/rlm/
├── rlm.go # Core RLM module implementation
├── config.go # Configuration options
├── repl_yaegi.go # Yaegi-based Go REPL
├── signature.go # DSPy signature definitions
├── token_tracker.go # Token usage tracking
└── rlm_test.go # Comprehensive testsRLM Module — Orchestrates the iteration loop, manages state, handles completion detection.
SubLLMClient — Interface for making sub-LLM calls from within the REPL:
type SubLLMClient interface {
Query(ctx context.Context, prompt string) (string, error)
QueryBatched(ctx context.Context, prompts []string) ([]string, error)
}TokenTracker — Accumulates token usage across all calls:
type TokenTracker struct {
RootInputTokens int
RootOutputTokens int
SubInputTokens int
SubOutputTokens int
}YaegiREPL — Sandboxed Go interpreter for code execution with security restrictions.
| Approach | Context Limit | Token Efficiency | Accuracy | Complexity |
|---|---|---|---|---|
| Full Context | Model limit | Low | High | Low |
| Chunking | Unlimited | Medium | Medium | Medium |
| RAG | Unlimited | High | Variable | High |
| RLM | Unlimited | High | High | Medium |
RLM combines the accuracy advantages of full-context processing with the efficiency of selective retrieval, without requiring external vector databases or retrieval infrastructure.
Install DSPy-Go:
go get github.com/XiaoConstantine/dspy-goImport the RLM module:
import "github.com/XiaoConstantine/dspy-go/pkg/modules/rlm"Create and use:
rlmModule := rlm.NewFromLLM(yourLLM)
result, _ := rlmModule.Process(ctx, map[string]any{
"context": yourLargeDocument,
"query": "Your question here",
})We’re actively developing RLM support in DSPy-Go with plans for:
RLM represents a paradigm shift in how we approach large context processing. By giving LLMs the ability to programmatically explore data through iterative REPL execution, we unlock:
The integration in DSPy-Go makes it easy to add RLM capabilities to your existing workflows while maintaining compatibility with our module, interceptor, and tracing systems.
Try RLM today and experience a new way of working with large contexts.
For questions and feedback, open an issue on GitHub or join our community discussions.