During a trip back to China, I finally had some quiet time to write down what I want to remember from building Peregrine: a small Zig + Metal inference engine for Qwen3.5 9B q4 on Apple Silicon.

I wanted two things out of this: get hands-on with Zig again, and learn inference by owning enough of the stack that the performance and memory tradeoffs were not hidden from me.

I still use hosted tools constantly: Codex, Claude Code, Amp, and other models are part of my normal workflow. But some recent constraints, including Codex reset needs and Fable accessibility work, made me wonder how much of my loop could move onto my own machine.

The hardware constraint matters here. My machine is an M3 Pro with 18GiB of memory. That is not enough headroom to pretend local inference is an unlimited GPU box, but it is enough to make the constraints interesting.

For this setup, the practical model choice became Qwen3.5 9B q4. Once that was fixed, the question I kept coming back to was:

What would a small inference engine look like if it only had to be good at this model, on this machine, for this workflow?

The mental picture I kept using was this:

prompt tokens
      │
      ▼
┌──────────────────────┐      ┌────────────────────┐
│ Zig runtime / planner│◀────▶│ q4 model tensors   │
│ buffers + scheduling │      │ Metal buffers      │
└──────────┬───────────┘      └────────────────────┘
           ▼
┌─────────────────────────┐
│ prefill                 │
│ prompt -> KV/state      │
└───────────┬─────────────┘
            ▼
┌─────────────────────────┐
│ decode loop             │
│ one token at a time     │
└───────────┬─────────────┘
            ▼
      output tokens

The notes I attach to that diagram are more useful to me than a generic list of “lessons learned”:

  • Zig runtime / planner: buffer lifetime and scheduling are part of the design, not cleanup work after the kernel is written.
  • Prefill: this is where time to first token is won or lost.
  • KV/recurrent state: the route is only useful if the state can be handed back into decode cleanly.
  • Decode: this is a different loop, with different measurements and different traps.

Across all of it, I kept coming back to two numbers: performance and memory. Can the engine process the prompt and decode quickly? And can it do that without quietly spending so much unified memory that the rest of the machine suffers?

Why Not Just Use llama.cpp or MLX?

This was never because llama.cpp or MLX are bad options. They are the obvious local inference paths:

  • llama.cpp
  • MLX

llama.cpp is the impressive general-purpose option. It is cross-platform, fast, mature, and supports a broad set of models and quantization formats. The reason I still wanted to build something myself is that I did not need a runtime for every model on every machine. I wanted to understand and optimize one path on my own hardware.

MLX is the natural Apple Silicon option, but it has a different kind of surface area. There is mlx-lm, mlx-vlm, and other MLX-adjacent tooling. It was not obvious to me which layer I should squeeze for my use case, or where I should put model-specific serving behavior without turning my own code into another framework.

So the point was not “build a better general engine.” The point was to learn by owning the whole route.

Runtime / Memory

Memory is where local inference stops being abstract for me. On Apple Silicon, unified memory makes the setup feel simple, but it also makes mistakes very visible. If I allocate too much temporary Metal scratch, the whole machine feels it.

The KV cache was the first place this became concrete. Longer context is not just “more tokens.” It changes the shape of the runtime. Full-attention prefill has scratch terms that grow badly with prompt length, so a schedule that looks good on a short prompt can be a bad default for a 9B long-prompt path.

The rule I want to remember is: make the default memory-bounded first, then make the faster schedule opt-in. This is not only an implementation detail. A local engine has to share the machine with everything else I am doing.

This is also where Zig paid off. Allocators, ownership, and buffer lifetimes are visible. That friction slowed me down at the beginning, but it made the memory contract hard to ignore.

Prefill

Prefill is where I fooled myself the most. It is the work over the prompt before the engine can emit the next token, so it dominates time to first token. It is also where a promising kernel can look much better than the route it belongs to.

FlashAttention-style prefill is the example I keep coming back to. The research idea is easy to say: avoid materializing the full score/probability matrix. In the engine, that sentence turns into blocked full-attention fusion, D256/D-split scouts, MPSGraph SDPA probes, and layout/scratch gates. Some probes computed the right thing and were still not wins, because they did not move the full prefill route without blowing up memory.

The same pattern showed up with FlashQLA and Qwen3.5’s recurrent path. The idea was to replace token-by-token recurrent work with chunked, matrix-shaped work. The concrete work became KKT solves, corrected output, context-parallel state commit, and state-handoff gates. The part I want to remember is not just the math; it is that the output has to flow back into the real engine.

The most useful prefill work ended up being less glamorous: layer-major execution, selective prepared RHS, graph-arena lifetime management, and queued or lookahead prepare. Those were not a single famous kernel. They were route-level changes that matched scheduling to memory lifetime.

The note to myself here is simple: a faster kernel is only a hint. It becomes real only when it survives the full route.

Decode

Decode is the repeated one-token path after prefill. I have to keep it mentally separate from prefill because the metric is different. A route can improve decode and still feel worse if prefill or startup gets slower. A prefix cache can make a repeated prompt feel much faster without improving the steady decode loop at all.

The decoding example I want to revisit later is speculative decoding. As a research idea, it has a clean sentence: do extra work to predict or verify more than one token. In the engine, it becomes MTP, DFlash, draft/verify scheduling, KV-cache handling, and verifier-tail plumbing.

I do not want to unpack all of that in this note. The important thing for me is that decode experiments need their own accounting. If I mix prefill, process wall time, prompt reuse, and steady decode into one number, I can convince myself of almost anything.

Measurement

The measurement side is part of the engine work, because it decides whether an optimization actually survived the route or only looked good in isolation.

The numbers I do not want to mix together again are:

  • generated throughput including prefill
  • prompt eval throughput
  • decode throughput
  • process wall time
  • peak memory
  • text or token equality against the reference path

The benchmark is not there to make the post look convincing. It is there to stop me from overclaiming to myself.

Why I Am Splitting the Inference Notes

The first version of this post tried to include everything: motivation, Peregrine, benchmark methodology, failed compression ideas, speculative decode, ICB experiments, GGUF packaging, and current bottlenecks.

That was too much. It made the draft read like a compressed changelog, and it also made the writing feel less like my own notes.

For myself, I think the split should be:

  1. This note: why I wanted a small local inference engine in Zig, and the mental model I used to organize the work.
  2. A methodology note: the benchmark gates, failed optimization attempts, and what counted as evidence.
  3. A Peregrine note: the distilled Qwen3.5 9B q4 engine surface and what makes local agent serving feel usable.

That split matches how the work felt in practice. This post is the mental model; the next one can go deeper on the validation loop.