Understanding llama.cpp context checkpoints for sliding window models

2026-10-30

I was using llama.cpp and trying to run Gemma 4 12B on it. I was stress testing the setup, with multiple requests running against it in parallel, and llama.cpp crashed on me.

I went into the logs, and it was an out of memory. I had 24 gigabytes of VRAM, so I was like, "Oh, that's weird. I didn't expect it to go over." I assumed, obviously, that I was running out of GPU memory.

So I started talking to Claude (like any person with a slight hint of AI psychosis nowadays): "Hey, what's going on? I don't get it." And Claude replies: no, that's not a GPU out of memory. It crashed out of RAM. And I was like, what the fuck? Why would it do that?

The machine had 24 gigabytes of VRAM, but only 16 of RAM, which I imagine is part of the reason it crashed, since 16 gigabytes isn't that much.

So we went into the logs, and trusty bot tells me there are two things that use a lot of RAM: the cache, and context checkpoints. Checkpoints are specific to models that have sliding window attention, or hybrid or recurrent layers. Gemma 4 is kind of a hybrid, because it interleaves the global layers, which have attention across every token in the context, and sliding window layers.

Trying to understand checkpoints and sliding windows

The prompt cache is easy enough to understand. llama.cpp keeps the last conversation in GPU memory, so if you continue it, it can quickly keep on going. If you switch to another conversation, the current one is moved to RAM and kept in a cache, in case a later request goes back to it. It can be loaded faster than doing the prefill step again.

On the other hand, the context checkpoints for the sliding window were harder for me to understand. I had an intuitive idea of how sliding window attention works. The idea is, well, you have a sliding window: as you process new tokens' KV values, you drop the older tokens.

But this concept of context checkpoints was weird to me. I couldn't understand what it was or how it worked. Talking to the bot, I understood intuitively that the idea is to take snapshots of your sliding window state and keep them in RAM so you can reuse them. But then I was like: how would you reuse it? You have this prefix cache; how do you go back? I couldn't understand the relationship between these checkpoints and what going back meant, since a model doesn't go back. You do inference: you have an input, it does stuff, and then you get output. It can't rewind, can it?

After a ton of back and forth with Claude, asking it to explain things to me, and it explaining things in that way that only an LLM can, I struggled for a good chunk of an evening, and then the next morning, trying to understand the concept, trying to visualize how these checkpoints came into play. Claude did try very hard to explain things to me, but there was a lot of jargon, and going back in time and stuff, and I just couldn't grok it.

At some point I was like, okay, why am I doing this? Opus 5.5 is apparently awesome at making animations. So I told it: "Take the thing you're telling me and explain it through an animation with diagrams. Explain what the layers do at each point, how you go back, how checkpoints are used, how they fill up the memory, and all that stuff."

Here's what I got out. It's pretty awesome, because I actually understood the thing. It was a mind-blowing realization. I was like: "Okay, I get it now 🎉." It's not such an alien concept. And when you think about how it works, it's a pretty awesome technique.

So in the end, a context checkpoint is a snapshot of the sliding window at strategic points in time, for example after a question. When a new request to continue a conversation comes, llama.cpp can check if it has a checkpoint to fill the sliding window with the correct values, and that way you can start from there and go faster. Otherwise you have to go through from the beginning and recompute the sliding window at point n.

And to do that, you really have to recompute it from the beginning, because the values in the sliding window at step n depend on everything that has happened before. Some of the information of the whole trajectory is inside those values. Not the tokens themselves, but the computed KV values for each, which depend on the ones that were computed just before.

So what caused my crash? Well, it was simply that on one side, the prompt cache was piling up to 8 GB of memory. And I had 2 slots, and the maximum number of context checkpoints in llama.cpp is 32 by default, and it's 32 PER slot (at least while they're both active). That, times the size of each checkpoint (320 MB for Gemma 4 12B), went over the 16 GB, and that crashed the process out of memory: not out of GPU memory.

So that was the mystery, and the solution was simple. For my use case, something like 4 checkpoints at most (--ctx-checkpoints 4) is more than enough to either go back to the system prompt and start a new conversation, or continue a past one and just start from the last question. With 4 checkpoints, llama.cpp seems to cover that already very well.

And on a final closing note: it's absolutely incredible what these language models can do for learning nowadays. It feels like having a very patient tutor, which today can not only explain things with words, but can also create rich animations and visualizations to explain complex concepts that don't translate well into just written words. I think that's an absolutely magical thing we have access to nowadays. Absolutely mind-blowing.