Running an AI assistant on the RPi5

Since early October, my day-to-day chat assistant has run on a Raspberry Pi 5 with 8 GB of RAM. It answers questions, searches the web and a local copy of Wikipedia, reads files I upload, does arithmetic with a calculator tool, writes and runs code in a sandbox, and replies in Greek when I write in Greek. Apart from the web searches themselves, nothing leaves the house.

Out of the box, it was barely usable. The model generated about 6 tokens a second, a new chat took 13–17 seconds before the first word appeared, and some follow-up questions took a minute and a half. This post covers every fix that mattered, the ones that didn’t, and the measurements behind each.

All numbers are my own measurements on this Pi, under its normal background load, unless a link says otherwise.

The setup

Hardware. A Raspberry Pi 5 with 8 GB of RAM: four Cortex-A76 cores at 2.4 GHz, LPDDR4X-4267 memory (specs), cooled by a PWM fan.

It isn’t a dedicated box. The same Pi runs about 60 other containers: Nextcloud, an RSS reader, backups, a Git server, search and more. Every fix below had to work alongside them, and some of the biggest gains came from dealing with them.

Software.

• Model: Gemma 4 E2B, the 2B-effective-parameter edge model, in Google’s .litertlm format (model).
• Runtime: Google’s LiteRT-LM 0.18.0 on the CPU, through its OpenAI-compatible serve command.
• Front end: LibreChat, which handles chats, presets, file uploads and memory.
• Tools: connected over MCP:
  • a local English Wikipedia served from a Kiwix ZIM file;
  • web search through a self-hosted SearXNG;
  • a calculator and date tool;
  • a shell sandbox with a Python interpreter;
  • offline translation with Argos Translate.
• File search: LibreChat’s rag_api with pgvector, using a second, small LiteRT-LM server for embeddings.

Picking the runtime

LiteRT-LM over llama.cpp. llama.cpp is the default answer for local models, but on the Pi it loses clearly for Gemma. Raspberry Pi’s own benchmark of Gemma 4 E2B measured LiteRT-LM at 99 tok/s prefill and 9 tok/s decode in 1.4 GB, against llama.cpp’s 24 and 4 tok/s in 4.4 GB. The .litertlm file also bundles the model’s multi-token-prediction drafter, which matters later.

E2B over E4B. The larger E4B fits in RAM, but it decoded at 3.4 tok/s in my benchmark, less than half of E2B’s 8.5. With thinking turned on, that’s too slow for a chat.

YNNPACK instead of XNNPACK. LiteRT normally runs CPU models on XNNPACK. YNNPACK is its newer, experimental successor, and on this Pi it made a big difference for E2B:

It also fixed memory. Under XNNPACK, a 16k or larger context was OOM-killed on the first request at a 7 GB limit; YNNPACK served a 32k context in about 4.6 GB.

The catch: serve never passes the YNNPACK option to the engine. Only the run and benchmark commands have the flag, and the config file ignores the key. My image patches one line into serve at build time, and the build fails loudly if upstream changes that line.

YNNPACK takes about 7 s to build its graphs when the engine starts. I profiled that, hoping a disk cache of packed weights could skip it. Packing the weights turned out to be only 1.4 s of the 7, and it happens once per engine start, not per chat. Not worth building.

Memory: hard limits and no swap

With 60 other containers, the model’s memory is under constant pressure. Early on, 3.6 GB of it had been swapped out, and every token paid for that.

• No swap for the model. The container has a 7 GB memory limit, with memswap_limit set equal to it. That gives the container no swap at all, so the model’s working set always stays in RAM.
• A reservation that did nothing. I’d also set mem_reservation: 4g, Docker’s soft memory guarantee. It turned out to be a no-op. Under cgroup v2 it becomes memory.low, and a cgroup’s protection is limited by its parents’ memory.low. Docker’s parent slice has 0, so the container’s effective protection is 0. Read the effective values in /sys/fs/cgroup, not the compose file.

Context size costs speed even when unused. I first set the context to the model’s maximum of 32k tokens. The engine allocates for the full context up front, and that costs speed on every request, even short ones:

I halved it to 16k, which still leaves room for a chat with search results and tool schemas.

CPU: priority, background load and the overclock

Background load matters more than clock speed. With the other 60 containers stopped, prefill went from 71 to 85 tok/s and decode from 6.6–7.0 to 8.2–8.5, at stock clocks. That’s a bigger gain than any overclock gave.

The reason is how decode works. All four threads move in lockstep, one token at a time, so if a neighbour takes even one core for a moment, the whole step waits.

CPU priority gets most of that back. I can’t stop the other services, but I can rank them. Every container gets a cgroup cpu.weight of 100 by default, on a scale from 1 to 10000. Giving the model container cpu_shares: 262144, which Docker maps to the maximum weight of 10000, gave:

Overclocking to 2.7 GHz. Raspberry Pi’s firmware starts throttling at 80°C, so I looked for the highest clock that never reaches it. To test safely, I used a one-shot tryboot.txt boot with a kernel watchdog, so a crash or hang reboots into the normal config. That saved three bad boots.

I settled on arm_freq=2700 at stock voltage. Ten minutes of all-core stress-ng peaked at 79.6°C with no throttle flags, and long chats peak around 71°C. It adds roughly 7% to prefill and 4% to decode.

For each clock, I checked stability two ways: stress-ng with result verification, and identical greedy chat output across clocks, byte for byte. A crash is obvious, but a silent miscalculation isn’t.

What didn’t help:

• CPU governor: ondemand, performance and schedutil measured the same, because ondemand already runs inference at full clock.
• Three threads instead of four: slower.
• fp16 activations: 53–59 tok/s prefill instead of 63–75.
• Changing the prefill chunk size: no effect.

Multi-token prediction, and a one-instruction bug fix

Gemma 4 ships with a small drafter model for multi-token prediction (MTP), a form of speculative decoding. The drafter proposes several tokens, and the main model checks them all in one step. Google reports 1.6× faster decoding for E2B.

On this Pi, it raised decode from 7.2–7.4 to 10.1–12.5 tok/s in real chats (+37–73%), with prefill unchanged. Earlier I’d tried speculative decoding on E4B under XNNPACK, and it made decode 20–30% slower. Test each combination; don’t assume.

The bug. In LiteRT-LM 0.18.0, MTP drops each reply’s stop token from the conversation’s memory whenever the drafter had guessed past it (#3439). In a single question, you’d never notice. But my server reuses conversations across turns (more on that below), so each affected turn left the model’s view of the history slightly corrupted.

I measured it by counting the conversation’s tokens after each of several greedy turns:

The fix. The decode loop rolls a speculative step back one token too far, taking the stop token with it. A comparison just before that instruction already leaves the missing one in the CPU’s carry flag, so changing one add instruction to adc (add with carry) corrects it.

My image patches that one instruction in the prebuilt library from PyPI. It checks the library’s SHA-256 first, so an upgrade fails loudly instead of patching the wrong bytes.

Why patch a binary instead of building from source? I did build LiteRT-LM from source: about 1.5 hours on the Pi, and the same speed as stock. But the open-source build has to use Google’s prebuilt tool-call constraint library, which is compiled against a different C++ standard library. It crashed on the first tool call. The PyPI library has it built in, so the binary patch it was.

The side effect. The drafter and verifier are greedy, so temperature, top_p and top_k no longer affect replies. For an assistant whose job is to be correct, I’ll take that.

The prefix cache: stop re-reading the whole chat

This is where most of the gains came from.

Upstream serve creates a fresh conversation for every request. Every message re-reads (prefills) the system prompt, every tool schema and the entire chat history. At 75 tok/s, a chat with a few search results in it means a minute or more of reading before the first new word.

The basic cache. My server patch keeps the conversation alive between requests. When the next request extends it, only the new messages are prefilled. Simple in principle. In practice, a series of things kept breaking the match.

Follow-ups after a search missed every time. My server rewrites some replies before sending them, for example writing the Sources line itself (see below). But the client sends back the rewritten reply as history, which didn’t match the raw text the cache had stored. Every follow-up to a search-backed answer re-prefilled the whole chat: 85 s for a one-line follow-up. The cache now matches on what the client actually received, so those turns hit.

Chat titles evicted the chat. After the first reply of every chat, LibreChat asks the model for a title. That request has no tools, so it looked like a different conversation and evicted the chat. Every second message then re-prefilled from scratch.

A small “title gate” now recognises LibreChat’s default title prompt and answers it straight away from the first message’s topic (“What is the capital of France?” becomes “Capital of France”), without touching the model. The second message’s first token went from 17.5 s to 3.5 s.

The memory agent did the same. LibreChat’s memory feature sends an extra request with every message to decide whether to save anything. On a server that handles one request at a time, that request ran ahead of the reply and replaced what was cached. In one measured question, about 30 of 64 seconds were memory overhead.

A “memory gate” now answers those requests without the model, unless the newest message actually asks to remember or forget something.

I also tried the cleaner fix: a scheduler that runs memory requests only when the engine is idle and cancels them when a chat request arrives. The scheduling worked, but the tests kept hitting faults in the MTP drafter (more on those below). Then a test running a second engine next to the 60 containers ran the Pi out of memory and hung it until it rebooted. Back to the gate.

Several chats at once. The cache originally held one conversation, so switching chats and coming back re-prefilled everything. It now keeps three, closing the least recently used first, with a cap on the memory they can hold. Going back to an older chat takes 4 s to the first token instead of 17.

New chats: snapshotting the system prompt

The prefix cache helps a chat once it exists. A new chat still prefilled its roughly 1,000-token opening, mostly system prompt, from scratch: 13–15 s before the first word.

Cloning. LiteRT-LM’s C API can clone a conversation, and a clone is byte-for-byte identical to the original. A clone costs about 1.1 s and 100 MB, against 12 s to rebuild.

The obvious plan: keep one conversation holding only the system prompt, and clone it for each new chat. The problem: conversations prefill their system prompt lazily, on the first message. So a snapshot of a conversation with no messages has nothing in it to reuse.

The fix. The engine has an internal option to prefill the system prompt when a conversation is created, but the 0.18 C API doesn’t expose it. It does expose a setter for a different option that serve never uses. So a second one-instruction patch makes that unused setter write the field I need instead. New chats now clone a prefilled snapshot and reach the first token in about 5–6 s, with thinking on.

Tool schemas are the hidden cost. Every tool’s name, description and JSON schema is part of the system prompt, so it all goes into the snapshot. That made the snapshot surprisingly expensive to build:

So I trimmed what the model sees:

• The search server’s suggestion and instance-info tools are hidden, since the model never needs them.
• The page-reader tool’s 70-token description, mostly about anti-bot proxies I don’t run, is now one line.

Any change to the tool list or preset still means a full rebuild.

Cold starts. After a restart, the first request had to load the engine, build the snapshot and warm up decoding: 36 s to the first token. The server now saves the recipe for the snapshot it keeps. On start, if the recipe is from the same day, it replays it against itself as a one-token request. The first real request after a restart now gets its first token in 4.7 s.

One snapshot, not several. I briefly kept three snapshots, one per tool set and preset, so switching between them wouldn’t rebuild. It cost too much: the container sat at 5.2 GB idle instead of about 3.7 GB, the Pi started swapping, and clones slowed from about 2 s to 5–6 s. It also made the next problem more likely. Back to one.

Drafter faults. With MTP on, a cloned conversation sometimes fails in the drafter before its first token: 2 of 8 requests in one probe run. It also happened with a single snapshot, so the extra snapshots weren’t the cause; they only made it likelier. The server catches the error and retries on a new clone, then on a fresh conversation, and that has always gone through.

Once, under heavy test churn, a clone gave a wrong answer with no error at all. No retry catches that. It’s the one known risk I’m living with.

Finally, closing conversations didn’t give memory back. glibc kept the freed heap, and the container sat at 5.4 GB with one chat and two snapshots. Over its memory cap, the server now trims the heap with malloc_trim, which recovers only 100–200 MB, and closes extra snapshots before cached chats.

Embeddings, and why not the GPU

serve handles one request at a time. If uploads were embedded on the chat server, every chat reply would queue behind them. So embeddings run on a second LiteRT-LM server with EmbeddingGemma 2 (270M), on the CPU, capped at 1 GiB of memory.

The GPU was much slower. The Pi 5 also has a GPU, the VideoCore VII. LiteRT-LM’s GPU backend reaches it through WebGPU, which runs on Vulkan, which here is Mesa’s V3DV driver. Inside Docker, the container needs only the render node, /dev/dri/renderD128. With Mesa 25.0.7, it ran and returned correct embeddings. It was also far slower:

The GPU run was also OOM-killed at the 1 GiB limit, while the CPU run peaked at 387 MiB. The Pi has no separate video memory, so the GPU’s roughly 580 MB of buffers are ordinary shared memory, and Linux counts them against the container’s memory limit.

Caveats. Each run was a fresh container, so every GPU number includes first-use setup. The times are end-to-end requests; the model card gives 161 ms of pure inference on the Pi 5 CPU. Raspberry Pi’s LiteRT guide also asks for V3D_WEBGPU_OVERRIDE=1 and their own Mesa build. I didn’t record whether the override was set or whether this was their Mesa build, so treat this as an unofficial setup.

Why so slow? I haven’t pinned it down. Two explanations fit:

• Shader compilation. V3DV compiles shaders the first time they’re used. A separate Pi 5 benchmark of a different embedding model, through llama.cpp’s Vulkan backend, found the GPU 25–44× slower and traced most of it to about 6 minutes of one-time shader compilation. Every one of my runs was a fresh container.
• Fixed input sizes. The model ships with fixed input sizes from 128 to 8192 tokens. The two short inputs both fit in 128, so equal times are expected; the 1500-character chunk needs 512.

Timing warm runs on one container would separate the two. Raspberry Pi’s guide already says to expect the CPU to win for now, and for chat it couldn’t help anyway. Decode is limited by memory bandwidth: each token reads about 0.75 GB of weights, and the GPU shares the CPU’s RAM.

Dropping the GPU also improved security. The chat container had been running with privileged: true purely to reach /dev/dri, which the render node alone would have covered. Now it needs no device access at all and runs with no-new-privileges.

Fixes that weren’t about speed

Speed made the assistant usable. These made it worth using:

• Thinking is visible. LiteRT-LM 0.17 generated Gemma’s thinking tokens but only counted them. A patch streams them as reasoning_content, which LibreChat shows as collapsible thoughts.
• Cut-off replies say so. Thinking counts against the reply’s token limit, and upstream marked every reply as finished normally. A reply that spent all 4,096 tokens thinking came back empty and looked complete. It now ends with finish_reason: "length" and a note that it was cut off.
• No leaked control tokens. Gemma’s internal markers, such as <channel|> and raw tool-call syntax, sometimes leaked into replies. The server strips them.
• The server writes the Sources line. Asking a 2B model to cite reliably in its own words never worked. The server now builds the Sources line from the actual Wikipedia, search and page-read results since the user’s message, and replaces any Sources block the model wrote.
• The checklist goes in every message. A small model follows the newest turn far more than a system prompt thousands of tokens back. One rule was mostly ignored in the system prompt but followed 8 times out of 8 when appended to the user’s message (thinking off). So the prompt’s short “Before answering” paragraph is copied into every user message. Because every message gets the same copy, the prefix cache still matches.
• Tools beat prompting for reasoning. I tried higher reasoning effort, self-check instructions and forced “Wait,” continuations. None improved results on hard maths questions. Giving the model a calculator and a sandbox did: 11 of 12 correct with tools, 4 of 6 without. In LiteRT-LM 0.18, every reasoning-effort level above “none” means the same unlimited thinking anyway.
• Help belongs in tool results, not descriptions. Hints in the sandbox tool’s description changed nothing (0 of 2). The same help placed in the tool’s result, such as suggesting package names after a failed install, worked.
• Other languages go through translation. The model works best in English, so the server translates other languages to English with Argos when it’s confident of the language, and translates the reply back, keeping code and links intact. Argos runs offline, but it makes word-sense errors: the Greek for “legs” once came back as “feet”.
• General fixes only. I tried patching individual failed prompts, with special-case regexes and pre-searching certain question types. Each fix helped the tested prompt and interfered with others. I reverted all of it. The rule now: a fix must help a prompt nobody has tried yet.

When the Pi isn’t enough

Some questions need a bigger model, so new chats default to Qwen3.5 9B on my workstation’s Intel Arc B570 GPU, at about 42 tok/s. LibreChat calls it directly. While the workstation is off, those chats fail and I switch to the Pi by hand.

Until October 11, a small router sat between LibreChat and the models. It passed each request to the workstation unchanged. If the workstation was off, its server was still loading, or it failed before replying, the router rewrote the request for the Pi’s preset (its system prompt, model and sampling, and only as many recent turns as fit the Pi’s 13k-token window) and sent it to the Pi, with a first line saying the Pi was answering. I’ve since dropped it.

Note: the router is still a good setup. With a big model on a machine that isn’t always on, you keep one chat, and when that machine is away the small model answers instead of an error.

The Pi is the assistant that’s always there; the workstation is a faster one when it happens to be on.

What didn’t work

Lessons

1. Measure end to end, under real load. The most expensive problems were cache misses and neighbouring containers, not the model’s raw speed.
2. Don’t re-read what you’ve already read. The prefix cache, title gate, memory gate and system-prompt snapshot together did more for responsiveness than every speed tweak combined.
3. Check the effective setting, not the config file. A memory reservation that was a no-op, an option serve silently ignored, a GPU backend that ran but crawled.
4. Small binary patches are fine, if they check what they patch. Two one-instruction patches fixed a correctness bug and unlocked system-prompt snapshots. Each checks the library’s hash first.
5. For a small model, the server does the reliable work. Citations, arithmetic, titles and translation are all done by code or tools around the model, not by the model itself.

The Pi 5 won’t win benchmarks against a desktop GPU. But with these fixes, a 2B model on a single-board computer runs a useful, private assistant around the clock, alongside everything else the board already does.

This post replaces an earlier one on the Pi 5’s GPU result, which is now the “Embeddings, and why not the GPU” section above.