[{"content":"Since early October, my day-to-day chat assistant has run on a Raspberry Pi 5 with 8 GB of RAM. It answers questions, searches the web and a local copy of Wikipedia, reads files I upload, does arithmetic with a calculator tool, writes and runs code in a sandbox, and replies in Greek when I write in Greek. Apart from the web searches themselves, nothing leaves the house.\nOut of the box, it was barely usable. The model generated about 6 tokens a second, a new chat took 13–17 seconds before the first word appeared, and some follow-up questions took a minute and a half. This post covers every fix that mattered, the ones that didn\u0026rsquo;t, and the measurements behind each.\nBefore After Generation (decode) 6.3 tok/s 10.1–12.5 tok/s Prompt processing (prefill) 60 tok/s ~75 tok/s, plus ~7% from the overclock First token, new chat 13–17 s ~5–6 s First token, second message of a chat 17.5 s 3.5 s First token, back to an older chat 17 s 4 s First token after a restart 36 s 4.7 s One-line follow-up to a search answer 85 s cache hit All numbers are my own measurements on this Pi, under its normal background load, unless a link says otherwise.\nThe setup Hardware. A Raspberry Pi 5 with 8 GB of RAM: four Cortex-A76 cores at 2.4 GHz, LPDDR4X-4267 memory (specs), cooled by a PWM fan.\nIt isn\u0026rsquo;t a dedicated box. The same Pi runs about 60 other containers: Nextcloud, an RSS reader, backups, a Git server, search and more. Every fix below had to work alongside them, and some of the biggest gains came from dealing with them.\nSoftware.\nModel: Gemma 4 E2B, the 2B-effective-parameter edge model, in Google\u0026rsquo;s .litertlm format (model). Runtime: Google\u0026rsquo;s LiteRT-LM 0.18.0 on the CPU, through its OpenAI-compatible serve command. Front end: LibreChat, which handles chats, presets, file uploads and memory. Tools: connected over MCP: a local English Wikipedia served from a Kiwix ZIM file; web search through a self-hosted SearXNG; a calculator and date tool; a shell sandbox with a Python interpreter; offline translation with Argos Translate. File search: LibreChat\u0026rsquo;s rag_api with pgvector, using a second, small LiteRT-LM server for embeddings. Picking the runtime LiteRT-LM over llama.cpp. llama.cpp is the default answer for local models, but on the Pi it loses clearly for Gemma. Raspberry Pi\u0026rsquo;s own benchmark of Gemma 4 E2B measured LiteRT-LM at 99 tok/s prefill and 9 tok/s decode in 1.4 GB, against llama.cpp\u0026rsquo;s 24 and 4 tok/s in 4.4 GB. The .litertlm file also bundles the model\u0026rsquo;s multi-token-prediction drafter, which matters later.\nE2B over E4B. The larger E4B fits in RAM, but it decoded at 3.4 tok/s in my benchmark, less than half of E2B\u0026rsquo;s 8.5. With thinking turned on, that\u0026rsquo;s too slow for a chat.\nYNNPACK instead of XNNPACK. LiteRT normally runs CPU models on XNNPACK. YNNPACK is its newer, experimental successor, and on this Pi it made a big difference for E2B:\nBackend (1024-token prompt, 256 generated, 4 threads) Prefill Decode XNNPACK 103 tok/s 8.5 tok/s YNNPACK 148 tok/s 9.1 tok/s It also fixed memory. Under XNNPACK, a 16k or larger context was OOM-killed on the first request at a 7 GB limit; YNNPACK served a 32k context in about 4.6 GB.\nThe catch: serve never passes the YNNPACK option to the engine. Only the run and benchmark commands have the flag, and the config file ignores the key. My image patches one line into serve at build time, and the build fails loudly if upstream changes that line.\nYNNPACK takes about 7 s to build its graphs when the engine starts. I profiled that, hoping a disk cache of packed weights could skip it. Packing the weights turned out to be only 1.4 s of the 7, and it happens once per engine start, not per chat. Not worth building.\nMemory: hard limits and no swap With 60 other containers, the model\u0026rsquo;s memory is under constant pressure. Early on, 3.6 GB of it had been swapped out, and every token paid for that.\nNo swap for the model. The container has a 7 GB memory limit, with memswap_limit set equal to it. That gives the container no swap at all, so the model\u0026rsquo;s working set always stays in RAM. A reservation that did nothing. I\u0026rsquo;d also set mem_reservation: 4g, Docker\u0026rsquo;s soft memory guarantee. It turned out to be a no-op. Under cgroup v2 it becomes memory.low, and a cgroup\u0026rsquo;s protection is limited by its parents\u0026rsquo; memory.low. Docker\u0026rsquo;s parent slice has 0, so the container\u0026rsquo;s effective protection is 0. Read the effective values in /sys/fs/cgroup, not the compose file. Context size costs speed even when unused. I first set the context to the model\u0026rsquo;s maximum of 32k tokens. The engine allocates for the full context up front, and that costs speed on every request, even short ones:\nContext allocated Prefill Decode 2k 121 tok/s 9.1 tok/s 8k 99 tok/s 8.4 tok/s 16k 75 tok/s 7.4 tok/s I halved it to 16k, which still leaves room for a chat with search results and tool schemas.\nCPU: priority, background load and the overclock Background load matters more than clock speed. With the other 60 containers stopped, prefill went from 71 to 85 tok/s and decode from 6.6–7.0 to 8.2–8.5, at stock clocks. That\u0026rsquo;s a bigger gain than any overclock gave.\nThe reason is how decode works. All four threads move in lockstep, one token at a time, so if a neighbour takes even one core for a moment, the whole step waits.\nCPU priority gets most of that back. I can\u0026rsquo;t stop the other services, but I can rank them. Every container gets a cgroup cpu.weight of 100 by default, on a scale from 1 to 10000. Giving the model container cpu_shares: 262144, which Docker maps to the maximum weight of 10000, gave:\nUnder normal background load Prefill Decode Default weight 60 tok/s 6.3 tok/s Weight 10000 75 tok/s 7.4 tok/s Overclocking to 2.7 GHz. Raspberry Pi\u0026rsquo;s firmware starts throttling at 80°C, so I looked for the highest clock that never reaches it. To test safely, I used a one-shot tryboot.txt boot with a kernel watchdog, so a crash or hang reboots into the normal config. That saved three bad boots.\nClock Result 2.6 GHz, stock voltage Stable 2.8 GHz, stock voltage LiteRT-LM crashed once (SIGBUS) 2.8 GHz, +50 mV Stable, but stress tests reached 84°C and were still rising 2.9 GHz, +50 mV Reached 85.6°C, clock dropped to 1.5–2.1 GHz 3.0 GHz Doesn\u0026rsquo;t boot I settled on arm_freq=2700 at stock voltage. Ten minutes of all-core stress-ng peaked at 79.6°C with no throttle flags, and long chats peak around 71°C. It adds roughly 7% to prefill and 4% to decode.\nFor each clock, I checked stability two ways: stress-ng with result verification, and identical greedy chat output across clocks, byte for byte. A crash is obvious, but a silent miscalculation isn\u0026rsquo;t.\nWhat didn\u0026rsquo;t help:\nCPU governor: ondemand, performance and schedutil measured the same, because ondemand already runs inference at full clock. Three threads instead of four: slower. fp16 activations: 53–59 tok/s prefill instead of 63–75. Changing the prefill chunk size: no effect. Multi-token prediction, and a one-instruction bug fix Gemma 4 ships with a small drafter model for multi-token prediction (MTP), a form of speculative decoding. The drafter proposes several tokens, and the main model checks them all in one step. Google reports 1.6× faster decoding for E2B.\nOn this Pi, it raised decode from 7.2–7.4 to 10.1–12.5 tok/s in real chats (+37–73%), with prefill unchanged. Earlier I\u0026rsquo;d tried speculative decoding on E4B under XNNPACK, and it made decode 20–30% slower. Test each combination; don\u0026rsquo;t assume.\nThe bug. In LiteRT-LM 0.18.0, MTP drops each reply\u0026rsquo;s stop token from the conversation\u0026rsquo;s memory whenever the drafter had guessed past it (#3439). In a single question, you\u0026rsquo;d never notice. But my server reuses conversations across turns (more on that below), so each affected turn left the model\u0026rsquo;s view of the history slightly corrupted.\nI measured it by counting the conversation\u0026rsquo;s tokens after each of several greedy turns:\nTokens after turns 1–4 MTP off 21, 39, 68, 88 MTP on 20, 37, 65, 84 MTP on, fixed 21, 39, 68, 88 The fix. The decode loop rolls a speculative step back one token too far, taking the stop token with it. A comparison just before that instruction already leaves the missing one in the CPU\u0026rsquo;s carry flag, so changing one add instruction to adc (add with carry) corrects it.\nMy image patches that one instruction in the prebuilt library from PyPI. It checks the library\u0026rsquo;s SHA-256 first, so an upgrade fails loudly instead of patching the wrong bytes.\nWhy patch a binary instead of building from source? I did build LiteRT-LM from source: about 1.5 hours on the Pi, and the same speed as stock. But the open-source build has to use Google\u0026rsquo;s prebuilt tool-call constraint library, which is compiled against a different C++ standard library. It crashed on the first tool call. The PyPI library has it built in, so the binary patch it was.\nThe side effect. The drafter and verifier are greedy, so temperature, top_p and top_k no longer affect replies. For an assistant whose job is to be correct, I\u0026rsquo;ll take that.\nThe prefix cache: stop re-reading the whole chat This is where most of the gains came from.\nUpstream serve creates a fresh conversation for every request. Every message re-reads (prefills) the system prompt, every tool schema and the entire chat history. At 75 tok/s, a chat with a few search results in it means a minute or more of reading before the first new word.\nThe basic cache. My server patch keeps the conversation alive between requests. When the next request extends it, only the new messages are prefilled. Simple in principle. In practice, a series of things kept breaking the match.\nFollow-ups after a search missed every time. My server rewrites some replies before sending them, for example writing the Sources line itself (see below). But the client sends back the rewritten reply as history, which didn\u0026rsquo;t match the raw text the cache had stored. Every follow-up to a search-backed answer re-prefilled the whole chat: 85 s for a one-line follow-up. The cache now matches on what the client actually received, so those turns hit.\nChat titles evicted the chat. After the first reply of every chat, LibreChat asks the model for a title. That request has no tools, so it looked like a different conversation and evicted the chat. Every second message then re-prefilled from scratch.\nA small \u0026ldquo;title gate\u0026rdquo; now recognises LibreChat\u0026rsquo;s default title prompt and answers it straight away from the first message\u0026rsquo;s topic (\u0026ldquo;What is the capital of France?\u0026rdquo; becomes \u0026ldquo;Capital of France\u0026rdquo;), without touching the model. The second message\u0026rsquo;s first token went from 17.5 s to 3.5 s.\nThe memory agent did the same. LibreChat\u0026rsquo;s memory feature sends an extra request with every message to decide whether to save anything. On a server that handles one request at a time, that request ran ahead of the reply and replaced what was cached. In one measured question, about 30 of 64 seconds were memory overhead.\nA \u0026ldquo;memory gate\u0026rdquo; now answers those requests without the model, unless the newest message actually asks to remember or forget something.\nI also tried the cleaner fix: a scheduler that runs memory requests only when the engine is idle and cancels them when a chat request arrives. The scheduling worked, but the tests kept hitting faults in the MTP drafter (more on those below). Then a test running a second engine next to the 60 containers ran the Pi out of memory and hung it until it rebooted. Back to the gate.\nSeveral chats at once. The cache originally held one conversation, so switching chats and coming back re-prefilled everything. It now keeps three, closing the least recently used first, with a cap on the memory they can hold. Going back to an older chat takes 4 s to the first token instead of 17.\nNew chats: snapshotting the system prompt The prefix cache helps a chat once it exists. A new chat still prefilled its roughly 1,000-token opening, mostly system prompt, from scratch: 13–15 s before the first word.\nCloning. LiteRT-LM\u0026rsquo;s C API can clone a conversation, and a clone is byte-for-byte identical to the original. A clone costs about 1.1 s and 100 MB, against 12 s to rebuild.\nThe obvious plan: keep one conversation holding only the system prompt, and clone it for each new chat. The problem: conversations prefill their system prompt lazily, on the first message. So a snapshot of a conversation with no messages has nothing in it to reuse.\nThe fix. The engine has an internal option to prefill the system prompt when a conversation is created, but the 0.18 C API doesn\u0026rsquo;t expose it. It does expose a setter for a different option that serve never uses. So a second one-instruction patch makes that unused setter write the field I need instead. New chats now clone a prefilled snapshot and reach the first token in about 5–6 s, with thinking on.\nTool schemas are the hidden cost. Every tool\u0026rsquo;s name, description and JSON schema is part of the system prompt, so it all goes into the snapshot. That made the snapshot surprisingly expensive to build:\nTools enabled Time to build the snapshot None 17 s Wikipedia, search, calculator 29 s + sandbox 36 s + browser automation (25 tools, ~5k tokens) 94 s So I trimmed what the model sees:\nThe search server\u0026rsquo;s suggestion and instance-info tools are hidden, since the model never needs them. The page-reader tool\u0026rsquo;s 70-token description, mostly about anti-bot proxies I don\u0026rsquo;t run, is now one line. Any change to the tool list or preset still means a full rebuild.\nCold starts. After a restart, the first request had to load the engine, build the snapshot and warm up decoding: 36 s to the first token. The server now saves the recipe for the snapshot it keeps. On start, if the recipe is from the same day, it replays it against itself as a one-token request. The first real request after a restart now gets its first token in 4.7 s.\nOne snapshot, not several. I briefly kept three snapshots, one per tool set and preset, so switching between them wouldn\u0026rsquo;t rebuild. It cost too much: the container sat at 5.2 GB idle instead of about 3.7 GB, the Pi started swapping, and clones slowed from about 2 s to 5–6 s. It also made the next problem more likely. Back to one.\nDrafter faults. With MTP on, a cloned conversation sometimes fails in the drafter before its first token: 2 of 8 requests in one probe run. It also happened with a single snapshot, so the extra snapshots weren\u0026rsquo;t the cause; they only made it likelier. The server catches the error and retries on a new clone, then on a fresh conversation, and that has always gone through.\nOnce, under heavy test churn, a clone gave a wrong answer with no error at all. No retry catches that. It\u0026rsquo;s the one known risk I\u0026rsquo;m living with.\nFinally, closing conversations didn\u0026rsquo;t give memory back. glibc kept the freed heap, and the container sat at 5.4 GB with one chat and two snapshots. Over its memory cap, the server now trims the heap with malloc_trim, which recovers only 100–200 MB, and closes extra snapshots before cached chats.\nEmbeddings, and why not the GPU serve handles one request at a time. If uploads were embedded on the chat server, every chat reply would queue behind them. So embeddings run on a second LiteRT-LM server with EmbeddingGemma 2 (270M), on the CPU, capped at 1 GiB of memory.\nThe GPU was much slower. The Pi 5 also has a GPU, the VideoCore VII. LiteRT-LM\u0026rsquo;s GPU backend reaches it through WebGPU, which runs on Vulkan, which here is Mesa\u0026rsquo;s V3DV driver. Inside Docker, the container needs only the render node, /dev/dri/renderD128. With Mesa 25.0.7, it ran and returned correct embeddings. It was also far slower:\nEmbedding input CPU GPU GPU vs CPU One-line query 0.9 s 38–65 s 42–72× slower ~300 characters 0.9 s 39 s 43× slower ~1500-character chunk 2.1 s timed out (\u0026gt; 2 min) — The GPU run was also OOM-killed at the 1 GiB limit, while the CPU run peaked at 387 MiB. The Pi has no separate video memory, so the GPU\u0026rsquo;s roughly 580 MB of buffers are ordinary shared memory, and Linux counts them against the container\u0026rsquo;s memory limit.\nCaveats. Each run was a fresh container, so every GPU number includes first-use setup. The times are end-to-end requests; the model card gives 161 ms of pure inference on the Pi 5 CPU. Raspberry Pi\u0026rsquo;s LiteRT guide also asks for V3D_WEBGPU_OVERRIDE=1 and their own Mesa build. I didn\u0026rsquo;t record whether the override was set or whether this was their Mesa build, so treat this as an unofficial setup.\nWhy so slow? I haven\u0026rsquo;t pinned it down. Two explanations fit:\nShader compilation. V3DV compiles shaders the first time they\u0026rsquo;re used. A separate Pi 5 benchmark of a different embedding model, through llama.cpp\u0026rsquo;s Vulkan backend, found the GPU 25–44× slower and traced most of it to about 6 minutes of one-time shader compilation. Every one of my runs was a fresh container. Fixed input sizes. The model ships with fixed input sizes from 128 to 8192 tokens. The two short inputs both fit in 128, so equal times are expected; the 1500-character chunk needs 512. Timing warm runs on one container would separate the two. Raspberry Pi\u0026rsquo;s guide already says to expect the CPU to win for now, and for chat it couldn\u0026rsquo;t help anyway. Decode is limited by memory bandwidth: each token reads about 0.75 GB of weights, and the GPU shares the CPU\u0026rsquo;s RAM.\nDropping the GPU also improved security. The chat container had been running with privileged: true purely to reach /dev/dri, which the render node alone would have covered. Now it needs no device access at all and runs with no-new-privileges.\nFixes that weren\u0026rsquo;t about speed Speed made the assistant usable. These made it worth using:\nThinking is visible. LiteRT-LM 0.17 generated Gemma\u0026rsquo;s thinking tokens but only counted them. A patch streams them as reasoning_content, which LibreChat shows as collapsible thoughts. Cut-off replies say so. Thinking counts against the reply\u0026rsquo;s token limit, and upstream marked every reply as finished normally. A reply that spent all 4,096 tokens thinking came back empty and looked complete. It now ends with finish_reason: \u0026quot;length\u0026quot; and a note that it was cut off. No leaked control tokens. Gemma\u0026rsquo;s internal markers, such as \u0026lt;channel|\u0026gt; and raw tool-call syntax, sometimes leaked into replies. The server strips them. The server writes the Sources line. Asking a 2B model to cite reliably in its own words never worked. The server now builds the Sources line from the actual Wikipedia, search and page-read results since the user\u0026rsquo;s message, and replaces any Sources block the model wrote. The checklist goes in every message. A small model follows the newest turn far more than a system prompt thousands of tokens back. One rule was mostly ignored in the system prompt but followed 8 times out of 8 when appended to the user\u0026rsquo;s message (thinking off). So the prompt\u0026rsquo;s short \u0026ldquo;Before answering\u0026rdquo; paragraph is copied into every user message. Because every message gets the same copy, the prefix cache still matches. Tools beat prompting for reasoning. I tried higher reasoning effort, self-check instructions and forced \u0026ldquo;Wait,\u0026rdquo; continuations. None improved results on hard maths questions. Giving the model a calculator and a sandbox did: 11 of 12 correct with tools, 4 of 6 without. In LiteRT-LM 0.18, every reasoning-effort level above \u0026ldquo;none\u0026rdquo; means the same unlimited thinking anyway. Help belongs in tool results, not descriptions. Hints in the sandbox tool\u0026rsquo;s description changed nothing (0 of 2). The same help placed in the tool\u0026rsquo;s result, such as suggesting package names after a failed install, worked. Other languages go through translation. The model works best in English, so the server translates other languages to English with Argos when it\u0026rsquo;s confident of the language, and translates the reply back, keeping code and links intact. Argos runs offline, but it makes word-sense errors: the Greek for \u0026ldquo;legs\u0026rdquo; once came back as \u0026ldquo;feet\u0026rdquo;. General fixes only. I tried patching individual failed prompts, with special-case regexes and pre-searching certain question types. Each fix helped the tested prompt and interfered with others. I reverted all of it. The rule now: a fix must help a prompt nobody has tried yet. When the Pi isn\u0026rsquo;t enough Some questions need a bigger model. When my workstation is on, chats go to Qwen3.5 9B on its Intel Arc B570 GPU, at about 42 tok/s. A small router between LibreChat and the models checks whether the workstation is reachable. If not, it rewrites the request to the Pi\u0026rsquo;s preset and sends it to the Pi.\nThe Pi is the assistant that\u0026rsquo;s always there; the workstation is a faster one when it happens to be on.\nWhat didn\u0026rsquo;t work Tried Result GPU (VideoCore VII) 40–70× slower for embeddings; OOM at 1 GiB llama.cpp instead of LiteRT-LM 24 / 4 tok/s vs 99 / 9 (Raspberry Pi\u0026rsquo;s benchmark) Gemma 4 E4B Less than half the decode speed of E2B Speculative decoding on E4B with XNNPACK Decode 20–30% slower fp16 activations Prefill 53–59 vs 63–75 tok/s Three threads Slower than four CPU governor changes No difference Prefill chunk size No effect YNNPACK weight cache Would save ~1.4 s, once per engine start LiteRT-LM built from source Same speed; crashed on the first tool call Overclock beyond 2.7 GHz Throttles at 2.8–2.9 GHz; no boot at 3.0 Three system-prompt snapshots +1.5 GB memory, swapping, more drafter faults Idle-time memory scheduler Drafter faults; an OOM that hung the Pi Higher reasoning effort, self-check prompts No better answers Per-prompt fixes Helped the tested prompt, hurt others Lessons Measure end to end, under real load. The most expensive problems were cache misses and neighbouring containers, not the model\u0026rsquo;s raw speed. Don\u0026rsquo;t re-read what you\u0026rsquo;ve already read. The prefix cache, title gate, memory gate and system-prompt snapshot together did more for responsiveness than every speed tweak combined. Check the effective setting, not the config file. A memory reservation that was a no-op, an option serve silently ignored, a GPU backend that ran but crawled. Small binary patches are fine, if they check what they patch. Two one-instruction patches fixed a correctness bug and unlocked system-prompt snapshots. Each checks the library\u0026rsquo;s hash first. For a small model, the server does the reliable work. Citations, arithmetic, titles and translation are all done by code or tools around the model, not by the model itself. The Pi 5 won\u0026rsquo;t win benchmarks against a desktop GPU. But with these fixes, a 2B model on a single-board computer runs a useful, private assistant around the clock, alongside everything else the board already does.\nThis post replaces an earlier one on the Pi 5\u0026rsquo;s GPU result, which is now the \u0026ldquo;Embeddings, and why not the GPU\u0026rdquo; section above.\n","permalink":"https://alevritis.com/posts/running-an-ai-assistant-on-the-rpi5/","summary":"A private chat assistant with web search, a local Wikipedia, file search and a code sandbox, running on a Raspberry Pi 5 next to 60 other containers. Everything it took to make it fast enough to use, with the numbers.","title":"Running an AI assistant on the RPi5"}]