A Raspberry Pi will never train a model, but running inference on a small model directly on a $70 board is genuinely practical today, and opens up a huge range of offline, low-power AI projects — from a voice assistant to a security camera that recognizes objects without phoning home.
What’s Realistic on Pi-Class Hardware
- Raspberry Pi 5 (8GB) — can run small quantized LLMs (1B-3B parameters) at usable, if slow, speeds via llama.cpp; also handles lightweight vision models comfortably.
- Raspberry Pi 5 + AI HAT+ (Hailo NPU) — adds a dedicated 13-26 TOPS neural accelerator, making real-time object detection and pose estimation genuinely fast.
Running a Tiny LLM with llama.cpp
sudo apt install -y build-essential cmake git git clone https://github.com/ggerganov/llama.cpp cd llama.cpp cmake -B build && cmake --build build --config Release -j4 # Download a small quantized model (e.g. Qwen2.5-1.5B, Q4_K_M) ./build/bin/llama-cli -m qwen2.5-1.5b-instruct-q4_k_m.gguf \ -p "Explain what a Raspberry Pi is in one sentence." -n 100
Object Detection with the AI HAT+
sudo apt install hailo-all # Run the bundled real-time object detection demo against the camera rpicam-hello -t 0 --post-process-file /usr/share/rpi-camera-assets/hailo_yolov8_inference.json
Managing Expectations
Without a dedicated accelerator, expect single-digit tokens per second from an LLM on a Pi — usable for occasional queries, not a chat interface you’d want to use constantly. With the Hailo NPU add-on, vision workloads run genuinely well in real time, since the workload characteristics (fixed-size convolutions) suit dedicated accelerators far better than variable-length text generation does.