Local AI means running an AI model on your own device instead of sending the work to a remote service. That might be a language model answering question on your laptop, a Raspberry Pi identifying objects through a camera, or a microcontroller recognizing hand movement from a motion sensor. The common thread is where the computation happens...on the edge!
But don't run to buy a new computer yet because you may not need one! Most modern laptops and desktops can handle small language models. And if a new computer isn't in your budget, there's lower-cost alternatives like a microcontroller or Raspberry Pi.
This guide will take you through the absolute basics of understanding local AI, the hardware requirements, and a very brief tutorial on how to setup your computer to chat with a local AI model. While not an exhaustive "getting started" guide, the information provided here is enough to get your feet wet.
Want to see local AI in action? Watch my YouTube video unboxing the Surface Laptop Ultra and trying local AI on the device.
Note: Local isn’t automatically private. Downloaded models can run offline, but connected tools, telemetry, cloud features, and saved chat logs have their own behavior. Check your app’s settings, keep model servers on localhost, and verify generated answers before relying on them.
A Little Vocabulary
Some of these technical terms are pretty dense, but don't let that intimidate you! It took me a couple of tries to wrap my head around a few of these concepts. It wasn't until I was given the "chef + recipe" analogy that everything started to click! So my gift to you is that same analogy in hopes that you're able to "get it" much quicker than me.
First, think of a recipe.
Imagine a chef developing a recipe through repeated rounds of cooking, tasting, and adjusting.
The finished recipe is like the model: something you can use after that development work. The adjustable quantities—how much salt, how much flour—are like parameters. The particular amounts the chef settles on are like weights, the numerical values learned during training.
Quantization is like rounding the measurements in a recipe: “about ⅓ cup” instead of “0.328 cup.” You keep less precise measurements, which may change how the dish turns out. Similarly, quantization stores model weights using fewer bits, reducing storage and memory needs but potentially changing output quality.
Basic terms to know
- Model
- A model is a learned system for a task, such as generating text or recognizing a sound.
- Weights
- Weights are the numerical values learned during training, stored in the files you download.
- Inference
- Using a trained model to produce an output. Running a local chat model is inference.
- LLM
- A large language model, generally containing billions of learned parameters. There’s no fixed parameter count that makes a model “large.”
- SLM
- A small language model, generally containing fewer parameters and requiring less memory and computation than larger models. “Small” is relative and doesn’t guarantee it will run on every device.
- Parameters
- The learned values inside a model. A label like 1B means roughly one billion parameters. More parameters usually mean more memory, not automatically a better fit for your task.
- Token
- A piece of text a model processes, often part of a word. Models read and generate tokens rather than counting whole words.
- Context window
- The amount of input and generated text a model can consider at once. Longer conversations and documents use more context and usually more memory.
- Quantization
- Storing model weights at lower numerical precision to reduce memory use. Labels such as Q4 or 4-bit describe that trade-off; smaller files can come with changes in output quality.
- RAM / VRAM / unified memory
- RAM is system working memory; VRAM is memory on a dedicated GPU. Unified memory is shared by the CPU and GPU on systems such as Apple Silicon Macs or Surface Laptop Ultra.
- CPU / GPU / NPU
- A general-purpose processor, a processor suited to parallel work, and a processor designed for neural-network workloads. Your software must support the specific hardware to use its acceleration.
- Runtime / GGUF / MLX
- A runtime loads a model and performs inference. GGUF is a model file format used by tools based on llama.cpp. MLX is an array framework used for local models on Apple Silicon. Be sure to match the model files to your tool.
- TinyML
- Small machine-learning models deployed to constrained devices, often for recognizing gestures, sounds, or sensor patterns. This is a different scale of work from running a general-purpose chatbot.
Hardware Requirements
For your first small text model, try your existing laptop or desktop. A compatible CPU can run inference without a dedicated GPU, although responses may be slower. Check that the runtime supports your operating system and processor first.
Memory is usually the first constraint. Model weights need space, but so do the runtime, the conversation’s working memory, your operating system, and other apps. A 4GB download does not mean a computer with 4GB of RAM can comfortably run it.
As a rough planning guide for quantized text models, 8GB of system RAM can be enough to experiment with small 1–3B models; 16GB provides more room, including some 7–8B models at modest context lengths; 32GB or more gives you more flexibility. However, keep in mind that these are starting points, not guarantees!
A supported GPU can improve speed when enough of the workload fits in its memory. Image generation, speech models, and long documents have different requirements from short chats. Use the specific model’s documentation rather than applying one memory rule to all local AI models.
- Memory
- Check available RAM or VRAM, not only the installed total. Close other heavy apps for a first test.
- Storage
- Use an SSD if possible. Leave space beyond the download size, and more if you plan to keep multiple models.
- Connection
- Internet is needed for setup and downloads. With model files and runtimes installed, supported inference can work offline.
- LM Studio
- LM Studio is a desktop app for downloading and chatting with local AI models. Check out the current requirements .
Before You Buy
Before spending any money, there's a couple of things to consider to ensure that you're getting just what you need. While there may be overlap between some local AI projects, not all projects will have the same requirements.
Choose a task before a machine.
A short text assistant, a camera classifier, and an image generator are different workloads. Pick the model and runtime first, then check their actual requirements. A machine marketed as an “AI PC” does not guarantee compatibility.
Read the memory specification carefully.
For a discrete GPU, look at VRAM capacity, not just the card name. For a unified-memory computer, leave room for the operating system and your other apps. Check whether memory is soldered or upgradeable before buying.
Verify the complete software path.
Check the exact operating system, processor architecture, GPU, drivers, and runtime support. CUDA, Metal, Vulkan, and ROCm support varies by tool and device. An NPU is useful only when your chosen software can actually use it.
Compare sustained use, not just a headline.
Look for tests using the same model, quantization, and context length you plan to use. Consider response speed, fan noise, cooling, battery life, and power draw. Don’t judge performance only by TOPS (trillions of operations per second). That number alone won’t tell you how quickly an agent will answer.
Budget for the rest of the setup.
Storage fills up quickly when you collect models. Small boards may also need a power supply, cooling, storage, a camera, cables, or an accelerator. Check return policies, used-hardware condition, and upgrade options before committing.
Raspberry Pi & Microcontrollers
Although we've seen prices for these devices increase in recent times, they're still infinitely cheaper than buying a computer (or just computer hardware in general) that's compatible for running local AI models. However, if you decide to go this route, the first order of business is to lower all expectations with regards to speed. These are slow. Or at least slower than running on a computer decked out with some of the specs I've mentioned above.
Due to their popularity, some retailers have placed restrictions on the quanity that can be purchased. However, don't let that discourage you! I've found that I often end up recycling mine across projects rather than hoarding a massive collection of them.
A Raspberry Pi 5 is essentially a tiny computer. It runs Linux, stores files, and can run compatible small AI models. It’s useful for a standalone build, but it won’t behave like a workstation with a large GPU. Start with a small quantized model, a compatible runtime such as llama.cpp, and again, realistic expectations about speed.
Microcontrollers are in a league of their own. They're not tiny computers, rather, think of them as devices that are made to execute one particular task. But fair warning, not all microcontrollers are equipped for running local AI models. For a greater chance of compatibility, I'd recommend an Arduino UNO Q which combines a Linux computer with a microcontroller on the same board.
What You’ll Need
Note: Some of the links in this list are affiliate links. What that means is that I'll earn a tiny morsel of commission if you choose to purchase directly via the link provided.
| Item | What it’s for | Where to buy |
|---|---|---|
| Raspberry Pi 5 | A Linux computer for small quantized language models and vision experiments. An 8GB or 16GB version offers more headroom; CPU-only chat can still be slow. | Amazon |
| 27W USB-C power supply | The recommended Pi 5 supply. Reliable power matters when peripherals and sustained processing are involved. | Raspberry Pi |
| Raspberry Pi Active Cooler | Active cooling for sustained Pi 5 workloads. Check clearance with your case and add-on boards. | Raspberry Pi |
| Raspberry Pi microSD cards | Boot storage for Raspberry Pi OS, applications, and small model files. Choose enough capacity for your downloads; an SSD is an optional upgrade. | Raspberry Pi |
| Camera Module 3 | An optional camera for vision projects. Pi 5 needs the appropriate mini camera cable; check what is included. | Raspberry Pi |
| Raspberry Pi AI HAT+ | An optional Hailo-8 / Hailo-8L accelerator for supported Pi 5 vision workloads. Not a general-purpose accelerator for arbitrary chat models. | Raspberry Pi |
| Arduino UNO Q | Combines a Linux computer with a microcontroller for local AI projects, such as recognizing objects or sounds and responding through sensors, lights, or motors. Model compatibility and performance depend on the available memory and software. | Amazon |
Find & Run Models
The model and the app that runs it are two different things. A model hub helps you find files. A runtime loads those files and performs inference. Some desktop tools handle both, which makes them a beginner-friendly place to start.
Before downloading, check the task, language, license, memory needs, and supported runtime. For everyday chat, an instruction-tuned or chat model is usually easier to use, like the Qwen2.5-1.5B-Instruct model in the tutorial below. By the way, open weights doesn't automatically mean unrestricted commercial use! Lean towards official publishers and trusted distributions, and don’t enable repository-provided code you haven’t reviewed.
- LM StudioDownload models and start chatting
Search for compatible models, download them, and chat through a graphical interface. This is the tool used in the tutorial below. Check its current system requirements before installing.
Read the LM Studio guide- Hugging FaceA place to discover model files
Explore models for text, images, audio, and more! Read the model card for intended use, limitations, license, and compatible tools.
Explore models- OllamaAn alternative runtime
Browse a library and run supported models from a terminal or connect them to a local application. Choose downloaded local models rather than cloud-hosted options for on-device inference.
Browse the model library- llama.cppA more hands-on option
Run supported GGUF models on a range of hardware. Useful when you want more control, including on a Linux single-board computer; expect more setup than a desktop app.
Read the setup documentation
Your First Local Model
This brief tutorial assumes you've checked your existing hardware specs and have confirmed that you're potentially able to run a local AI model. If you skipped that step and came straight here, well, here goes nothing!
We’ll use Qwen2.5-1.5B-Instruct in its Q4_K_M version, a small model made for following instructions. A laptop with 8GB of RAM may be able to run it for short chats without a dedicated GPU, provided LM Studio supports its operating system and processor and enough memory is available. And by the way, close memory-heavy apps before trying it!
Install LM Studio.
Download LM Studio for your operating system, checking its system requirements first. Open the app and finish the initial setup, including any runtime downloads it requests. Keep your internet connection on for these steps.
Find the model.
Open LM Studio’s Discover tab and search for Qwen/Qwen2.5-1.5B-Instruct-GGUF. You can also paste the model page’s URL into search. Choose the repository published by Qwen, the model’s creator, and review its model card and license.
Download the Q4_K_M option.
Select qwen2.5-1.5b-instruct-q4_k_m.gguf, a roughly 1.12GB download. Q4_K_M is a 4-bit quantized version that keeps the download and memory needs smaller. Download just this option, not every variant.
Load the model.
Go to Chat, and open the model loader. Select Qwen2.5-1.5B-Instruct-GGUF to load it and wait until it is ready. Loading puts the model into working memory. Leave the other settings alone for this first test.
Try a simple prompt.
Type this into the message box and send it: What are the two leads called on a LED?
Try it offline.
Once the model and its runtime are installed, disconnect from the internet and send another message in the same local chat. It should still work without web tools or external integrations.
Note: When you’re done, use the model selector’s unload/eject control to release memory. The downloaded files remain available for next time; manage or remove downloads in My Models.
If it doesn’t load or feels slow
Close other memory-heavy apps and try again. If needed, lower the context length in the model’s load settings or try the smaller Qwen2.5-0.5B-Instruct model from the Qwen publisher. Check LM Studio’s runtime (within Settings) and hardware support for your machine. A smaller model can be easier to run, but its answers may be less capable.
Menu labels can change between versions. See LM Studio’s model download guide and offline documentation for current details.
Projects to Build
These projects are suggested starting points to dive into using local models within your own projects.
Notes Assistant
Turn hand-written notes into a digital archive that an AI assistant can answer questions about and organize. Start with text you wrote and compare the result with the original.
Draws on: local text inference, prompting, checking outputs.
Flower Identifier
Run a small image-classification model locally to identify a few types of flowers from photos. Start with three clearly different flowers and test with photos outside the training set. Show “not sure” when confidence is low.
Draws on: image classification, training data, confidence thresholds.
Desk-Side Voice Notebook
Use a local speech-to-text model to transcribe a short recording, then have a local language model organize the notes. Ask permission before recording anyone else.
Draws on: audio inference, model pipelines, privacy.
Parts-Sorting Camera
Train or adapt a small vision classifier to distinguish a few clearly different objects in your toolbox. Run it on a Raspberry Pi with a compatible camera and leave an “unknown” option for uncertain results.
Draws on: computer vision, Raspberry Pi, confidence thresholds.
Gesture-Controlled Desk Light
Use a compatible sensor-equipped microcontroller and a TinyML model to recognize a few gestures, then change an LED pattern. Keep a regular button as a fallback.
Draws on: TinyML, sensor data, microcontrollers.
Sound-Aware Desk Display
Train a small audio model to recognize a few everyday sounds, such as a doorbell or a kitchen timer, then light up a matching icon on a display. Process audio locally without saving recordings, and include an “unknown” result for unfamiliar sounds.
Draws on: audio classification, local inference, hardware interfaces.
Further Reading
- AI Projects with Raspberry Pi