Local LLMs run open-source models on personal computers or local servers, without sending data to the cloud. This approach offers clear advantages in privacy protection, cost control, and offline use. With high-quality open-source models like Llama 3, Mistral, Gemma, and Qwen, and mature quantization technology (GGUF format), 4-bit quantized 7B models run smoothly on ordinary laptops.
## Ollama: Lowest-Barrier Local LLM Tool
**Ollama** (ollama.com) is the most popular local LLM runtime, supporting macOS (Apple Silicon and Intel), Linux, and Windows. Installation is minimal: download the package → run a model with one command.
“`bash
ollama run llama3.2
ollama run qwen2.5:7b
ollama run deepseek-r1:8b
“`
Ollama automatically manages model downloads, versioning, and GPU/CPU resource allocation, providing a local REST API (default `http://localhost:11434`) compatible with the OpenAI API format — directly replaceable as a local development/debugging substitute.
## Hardware Requirements and Model Selection
**Apple Silicon (M-series Mac)**: Metal GPU acceleration; unified memory architecture lets 16GB RAM run 13B models smoothly, 32GB run 34B models. Among the best platforms for personal local LLM today.
**NVIDIA GPU**: VRAM is the bottleneck. 8GB VRAM handles 4-bit quantized 7B models; 16GB handles 13B; 24GB handles 34B; two 24GB cards can run 70B (RTX 3090/4090).
**CPU-only**: feasible but slow. 16GB RAM can run a 7B model (approximately 5–15 tokens/second — close to human reading speed but not fluid).
**Model recommendations**: general tasks: Llama 3.2 (Meta), Qwen2.5 (Alibaba); coding: DeepSeek-Coder-V2, Codestral (Mistral); Chinese language: Qwen2.5, Yi (01.AI); reasoning: DeepSeek-R1 distill series.
## Integration with Development Tools
Ollama’s OpenAI-compatible API connects directly to: Cursor/Continue (VS Code AI coding plugins), Open WebUI (local ChatGPT interface), LangChain/LlamaIndex (AI application frameworks). This makes building complete RAG applications and AI agents locally — with no cloud service dependency — fully practical.
See [RAG Knowledge Retrieval](https://sunqi.org/rag-knowledge-retrieval-en/) and [Ollama official site](https://ollama.com/).
