Open source LLMs — Llama 3, Mistral, Qwen, Gemma — can run on your local machine with no internet connection and no API cost. For researchers handling sensitive data, working in regions with unreliable internet, or wanting to customize models, local AI is a real option. But the tradeoffs are real too.
Why Local AI for Researchers
Data privacy: confidential patient data, proprietary datasets, pre-publication findings cannot ethically be sent to commercial API services. Local models process everything on your machine with no data leaving. Cost: once set up, zero marginal cost per query — important for intensive batch processing of documents. Customization: fine-tune a model on domain-specific papers to improve performance on your specific field.
Hardware Requirements
The required hardware depends on model size. 7B parameter models (Mistral 7B, Llama 3 8B) run on a 16GB RAM machine with CPU-only inference — slowly (1–3 tokens/second). 13B parameter models need a GPU with 8–12GB VRAM for reasonable speed. 70B parameter models (Llama 3 70B) require 40+ GB VRAM — typically A100 or H100 GPUs, only available via HPC clusters.
Getting Started with Ollama
Ollama (ollama.com) is the easiest way to run local models. Install it on Mac or Linux, then run: `ollama pull llama3` and `ollama run llama3`. You now have a local chatbot. Ollama also exposes an OpenAI-compatible API, meaning Cursor and other tools that support custom API endpoints can route to your local model instead of OpenAI.
Realistic Performance Comparison
Llama 3 8B on a consumer laptop: good at summarization, acceptable at question answering over documents, poor at complex reasoning. Llama 3 70B on HPC: close to GPT-3.5 quality on most tasks, significantly behind GPT-4 and Claude Sonnet on complex writing and reasoning. The gap with frontier models is real for research tasks that require nuanced judgment.
Best Use Cases for Local Models
Document classification (labeling large datasets by category), data extraction from structured reports, batch summarization of proprietary documents, preprocessing tasks that don’t require nuanced judgment. Not recommended: complex synthesis, argument evaluation, writing assistance where quality matters.
