Local LLM Routing: When Ollama Answers and When Cursor Takes Over
Cloud AI assistants are powerful and expensive. Local models are private and limited. I built a smart-router MCP server that picks the right tier per prompt instead of defaulting everything to the most capable (and costly) model. But first: auto-routing does not make Cursor chat private. Your message always reaches the cloud before any router runs.
Routine prompts — summaries, formatting, quick lookups — go to Ollama running llama3.1:8b in Docker on localhost. Medium tasks — fix, implement, write tests — route to Groq when an API key is configured, with fallback to Ollama. Hard problems — architecture, security audits, large refactors — delegate back to the Cursor main agent.
The config lives in models.json. Each tier has a provider, model name, temperature, and timeout. Swapping models means editing one file, not refactoring application code.