znazz75.de

Building a Local AI Coding Assistant with Open-Source Models

Claude
,

You don’t need a cloud subscription to get AI-assisted coding in your editor. Open-weight coding models have gotten genuinely good, and running them locally means your code never leaves your machine. Here’s how to wire one up in VS Code.

Picking a Model

For local coding assistance, code-specialized models punch well above their parameter count compared to general chat models:

  • Qwen2.5-Coder (7B/14B/32B) — currently one of the strongest open coding model families, competitive with much larger general models on coding benchmarks.
  • DeepSeek-Coder-V2-Lite — strong at multi-file reasoning and larger context windows.
  • CodeGemma / StarCoder2 — lighter options if you’re limited to 8GB of VRAM or less.

Serving the Model

ollama pull qwen2.5-coder:14b
ollama serve &
curl http://localhost:11434/api/tags

Connecting VS Code with Continue

The Continue extension gives you inline chat and completions against any OpenAI-compatible or Ollama backend, right inside VS Code. Install it from the Extensions marketplace, then point it at your local model:

{
  "models": [
    {
      "title": "Qwen2.5 Coder (local)",
      "provider": "ollama",
      "model": "qwen2.5-coder:14b",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5 Coder 7B (autocomplete)",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b",
    "apiBase": "http://localhost:11434"
  }
}

Scripting Your Own Code-Review Assistant

Because everything runs over a local HTTP API, it’s trivial to script your own tools around it — for example, a pre-commit hook that asks the model to flag obvious issues in your diff:

import subprocess
import requests

def get_staged_diff():
    return subprocess.run(
        ["git", "diff", "--cached"], capture_output=True, text=True
    ).stdout

def review(diff, model="qwen2.5-coder:14b"):
    if not diff.strip():
        return "No staged changes to review."
    prompt = (
        "Review this diff for bugs, missing error handling, and unclear "
        "naming. Be concise, list only real issues:\n\n" + diff
    )
    resp = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": model, "prompt": prompt, "stream": False},
    )
    return resp.json()["response"]

if __name__ == "__main__":
    print(review(get_staged_diff()))

Hardware Expectations

A 14B coding model in 4-bit quantization needs roughly 10-12GB of VRAM for smooth completions. If you’re VRAM-limited, drop to the 7B variant for autocomplete and reserve a larger model, or an API-based one, for occasional deeper chat-based questions.

Why Bother, When Cloud Assistants Exist

Local assistants win on three fronts: your code never leaves your machine (important for proprietary or regulated codebases), there’s no per-token cost, and it works fully offline — useful on flights, in restricted networks, or simply to avoid rate limits during a long refactor.

, ,

Recommended

Categories