CodingAgent
■ LOCAL LLMs // SOVEREIGN INFERENCE
Local runtimes are first-class targets: private repositories route strictly to local models, with hardware-aware dispatch across VRAM, context size and task history.
LOCAL
Developer-Local Inference
Instant local model serving with automatic GPU acceleration across Metal, CUDA, and ROCm. Perfect for single-developer workstations.
Specs
Fast quantization (Q4/Q5), 32k context, automatic model pulling
Hardware
16GB-64GB unified memory or 12GB+ VRAM GPU
Privacy
100% offline, zero network telemetry
LOCAL
High-Throughput Self-Hosted Serving
Continuous batching and PagedAttention for serving multiple concurrent agent subroutines across shared on-premise GPU clusters.
Specs
PagedAttention v2, tensor parallelism, OpenAI API compatibility
Hardware
NVIDIA A100 / H100 or multi-RTX 4090 cluster
Privacy
On-premise enterprise VPC isolation
LOCAL
Portable GGUF Bare-Metal Runtime
Lightweight C/C++ engine executing quantized GGUF weights with zero dependencies and efficient CPU/RAM fallback.
Specs
GGUF format, mmap zero-copy loading, CPU NEON/AVX2 acceleration
Hardware
Runs on standard laptops, edge devices, and CI runners
Privacy
Air-gapped verified, zero external dependencies
LOCAL
Desktop Model Endpoint
Visual desktop application providing interactive model experimentation, hardware utilization monitoring, and local HTTP endpoint.
Specs
GUI catalog, HuggingFace direct download, local port 1234 API
Hardware
Apple Silicon Mac or Windows/Linux gaming PC
Privacy
Localhost loopback only
Four rules decide where inference runs: the repository's privacy classification, real GPU memory headroom, the required context window, and each model's empirical verification history.
Privacy Tier
POLICY
If code is labeled Confidential or Air-Gapped, routing locks strictly to local models; external cloud APIs are hard-disabled.
VRAM Fit
RULE
Inspects real-time GPU memory headroom: dispatches 7B-14B models for fast localized edits, reserving 32B+ for complex refactors.
Context Size
RULE
Evaluates required context length from AST search. Short fixes fit standard 8k windows; deep call graph migrations use 32k-128k.
Quality & Task History
RULE
Routes based on historical verification pass rates for specific languages (e.g. Qwen2.5-Coder for Python, DeepSeek-Coder for TypeScript).