Local AI & On-Device Inference

Run open-weight LLMs locally with Ollama 5.x: model pulling, Modelfile customisation, REST API, GPU acceleration, multi-model management, and zero-cloud inference verification.

Maximum sovereignty with llama.cpp: compile from source, GGUF model formats, quantisation levels (Q4_K_M, Q8_0), CLI inference, server mode, and performance tuning for CPU and GPU.

Run AI models entirely on local hardware: Apple Silicon with MLX framework, NVIDIA CUDA with TensorRT, and AMD ROCm. Covers memory requirements, throughput benchmarks, and chip selection.

End-to-end sovereign AI stack builds: Ollama + Open WebUI + pgvector + LangChain running 100% on-device. Full companion repo, tested code, and sovereignty audit included.

Benchmark sovereign local LLM performance: tokens/second, memory usage, first-token latency, and quality metrics across Ollama models on Apple Silicon, NVIDIA, and AMD hardware.