A growing number of customers no longer want to send every prompt to a third-party API. They want the model on a machine they control. In 2026 that request usually arrives as a ticket about RAM, swap, or why the process keeps getting killed.
This is not a guide to training models. It is a practical look at what happens when someone tries to run inference on a regular KVM VPS.
What people are actually installing
The most common stack we see is Ollama or a lightweight vLLM setup on a Linux VPS. Customers pull a quantized model (Q4 or Q5 is typical) and expect it to answer questions without leaving the box. Some add a simple web UI in front. A smaller group tries to expose an OpenAI-compatible endpoint for their own tools.
The hardware reality is blunt. A 7B or 8B quantized model can run on 8–16 GB of RAM if you are careful. Anything larger starts demanding more memory or a GPU that most general-purpose VPS plans do not include. Customers who order a $10 plan and then try to load a 70B model quickly discover the limits.
The tickets that follow
- “OOM killer just took my process.”
- “The model answers once and then the whole server feels slow.”
- “Can I add a GPU later?”
- “Swap is thrashing and the site on the same box is timing out.”
Running a local model on the same VPS that also serves a website is a frequent source of pain. The model wants memory and CPU; the web stack wants consistent latency. Mixing them without isolation usually ends in a support conversation about cgroups, separate instances, or simply moving the model to its own machine.
What works in practice
Give the model its own VPS when possible. Prefer plans with higher RAM over higher core counts for pure inference. Use quantized weights. Keep an eye on swap and set reasonable limits so one runaway process cannot take the whole node down. Snapshots before major model upgrades remain useful; a bad pull or a broken GGUF file is easier to roll back than to debug at 2 a.m.
GPU instances exist, but they change the price and the operational model. Most of the tickets we still see are from people trying to stay on CPU-only plans.
The honest boundary
A VPS is a good place to run a modest local model for private use, internal tools, or low-traffic applications. It is a poor substitute for a dedicated inference cluster when the load grows. Customers who understand that distinction open fewer emergency tickets and keep the rest of their services stable.
Written at the desk
ChadBe the first to weigh in.