Running local AI no longer automatically means buying a workstation with a giant GPU. For lightweight inference, embeddings, classification, private assistants, and development work, a normal KVM VPS can be useful — provided you size it for the model you intend to run instead of the model you saw in a benchmark headline.
The biggest planning mistake is treating “AI” as a single workload. A 2 billion-parameter quantized model, a 7 billion-parameter model, a vector database, and an image generator have radically different memory and compute needs. The right question is not “Can this VPS run AI?” It is “What has to stay resident in memory, how quickly must it answer, and how many requests arrive at once?”
The short version: RAM sets the ceiling
| VPS memory | Reasonable local AI use | What to avoid |
|---|---|---|
| 4 GB | Small quantized text models, embeddings, lightweight classification, API experiments | Large context windows, multiple models loaded together, image generation |
| 8 GB | More capable quantized text models, RAG prototypes, single-user assistants | High concurrency, large unquantized models, serious GPU-style workloads |
| 16 GB | Comfortable CPU inference for modest models, larger vector indexes, multi-service stacks | Expecting workstation-GPU latency from CPU-only inference |
These are planning ranges, not compatibility guarantees. Model file size is only part of memory use. The runtime needs working memory, the context window consumes memory, and the operating system, database, reverse proxy, monitoring, and your application all need room too.
Why quantization changes the math
Model weights are numbers. Storing those numbers at lower precision can shrink the memory footprint dramatically, which is why quantized models are the practical route on ordinary servers. The trade is some loss of precision and, depending on the format and runtime, a different speed profile. For a private support assistant or coding helper, that trade may be perfectly acceptable. For a scientific workload where small numerical differences matter, it may not be.
A useful rule is to leave operating headroom rather than filling RAM to the last megabyte. Linux can use spare memory for filesystem cache, and AI runtimes often allocate additional buffers under load. A model that technically starts at 3.8 GB on a 4 GB server is not a production plan; it is a screenshot waiting to become an out-of-memory kill.
What 4 GB is actually good for
A 4 GB VPS is a good laboratory. It can host a small local model behind a private API, generate embeddings for a modest document set, classify tickets, summarize short text, or act as an orchestration node that calls larger hosted models only when needed. It is also enough to learn the operational pieces: process supervision, TLS, authentication, rate limits, logs, backups, and model updates.
LogicWeb note: If the AI endpoint is private, bind the model runtime to localhost and put authentication in front of the application. Do not expose a raw model port to the public internet simply because it answered on your laptop.
What changes at 8 GB
Eight gigabytes gives you breathing room. You can keep a modest quantized model resident while also running an application server, a small vector database, Redis, and a reverse proxy. This is where retrieval-augmented generation prototypes become much less annoying because the model is no longer competing with every supporting service for the last few hundred megabytes.
For a single user or low-volume internal tool, CPU inference can be acceptable even when it is not instant. If your product promise depends on dozens of simultaneous low-latency generations, however, the bottleneck becomes compute and concurrency rather than simple model fit. More RAM does not turn a general-purpose CPU into a large accelerator.
What 16 GB buys you
Sixteen gigabytes expands the set of models and the amount of context you can use comfortably. It also lets you run the boring infrastructure that makes an AI service reliable: a database, queue worker, observability agent, backups, and a staging copy. In practice, that operational headroom is often more valuable than squeezing in the next model size.
This tier is also a sensible place to split responsibilities. Keep the web app and retrieval system on one VPS, use a second service for inference when needed, and design the application so a model can be replaced without rebuilding the rest of the stack.
CPU, disk, and bandwidth still matter
- CPU: Local inference is compute-heavy. Fast cores can matter more than a large core count for low-concurrency requests.
- NVMe: Models must be loaded from disk, vector indexes perform frequent reads, and container images accumulate quickly. NVMe reduces the waiting around the edges.
- Bandwidth: Pulling multi-gigabyte model files is not free. Repeated rebuilds can consume more transfer than the application itself.
- Backups: Back up your prompts, application code, configuration, and data. You usually do not need to back up model files that can be pulled again from a trusted source.
A sensible deployment pattern
- Start with the smallest model that can perform the task, not the largest model that can boot.
- Keep the inference service private and expose only your authenticated application or API.
- Measure resident memory after the model is warm, not just at startup.
- Load-test with the number of simultaneous requests you actually expect.
- Snapshot before runtime or model upgrades, and keep application data backed up separately.
LogicWeb’s KVM VPS plans are a natural fit for this kind of experiment because you control the guest OS and can reinstall or resize the stack as the workload becomes clearer. The smallest plan currently starts with 4 GB RAM, so it is enough to build the first version without pretending it is a GPU server.
FAQ
Can a CPU-only VPS run an LLM?
Yes, small and quantized language models can run on CPU-only systems. The main tradeoff is latency and concurrency compared with GPU-backed inference.
Is 4 GB enough for Ollama?
It can be enough for small quantized models and development. Leave memory for the operating system and supporting services rather than selecting a model that consumes nearly all available RAM.
Should I use a VPS or an AI API?
Use a VPS when privacy, predictable infrastructure cost, local control, or offline-style operation matters. Use an API when you need very large models, high throughput, or do not want to operate inference infrastructure.
Written at the desk
ChadBe the first to weigh in.