Minimum system requirements

[zulhfreelancer]

Just wondering, if I want to run ollama and ollama-web (using Docker) on a single machine, what's the minimum requirement (CPU, RAM, disk)? Is GPU required?

[justinh-rahb]

CPU:

• Optimal: Aim for an 11th Gen Intel CPU or Zen4-based AMD CPU, beneficial for its AVX512 support which accelerates matrix multiplication operations needed by AI models. CPU instruction set features matter more than core counts, with DDR5 support in newer CPUs also important for performance due to increased memory bandwidth.

RAM:

• Minimum for a Decent Experience: 16GB is the starting point for running models like the 7B parameters effectively. It's enough for either running smaller models comfortably or managing larger ones with caution.

Disk Space:

• Practical Minimum: About 50GB should suffice, primarily to accommodate the Docker container size (around 2GB+ for ollama-webui) and model files, without needing a large buffer beyond the essentials.

GPU:

• Not Mandatory but Recommended for Enhanced Performance: A GPU can significantly improve model inference performance. However, the ability to run quantized models and the requirement for VRAM depends on the GPU generation. For instance, a 7B model at FP16 might require around 26GB of VRAM, which is beyond the capacity of many consumer-grade GPUs.
  • For Running Quantized Models: GPUs that support 4-bit quantized formats can much more efficiently handle large models, needing significantly less VRAM:
    • 7B model requires ~4 GB
    • 13B model requires ~8 GB
    • 30B model needs ~16 GB
    • 65B model needs ~32 GB

Considerations for Larger Models:

• Running 13B+ and MoE Models: Only advisable if you have a Mac, or a particularly large GPU or one that supports quant formats effectively. The memory and computational requirements for these larger models are substantially higher, making them impractical for most consumer-level hardware unless specifically configured for AI tasks.

Summary:

For optimal performance with ollama and ollama-webui, consider a system with an Intel/AMD CPU supporting AVX512 or DDR5 for speed and efficiency in computation, at least 16GB of RAM, and around 50GB of available disk space. A GPU is not required but recommended for performance boosts, especially with models at the 7B parameter level or higher. Particularly large or quant-supporting GPUs are considered essential for running larger models like 13B parameters efficiently, keeping VRAM requirements in mind for different model sizes and quantization levels.

[zulhfreelancer]

Thank you!

[nextideas]

Just wondering, if I want to run ollama and ollama-web (using Docker) on a single machine, what's the minimum requirement (CPU, RAM, disk)? Is GPU required?

Hi Zul,
if i use 2 RTX 4060 with 16GB each, this is equals a 32gb of vram?
to use a 65B model, for eg?