Sizing GPU infrastructure for local AI: plan model, load and operations together
By Kevin Kröger, Geschäftsführer, Software und Plattformbetrieb
The right GPU is not determined solely by the number of model parameters. The decisive factors are number format and quantization, context length, parallel requests, desired response time, additional models, startup, failure scenario and actual usage times. Before purchasing, a representative load test should demonstrate the required storage reserve and the achievable throughput.
What task determines the infrastructure?
Starts with the user journey: How long is the input and output, how quickly does the first response have to appear, how many people are working at the same time and what error sequence does a delay cause? Document search, batch processing and interactive assistant create different load profiles. In addition, embedding, image recognition, speech recognition or post-evaluation may require their own resources. Only this workload decides whether a single GPU, multiple cards or a split-time setup makes sense.
Why is graphics memory so important?
Model weights, intermediate results, context and parallel queries consume memory. A theoretically suitable model size can no longer run stably under real parallelism. Quantization reduces memory requirements but, depending on the process, changes speed, compatibility and possibly quality. Hugging Face's documentation shows that different quantization methods have different requirements and runtime properties. Therefore, exactly the planned model version must be tested with the intended inference software.
How is a sensible load test set up?
Uses real length distributions and not just short sample prompts. Measures time to first token, issuance rate, total duration, memory, energy, errors, and queuing as concurrency increases. Tests normal operation, peak load and a longer continuous phase. Repeat the test with the technically accepted model variants. A good report shows not just a maximum value, but the range in which response time and stability meet the agreed limit.
What reserve is required in operation?
Plan space for security updates, new model versions, longer contexts, and temporary double loading when switching. Clarifies whether a disruption may lead to slower operation, failure or a controlled switch to another platform. Redundancy can mean a second GPU, a second host, or a deliberately limited replacement process. The economic solution depends on the accepted recovery time and must consider application, model, data index and identity together.
When do co-hosting and hybrid operation make sense?
Own hardware can be operated in co-hosting if the investment and technical control should remain with the company, but power, network, access and monitoring must be organized professionally. A hybrid structure separates confidential or constant load from rare, specialized tasks. The application must transparently decide which path to use. An external replacement must not circumvent data rules unnoticed. Operating costs, energy, readiness and renewal are included in the same calculation as the hardware.
From the answer to implementation
Related service
View the scope, delivery model and responsible contacts.
Open →Related product
See a practical product path connected to this topic.
Open →Working checklist
Prepare the next decision with a structured checklist.
Open →All specialist articles
Continue with reviewed answers from the same practice areas.
Open →Sources and basis
The central statements in this article were reviewed against the following primary sources.
Frequently asked questions
- Can GPU be calculated from model size?
- Just roughly. Quantization, context, parallelism, inference software and additional models significantly change the actual storage requirements and throughput.
- Is a larger GPU always more economical?
- No. Unused capacity also incurs costs. What matters are real load, growth, reserves and the costs of a bottleneck.
- Does local AI have to be permanently online?
- That depends on the process. For some tasks, a planned launch is sufficient, others require a monitored service with a defined restart.