Preparing IT infrastructure for local LLMs
1. The strategic trade-off: when is on-premise necessary?
Before an organization decides to invest in physical hardware for running large language models (LLMs) locally, the fundamental question must be answered: what specific problem makes local infrastructure necessary? Many organizations start this journey from a general sense of data security, but a successful implementation requires a sharp analysis of data classification, laws and regulations, latency, and cost structures.
In heavily regulated sectors, such as healthcare, financial services, and government, the processing of personal data or intellectual property is subject to strict legislation. When data must never leave the internal network under any circumstances, public API providers are immediately ruled out. The introduction of legislation such as the GDPR, NIS2, and the AI Act forces organizations to maintain full control over the data flow. Hosting models locally guarantees that no data is shared with external parties for training purposes.
In industrial processes or real-time customer systems, the delay of an external network connection can be unacceptable. Local inference, if set up correctly, eliminates dependence on internet connections and reduces latency to the minimum the physical hardware allows. This is particularly relevant when LLMs are integrated into complex workflows with microservices that call each other in sequence.
Although the initial capital investments (CapEx) for owned hardware are significant, they offer a predictable cost structure in the long run. With a constantly high volume of transactions, public API costs can become unmanageable. An on-premise infrastructure flattens this cost curve: the costs for power, cooling, and management are largely linear and predictable, regardless of the number of tokens generated.
However, it's important to determine whether a fully physical on-premise installation is necessary, or whether a private cloud variant is sufficient. A dedicated cluster within a European cloud provider often offers the same guarantees around data sovereignty and network isolation, without the operational burden of physical hardware management. In the strategic trade-off between build or buy this nuance plays a central role.
2. Capacity planning: from usage to resources
Sizing a local AI infrastructure is complex because performance depends on an interplay of variables. Without zooming in directly on specific chip models, an organization can roughly calculate the required capacity based on the expected workload.
Capacity needs are determined by three main factors: expected concurrent usage, prompt length, and desired response time. Concurrent usage refers to the maximum number of users or systems submitting a request at exactly the same moment. The length of the input (prompts) and the desired length of the output (generation) determine processing time. The input phase (prefill) is compute-intensive, while the output phase (decode) is memory-bandwidth-intensive.
With these variables, total token throughput per second can be estimated. This throughput determines the required compute power of the accelerators. Because hardware vendors' theoretical limits are rarely reached in practice due to overhead in the software stack, a generous safety margin is necessary. It's advisable not to fully utilize the theoretical capacity in the initial design.
Because every model and every specific implementation (such as Retrieval-Augmented Generation, or RAG) has unique performance characteristics, a theoretical calculation is never conclusive. A controlled pilot is necessary to validate whether the hardware choices match actual user behavior. During this phase, metrics such as memory usage and processing speed are measured under realistic load. Moving from a pilot to production requires this empirical data to prevent costly miscalculations in the final hardware purchase.
3. Working memory on the accelerator: the hard limit
When running LLMs, the accelerator's working memory (VRAM or High Bandwidth Memory) is the most critical factor. If a model doesn't fit in memory, inference cannot take place, or speed drops to an unusable level. The size of this memory directly determines which model size can be deployed.
The basic memory requirement is determined by the number of parameters in the model and the precision at which these parameters are stored. By default, models are trained at 16-bit precision (FP16), which means each parameter takes up 2 bytes of memory. A model with 70 billion parameters therefore requires a minimum of 140 gigabytes of VRAM, purely to load the model into memory.
To reduce the memory footprint, quantization can be applied. This converts the parameters to a lower precision, such as 8-bit (INT8) or 4-bit (INT4). This significantly reduces the memory required, at the cost of a minimal loss in model quality. Quantization makes it possible to run larger models on less or lighter hardware, drastically increasing efficiency. More details on how this works at the individual level can be found in the guide on hardware for local LLMs.
In addition to the model itself, actively processing prompts requires memory. The Key-Value (KV) cache stores the intermediate calculations of earlier tokens in a conversation so they don't need to be recalculated for every newly generated token. This memory usage grows linearly with context length and the number of concurrent users. For long documents or extended chat sessions, the KV cache can become larger than the model itself. If the memory limit is reached, the system will have to refuse requests or truncate the context, which disrupts operation.
4. The rest of the machine: storage, network, and facilities
When designing an on-premise infrastructure for AI, attention often focuses exclusively on the accelerators. The surrounding infrastructure, however, is at least as important for preventing bottlenecks. This concerns storage speed, network bandwidth within the rack, and the server room facilities.
Model files are large and need to be written from storage to the accelerator's memory when services start up or a model is swapped. Traditional hard drives or slow network-attached storage (NAS) lead to long wait times here. Using fast, directly attached PCIe NVMe SSDs is necessary. In production environments where redundancy and fast recovery matter, the storage infrastructure must be designed for sequential read speeds of several gigabytes per second.
When a model is too large for a single accelerator, it must be split across multiple units (tensor parallelism) or even across multiple physical servers (pipeline parallelism). Communication between these units during computation is extremely intensive. If the network connection between the cards or servers becomes a bottleneck, the accelerators' compute power will go unused. Within the rack, using dedicated, low-latency connections with Remote Direct Memory Access (RDMA) is a requirement for scalable performance.
AI infrastructure also has a high power density. A single server rack with multiple accelerators can consume tens of kilowatts of power. This places specific demands on power supply and redundant power provisioning (UPS) in the server room. The heat generated must also be removed efficiently. Traditional air cooling is often insufficient at high densities, forcing organizations to invest in liquid cooling (direct-to-chip or immersion cooling) to prevent overheating and performance degradation.
5. Inference versus fine-tuning: divergent infrastructure requirements
The demands placed on the infrastructure depend heavily on the activity: using an existing model (inference) or adapting a model with proprietary data (fine-tuning). The differences in compute power and memory load are fundamental.
During inference, the system only needs to perform a forward pass. The model parameters remain unchanged. With fine-tuning, on the other hand, the gradients and optimizer statistics must also be stored in memory for the backward pass. This means fine-tuning requires up to three to four times more VRAM per model parameter than inference. In addition, significantly more compute power is needed to complete the calculations within an acceptable time.
Because of these exponentially higher requirements, many organizations choose to run only inference on-premise. Fine-tuning of models then happens periodically in a controlled cloud environment, after which the trained model weights are exported to the local infrastructure. This minimizes the need for extremely expensive training hardware that would sit unused much of the time. The trade-off between these scenarios directly touches on the total cost of ownership of open versus closed systems, as described in the analysis of the TCO of open vs. closed.
6. The software layer: from drivers to model registry
A stable AI infrastructure rests on a well-structured software stack. Without the right abstraction layers, managing local models is complex and error-prone.
+--------------------------------------------------+
| API Gateway |
+--------------------------------------------------+
| Inferentie-engine |
+--------------------------------------------------+
| Containerplatform (K8s) & Versneller-drivers |
+--------------------------------------------------+
| Fysieke Hardware |
+--------------------------------------------------+Although bare-metal installations offer minimal overhead, containerization (for example via Kubernetes with specific runtime plugins for hardware accelerators) is preferred in enterprise environments. Containers make it possible to dynamically allocate resources, isolate applications, and quickly move or scale workloads.
The interface between hardware and software is formed by the drivers and compute toolkits. Managing these is notorious for compatibility issues. An update to an inference engine may require a newer driver version, which in turn must be compatible with the underlying operating system. This requires a tight change management process and thorough regression testing in a staging environment.
The inference engine (such as vLLM or TensorRT-LLM) optimizes model execution through techniques such as continuous batching and advanced KV cache management. This engine is exposed via an API gateway that handles authentication, rate limiting, and logging. For stable integration, it's essential to place these local models behind an API so that internal applications can communicate with the AI infrastructure via standardized protocols.
A model registry acts as the 'single source of truth' for model files. It guarantees that only approved, scanned, and verified versions of model weights are put into production. This prevents individual developers from downloading unauthorized models directly from public repositories and running them on the infrastructure.
7. Availability, redundancy, and maintenance
Guaranteeing high availability for local LLMs requires specific measures that differ from traditional IT services. The size of the models and the specific hardware make standard failover scenarios challenging.
Updating a model is not comparable to a software update of a few megabytes. Because models are enormous, loading them can take several minutes. To prevent downtime, updates must be carried out via rolling deployments, in which a new instance of the model is started and tested before the old instance is decommissioned. Should the new model exhibit unexpected behavior in production (such as regression or incorrect output), the API gateway must be able to immediately fall back to the previous version via an automated rollback mechanism.
Even the best-designed local infrastructure can face hardware failures or unexpected spikes in demand. A hybrid architecture, in which the local API gateway temporarily forwards requests to a secure, compliant private or public cloud environment during overload or outages, acts as a safety valve. This guarantees business continuity without the local infrastructure having to be oversized for rare peak moments.
8. Security and network segmentation
A common mistake is assuming that an internal AI service is inherently secure. Within a modern IT landscape, the local LLM infrastructure must also be set up according to zero-trust principles. This protects the systems against unauthorized access and data leaks.
The inference servers should be placed in a separate network segment (VLAN), isolated from general office automation and other application servers. Direct access to the servers hosting the models must be blocked; all communication runs through the API gateway. The connections between client applications, the gateway, and the inference servers must be encrypted using mTLS (mutual TLS).
Not every employee or internal system should have access to every model. Some models are trained on sensitive HR data or financial information, while others are intended for general text processing. Implementing Role-Based Access Control (RBAC) on the API gateway enforces that clients must authenticate and can only call the endpoints approved for them.
Although the data stays within the organization's own network, detailed logging of who sends which data to the model is necessary. To meet compliance requirements, however, these logs must be anonymized or pseudonymized, so that no sensitive personal data ends up in the general system logs of the AI infrastructure.
9. Management overhead and team competencies: the hidden costs
The acquisition costs of hardware are visible, but in practice the operational management overhead is often the biggest cost item. Managing a local AI infrastructure requires specialist knowledge that is scarce in the labor market.
Keeping a local AI platform structurally up and running requires several disciplines. Infrastructure specialists are responsible for the physical servers, network connections, power, and cooling. MLOps engineers form the bridge between data science and IT operations; they manage the inference engines, container orchestration, and the model registry. Security officers oversee compliance with security guidelines and monitor data flows. Finally, AI validators monitor the quality and accuracy of model outputs to catch drift and bias early.
Many organizations underestimate the complexity of maintaining this stack. When failures occur in the drivers or hardware accelerators, a regular system administrator cannot resolve them. The structural costs of training and hiring this staff must be fully factored into the business case. This aspect forms an important part of a realistic AI implementation plan for SMEs.
10. A realistic rollout sequence
Building a large-scale on-premise production environment right away carries unacceptable risks. A phased approach ensures that investments are only made once the underlying assumptions have been proven.
| Phase | Objective | Infrastructure | Decision point for next phase |
|---|---|---|---|
| 1. Proof of Concept | Functional validation of the selected model | Temporary cloud or existing local hardware | Does the model meet the stated functional requirements? |
| 2. Controlled Pilot | Measuring system performance and resource usage | Limited on-premise test node | Are the latency and memory characteristics acceptable under load? |
| 3. Production Setup | Scaling, redundancy, and full integration | Fully redundant on-premise cluster | Is the management organization ready for operational go-live? |
In the Proof of Concept (PoC) phase, functional operation is central. The focus is on investigating whether the model is capable of performing the desired tasks. This can often be done cost-effectively on cloud infrastructure, since data security in this exploratory phase can be handled using anonymized data.
Once functionality has been proven, the move is made to a local test environment (the pilot) with representative data. Here, actual system performance is measured: how does memory behave under load? How many tokens per second are actually generated? This data is crucial for the final sizing of the hardware.
Only after a successful pilot is investment made in the full on-premise production infrastructure, including redundancy, backup provisions, and integrations with the central monitoring and security systems. The transition to this phase is only complete once the management team is also fully set up and trained.
