How to Build a Local AI Server: A Practical Guide for SMEs

A local AI server is a machine hosted on your premises that runs AI models directly on company data, without sending anything to the cloud. For an SME you need a multi-core CPU, one or more GPUs with adequate memory, NVMe storage in RAID and a reliable network. The result: your data stays in-house and your costs become predictable.

At TN Solutions we design and install these systems as a System Integrator with more than 25 years of experience. In this guide we look at when an on-premise AI server makes sense, how to size hardware and storage, how to configure it with Proxmox and how to secure it under the NIS2 rules. The goal is concrete: bring AI into your business with no surprises.

Why run an AI server on-premise

Running AI locally, rather than on pay-as-you-go cloud services, makes sense when three factors weigh heavily: data confidentiality, cost predictability and operational continuity. A model that analyses contracts, patient records or production documents is processing sensitive information. Keeping it inside the corporate perimeter reduces risk and simplifies GDPR compliance.

On the economic side, cloud AI services are billed per token or per GPU hour. With heavy, continuous use, the monthly bill climbs fast. A dedicated server carries a defined upfront cost and, once you pass the break-even point, inference becomes effectively free. Then there is continuity: a local server keeps working even over unstable connections and does not depend on an external provider's policies.

The trade-off is management. Hardware, updates and security remain your responsibility. That is why it pays to approach the project with a partner who handles the infrastructure end to end, as described on our AI for business page.

What hardware you need: GPU, CPU, RAM and power

The heart of an AI server is the GPU, because deep learning models perform billions of operations in parallel. The most important variable is not just raw compute, but VRAM, the card's onboard memory: it determines how large a model you can load.

As a practical reference for inference (using an already-trained model):

• Models up to 8 billion parameters: 16-24 GB of VRAM is enough, for example a 24 GB professional card.
• Models of 13-30 billion parameters: you need 24-48 GB, often with 4-bit quantisation to keep the footprint down.
• Models above 70 billion parameters: you work with multiple GPUs and 80 GB or more of combined memory.

The other components are sized around the GPU. A CPU with a good core count (16 or more) handles data pre-processing and I/O. System RAM should be at least equal to total VRAM, ideally double: 128 GB is a sensible starting point. Watch the power supply: AI GPUs draw a lot, so you need redundant PSUs and proper thermal sizing of the room or rack, with dedicated cooling.

New server or refurbished

You don't always need to start from scratch. A recent business server can be upgraded with GPUs and storage suited to AI, cutting the investment. We assess hardware condition and warranties case by case, work that dovetails with our server support and the analysis of your existing infrastructure.

Storage and RAID: protecting data and models

AI models, datasets and embeddings take up a lot of space and are read constantly, so storage must be fast and reliable. The recommended choice is NVMe drives for low latency, arranged in RAID to combine performance with data protection.

The configurations most used in an AI server:

• RAID 1 (mirroring): two drives that copy each other, ideal for the operating system and critical models. If one drive fails, the other keeps working.
• RAID 10: combines mirroring and striping, delivering high read and write performance with fault tolerance. It is the typical choice for working datasets.
• RAID 5/6: maximises usable space with distributed parity, better suited to cold archives than real-time inference because writes are slower.

RAID protects against a single drive failure, but it is not a backup: it does not guard against accidental deletion, ransomware or human error. You always need a 3-2-1 strategy, with separate and tested copies. If an array degrades or a disk fails, recovery must be handled by people who know the controllers, the kind of intervention we run through our data recovery centre without improvising.

Virtualisation with Proxmox and high availability

Dedicating a physical machine to a single task is often wasteful. With Proxmox VE, an open-source hypervisor based on KVM, you isolate AI workloads in virtual machines or containers while keeping other business services on the same infrastructure. The practical benefit is flexibility: snapshots before an update, scheduled backups, workload migration between nodes.

To let a VM use the GPU you configure PCI passthrough, which assigns the card directly to the virtual machine with near-native performance. In multi-user scenarios you can consider GPU partitioning, where the hardware supports it.

Where continuity is critical you build a Proxmox cluster across several nodes with high availability: if one node goes down, the VMs restart on another. To distribute requests to the inference service you use a load balancer such as HAProxy, which routes traffic across multiple backends and manages health checks. That way an update or a failure does not block access to the AI. These themes are central to our work on server virtualisation.

Security and compliance: GDPR and the NIS2 Directive

A server processing company data is a target: it must be protected by design, not after the fact. The NIS2 Directive, now transposed across the EU, raises cybersecurity obligations for a wide range of organisations, including many SMEs in supply chains, and requires technical measures, incident management and documented governance.

The baseline measures on an on-premise AI server:

• Network segmentation: the AI server on a dedicated VLAN, isolated from the user network, with a firewall controlling traffic.
• Least-privilege access: strong authentication, defined roles and tracking of who accesses models and data.
• Updates and hardening: operating system, GPU drivers and AI runtime kept up to date, unnecessary services disabled.
• Encrypted, tested backups: verified offline copies checked periodically, so you can restore quickly after an incident.

As a System Integrator certified to ISO 9001 and ISO 27001, we fold AI into the wider IT security strategy: adopting AI must strengthen data protection, not open new gaps.

From machine to solution: the Hector platform

A server on its own produces no value: you need the software that turns compute power into business results. Hector is the AI platform from TN Solutions that runs on this infrastructure. It lets you build assistants on your internal knowledge base, automate document processing and connect models to the business applications you already use.

The advantage of running Hector on a local server is data sovereignty: information never leaves the company and stays under the control of the people who own it, in line with our ISO certifications. Above all, you have a single point of contact who designs the hardware, virtualises it, secures it and maintains the platform over time, with no buck-passing between different suppliers.

Let's talk about your AI server

Are you considering bringing AI into your business on infrastructure you own? Our specialists size hardware, storage and security around your real use case and see you through to production with the Hector platform.

Call TN Solutions on 02 9517550 or reach us via the contact page: together we'll design a concrete, secure and measurable AI server.