On-premises AI
AI without sending your data to the cloud
Most AI advice assumes you're happy to upload your work to somebody else's servers. Plenty of businesses aren't — and they've been told, wrongly, that this means sitting AI out.
There's a conversation we have often. A business owner describes something AI would obviously help with — quotes taking three days to go out, a filing cabinet nobody can search, the same spreadsheet rebuilt every Monday. Then they say some version of: but we can't put our files into one of those things.
They're usually right that they can't. A machine shop's drawings, a law firm's client files, a fabricator's bid pricing, a clinic's patient records — these are not documents you paste into a consumer chatbot. What's wrong is the conclusion. Not being able to use the cloud doesn't mean not being able to use AI. It means running it somewhere else.
What "on-premises AI" actually means
An AI model is a very large file containing patterns learned from training. Services like ChatGPT and Claude keep that file on their own servers; you send text in, an answer comes back, and your text has travelled to a company you don't control.
There is another category — open-weight models — where the file itself is published and can be downloaded. Meta's Llama family, Mistral's models, Alibaba's Qwen, Google's Gemma and DeepSeek's releases are the best known. Once you have the file, you can run it on a computer you own. There's no account, no per-message fee, and no outbound connection.
That's the whole idea. The model sits on a machine in your building, your software talks to it over your own network, and your documents never cross the boundary. People call this on-premises, self-hosted, local, or — when the machine has no internet connection at all — air-gapped.
The question was never "cloud or nothing." It's where the model runs — and that's a decision you get to make per workflow, not once for the whole business.
The honest version of "is it as good?"
You'll hear two dishonest answers. Vendors of cloud AI imply open models are toys. Open-source enthusiasts insist the gap has closed entirely. Neither is true, and the useful answer depends on what you're asking the model to do.
For the work small businesses actually need, open models are good enough that most teams can't tell the difference:
- Reading documents and pulling out figures — invoices, purchase orders, delivery notes, forms, scanned paperwork. This is pattern recognition against a known structure, and local models do it well.
- Drafting and rewriting — quotes, emails, job descriptions, standard letters, meeting summaries.
- Sorting and routing — deciding what an incoming message is about and who should see it.
- Searching your own material — answering "what did we quote them last time?" from your own files, with a link to the source document.
Where the largest commercial models still lead is genuinely hard reasoning: long multi-step analysis, novel programming, subtle judgement across many documents at once. If that's your core use case, be honest about it — a local model may frustrate you.
But notice that the demanding cases are rarely the confidential ones. The work that must stay in-house is usually document handling, extraction, and search — precisely where the gap is narrowest.
How to tell whether you need this
Most businesses don't. If your AI use is scheduling, general admin, marketing copy, and customer email, a commercial cloud service under a proper business agreement is cheaper, faster to set up, and perfectly appropriate. We say so regularly.
On-premises earns its cost when one of these is true:
- You hold trade secrets. Proprietary designs, tooling, formulas, process knowledge — the things that took years to develop and would take a competitor months to replicate if they saw them.
- Your contracts restrict processing. Defence, aerospace, medical, and legal work frequently carries terms about where client material may go. Those terms often predate AI and were not written with it in mind, which makes them stricter, not looser.
- Your pricing is your edge. If bid and quote data leaking would cost you real work, that data has a value a competitor would pay for.
- Your staff are already doing it anyway. This is the one owners underestimate. Somebody in your business has already pasted a customer contract into a free AI tool to summarise it. The exposure exists whether or not you sanctioned it. A sanctioned internal system is how you stop it.
A note on terms of service. The major AI providers' business and enterprise tiers do commit that your inputs aren't used to train their models, and those commitments are real. But "not used for training" is not the same as "never stored" or "never seen by a subcontractor." If your obligation is contractual rather than reputational, read what you're actually being promised — and check whether it satisfies the terms your own clients imposed on you.
What it costs, roughly
The shape of the cost is different from cloud AI, which is why comparisons get muddled. Cloud is a small recurring bill that scales with use. On-premises is a hardware purchase up front and very little afterwards.
For a small business the hardware is usually one workstation-class machine with a modern GPU — not a rack, not a data centre. A single well-specified machine typically serves a team of five to twenty people doing document and drafting work. Expect it to sit in the low thousands rather than the tens of thousands, though prices move and should be checked at the time you buy.
After that, running it costs electricity. There is no per-message charge, which means the economics invert at volume: heavy use makes local cheaper, where in the cloud it makes the bill grow.
We've written a fuller breakdown in what on-premises AI actually costs, including where the hidden expenses sit — which, spoiler, is not the hardware.
Usually the answer is a split
Very few businesses need everything in-house, and treating it as all-or-nothing leads people to overspend or give up entirely.
The common outcome is a line drawn through the middle. The appointment reminders and the marketing copy run in the cloud, where it's cheap and setup is trivial. The drawings, the client files, and the bid pricing never leave the building. Two systems, one boundary, deliberately chosen.
Drawing that line correctly is most of the value of the exercise. It requires knowing which of your workflows touch protected material and which merely feel sensitive — and those are not the same list. Owners are often surprised in both directions.
What to do about it
If you've been avoiding AI because your work is confidential, the useful first step is not shopping for hardware. It's an inventory: what work would AI genuinely help with, and which parts of it touch material that can't leave?
That's a conversation, not a purchase. It's also exactly what our free audit covers — and if the honest answer is that a cloud service under a business agreement serves you fine, that's what we'll say.
Related reading: Self-hosted AI for manufacturers covers the specific case of shops running CAD, ERP, and job data under customer confidentiality terms. What on-premises AI actually costs covers hardware sizing and the expenses people miss.
M Kiln AI builds AI systems for small and medium businesses in Rochester, New York and across the United States remotely — in the cloud, on isolated infrastructure, or entirely on your own hardware. We'll tell you which one you need before you spend anything.