First, what are we actually comparing?

An LLM, or large language model, is the software behind tools like ChatGPT, Claude and Gemini. You send it text (a question, a document, an instruction) and it sends text back. There are two broad ways to get that ability into your own product.

The first is a hosted API. API stands for application programming interface, which is just a doorway one piece of software uses to talk to another. Companies like OpenAI, Anthropic and Google run very large models on their own hardware. Your app sends a request over the internet, their model does the work, and the answer comes back in a second or two. You pay per use, measured in tokens. A token is a chunk of text, roughly three quarters of an English word, so 1,000 tokens is about 750 words.

The second is an open model that you host yourself. Meta's Llama, Mistral, Alibaba's Qwen, DeepSeek and Google's Gemma families all publish their model files. You download them and run them on servers you rent or own. Nobody charges you per token. You pay for the machines instead, usually graphics processing units (GPUs), the specialized chips that do the heavy math these models need.

A quick note on the word "open-source." Most of these models are really "open-weight." The weights are the billions of numbers a model learned during training, and those get published, while the training data usually stays private. For a product decision, what matters is simpler: can you run it on your own hardware and change it? Here, yes.

There's also a middle path people forget about. Several companies host open models for you and charge per token, much like a closed API. Together AI, Fireworks and Groq do this, and so do the big clouds through services like AWS Bedrock, Google Vertex AI and Azure. You get open-model pricing and the freedom to leave, without running servers yourself. Most generative AI development today starts with one of these three setups, and the middle option changes the cost math in ways we'll get to shortly.

The short version, side by side

Hosted API vs self-hosted open model at a glance

Tables like this make the choice look tidy. Real projects are messier, and the rest of this article is about the rows where the honest answer is "it depends," and what it depends on.

What the market is actually doing

Read together, those numbers show a split. Most companies pay for closed APIs because they want top quality without managing hardware. The Gartner figure on domain-specific models points the other way: smaller models built for one industry or task, which is exactly where open models do well. Frontier APIs take the hard, open-ended work, and smaller, cheaper models take narrow jobs that run millions of times a day.

The money question

Cost is where most teams start, and it's where most of the confusion lives. LLM application costs grow in completely different shapes on the two paths, and the shape matters more than the headline price.

How an API bill grows

With an API, cost follows usage in a straight line. Double your users and you roughly double your bill. Here's a simple example with made-up but realistic numbers.

Say your support assistant handles 50,000 conversations a month. Each one uses about 3,000 input tokens (the customer's messages, your instructions and a few help articles you include for context) and 500 output tokens (the reply). Suppose your model charges $3 per million input tokens and $15 per million output tokens.

Sample monthly API bill at two traffic levels

Now add an AI agent that calls the model eight times to finish one task instead of once, and the bill multiplies again.

This explains a common puzzle. The Stanford AI Index 2025 reported that the price of GPT-3.5-level quality fell from about $20 per million tokens in November 2022 to $0.07 by October 2024, a 280-fold drop. Yet plenty of companies watched their AI bills rise over the same period. Unit prices fell. Usage grew faster.

How a self-hosting bill grows

Self-hosting flips the shape. You pay for GPUs by the hour, whether they're busy at noon or sitting idle at 3 a.m. A high-end data center GPU typically rents for a few dollars an hour, and a mid-sized open model often needs one to four of them to run at a decent speed.

Two GPUs at around $3 an hour, running all month, cost about $4,400. You'll probably want a second machine so one failure doesn't take your product down, which brings it close to $8,800.

At 50,000 conversations a month, that's ten times the $825 API bill. At 2 million conversations, if the hardware can keep up, it's far less than $33,000. Somewhere in between sits a break-even point, and it's different for every product because it depends on prompt length, reply length, model size and how evenly your traffic is spread across the day.

The costs nobody puts in the first spreadsheet

The full cost picture

Founders underestimate the engineering line most. A person who can keep a GPU cluster healthy and get up when it breaks costs more per year than most early-stage API bills. If your monthly API spend is under roughly $10,000, self-hosting just to save money rarely works out once you count salaries. Privacy or control might still justify it. Savings alone usually won't.

Where the decision actually gets made

Price and quality dominate early planning, but teams usually change approaches because of problems that show up after launch. Five come up again and again.

Data gaps: what the model doesn't know, and what you can't send it

Every model has blind spots. It doesn't know your product catalog, your internal policies or the pricing change you made last Tuesday. There are two common fixes.

Retrieval means looking up the relevant documents at the moment someone asks a question and adding them to the prompt. You'll often see this called RAG, short for retrieval-augmented generation. Fine-tuning means training the model a bit further on your own examples, so a style or a body of knowledge becomes part of the model itself.

Retrieval works the same on both paths. Fine-tuning is where they split. Some API vendors let you fine-tune selected models, but only with the options they offer, and the tuned model still lives on their servers. With an open model you can fine-tune however you like, including cheaper methods like LoRA, which trains a small add-on layer instead of changing the whole model.

The second kind of data gap is legal. A hospital, a bank or a government contractor may not be allowed to send certain records to an outside company at all, whatever the vendor promises. Business API plans usually offer commitments not to train on your data, a choice of storage region and short retention periods, and for many companies that's enough. For others the rule is plain: this data never leaves our network. That single rule can settle the whole question.

There's a quieter gap too: visibility. An API returns an answer and a usage count. Some vendors also share per-word probability scores, which show how confident the model was, and some don't. Run the model yourself and you can inspect everything, which helps you catch it guessing before a customer does.

Conflicting signals: when the evidence doesn't agree

Here's something that happens all the time. A new open model tops a public leaderboard. Your developer tries it and says it's great. Your support lead tries it on real tickets and says it's worse than what you have now. Who's right?

Often both of them. Public benchmarks test general skills like math puzzles, science questions and trivia. Your product needs something narrower, such as following your refund policy exactly, or replying in Hindi without drifting into English halfway through. A model can score higher on general tests and still be worse at your particular job.

The fix is your own evaluation set: a few hundred real examples from your product, each paired with the kind of answer you'd accept. Run every candidate model through it and score the results, partly with automated checks and partly by having a person review a sample. It sounds tedious, and it's the most useful thing you can build in LLM development, because it turns opinions into numbers you can compare.

Conflicting signals also appear over time. API vendors adjust their models and the systems around them, so behavior can shift even when you haven't touched anything, and users say the assistant "got worse" while your dashboards look normal. With a self-hosted model the weights don't change unless you change them, which makes it much easier to tell whether a problem came from the model or from your own code.

Real-time decisions: speed, and where each request goes

Some products can wait five seconds for an answer. A voice assistant, live autocomplete or a fraud check at checkout cannot.

Two measurements matter here. Time to first token is how long before the first word appears. Throughput is how fast the rest of the reply arrives. APIs are generally quick, but you share their servers with a huge number of other customers, so response times can swing during busy hours. Distance adds delay as well. If your users are in Southeast Asia and the vendor's nearest servers are in the United States, that round trip is part of every single request.

Self-hosting lets you place a small model right next to your app. A model with 7 or 8 billion parameters (parameters are those learned numbers again; more usually means smarter but slower) can answer simple questions in a fraction of a second on one GPU.

This leads to a pattern many mature products use: routing. A small, fast model reads each request first and decides where it should go. Simple questions like "What are your opening hours?" get answered by the cheap model. Harder ones go to a stronger, pricier model through an API. The router decides in milliseconds, so it's usually a small classifier (a model that sorts inputs into categories) with a rule for when it isn't sure. Most teams send uncertain cases to the stronger model, trading a little extra cost for fewer bad answers.

Exceptions and edge cases

The odd cases are where products earn or lose trust. These are the ones that catch teams out most often:

• Refusals. API models have safety filters, and sometimes they refuse perfectly legitimate requests. A pharmacy app asking about drug interactions or a security tool analyzing suspicious files can run into this. With an open model you can adjust that behavior, which also means the responsibility for it moves to you.
• Broken formatting. If your code expects the reply in a strict format like JSON, the model will occasionally produce something slightly off, and your app crashes on it. Many APIs now offer structured output modes that force valid formatting. Open-model serving tools have a similar feature called constrained decoding, but someone has to set it up.
• Long documents. Models advertise very large context windows (how much text they can read at once), but accuracy often drops for details buried in the middle of long inputs. Test with your real document lengths, not the maximum listed on the spec sheet.
• Model retirement. API vendors retire older versions, usually with a few months' notice, and any tuning around one version's quirks gets redone on their timeline.
• Languages. Quality in less common languages varies a lot between models. Excellent English doesn't guarantee decent Marathi or Swahili.
• Licenses. Some open-model licenses restrict use above a certain number of monthly users or rule out particular uses. Read the license before you build a business on top of it.

Under pressure: what happens at scale

Most demos work. The real test comes when traffic jumps tenfold in an hour.

On an API, the first wall you hit is the rate limit. Vendors cap how many requests and tokens you can use per minute, and the cap depends on your account level. Go over it and you get errors, usually an HTTP 429 code that means "too many requests." Well-built systems handle this with retries that wait a little longer each time, a queue for work that isn't urgent, and a backup provider for when the main one struggles. Every major vendor has also had outages lasting hours, and a product with no fallback goes down with them.

Self-hosted systems fail in different ways. A GPU has a fixed amount of memory, and every active conversation uses some of it for what's called the KV cache, a kind of scratchpad holding the conversation so far. Long conversations and many users at once eat that memory quickly. When it runs out, requests start queuing and response times climb from one second to twenty. Serving software such as vLLM helps by packing many requests together efficiently, a method called continuous batching, but it can't create memory that isn't there.

Adding capacity is slow too. Starting a GPU server and loading a model that weighs tens of gigabytes can take several minutes, and popular GPU types are sometimes sold out in a region right when you need them. Teams that self-host at scale usually keep spare capacity running, which brings back the idle-cost problem from earlier.

There's an upside hiding in all this. At very high, steady volume, self-hosting gets cheaper per request as you pack more work onto each GPU, while API pricing stays flat per token no matter how much you use. This is where LLM application costs start to favor running your own hardware. It's why companies with huge, predictable workloads, like sorting millions of documents overnight, often move those jobs in-house while keeping APIs for everything else.

Three products, three different answers

A two-person startup building a writing assistant

They have no users, no GPU experience, and one urgent question: will anyone pay for this? An API is the obvious choice, with a prototype possible in a weekend and a small bill. The one thing worth doing early is putting a thin layer of their own code between the app and the vendor, so switching models later means editing one file instead of fifty.

A mid-sized clinic network summarizing patient intake notes

They want to summarize intake forms and flag cases that look urgent. The data is sensitive, the compliance team is careful, and the task is narrow and repetitive. A mid-sized open model, fine-tuned on a few thousand reviewed examples and hosted inside their own cloud account, fits well. They don't need the smartest model in the world. They need one that does a single job reliably and never sends records outside. A nurse still reviews every flagged case, because software shouldn't have the final word on who needs care first.

A growing software company with a busy support chatbot

This company uses an API for about a million conversations a month, and the bill now comes up in board meetings. Moving everything to self-hosting would be a big, risky project, so they route instead. Routine questions go to an open model hosted by a third-party provider, and the tricky 20% go to a frontier API. Costs drop, and customers with hard questions still get the stronger model.

Notice that the third answer is a mix. For most products past the early stage, LLM APIs vs open-source models stops being an either-or choice and turns into a question of which requests go where.

The team behind the choice

For an API-based product, you need solid application developers who understand prompt design, retrieval, testing and cost tracking. That's a manageable hiring lift. In API-based LLM development, many good web and backend developers pick up the specifics within a few months.

Self-hosting adds a second layer: people comfortable with GPU servers, model serving software, containers, monitoring and fine-tuning. Those people are scarce and expensive. When companies hire AI developers for this work, a common mistake is hiring a researcher who can train models but has never kept a production system alive at 2 a.m. For most products, you want the second skill more than the first.

A few questions worth putting to any candidate or agency:

•       How would you measure whether a new model is better for our product specifically?

•       What happens in your design when the model provider goes down or rate-limits us?

•       How would you estimate our monthly bill at ten times today's traffic?

•       Have you run a model in production, and what broke first?

Plenty of teams bring in outside help for the first build, then decide whether to hire AI developers in-house once they know which path they're on. That order makes sense, because the path decides the hire.

A quick way to decide

Work through these questions in order. The first strong "yes" usually settles it.

1.      Are you legally or contractually barred from sending this data to an outside company? If yes, self-host an open model, or run one inside your own cloud account.

2.      Are you still testing whether the product is useful at all? If yes, start with an API. Learning quickly matters more than saving money right now.

3.      Does the task need the strongest reasoning available, such as complex analysis, multi-step agents or difficult code? If yes, a frontier API will probably beat open models for now.

4.      Is the task narrow, repetitive and high-volume, like tagging, pulling fields out of documents or sorting messages? If yes, a small open model, possibly fine-tuned, is often cheaper and just as accurate.

5.      Is your monthly API bill higher than the cost of one or two infrastructure engineers? If yes, price out self-hosting or third-party open-model hosting for part of your traffic.

6.      Do you need responses in well under a second? If yes, look at a small self-hosted model placed close to your app.

If none of these gives a clear yes, start with an API and begin building your evaluation set. In three months you'll know far more.

What changes heading into 2027

Prices will keep dropping. Epoch AI's September 2026 analysis puts the fall in the price of a fixed performance level at roughly 13 times per year since 2023. That helps both paths. API bills shrink for the same work, and small open models reach quality levels that used to require much larger ones.

The gap at the very top may not close. Menlo Ventures' 2025 data put open models 9 to 12 months behind the best closed ones, and the frontier keeps moving too.

Specialized models are growing fast. Gartner's forecast of 210% growth in spending on domain-specific models suggests many companies want models built for their industry rather than general-purpose ones. That's familiar ground for open models, and it's pulling more generative AI development toward small, focused systems.

Agents raise the stakes on cost. When one user request triggers ten or twenty model calls, small per-token differences add up quickly. LLM application costs for agent products need to be modeled per completed task, not per request, on either path. One busy agent can burn as many tokens as dozens of chat users.

Picking your starting point

There's no universal winner in the LLM APIs vs open-source models question, and vendors on both sides have reasons to tell you otherwise. What does exist is a sensible order of operations. Start where you can learn fastest, which for most teams means an API. Measure real usage, real costs and real quality on your own examples. Then move specific workloads to open models when a clear reason appears: a privacy rule, a bill that has grown past a salary, a speed target, or a narrow task a small model handles well.

The teams that handle this well tend to share one habit. They build their product so the model underneath can change without a rewrite. In generative AI development, that flexibility is worth more than getting the first pick exactly right.