Contents
The right AI model is the one that handles your real tasks at an acceptable cost and privacy level, not the one at the top of a leaderboard. Test two or three models on 30 to 100 examples from your own operations, check the contract and processing region, and build an architecture that lets you switch.
AI model leaderboards change every quarter. If I recommended a specific model today, this page would be wrong in six months. Instead, here are the criteria I use to choose, the testing method, and the architecture that lets you change models without rebuilding the application. Any product mentions below are dated: check them again when you start your project.
The eight criteria that actually matter
| Criterion | The question to ask | How to check |
|---|---|---|
| Quality on your tasks | Does the model handle your real cases? | A test set of 30 to 100 examples from your operations |
| Cost per task | What does one processed document or one answer cost? | Provider’s price list × volume measured during the test |
| Speed | Does the answer come back fast enough for the use case? | Response time measured under real conditions |
| Privacy | Is the data you send used to train the model? | Contract, API terms of use, data processing agreement |
| Processing region | Where is the data processed and stored? | Provider’s region options, list of sub-processors |
| Context size | Can the model read a long document in one pass? | Technical docs, a test on your largest documents |
| Stability | How long will this version be available? | The provider’s published deprecation policy |
| Ability to switch | Can you change providers without rewriting everything? | The application’s architecture (see below) |
The first criterion outweighs all the others. A model that shines on public benchmarks can still stumble on your industry vocabulary, your purchase order formats or your internal abbreviations.
Test on your own data
This is the step people skip most often, and the one that prevents the most mistakes. The method I use:
- Build a test set: 30 to 100 real, representative examples, including the hard ones (poorly scanned documents, ambiguous requests). Data is pseudonymized if it contains personal information.
- Define the correct answer for each example, with the person who does the task today.
- Run two or three models on the same set, with the same instructions.
- Score them: share of correct answers, serious errors, response time, total cost of the test.
- Keep the test set. It lets you revalidate each new model version in a few hours.
A few points of extra quality can justify a higher cost when errors are expensive. Conversely, a model that costs half as much and is only slightly less accurate is often the right choice for high-volume sorting.
Privacy, contract and region: check them in writing
Before choosing a model, I read three documents: the API terms of use, the data processing agreement (DPA) and the list of sub-processors. What to check:
- reuse for training: as an example, OpenAI states that data sent through its API and business plans is not used to train its models unless the customer explicitly opts in (page checked October 27, 2026);
- the region: since February 2025, the same provider has offered in-region processing of API requests in Europe for eligible customers, set when a project is created (same source, same date);
- how long the provider retains requests and responses;
- the law the provider is subject to, which matters as much as where the servers are.
These terms change. Check them again at every renewal. Sorting the data you can send is covered in what data to share with AI.
Proprietary model via API or self-hosted open-weight model
| Proprietary model via API | Open-weight model hosted by you or a host | |
|---|---|---|
| Getting started | Fast | Slower (server, deployment) |
| Cost | Pay per use, predictable if volume is | Fixed server cost, pays off at high volume |
| Data | Sent to the provider, under its contract | Stays in the infrastructure you chose |
| Maintenance | Handled by the provider | On you or your development partner |
| Quality | Often highest on complex tasks | Good enough for many focused tasks |
For a first version, I usually start with a proprietary model through an API, with reuse turned off and a suitable region. My usual stack relies on OpenAI, but the choice is made project by project, based on the tests. A self-hosted open-weight model makes sense when the data is highly sensitive or when volume makes per-request pricing too expensive.
An architecture that lets you switch models
Today’s model will not be the one you use in two years. The application should therefore:
- isolate AI calls in a dedicated layer, so the rest of the code does not depend on one provider;
- store the instructions (prompts) as versioned settings, not scattered through the code;
- log inputs, outputs and costs, so you can compare two models on real data;
- plan a fallback: if a provider is down, the task goes to manual handling or to a second model.
When you should not choose a model at all
If your need is covered by off-the-shelf software that already includes AI (your CRM, your help desk tool, your office suite), start there. Choosing a model only makes sense if you are building a custom feature: processing your own documents, automating a process specific to your company, or an assistant connected to your data. And if a simple rule does the job, you do not need AI.
Where to start
Choosing the model is part of the scoping for every AI integration project: I build the test set with your team, compare models and hand you the results, with costs, before development starts. The overall budget is broken down in how much AI integration costs, and the estimator gives you a range for a business tool with built-in AI.
Frequently asked questions
Should we pick the most powerful model on the market?
Rarely. To sort emails, extract fields from an invoice or summarize a request, a smaller model often gives equivalent results, faster and for far less money. Keep the most powerful model for tasks where your tests show a real quality gap.
Is an open-weight model safer for our data?
It can be if you host it yourself or with a host you trust, since the data never leaves your infrastructure. In return, you take on hosting, updates and server security. A proprietary model through an API, with a solid contract and a European region, is often simpler for a first version.
How long does a model stay relevant?
Providers release new versions several times a year and retire old ones. Expect to change versions at least once a year, which is why keeping your test set matters: it lets you revalidate quickly.
Can we use several models in the same application?
Yes, and it often makes sense: an inexpensive model for sorting and extraction, a more capable one for drafting or complex analysis. The application picks the model based on the task.