How we choose a model

The frame before the model

Five constraints decide; the ranking comes after.

Choosing a model in 2026 doesn't start with a leaderboard: it starts with your constraints. A model that tops an English benchmark can fail your operation's Spanish, cost twice as much per token, or be unable to run where your compliance requires.

The model evaluation framework at Slash AI Lab has five axes: language, code, agents, cost and privacy. We measure them with the client's prompts, not generic tasks, against the best generic model available as a baseline.

How the evaluation runs

Reproducible by design: anyone can re-run it.

We gather 30 fixed prompts per client or industry (real business tasks, not toys) and run them against each candidate in the cloud and, where it applies, in local inference.

Each run measures four things: quality (against agreed criteria), observed failures (what breaks when things go wrong), cost per inference and p95 latency. The output is a per-axis comparison, not a single score.

• Prompts: 30 fixed per client, versioned.
• Engines: 5 finalists per run, picked from the candidates.
• Languages: business Spanish, French and English.
• Output: a per-axis comparison with a reproducible recommendation.

Failures outweigh successes

Production trust breaks on errors, not on hits.

Success benchmarks reward the easy cases. A deployment is decided by what happens when the model is wrong: does it answer with false confidence, invent sources, drop the client's language under pressure?

So the evaluation includes hostile cases: ambiguous prompts, dirty data, broken tools and out-of-scope questions. If the model fails dangerously, it goes back to design, whatever its ranking says.

When a new model ships

A launch is a hypothesis, not a migration.

In 2026 the big labs ship new versions every few weeks. An announcement like Claude Opus 5.5 (September 22, 2026) changes the candidate list, not the decision: the new model goes through the same 30 prompts before it touches production.

What we look at first is cost per solved task, not price per token. A model that costs more per token can come out cheaper if it finishes the job in fewer attempts, and the other way round.

What this study is not

Declared limitations, no smoke.

It isn't a paid or sponsored ranking; it doesn't cover models we can't put under real load; and it doesn't replace an evaluation on your own prompts. It's the public base of the method Slash runs on every project.