How to tell whether your AI provider swapped the model

You never see the model you pay for

When you call a model through an API, you send text and get text back. You cannot see the weights, the hardware or the settings. A provider can quietly serve something cheaper and keep billing the same:

• A compressed copy. "Quantisation" stores the model's numbers with fewer bits, for example 4 bits instead of 8. It needs about half the memory and is faster, and it is slightly worse.
• A smaller sibling. The 1-billion-parameter version instead of the 4-billion one.
• A different model altogether.
• A mix. The real model for some requests, a cheaper one for the rest, for example on busy days.

This is not hypothetical. A 2026 audit of paid "shadow" API resellers (Zhang et al., arXiv:2603.01919) found that 11 of 24 endpoints failed model-identity checks; one endpoint sold as Gemini answered like Qwen2.5-7B.

Checking the weights would settle it, but you never get them. So the only evidence you have is the answers.

Record answers on good days, compare later

Nothing runs on the provider's side and nobody needs to cooperate. tracta needs only your API key, which stays on your machine, and a folder for the saved answers.

Questions at the edge of what the model knows

tracta asks two kinds of question: element facts ("Which element has atomic number 13?", and the reverse) and three-digit by two-digit multiplications ("What is 687 × 73?"). Each answer is one number or one word, at most 16 tokens.

These sit near the edge of what a small model knows. On such questions a model is often wrong, but it is consistently wrong. Change the model, or compress it, and the wrong answers change too. Here is Gemma 3 1B from tracta's experiments:

Across all 436 questions, the 4-bit copy of Gemma 1B gave a different answer on 242. Asked again, the unchanged 8-bit model gave a different answer on none of them.

Each user's multiplications and question order come from a private random seed, so a provider cannot keep one public list of tracta questions to recognise.

Providers wobble, so tracta measures the wobble

Even an honest provider does not always give identical answers. Batching many users' requests together, different hardware, or a software update can nudge a few answers. A detector that ignored this would raise false alarms. So tracta does not assume zero wobble; it measures it from your own two baselines.

Picture each question as one square, and colour it when its answer differs from the first baseline:

A fair comparison between today and the second baseline

tracta compares today with the second baseline, question by question. If the provider has not changed, today is just another ordinary day, no different from the second baseline. So on questions where exactly one of the two moved, either is equally likely to be the one.

That turns the check into counting. Suppose 10 questions moved only today and none moved only in the second baseline. If nothing had changed, each of those 10 would have been a 50-50 case, and all 10 landing on "today" has a probability of 1 in 1,024, or 0.001. That probability is the p-value. The smaller it is, the harder it is to explain the change as ordinary wobble.

A false-alarm budget that does not run out

If you checked every day with a fixed 5% false-alarm rate, then over 30 days you would expect at least one false alarm 79% of the time. tracta prevents that by giving every check a smaller share of one total budget, 5% by default:

A check says "changed" only when its p-value is below its share. The shares add up to less than 5% forever, so the chance of ever raising a false alarm on an unchanged provider stays under 5%, however often you check. The cost is that later checks need stronger evidence.

Tested on ten published models

tracta was tested on ten freely published models from 0.35 to 4 billion parameters, served from a local computer so the "swaps" were known. For each, the chart shows how many of the 436 answers moved:

Every 4-bit copy and every swap moved more answers than serving the same model again (0 to 0.5%). Serving the same file through different software, measured for four models, moved 10 to 15%. One case is open: Qwen3.5-4B's 4-bit copy moved only 4.6%, and its serving-software wobble could not be measured.

The test on Anthropic's Claude API

The tool was then pointed at a real hosted API: two baselines on Claude Haiku 5.5, a third Haiku run, and a run that quietly called Claude Sonnet 5.5 instead.

The swap was caught, but by a thin margin. Strong models answer nearly all these questions correctly, so two of them rarely disagree. Nine of the ten differences came from Sonnet writing out its working ("687 × 73 = 687×70 + …") and being cut off at 16 tokens. For frontier-class models, tracta needs harder questions; that is the next piece of work.

What a verdict does and does not mean

• "Changed" is not "cheated". An announced update, a new default system prompt or new settings also change answers. The report says the answers moved more than the provider's own wobble, nothing about intent.
• A careful provider could evade it. A provider that recognises tracta's requests can serve the real model to them. Private seeds make that harder, not impossible.
• It only sees what these questions see. A swap that leaves short facts and sums alone, such as between two strong models that both know them, is hard to detect.
• It compares with your own past. A change before your first baseline is invisible.

Three commands

tracta baseline --name myprovider --endpoint https://api.example.com/v1 --model some-model --key-env EXAMPLE_API_KEY
tracta baseline --name myprovider     # again, ideally on another day
tracta check --name myprovider        # writes a report; exit 0 no change, 2 changed, 3 no verdict

tracta is one program with no account and no server, for macOS, Linux and Windows, under the MIT licence. It works with any provider that speaks the common OpenAI chat format, and with the Claude API. Each check writes a report with the verdict, the numbers behind it and the questions whose answers moved (an example report).

Download the binary for your platform from the latest GitHub release and check it against SHA256SUMS, or build it with go install github.com/NakliTechie/tracta/cmd/tracta@latest. The source, the method and every experiment, with the pinned model files, are in the repository.

baseline

A saved set of answers from a day you trust the provider.

quantisation

Storing a model's numbers with fewer bits. Smaller and faster, slightly less accurate.

Q8_0, Q4_K_M

Common 8-bit and 4-bit formats for published model files.

greedy answer

The model's single most likely answer, with no randomness added.

wobble (noise)

Answers that change between two runs of the same unchanged model.

p-value

How likely a result at least this extreme would be if nothing had changed.

false alarm

A "changed" verdict when the provider did not change anything.