The model changed and nothing failed

A pinned model is not a frozen model. The name in your config stays the same, the requests keep returning 200, and your tests stay green because your tests mock the model. Meanwhile the answers change shape. JSON comes back wrapped in a Markdown fence and the parser throws in production. A classifier that returned one word starts returning a sentence. A prompt that used to decline a question starts answering it. Nothing in the stack is built to notice any of that, so the first report comes from a customer.

Pinning is weaker than it sounds

Providers publish dated snapshots, and pinning one is the right move. Most teams do not. They ship the friendly alias because it is what the docs show, and an alias points at whatever is current by design. Even a hard pin only covers one of the inputs. Your own prompt gets edited. A gateway reroutes to a different provider. A fallback model serves the request during an incident and nobody diffs what it sent back. The model is one moving part in a stack full of them, and none of them announce a change.

You cannot diff the responses

The obvious approach is to save the old answer and compare strings. It does not survive contact with a real model. Responses vary between identical calls, so byte equality fails constantly while telling you nothing about whether behaviour actually moved. Temperature is not the escape hatch either. The current Claude models reject sampling parameters outright, so there is no zero to turn the knob down to.

So modeldrift does not store answers. It reduces each one to a fingerprint: a set of named features, grouped into five aspects that are compared separately.

• format: is it fenced, does it open with a preamble, how many bullets and lines
• structure: does it parse as JSON, what are the top-level keys and their types
• policy: was it refused, was it hedged
• length: characters and words, the drift that costs money rather than breaking anything
• lexicon: the vocabulary and the opening words, which is the answer itself changing

Each probe declares which aspects it cares about. The probe guarding a JSON contract watches structure and format and ignores wording entirely, so a synonym swap never wakes it up. The probe watching verbosity ignores everything except length. Aspects that are compared together are aspects that mask each other.

Measure the noise before you measure the change

This is the decision the whole tool rests on. Recording a baseline runs every probe five times and keeps every fingerprint. The spread between those repeats is the model variance, measured per aspect instead of assumed.

$ modeldrift baseline --suite suites/core.json --samples 5  probe                    noise floor by aspect  json-contract            format 0.00  structure 0.00  single-word-obedience    format 0.00  length 0.00  lexicon 0.00  open-prose               length 0.03  lexicon 0.08

Open prose wanders on its own. A JSON contract does not. At check time the distance from the baseline is measured the same way and that floor is subtracted, so the prose probe has to move a long way before it counts while the JSON probe trips on the first fence. Skip this step and the tool reports noise, you stop reading it, and it may as well not be installed.

What it looks like when it fires

$ modeldrift check --baseline baselines/production.json  probe                     drift  limit  json-contract              0.17   0.10  DRIFT format/fenced false -> true                                                format/lines 4 -> 6  single-word-obedience      0.00   0.10  ok  dosage-boundary            0.00   0.20  ok  open-prose                 0.00   0.30  ok  suite drift 0.02   1 of 8 probes over their limit

One probe fired and seven did not. Note the suite average: 0.02. Wire the alarm to that number and you miss this entirely, which is why the limit that matters is per probe and the average is only a summary. The reason line is the real deliverable. Not a score, but a sentence: this response is fenced now and it was not before.

A monitor that alarms on everything gets muted within a week. Suppressing false alarms is not a refinement here, it is the product.

What it is not

This is not an eval harness and does not replace one. promptfoo, Braintrust and LangSmith score a model against an answer key you wrote, which is how you decide whether output is good. modeldrift has no answer key and no opinion about quality. It knows what your model did on the day you shipped and whether it still does that. Those are different jobs. An eval tells you the new behaviour is worse. This tells you there is new behaviour at all, which is the part nobody is watching.

It is also honest about its own demo. The tool ships with a deterministic local stub so the whole thing runs without a key and without spending anything. The stub is a test fixture. It is not a recording of any provider, and nothing in the repo claims that a named model has drifted.

Running it

pip install git+https://github.com/agentdynarq/modeldrift.gitmodeldrift baseline --suite suites/core.json \  --backend anthropic --model claude-sonnet-5 \  --samples 5 --out baselines/production.jsonmodeldrift check --baseline baselines/production.json \  --backend anthropic --model claude-sonnet-5

Exit 0 stable, 1 drift, 2 the run itself failed, so a CI step needs no parsing. The baseline is committed like any other fixture, and the suite is hashed into it: edit a prompt and the next check refuses to run rather than quietly comparing against moved goalposts. The repo ships a weekly workflow that opens an issue when the pinned model moves, and it skips itself when no key is configured, so a fork never fails and never spends money. Pure standard library, zero runtime dependencies, 94 tests, MIT.