I keep a short list of operating rules for my own stack. One of them was about vendor concentration, and it read roughly: if any single paid AI vendor accounts for more than half your dependency surface, that's a risk worth acting on.
It sounds sensible. It is the kind of thing you write down once, feel good about, and never test. So this week I finally measured the actual surface instead of estimating it.
What the measurement said
Three things in my stack call a model at runtime. Two of them call Anthropic. One is a local content pipeline with a model name hardcoded in a config line, the other is my whole knowledge vault, which is driven by Claude Code across sixty-odd skills, a set of hooks, and six scheduled jobs. On top of that, one hundred percent of my development layer is the same vendor. I don't write code in an editor anymore; I write it in an agent.
So the number came out above ninety percent. Not "creeping toward the threshold." Not "worth watching." Above ninety, and structurally so.
Then I looked back at the rule and realized what I was holding:
- The rule fires on day one.
- It fires on every project.
- It fires tomorrow, and next month, regardless of what I do.
A test that returns the same answer in every possible state of the world carries no information. It isn't a threshold; it's a constant wearing a threshold's clothes. I had been carrying it around for months as though it were a live check.
The rule was also pointed at the wrong event
The second problem was worse than the first. A ratio rule implicitly models concentration as something that accumulates, you add a dependency, then another, and one day you cross a line and the alarm goes off. That story assumes there was a period during which the number was low and rising.
There was no such period. The concentration didn't arrive by accumulation; it arrived with the harness, in one step, the day I moved my development into an agent. By the time there was anything to measure, the measurement was already maxed out. A rule watching for a gradual crossing can never fire usefully on a discontinuity that already happened.
What replaced it
I threw out the ratio and wrote down a list instead. Seven units, the six scheduled jobs plus the one pipeline that calls a model in production. Each unit gets exactly one binary question:
Does this still run tomorrow if my access to this vendor ends today?
That's it. No percentages. The answer per unit is yes or no, and, this is the part that makes it real, it's established by drill, not by estimate. You don't get to reason about whether the job would survive. You cut the dependency and watch what happens.
The properties I wanted, which the ratio never had:
| Ratio rule | Survival list | |
|---|---|---|
| Answer varies with state | No, always "over" | Yes, per unit, yes/no |
| Moves when you do work | No | Yes, fixing a unit flips it |
| Verified how | Arithmetic on a guess | A drill you actually run |
| Tells you what to do next | No | Yes, the failing unit |
The ratio cannot improve, because it's dominated by the development layer, which I am not going to un-adopt. The list improves one unit at a time, and each flip is a thing I did.
The remedy is not a second vendor
The instinct, once you see a concentration number that high, is to go shopping for a fallback. I want to be precise about why that's wrong here, because it's where my original rule quietly smuggled in a bad assumption.
For the one production pipeline, "swap the vendor" is a config change. Real, cheap, a genuine second source.
For the agent harness, "swap the vendor" is a different agent harness and a rewrite of sixty skills. Those two remediations are not the same action, and they are not comparable in cost, but the ratio rule treated them as one undifferentiated block of "dependency," which is exactly how it produced a recommendation that would have taken months of rewriting to act on.
What I'm doing instead, for the units that fail the drill, is making them not need a frontier model at all:
- Move the deterministic work out of the model. Parsing, extraction, arithmetic, reconciliation, these are not judgment tasks, and a model doing them is a liability dressed as convenience.
- Keep the model for the part that is actually judgment.
- Use "a small local model could drive this" as the acceptance test.
I'd already done this once, on a document-processing skill, for unrelated reasons: I pulled the parsing into a local deterministic step that refuses to continue unless the extracted rows reconcile against the document's own totals, and left the model with only the interpretive half. That skill is now vendor-agnostic as a side effect. It turns out the fix for "this component depends on one expensive vendor" and the fix for "this component silently produces confident garbage" are the same fix.
The general form
I'm keeping the specific list, but the transferable part is smaller and applies to any metric you're tempted to write down:
Before adopting a threshold, ask what states of the world it distinguishes. If you can't name a plausible state where it returns a different answer than it does today, you haven't written a check. You've written an opinion in the grammar of a check. Opinions are fine. But they don't belong on a dashboard, because a dashboard's whole job is to change when reality does.
And the follow-up question, which is the one that actually saved me here: is this measuring something that accumulates, or something that already happened in one step? Ratios are for the first kind. For the second kind you don't want a gauge at all. You want a list of things that could break, and a habit of pulling the plug on them one at a time to find out.