vastops: a self-hardening pipeline for training small models on rented GPUs

5. The gate is a contract the framework can read. It used to be a convention: the project echoed == GATE == into its log and a deploy script grepped for it, which meant the framework could not distinguish a finished run from a good one, and an eval that died before printing its number read as silence rather than as failure. Thresholds are now declared in the project's vastops.toml, the eval writes a metrics JSON, and vastops gate judges it and exits non-zero. Two behaviours are deliberate and tested: a metric the gate names but the eval never produced is a failure, not a skipped check, and a project with no gate at all exits with its own distinct code, because nothing-was-checked is not a pass. Both are the same rule applied to a gate that guards against silence as well as against bad numbers.

This is not theater, and it has now refused two models. ops-doctor v3 improved on the metric its enrichment targeted while silently losing 45 points on ssh-unreachable hard negatives; it evaluated as a candidate, failed head-to-head against v2, and never reached the live tag. The postmortem (checkpoint selection by eval-loss, which does not track label accuracy) became a recipe rule that v4 shipped with. sign-fielder v1 beat its prompted baseline by 2.5x and the gate refused it anyway, 2.6 points short on signature recall. Triage traced both failing metrics to a single cause: the output format nested the field grammar's braces inside JSON strings, and the six garbled replies held exactly the signatures the bar missed. v2 switched to a collision-free line format and cleared the identical bar the same evening. A gate that only ever passes things is decoration; this one earns its keep by saying no. A skeptic will notice that the early refusals were both fixed within a day and read the gate as a delay, not a barrier -- but the gate's product is the forced diagnosis. Both root causes (checkpoint selection by eval-loss, a brace collision in the output format) would have shipped silently without it, and one of the later refusals has not been fixed at all: the 3B edge adapter stays refused, because the honest answer so far is that the capacity isn't there.

The ceremony also grew this week. Ten canary inputs with exact expected outputs run before the statistical eval, because a rare catastrophic regression does not move an n=86 aggregate but trips an exact expectation immediately. That stage has since earned its keep for real: an edge-serving candidate passed the statistical bar outright (F1 0.924, signature recall 0.979) while failing the same multi-party canary four times out of four. The aggregate had averaged the miss away; the canary refused the model. The canaries are also the only stage computed entirely on the operator's own box against exact expectations, which makes them the de facto anti-tamper check on artifacts pulled from untrusted hosts -- a job ten inputs is thin for, and one that costs nearly nothing to grow for grammar-constrained tasks. A hand-labeled out-of-distribution slice reports alongside the bars (advisory until calibrated, then gating). And promotions over an existing incumbent now have a paired-statistics stage available (McNemar on discordant items, bootstrap intervals on per-document deltas) that returns three verdicts, better, worse, or unresolved, because at these sample sizes a naive comparison routinely reports noise as a winner [13]. Unresolved means the absolute bars decide; the paired test only has authority when it has resolution.

Results

The substrate, measured

"Dependable" is a measurable claim, so the substrate numbers come first. From the ledger, 2026-07-11 through 2026-09-15:

Generated by scripts/ledger-stats.py, apart from the two gate rows, which have no derivable source: the ledger records instance lifecycle, not gate verdicts, so those two are hand-counted through 2026-09-09 and are exactly the kind of number this table exists to distrust. Two instances with no stop or destroy event are excluded from spend rather than guessed at. Two figures moved when the table stopped being typed: the earlier draft counted create events where this counts distinct instances (100 events, 95 instances, in the July window), and it included open-ended rentals in spend where this excludes them ($19.93 became $17.98 for that same window). The correction is small and it is the point: a number a command reproduces cannot drift.

Roughly two in three creates died before any training happened, and on the two documented platform-wide bad days the failure rate hit three in four. That is not the pipeline failing; that is the marketplace being what it is, made cheap: the number to judge is the cost of those deaths, $4.96 across the whole history at under thirteen minutes per bad host, because detection is fail-fast rather than poll-and-hope. Those bad days are why a second provider went from queued to shipped, and RunPod rentals now sit in the same ledger under the same rules.

The models, measured

All rows are held-out and gated on the worst of two seeds, except ops-doctor v4, whose promotion was a single-seed head-to-head against its shipping incumbent (n=300). "Baseline" unless stated is qwen3.5:35b-a3b given the same task with a carefully written prompt: a 35B-total-parameter mixture-of-experts that activates roughly 3B parameters per token, and the strongest generalist the 24GB serving box runs -- which is the honest constraint, because the practical alternative to each fine-tune is a prompt against this box, not a paid API. The baseline got the same output-format iteration the fine-tunes did (sign-fielder's prompted baseline was measured in both the JSON and line formats, and the better number is the one reported). To be precise about sizes: against the 0.6B fine-tune the baseline is 58x the total parameters; against the 8B it is 4x the total parameters but fewer active parameters per token. The honest reading is not "David beat Goliath" at every row. It is that a model trained on the task beats a much broader model prompted into it, whatever the parameter ratio, and the frontier-scale comparisons (GPT-4-class) come from the published record cited in the thesis section, not from this table.

¹ ops-doctor's reference is its shipping incumbent, not the prompted 35B -- by v4, the incumbent was the harder bar to clear.

² A perfect score on a simulator-generated eval means the eval has nothing left to teach; the real-corpus retrain replaces this number.

³ Measured with an embedding-similarity sweep. Validation is deployment-shaped for a template-driven product; the OOD slice is the number that keeps it honest, and the regex incumbent still beats the fine-tune on that slice.

Three patterns repeat. The fine-tune beats the prompted baseline on the three axes the bars declare: task accuracy, the task's costly error, and latency. The training cost rounds to pocket change. And the asymmetric error that matters for the task (cutting a user off mid-sentence, calling a dead run healthy, missing a signature line) is a named, gated metric, not an afterthought.

The comparator question, answered

A fair objection to the table above: beating a prompted 35B does not establish that a fine-tune beats a good deterministic parser or a right-sized encoder. So the newest task ran as a four-way fair fight on identical rows: PII span detection for the e-sign product, on a dataset whose training and held-out format pools are deliberately disjoint.

Nobody wins all three columns, and that is the finding. Rules are perfect on the surface forms their author has seen and structurally blind past them. The fifteen-cent encoder dominates the deployed distribution -- and scores zero on the highest-severity classes in unseen formats, a perfect surface-form memorizer exposed by the format-disjoint split built to catch exactly that. Only the prompted generalist degrades gracefully under drift, and it is mediocre everywhere at twelve times the encoder's latency, with 5-9% invalid outputs. The gate refused the encoder despite its near-perfect validation numbers, on canaries; that refusal, of our own best candidate, is the strongest evidence in this paper that the ceremony measures models rather than flattering them. The production answer is a cascade, not a winner: rules for the known, a specialist for the deployed distribution once format augmentation closes its drift gap, and the generalist as the fallback for the genuinely new.

The honest numbers ship too, and they drive retrains. ambie-eou v1 reported its external out-of-distribution slice at 80.9% (an independent team's test set, versus 99.3% in-distribution) and diagnosed why: inputs with no silence signal inherited a prior from an injected placeholder value. v2 trained with the silence field randomly absent, which is standard feature dropout, and re-measured both versions on the identical, honestly-reshaped slice: v1 scored 82.4% with 29.8% false interruptions, v2 scored 90.0% with 16.4%. The out-of-distribution number improved because it was reported instead of hidden. log-doctor's README carries a same-simulator caveat. sign-fielder's record carries two of its own: an advisory out-of-distribution slice (67.4% structural pass on independent upload-style documents, against a regex incumbent that scores 100% on what is effectively its own tuning set), and a measured 51% semantic-sibling rate between its train and validation sets -- quantified with an embedding-similarity sweep and interpreted in the open: validation measures in-distribution performance, which is deployment-shaped for a template-driven product, and the out-of-distribution slice is the number that keeps it honest. Eval theater is a failure mode this pipeline treats like any other: something to detect and close.

The route question, and the experiment that settles it

The comparator question above asks which model wins. A different question sits underneath it: which route to a specialist wins. A note circulated in September 2026 argued the fashionable one, structured pruning of a 7-8B model followed by knowledge distillation, and it is worth being precise about why this pipeline does not take that route.

Five routes exist. Fine-tune the smallest architecture that already matches the output shape. Fine-tune an already-small current general model. Put a LoRA adapter on a base that is already resident. Quantise. Prune a bigger model and distil it back. Six independent expert reviews and five literature passes, run against this question in September 2026, put prune-and-distil last on all six, and the published evidence is what makes that more than opinion. Token-matched, pruning loses: at equal total token budget, from-scratch training beat coarse structured pruning of Llama-3.1-8B by 0.40 perplexity on WikiText-2, and the paper's own reading is that architecture transfers while weights largely do not [14]. On the canonical demo task, aggressively pruning GPT-OSS-20B's experts to about 5B and recovering lands competitive with rather than ahead of NLLB-200-3.3B and TranslateGemma-4B [15] -- while Helsinki-NLP/opus-mt-es-en does Spanish to English at 75M parameters under Apache-2.0. And the money is not close: a genuine ten-billion-token recovery pass for an 8B-to-4B prune is a several-hundred-dollar rental, where specialising a 1-3B model on a hundred million tokens is a single-digit-dollar one. That is two orders of magnitude spent to arrive somewhere worse.

Three specific claims in that note are wrong in ways worth recording, because they are the kind that survive by sounding plausible. Attention probabilities are softmax-normalised and sum to one for every query, so a head's average weight is about one over the context length even when the head is vital; "heads with near-zero attention weights are dead" is not a measurement. Under grouped-query attention, dropping a query head does not remove its shared key-value projection or shrink the KV cache, and NVIDIA's own Minitron ablation kept attention intact and pruned only MLP and embedding channels because that scored better. And the memory wall in logit distillation is not two sets of weights, it is the [batch, sequence, vocabulary] logit tensor and everything the KL loss materialises from it: measured, an 8B teacher into a 3.2B student on one H200 at 8k context peaks near 103GB online with a dense KL and near 58GB with a fused chunked one [16]. Modern vocabularies make this worse rather than better, which points at the one idea the note missed entirely -- embedding plus untied lm_head on an 8B Llama is about 1.05B parameters, roughly 13% of the model, spent on tokens a single-task specialist will never emit. Trimming the vocabulary to the task's observed token set is the cheapest structured prune available, and almost nobody does it.

None of that is a reason to take the conclusion on trust, so examples/tool-caller/ is the experiment that checks it on our own bench. One task, one corpus, one evaluator, several methods: prompt the bigger model, fine-tune a small one, adapter on a shared base, quantise, and prune-then-recover, all scored by the same harness on the same splits so a difference between arms is a difference between methods. The task is schema-constrained tool calling, chosen because the output contract is rigid and the metric is exact match on a parsed call AST rather than a judge model. Everything in it is Apache-2.0 and ungated, so the matrix reproduces without a licence click.

The split that will decide it is valood, which is tool-disjoint: every record offers at least one function whose schema was never shown during training, mixed in with familiar distractors, and 483 of them require calling an unseen tool. It is the direct descendant of the format-disjoint split that caught the sign-pii encoder as a surface-form memoriser, and the gap between val and valood is the number this example exists to produce. A separate canary split holds only no-call cases, where emitting any call at all is the expensive failure.

What the bench actually said

It ran on 2026-09-09. Every arm trained on the same corpus at the same budget (1,200 optimizer steps at an effective batch of 32, so roughly 38,400 examples each), scored by the same evaluator on the same splits. The whole matrix, ten scored arms, cost $2.63 of rented GPU: three creates on RTX 5090s, of which two never delivered a working GPU and one of those had to be destroyed by hand after an external timeout orphaned it. That is the single-digit-dollar figure asserted above, measured rather than estimated, with the marketplace's failure rate already inside it.

Seven of the ten scored arms are shown; the remaining three (base-1.7b, base-fg270m, q4-trim-sft-1.7b) are in examples/tool-caller/results/metrics/ and none of them changes the ranking.

The thesis held. A fine-tuned 0.6B beat a prompted 4B, a model 6.7x its size, on every split, and the paired McNemar on identical items says BETTER on val (+36/-21) and BETTER out of distribution (+44/-24), not noise. That is the controlled version of this paper's central bet: same family, same corpus, same evaluator, same output contract, with specialisation the only variable.

Five things the bench said that we did not expect, and three of them corrected a claim already made in this paper.

Zero-shot ranking does not predict fine-tuned ranking. base-1.7b was the worst zero-shot Qwen (0.653, below the 0.6B's 0.708) and sft-1.7b is the best trained arm. Selecting a base model by its off-the-shelf score would have chosen wrong.

Vocabulary trimming is not the cheap lever it is described as. The idea is sound on paper: 22,992 of 151,643 tokens are ever used by this corpus, and the embedding is a quarter of a 0.6B. Measured, trimming without a recovery pass took the 1.7B from 0.874 to 0.176, with output validity collapsing to 0.065 and latency rising tenfold because the damaged model never emits a stop token. The cause is the failure mode the design predicted and the write-up still underweighted: the kept set comes from the training corpus, so eval prompts contain tokens that become UNK on input, not merely unemittable on output. Trimming plus a recovery pass may well work; trimming alone is destructive.

Quantisation trades speed for memory, not speed for size. q4-sft-1.7b uses 2.2x less VRAM and is 31% slower at batch 1 (1638ms against 1254ms), because dequantisation is work paid per token with no throughput to amortise it against.

Prefill does not dominate; decode does, by thirty times. This bench was built expecting a few-thousand-token JSON Schema prompt to dominate a few-dozen-token answer. Prefill measures 27-54ms against 1131-1680ms end to end. Shorter prompts will not help this task; fewer output tokens or faster decoding will.

Fine-tuning bought format compliance for free. sft-1.7b reached 1.0000 output validity with no constrained decoding, which is the same gap the constraint-tax literature measures from the other side.

And the pruning arm never ran, because the planner refused it. Asked for 350M from a 1.7B parent, it reported that no honest plan reaches that target without cutting the MLP below a quarter of its width, and stopped. The guard was written that morning against a different model and fired correctly on a request made in good faith hours later.

The prompt is part of the model's cost, and it is not redundant

A second experiment, 2026-09-10, on the same harness. It exists because the

first one measured the wrong resource.

VRAM in the table above is what a model costs to hold. What bounds a serving

fleet is what a request costs to run, and that is KV cache. Qwen3-0.6B and

Qwen3-1.7B both spend 112 KB per token (28 layers, 8 KV heads, head_dim 128),

so on a 32GB card the 0.6B's 2.2GB weight advantage is worth about 8% more

concurrent requests rather than the 2.9x its parameter count advertises.

Against the 4B the honest multiple is roughly 1.7x, not 5.2x. Anyone sizing a

fleet from the weights column would overestimate it badly.

KV cache scales with prompt length, and for this task the prompt is mostly

JSON Schema describing tools the model has seen tens of thousands of times in

training. So: is the schema in the prompt buying anything the weights do not

already hold? Five arms, $1.25, same seed and budget as the control.

It is buying something. Every arm is WORSE on a paired McNemar over identical

items, none of the gaps is near the bench's ~5 point resolution, and the

signature arm -- the best of them -- lands below the prompted-4B bar the

gate is set at. It would not ship.

Three things fall out that the token counts alone do not predict.

The catalogue is not in the weights. Truncating the prompt on an

already-trained model costs 8.8 points at signature and 33.3 at names, and

retraining at the short prompt recovers only part of that. The model reads its

schema at inference; fine-tuning on a fixed tool set does not memorise it.

Damage is wildly disproportionate to what you save. Dropping the prose

descriptions costs 209 tokens for 7.1 points. Dropping the argument names and

types costs 34 further tokens for another 11.9. Per token removed, the second

cut is about 10x worse. The prose is cheap; the signature is load-bearing. A

plan built on "the prompt is mostly boilerplate, cut it" would have cut the

wrong half.

And it buys no latency at all. 521 / 517 / 536 ms p50 across a 2.9x range of

prompt length, prefill flat at 14-15ms. That is the overhead floor above,

confirmed from the opposite direction: shrinking the model did not move the

clock, and neither does shrinking the prompt.

So the real trade is 3.4x concurrency for 7.1 points of accuracy, and at that

exchange rate renting 3.4x the GPUs is cheaper than losing seven points. The

lever that does survive is orthogonal and unglamorous: an fp8 KV cache halves

KV per token for one flag, no accuracy cost, and no experiment needed to know

it works. The failure mode this experiment was really testing for is the one

where an elaborate idea is preferred to a boring flag, and the boring flag won

again.

What this bench does NOT establish

The latency column measures the harness at least as much as the model. Every

p50 above comes from HuggingFace generate() at batch 1, and decode is 97% of

that wall clock (42 output tokens against 31ms of prefill). Check it against the

hardware: sft-0.6b is 1.2GB of bf16 weights and an RTX 5090 moves about

1.8 TB/s, so reading every weight once takes 0.67ms -- against a measured

30ms per token, which is 45x off the roofline. The 1.72B model decodes at

29.1ms/token and the 0.60B at 30.0ms/token, near enough identical despite a

2.9x difference in parameters. That is the signature of a fixed per-token

overhead -- Python loop, sampling, cache management -- setting the clock while

the GPU idles. So these numbers rank the arms fairly against each other,

because every arm paid the same overhead, but they must not be read as

"a smaller model is faster": at this size class, on this serving path, it

mostly is not. A real latency claim needs a serving runtime that removes the

overhead floor (vLLM, llama.cpp, TensorRT-LLM), and we have not run one.

And the overhead floor is not even a constant. sft-1.7b was benched again

on 2026-09-10, same weights, same dtype, same GPU model, output length within

10%: it came back at 13.5ms per token against 29.1 the day before, a 2.2x

swing attributable to a different physical host and transformers moving 5.16

to 5.17. So the latency figures are comparable BETWEEN arms of one run, which

is what the table uses them for, and are not portable across runs or

quotable as absolutes. Note the contrast with the accuracy numbers, which

reproduced on that same re-run to within the bench's own resolution (val

0.8875 against 0.8736). The eval is stable; the clock is not.

And VRAM is not concurrency, which the prompt-length experiment above

measures rather than asserts: the weights column is what a model costs to

hold, not what a request costs to serve.

The baseline is zero-shot, not few-shot. A few-shot prompted 4B might close some of the two-point gap, and this experiment does not test that. It is the same gap the 2026 literature has, and naming it is better than letting the headline travel further than it earned.

It is also one task, on a public corpus old enough to be in every pretraining mix here. The small val-to-OOD gaps argue against memorisation dominating, but do not eliminate it. And the arms differ in learning rate as well as in method, so a result inside a couple of points carries a hyperparameter choice with it, which is what the paired test is there to stop anyone over-reading.

This is the honest shape of a claim about method: not "we chose the best route", but "here is the bench on which the routes are ranked, and here is what it says". docs/COMPRESSION-AND-SPECIALISATION.md carries the standing decision, including the narrow conditions under which prune-and-distil is right, which are Minitron's and not ours: you already own a large parent, you want a family of sizes, and the cost amortises over volume.

The gate follows the model to the serving surface

When a model leaves the training box, the rules go with it. The production field-detection model now serves over two paths. The primary is an authenticated tunnel from the product's edge functions to the home inference box: the exact gated artifact, same weights, same decoding configuration, behind mutual service-token auth. The fallback tier is a LoRA adapter hosted natively on the edge provider's own base model, retrained from the same dataset because an adapter is dimension-specific to its base. The pipeline refuses to treat these as the same model: the tunnel serves what the gate approved, while the edge sibling must clear the identical bars on its own edge-served outputs before the router may fall back to it.

That rule has done real work. The edge sibling has been refused three times, each for a different reason. First the canary caught the multi-party miss a passing aggregate had hidden (the incident in the gate section above). Then the provider's documented LoRA rank limit of 32 turned out to be a real limit of 16, discovered when a rank-32 adapter uploaded cleanly and then failed every inference call; a local validator now rejects over-cap adapters at upload so no future GPU run re-buys that lesson. Third, with the strongest recipe the platform legally serves (rank 16, all linear modules) and doubled multi-party training data, the 3B base still missed the same signature block every time: a genuine capacity wall, not a data problem. Trust-nothing turns out to apply to serving vendors as much as to marketplace hosts, and the same loop metabolizes both: undeletable fine-tunes, required config fields the trainer omits, and a roughly 40% transient failure rate on freshly-uploaded adapters are all encoded as code, not folklore.

The capacity lever worked. The same recipe retargeted at the provider's strongest tunable base, a 7B at 2.3x the parameters and about seventy cents of GPU, went ten-for-ten on the canaries including both historical failers, cleared the statistical bars on the worst seed, and in the paired head-to-head against the exact-model tunnel incumbent came out measurably better on document F1 (bootstrap CI95 [+0.018, +0.056], entirely positive), with the signature discordants unresolved at n=2, which is what honest resolution reporting looks like at that count. The edge tier has its fallback; go-live is one flag-gated router line and a secret, on the operator's schedule. Three model sizes, five dead attempts, and one vendor documentation error separated the idea from the artifact, and the gate is what kept every intermediate from shipping.

The bet: fleets of specialists, not one know-it-all

The results table is not a run of luck. It is the local evidence for a thesis this operation has held since the AMBIE autonomous-ASR design: production AI work decomposes into narrow, stable, high-volume tasks, and for those tasks a small model trained on exactly that task beats a giant general model on accuracy, latency, and cost at the same time. The AMBIE spec drew it before this pipeline existed: a router in front of a fleet of specialized recognizers, one tuned for industrial noise, one for medical vocabulary, one for legal, with the general model as the fallback rather than the workhorse. vastops is that idea built out as infrastructure, one specialist at a time.

The external evidence has caught up, and it needed refreshing: the citations this paper leaned on were measured against GPT-4, a baseline two generations stale. The current one is better for us. "Sub-Billion, Super-Frontier" (June 2026) fine-tuned a Qwen2.5-0.5B on relation extraction and beat zero-shot GPT-5.4 (0.83 vs 0.69 micro-F1) and Claude Sonnet 4.6 (0.66), rising to 0.92 against 0.83 in the literary domain. An in-domain RoBERTa baseline also beats both frontier models, so the authors attribute the win to task adaptation rather than to anything about model type [14]. On tool calling specifically, fine-tuned 3B variants match 7B ones, and fine-tuning lifted tool-selection recall 45 points [15]. NVIDIA's position paper still measures serving a 7B model at 10–30x cheaper than a 70–175B one and estimates 40–70% of three real agent frameworks' calls are narrow enough for a specialist [1]. Predibase's LoRA Land remains the largest breadth result, at 310 adapters over 31 tasks, but its GPT-4 comparison should now be read as historical [2]. A fine-tuned DeBERTa-large, roughly a third of a billion parameters, scores 87.8% on legal clause classification where GPT-4 zero-shot scores 67.2% [3]. Meta's Llama Guard 3, an 8B guardrail model, scores 0.939 F1 on its hazard benchmark where GPT-4 scores 0.805 [4]. LiveKit runs a half-billion-parameter turn detector (a Qwen2.5-0.5B fine-tune) in production on CPU, and a single version bump cut its false interruptions by ~39% [5]. Our own table sits comfortably in this company -- as consistent-with, not proof-of: every fine-tune beat the strongest prompted model our box serves on its declared axes, including a 0.6B that answers in 140ms where a model with 58x its parameters took half a second to be wrong more often.

The part of the thesis that sounds like arithmetic but changes the architecture is VRAM co-residency. A 4-bit 4B model occupies about three gigabytes; a 24GB consumer card holds a handful of distinct specialists at once, and adapter-based serving pushes the marginal specialist toward zero, since one resident base model can host thousands of task adapters that page in per request. That was a research claim when this paper first cited S-LoRA [6]; by 2026 it is shipped infrastructure -- vLLM's multi-LoRA is generally available and runs behind AWS SageMaker and Bedrock, benchmarked with eight adapters resident at once [18]. The operational friction that remains is cold-loading a new adapter and mismatched ranks across adapters, not the idea. One box on a home LAN already serves this operation's whole fleet: two ops models, a turn detector, a triage model, and a field detector, with room to keep going. The giant-model equivalent of that fleet does not fit on the card at any quantization, and renting it means paying know-it-all prices for skills the task never uses.

This is worth distinguishing from mixture-of-experts, which routes among experts inside a single jointly-trained network and ships as one deployment. A fleet is looser and more operable: each specialist is trained, evaluated, gated, promoted, and rolled back independently. When the field-detection grammar changes, the turn detector doesn't retrain, and a regression in one specialist cannot leak into another. The router is dumb and explicit instead of learned, which is precisely why it is debuggable.

The case against, argued properly

The loudest 2026 counter-signal is that OpenAI deprecated self-serve fine-tuning in May 2026, closing it to new organisations immediately and ending all self-serve jobs by January 2027. Their stated reason goes straight at this thesis: newer base models follow instructions and formats much better, prompt-based approaches are cheaper and faster, and they see fewer use cases that require fine-tuning [16].

That evidence is real and it is about a different thing. It concerns fine-tuning a closed-weight frontier model through somebody's API, where you pay a premium for a checkpoint you do not hold and cannot serve yourself. This pipeline fine-tunes open weights on rented hardware and serves them on a box in the house. The economics and the failure modes are not the same, and a reader should not let one stand for the other. But the underlying claim, that better base models erode the need for adaptation, applies to both, and it is the strongest argument against this whole enterprise.

Three more erosion points, named rather than dodged:

• Volume. Prompt caching cut input costs substantially, moving the fine-tuning break-even from roughly ten thousand requests a day into the tens of thousands. Below that, a frontier model with a good prompt is cheaper once engineering time is counted. Practitioner synthesis rather than a controlled study, and should be read as such.

• Labelled data. Every win cited above is fine-tuned-on-labels versus zero-shot-frontier. Without in-domain labels there is no specialist, and the pipeline's answer of generating them from a simulator carries its own circularity, named elsewhere in this paper.

• Format compliance is no longer the argument. Constrained decoding is cheap and ubiquitous now: XGrammar is the default backend in vLLM, SGLang and TensorRT-LLM at under 40 microseconds a token. "The small model emits valid JSON" is not a differentiator. Frontier models get the format right.

That last point deserves its inversion, because the evidence cuts the other way too. "The Constraint Tax" measured hard-schema constrained decoding on small models: validity rises from 61.5% to 100% while answer accuracy falls from 19.7% to 11.0%, and a Qwen2.5-1.5B on a tool-call task drops from 91.5% executable accuracy under plain prompting to 48.0% under a hard schema [17]. Constraining the grammar does not fix wrong answers; it makes wrong answers well-formed. And at the frontier, format validity and semantic correctness have plainly decoupled: a model can hit 99.97% JSON pass rate and 48.6% fully-correct responses.

Our own numbers corroborate the useful half of that. The fine-tuned 1.7B tool-caller reached 1.0000 output-validity with no constrained decoding at all -- fine-tuning bought format compliance for free, without paying the constraint tax. Anyone reaching for a grammar to fix a small model's output should measure what it costs them in accuracy first.

Finally, the honest gap: nobody has published a rigorous "we built the specialist and prompting won" postmortem with numbers. The 2026 case against is structural and economic, not an empirical failure. This paper will not invent one to knock down.

The honest boundary: specialists win when the task is narrow, stable, and called often. Open-ended reasoning, low-volume work, and fast-shifting requirements still belong to generalists, where long context and in-context learning inject task knowledge without a retrain. The production answer is heterogeneous, and this pipeline's own monitoring ladder is the pattern in miniature: local specialists handle the routine calls for electricity, and a frontier model sees only the ambiguous residue. The real moat is not any single small model. It is the factory: a fleet strategy is only viable if training a new specialist costs an evening and a quarter, retrains are cheap enough to run on every material research delta, and a gate keeps each one honest. That is what this pipeline is for.

How the pipeline improves itself

This is the core claim, and it rests on four loops rather than one.

Loop 1: every incident becomes code, the same day. The rule is that an operational lesson may not live in memory or in a doc alone; it must land as a commit that makes the failure structurally impossible or automatically handled. A sample from one week: a WAN flap false-killed a healthy run, so the watch loop got a six-cycle grace window and now distinguishes SSH-down from session-ended, corroborating against the marketplace API before declaring death. A shell one-liner bound its log redirect to the last command only, so a pip failure was invisible and healthy hosts got blamed; the train command is now group-wrapped, and that's a code-reviewed invariant. A broken direct port on one host's NAT produced an unreachable instance, so SSH and rsync both learned a proxy fallback, proven live the same week.

The loop found a class it had not seen before on 2026-09-09, and the class is worth naming: a guardrail that fails closed, silently. Three separate filters were doing nothing, and each looked like a fact about the world rather than a bug. -- gpu RTX_5090,RTX_4090,RTX_3090 emitted gpu_name=RTX_5090,RTX_4090,RTX_3090, which vast reads as one literal SKU name, so every multi-SKU pool probe the daily research protocol prescribes had been returning zero offers and reading as a thin market. -- ram 24 emitted gpu_ram>=24576, because the offer a search returns reports MB while the query it takes is in GB, so the memory floor had never matched a single host in the framework's life; live, gpu_ram>=24 matches sixty RTX 3090 offers and gpu_ram>=23552 matches none. And the spot checkpoint lifeboat, the thing that turns a dead interruptible instance into a resume instead of a loss, hardcoded /out/checkpoint- while vastops pull honoured the project's configured checkpoint path, so any project that wrote anywhere else had a lifeboat protecting nothing and saying nothing about it.

None of the three would have shown up in a test suite at 100% branch coverage, because all three werecorrect code computing the wrong thing, and the two query bugs could only be caught against the live marketplace. The general lesson is that a protective filter needs a test that asserts it matches something real, not merely that it emitted the string the author intended. All three are now fixed with tests that name the live evidence, and two of the tests that encoded the wrong unit were corrected rather than deleted, because a test asserting a bug is worse than no test.

One day shows the loop closing in real time, twice. A variable-length batch OOM'd a 24GB card at step 55. The pipeline stopped billing within one poll cycle, kept the disk, and the run resumed on the same host twenty-two minutes later with a halved micro-batch. Then the incident itself got generalized into vastops heal: the watch loop now classifies a dead session from its log tail using the same failure vocabulary the monitors use, mutates the config (halve the micro-batch and double the gradient accumulation after an OOM, preserving the effective batch size and the learning-rate schedule), re-pushes, and relaunches on the same instance, capped at two attempts. OOM is the only class that gets a remedy: NaN, like every unknown failure, stops and preserves the evidence, because a NaN has four plausible root causes and a config mutation that suppresses all of them is evidence-burying, not healing. Replayed against the morning's crash log, heal derived the exact fix the operator had applied by hand. Each applied remedy is appended to a lessons ledger with its hardware and config shape, and every launch first prints what past runs on similar shapes needed, so the same wall does not get hit twice. The second fix from the same day: a resumed instance's SSH port mapping changes and comes up slowly after a stop/start cycle, which once burned a push's retries on a dead cached port. The pipeline now waits on an endpoint-re-resolving SSH probe before it syncs.