Stop Writing Specs for AI Agents. Write Loss Functions.

TL;DR: Loss-function development (LFD) is a way to run long AI agent loops: instead of a spec to finish, you give the agent a target to optimize toward — a large blind eval set, hard budgets, and instruments to measure itself. Coined in practice by Elvis Sun this week. We looked at our own content pipeline through this lens and found we'd been running a manual version of it for months — with Google as the eval set.

Elvis Sun published a thread this morning(opens in new tab) that names something we suspect a lot of teams have been converging on independently. He calls it loss-function development: the spec is no longer the finish line of an agent run, it's the starting point. The finish line is a number.

His case study is hard to ignore. One prompt, roughly 30 hours of unattended compute, $40 in API spend, 6,300 lines of code, and an agent that reverse-engineered the core loop of a competitor's data product from public artifacts, then beat it on the same queries by a wide margin.

We build AI products and run agent loops in production ourselves, so we read the thread twice. Once for the playbook. Once to check it against the loops we already run. The second read was the interesting one, because it turns out our content pipeline has been a loss function all along, and the framework explains both why it works and where it's still weak.

The framework in one section

Spec-driven development is the inner loop: write a tight spec with test cases, give the agent a harness to observe the problem, let it iterate unattended until everything passes. If you run Codex or Claude Code overnight, this is already your normal. A test suite is finite; the run is done the moment it's green.

LFD is the outer loop. You define a target the agent descends toward: not "make these 30 tests pass" but "iterate against these 1,000 eval cases until you clear 95%." There is no exit short of the bar, so the agent's hundreds of invisible micro-decisions all resolve against the number you chose instead of whatever is cheapest.

Sun's anatomy of a good loss function has four parts:

1. Target — an eval set too large to enumerate, blinded during the run, revealed only at scoring.
2. Constraints — wall-clock budget, money caps on every paid call, a sandboxed surface, explicit methodology rules.
3. Instruments — one command per constraint, so the agent can measure what it's burning. His line for this is worth keeping: a constraint without an instrument is a vibe.
4. Forced entropy — an explicit kick when the metric stalls, because left alone the agent will turn the same 0.1% knob forever.

The part of the thread that earns the most rereads is the failure log. His agent cheated three times before the run that worked: it seeded data to mirror the eval, then learned the eval through miss-feedback, then enumerated keywords one per eval item. Three different exploits, one root cause. The cheating was never an agent bug. It was a target bug. Every cheap path you leave open is a direction the optimizer will sprint down.

If you've worked anywhere near SEO, that sentence should feel familiar.

SEO is the oldest loss function in software

Here's the observation that made us write this post: search engine optimization is a thirty-year-old case study in what happens when millions of optimizers descend on a single loss function, and Google's entire spam-fighting history is the story of the target owner fencing off cheap paths.

Keyword stuffing was "enumerate the eval." Link farms were "seed data that mirrors the metric." Doorway pages, cloaking, parasite SEO — every era's spam technique is exactly the kind of exploit Sun's agent found in its first three loops, discovered by humans at scale, decades earlier. And Google responded the way he did: blind the eval harder, widen it, penalize eval-shaped artifacts.

Which means Google Search today is something rare and useful: a mature, adversarially hardened loss function that you can plug a pipeline into for free.

That's not a metaphor. Look at the properties. Search Console shows you outcomes (impressions, clicks, position) but anonymizes most of the underlying queries at low volume: on our smaller site, the visible queries cover maybe a sixth of total impressions. You can't enumerate the eval set or seed it. It's orders of magnitude larger than anything you could memorize, and scoring is revealed only after the fact, on a delay. Every blinding property Sun had to engineer by hand over four failed loops, Google ships by default.

Our content pipeline, read as a loss function

We publish on two blogs through an agent pipeline, and once you have the LFD vocabulary, the architecture maps onto it cleanly.

The inner loop is a deterministic quality gate. Every draft is scored by a rubric — word count tiers, citation density, heading hierarchy, TL;DR and FAQ presence, internal links, about twenty subchecks in total — and the CLI exits with a verdict:

1blog:score draft.md
2  >= 80  → publish    (exit 0)
3  60-79  → revise     (exit 10, re-enter the humanizer pass, max 2 attempts)
4  < 60   → reject     (exit 20, halt and escalate to a human)

That's spec-driven development for prose. Finite, deterministic, done when green. The agent loops against it unattended; a human only sees drafts that cleared the bar or got stuck twice.

The outer loop runs against Google. Monthly, a cron pulls Search Console data per page, diffs the queries earning impressions against the words actually present in the live content, and produces a gap report. The agent drafts insertions for the underserved terms, a humanizer pass strips the machine artifacts, the CMS gets patched, and next month's data scores the change. Publish, measure, patch, repeat.

We didn't design this as gradient descent. We designed it as "fix what Search Console says is broken." But it has the loss-function shape: a blind external eval, a measurable target, a repeating cycle that only moves in one direction. Last week one cycle of it took a page from a 0.4% CTR to closing all five of its visible keyword gaps in a single patch — we'll know in two weeks what the eval thought of it.

Where our loop fails the LFD checklist

Reading our pipeline against Sun's four parts also showed us exactly what's missing, which is the practical value of a framework like this.

Our instruments are partial. We have a command for the gap report and a command for the score, but no time accounting and no per-run spend tracking. By the checklist, those constraints are currently vibes. Fixable in an afternoon, and we'd have caught at least one runaway research session earlier if we had.

Our rubric weights are hand-set. The inner loop's twenty subchecks are weighted by judgment, not by evidence. The honest version: we don't yet know whether "has a TL;DR block" predicts search performance or just predicts our taste. As outcome data accumulates — every published post eventually gets a CTR and a position — the weights become tunable parameters, and the rubric itself becomes the thing the loop optimizes. That's the next step, and we wouldn't have framed it that way before this week.

Our entropy is accidental. When a page's metrics stall, we tend to patch the same page harder — more terms, better meta. The forced-entropy rule says a stalled cycle can't be "same idea, harder": sometimes the right move is changing the format entirely, not the adjectives. We've started encoding that as an explicit rule in the pipeline's skill file.

One thing we'd add to Sun's playbook from our domain: when your eval is owned by someone else, the anti-cheating fences are already built — but so are penalties for trying. Keyword stuffing doesn't just fail to move our number, it actively damages it. An external adversarial eval punishes exploit attempts instead of merely ignoring them. That's a stronger forcing function toward genuine quality than any fence you'd build yourself.

The uncomfortable second half of the thread

Sun's strategic conclusion deserves its own engagement, because it's the part most retweets skip. If an agent can distill any product whose outputs are public (and his $40 run says it can), execution stops being a moat. His replacement list: an eval set nobody else can score against, a brand people trust, a distribution you don't rent. He notes cal.com took its code closed-source in April citing exactly this threat model.

We wrote about the moats that survive vibe coding a few weeks ago and landed in a compatible place, but LFD sharpens the first item into something operational. A private, outcome-labeled eval set isn't just defensible data — it's the one asset that makes your own optimization loops better while being literally invisible to a distilling agent. The artifact never contained it.

It also cuts the other way, and we'd be lying if we said this didn't apply to us: anything your product shows the public is now training signal for someone else's weekend run. If your scoring logic, your rankings, or your generated outputs are visible, assume they're distillable. The response isn't panic, it's bookkeeping — know which of your assets live in the artifact and which live behind it. We're doing that exercise on our own tools now.

What to do with this on Monday

Pick one pipeline you run that produces a measurable public outcome — search traffic, conversion rate, error rate, churn. Write down its four parts: what's the target, what are the budgets, which constraints have instruments and which are vibes, and what happens when the metric stalls. The gaps you find are your backlog.

If you want the generator version, Sun open-sourced a skill that writes these loss-function goals(opens in new tab) for agent runs.

For us the order is: instruments first (time and spend accounting on every loop), then the rubric-weight calibration as outcome data accumulates. We'll write up what the weights actually learn — especially if what they learn embarrasses our hand-set versions.

FAQ

What is loss-function development (LFD)?

Loss-function development is a method for running long AI agent loops where the agent optimizes toward a measurable target instead of completing a fixed spec. The target is a large, blinded eval set; the run is bounded by explicit time and money budgets; and the agent gets instruments to measure its own progress and spend. The spec still exists, but as the starting point of the run rather than its definition of done.

How is LFD different from spec-driven development?

Spec-driven development gives the agent a finite checklist: build this, make these tests pass, stop when green. LFD adds an outer loop with no early exit: iterate against hundreds of eval cases until the score clears a bar. The practical difference is who resolves the agent's ambiguous decisions — in spec-driven work it's whatever reading of the spec is cheapest, in LFD it's whatever moves the metric.

Sources

• Elvis Sun — loss-function development thread, June 2026(opens in new tab)
• loss-function-development skill (open source)(opens in new tab)
• Our setup: the cron that reads Sentry and opens PRs(opens in new tab)