Public Radar sends daily email digests of public-data events: FDA recalls, patents and trademarks, SEC insider and 8-K filings, clinical trials. The hard part is one question asked of every record: does this actually matter to this subscriber?
I rebuilt the component that answers it for FDA recalls: tool calling, a strict output schema, verbatim evidence checked against the source. Then I measured it against 20 recalls I'd labelled by hand. It scored an F1 of 0.824.
A function that returns True for every item scored 0.919.
To be clear up front: all of this lives on a local feature branch. Production still runs the old scorer, and the larger set that would tell me whether the new one earns its keep isn't labelled yet. This is what I learned building the eval before shipping, which turned out to be the more useful thing.
The pipeline, and the question it answers
Each public data source has an adapter. A keyword prefilter keeps any record that mentions a subscriber's watchlist term, then Claude Haiku 4.5 scores each survivor for relevance and severity and writes a short brief. Anything scoring 55 or above goes in the digest.
The prefilter is a naive substring test, so a watchlist term in a firm's name or a diagnostic test's name gets through. Sorting those out is the scorer's entire job.
The old scorer failed silently
The version on main asks the model for a JSON array in prose and scrapes it out with a regex. Simplified:
# core/ai.py on main (simplified)
match = re.search(r"\[.*\]", text, re.DOTALL)
try:
arr = json.loads(match.group(0)) if match else []
except json.JSONDecodeError:
arr = []
out = []
for i, item in enumerate(items):
# results are mapped back to items by position in the batch
data = arr[i] if i < len(arr) else {}
out.append(_coerce(item, sub, data)) # missing keys become relevance 0
Two failure modes hide in there. If the reply is garbled, every item in the batch gets relevance 0, which looks exactly like a quiet day, and nothing retries. And because results are matched to items by position, a reply that comes back one object short silently zeroes the tail of the batch.
The docstring called relevance 0 "a safe default". It was safe in the sense that it never sent a bad alert. It also never told anyone a good one had been dropped.
The rebuild, briefly
The new scorer is a per-item tool-calling loop. The model can call search_openfda to check a firm's or ingredient's recall history, and must return an object matching a JSON schema. A validator checks what the schema can't express, and on a violation the model is re-asked with the exact rule it broke, at most twice. Failures still end at relevance 0, but now with a recorded reason.
The schema has eleven required fields. The decisions that mattered:
item_idis echoed back, so a score can't be attached to the wrong record.confidenceis a low/medium/high enum, not a float. Model-reported 0–1 confidences aren't calibrated, and a float invites people to treat them as if they were.match_basissays why the item matched, and includesnone: the term is present but the match is spurious.evidence[]holds quotes that are checked verbatim against the record or a tool result. A paraphrase fails validation.- There is no
should_alertfield. The model scores, the code decides, and the threshold stays a knob I can move.
# core/score_schema.py (one of the eleven fields)
"match_basis": {
"type": "string", "enum": list(MATCH_BASES),
"description": "Why this item matched the watchlist. 'none' means the "
"term is present but the match is spurious — the "
"keyword prefilter fired on something incidental.",
},
A new scoring_decisions table records every decision, including suppressed ones. The old alerts table only held sent items: no negatives to measure against, and no relevance score at all.
Then I measured it
Phase 1 of the eval was a smoke test: 20 real openFDA recalls that survive the prefilter for the live watchlist, shuffled and labelled blind: nothing in the labelling sheet hinted at which items were expected to matter.
The harness prints every metric next to three trivial baselines, because a number on its own tells you nothing. Simplified:
# eval/score_label_pool.py (simplified)
truth = [r["label"]["should_alert"] == "y" for r in labelled]
model = [r["relevance"] >= threshold for r in labelled]
baselines = {
"always-alert": [True] * len(labelled),
"always-suppress": [False] * len(labelled),
"keyword-only": [keyword_baseline(r) for r in labelled],
}
The keyword baseline alerts if a watchlist term appears in the product description. Its docstring sets the bar: if an LLM call can't beat one in test, it isn't earning its cost.
Here's what came back at the default threshold of 55:
P R F1 TP FP FN TN
scorer 0.824 0.824 0.824 14 3 3 0
always-alert 0.850 1.000 0.919 17 3 0 0
always-suppress 0.000 0.000 0.000 0 0 17 3
keyword-only 0.667 0.353 0.462 6 3 11 0
The set was 17 alerts and 3 non-alerts, so a constant that says yes to everything gets perfect recall and 85% precision. The scorer got none of the three negatives right and missed three real alerts on top.
Was 55 just the wrong cut-off? The harness treats the threshold as a parameter and sweeps it:
threshold 0-45 F1 0.919 every item clears it: this is always-alert
threshold 50-65 F1 0.824
threshold 70 F1 0.848
threshold 80 F1 0.640
No threshold beats the constant; the best the scorer can do is become it. That's a clean negative result, and I'd rather have it now than after a month of digests.
Why it lost
The misses were all one kind of recall
All three false negatives scored exactly 45, and all three were mislabel recalls. The clearest was a repackager recall where tablets of a prescription antipsychotic "may have potentially been mislabeled as" a common over-the-counter supplement. In plain terms: bottles labelled as the supplement may have contained a psychiatric drug. If you watch that supplement, that is precisely the recall you want to see.
The model understood the facts. Its rationale says some units ended up in the supplement's bottles. It then scored the item as moderate because the watchlist term was "the MISLABEL destination", not the product being recalled. It wasn't even consistent: the set had six mislabel recalls, and it alerted on the other three. Two recalls from the same repackager, with near-identical wording and the same watchlist term, scored 45 and 78.
The missing piece was a domain rule: a recall of bottles labelled X that may contain something else still matters to someone watching X. That rule existed only in my head. Worse, when I reread the system prompt, it listed "the thing a product was mislabelled as" among its examples of a watchlist term appearing incidentally. I had written the opposite of my own business rule into the prompt. No schema catches that. Only labels do.
The three negatives all got through
The set had just three non-alerts, and the scorer alerted on all of them. One was a textbook hard negative: a diagnostic test for one of the watchlist substances. A test for X isn't X, and the scorer gave it 65.
The other two were closer calls. Reading my notes back, at least one was me labelling on severity ("sounds bad, but i dont know if it would cause illness?") rather than on whether the term matched the product. The Phase 2 labelling rubric now opens with "alert ≠ severity" for exactly that reason. Eval sets have bugs too.
Three of four judgement fields collapsed
The fields meant to explain each decision barely moved:
match_basis:ingredienton 20 of 20 items, including the diagnostic test.confidence:highon 20 of 20.severity:medium16 times,high4 times,lownever.
The harness compares each field against a majority-class baseline, the only fair way to read agreement on a skewed set. match_basis agreed with my labels on 16 of 20 items, exactly what always guessing the most common label gets. confidence matched its baseline too, 13 of 17. severity agreed on 7 of 17, worse than the baseline's 8. A field that never changes carries no information.
The watchlist was mostly to blame
The uncomfortable conclusion: this test mostly measured the watchlist, not the model. The live watchlist is a handful of specific chemical names that rarely appear incidentally in a recall record. When I later sourced negatives exhaustively, those terms produced four distinct hard negatives across the entire openFDA enforcement corpus. Not four in my sample; four in existence. For that watchlist, about 1% of prefilter survivors are genuine hard negatives, so "alert on everything" is very nearly the correct policy, and a skewed pool flatters the constant arithmetically on top of that.
That doesn't excuse the scorer; the collapsed fields and the mislabel misses are real. But the honest framing is narrower: on that watchlist the alert/suppress decision is close to trivial, so the scorer's value has to come from severity, evidence and the brief rather than from suppression. Twenty items from one watchlist couldn't tell those apart.
Building a set that can answer the question
Phase 2 is a 150-item set, designed and sourced but not yet labelled, so there are no Phase 2 numbers. The design is the point:
- Items are (record, subscriber) pairs across two personas: the live watchlist and a broader eight-term supplement-brand watchlist. The same recall means different things to different watchlists, and the second persona exists because that's where the negatives are.
- It's 40% negative, split 100 dev / 50 test, with both halves stratified identically. I iterate prompts on dev. Test gets looked at once per real prompt version.
- Eight strata, sampled on observable structure rather than guessed labels. Four positive (clear ingredient, manufacturing deviation, mislabel target, claim or product type) and four hard negative (firm name only, a test for the analyte, negation or "-free", the term in an unrelated context).
- At most four items per recalling firm per stratum, so one firm's recall family can't pose as many independent judgements.
- Near-duplicate clustering. openFDA returns one recall event as many near-identical distributor records, so the 150 representatives stand for 274 raw records.
The eval README leads with its limitations, because two decide what the numbers may mean. The mislabel stratum is 8 items from 3 firms, and for these watchlists the whole corpus has only about two distinct firms with that pattern, so a passing number there means "handled that one repackager's pattern", not "learned the rule". And negatives are over-sampled roughly 3–40x relative to production, depending on the persona, so precision and recall on this set measure discrimination, not production error rates.
Smaller things the data changed
The evidence validator originally rejected quotes under 8 characters. Two of the three Phase 1 retries were that floor rejecting "Class I" (7 characters) while "Class II" (8) passed, a coin-flip on recall severity. The floor is now 6, and a quote reproducing an entire field exactly is exempt.
The operational numbers from the 20-item run:
tokens per item 8,636 in / 847 out
tool calls 1.75 per item
latency 10.7 s average, 21 s max
retries 3 of 20 items, 0 hard failures
cost about $0.013 per item (plan estimate: about $0.005)
The plan assumed roughly 3,000 input tokens per item; the real figure was nearly three times that. A cent per item is still affordable for a daily digest. The limit that bites first is tool-call latency, not tokens. One tool call also failed without a captured reason, so the harness now records every tool error payload.
Lessons learned
1. Compute the constant before you celebrate. Trivial baselines cost nothing to run. If your model can't beat them, you want to know before your users do. My harness prints them automatically, so I can't forget.
2. The negative class is the binding constraint. Hard negatives are rare, and they're the only place a scorer can show its value. Design the set around finding them, and put the base rate next to every metric.
3. Treat the threshold as a parameter. A single operating point hides whether the score carries any signal. A sweep answers it.
4. Watch for fields that never move. A structured schema looks rigorous even when three of its judgement fields are constants. Compare each against a majority-class baseline. Zero lift means zero information.
5. Read the rationales on the misses. The model had the facts right. It lacked a business rule I'd never written down, and my prompt pointed the other way.
Next: label the 150, put the mislabel rule into the prompt, and iterate on dev. The bar is already written down: the new scorer only counts as working if it beats the best trivial baseline on the held-out test split. If it doesn't, I'll have learned that cheaply too.
Putting an LLM in the decision path and want an eval that tells you the truth? Let's talk — or see what Public Radar does.