Staged extraction vs. a single large model

Hand a hard NLP problem to a team today and the reflex is to reach for the biggest model available and let it sort everything out. On French clinical text that reflex is expensive, slow, and hard to audit — and, it turns out, not even the most accurate. DEMNE makes the opposite bet: send each entity type to the lightest tool that can handle it, and escalate only when the text truly demands it.

A staircase, not a hammer

The pipeline has three tiers. Deterministic rules — regex and curated lexicons — take the entities written in a few stable ways: dates, dosages, standard biomarker abbreviations. A transformer fine-tuned on French oncology text handles the entities that need real semantic and domain understanding. An LLM sits at the top, reserved for the genuinely ambiguous cases that survive both earlier stages. Nothing climbs higher than it has to.

Why not just the LLM?

Because one big model pays the LLM’s price on every entity, including the ones a regex would have nailed instantly. It is slower, it needs a GPU budget most hospitals do not have, and every decision is a black box you still have to defend to a clinician or an auditor. And on the routing task itself the always-LLM baseline was the worst performer, agreeing with the expert tier only 0.148 of the time. The heaviest tool is not automatically the most reliable.

The cost of routing wrong

Staged extraction only pays off if the routing is good, so that is where DEMNE spends its care. Five metrics measured from the annotations feed a deterministic decision graph whose 17 thresholds are calibrated by a multi-objective search. The objective is deliberately asymmetric: sending an entity to a method that is too weak is penalised twice as hard as sending it to one that is merely too expensive — because a missed extraction in a clinical report costs more than a wasted GPU second.

What the numbers say

On the oncology calibration set the routing matches the expert reference tier 0.954 of the time, against 0.729 for always-rules and 0.148 for always-LLM. Applied untouched to 300 unseen AP-HP reports, it routed 99.5% of mentions to deterministic rules, which reached a mean F1 of 0.941 — where a transformer fine-tuned under a realistic hospital budget managed 0.358. For the standardized entities that dominate these reports, the cheap tier was also the accurate one.

When a single model is the right answer

Staging is not free: it adds a routing layer to build, measure and maintain, and the thresholds must be recalibrated for each new domain. If the text is uniform, the entities are all equally hard, or there is budget and audit tolerance for one large model, the staircase is just overhead. The claim here is narrow — when cost, latency and traceability matter at once, as they do in clinical NLP, deciding per entity beats defaulting to the heaviest model and hoping.

The code and the full method are on GitHub; the manuscript is in preparation.