Why French clinical NER breaks general-purpose models

Pull up a French oncology report and most off-the-shelf NER models stumble within the first sentence. The vocabulary is dense and abbreviated (ER, PR, HER2, Ki67, FISH), the formatting shifts from one hospital to the next, and the biomarker terminology that matters is almost absent from the corpora these models were trained on. The usual reflex — throw a large model at it — works, but it is expensive, opaque, and hard to defend to a clinician or an auditor.

Not every entity needs an LLM

DEMNE starts from a blunt observation: the entities in these reports are not equally hard. A biopsy mention is written in a handful of stable ways; a drug name can appear in a hundred. So instead of one model for everything, DEMNE measures each entity type and sends it to the lightest method that is reliable enough — a deterministic rule, a fine-tuned transformer, or an LLM only as a last resort.

The measurement is five corpus metrics computed from the annotations: how templateable the surface form is, how homogeneous the vocabulary is, whether a small keyword set is discriminative, how often the mention sits under negation or contradiction, and whether there is enough labelled data to train a supervised model at all. A deterministic decision graph turns those numbers into a tier, and its 17 thresholds are calibrated with a multi-objective search that penalises sending an entity to a method that is too weak twice as hard as sending it to one that is merely too expensive.

What it buys

On the oncology calibration set (59 entity types), the routing agrees with the expert reference tier with a concordance of 0.954 and a leave-one-corpus-out estimate of 0.866 — well above any fixed-tier baseline (always-rules reaches 0.729, always-LLM 0.148). The more convincing test is external: applied without any retuning to 300 AP-HP CT-scan reports from cancer sites the model had never seen, DEMNE routed 99.5% of highlighted mentions (411 of 413) to deterministic rules. Those rules reached a mean F1 of 0.941, while a transformer fine-tuned on the same entities under a realistic hospital budget managed only 0.358.

In other words: for the standardized entities that dominate oncology reporting, rules were not just cheaper — they were better, and every rule decision traces back to an explicit condition, which is exactly what Articles 12 and 14 of the EU AI Act ask for.

Where it stops

I want to be honest about the limits. The reference standard is small — 92 entity types, 59 of them in oncology — so single-corpus scores are optimistic and only relative comparisons hold. The LLM tier was never evaluated end to end on the external cohort; its one routed entity was too rare. And the calibration does not transfer across domains: move to a different clinical genre or language and concordance drops to 0.45–0.73, so the thresholds have to be refit. DEMNE is a proof of feasibility, not a finished product.

Why it matters

The point is not that rules beat transformers. It is that “which method” is a question you can answer per entity, from measurable properties of the text, instead of defaulting to the heaviest model and hoping. For clinical NLP — where cost, auditability and trust all matter at once — that framing is worth more than another percentage point on a benchmark.

The code and the full method are on GitHub; the manuscript is in preparation.