Your threshold is a function of the question

Most retrieval systems end up with a number in them. Somewhere there is a constant, usually around 0.8, and content scoring below it is treated as not relevant enough to use. The team arrived at it honestly: they ran the test questions, watched where the good answers sat, and drew the line just under them.

The trouble is what those test questions look like. They were written by people who already knew the answer.

A test set written by your team is a test of your team's ability to describe the problem. Customers have never seen the article, and it shows in how they ask.

The same document, two scores

Take a support article about a laptop overheating under sustained load. The question it was tested against reads something like "my laptop overheats when I run video calls, what should I do?" That is a good question. It names the device, the symptom and the trigger. Against the right article it might score 0.81, comfortably over the gate.

The customer types "its getting hot".

Same person, same problem, same article, and now it scores something like 0.61. The gate rejects it, and the system says it cannot help, about content it holds and could have used. The customer hears "they don't have anything on this", which is both wrong and unfalsifiable from where they are sitting. Nothing errors. Nothing is logged as a failure. It is one of the quietest ways an otherwise good retrieval system underperforms.

Why the number moves

It is tempting to explain this as fewer matching terms, and for keyword search that is literally what happens. Dense retrieval does not work that way. The short query is not matching less of the document; it is pointing somewhere less specific. "Its getting hot" sits in a generic neighbourhood shared by every heat-related thing in the corpus, rather than aligning with one article about thermal throttling under load. Fewer constraints, vaguer direction, lower similarity to any single document.

Two things usually make it worse. Most embedding models are trained on symmetric similarity, sentence against sentence, and then used asymmetrically, three words against a long passage; the mismatch punishes short queries specifically. And the longer your chunks, the more a brief query is diluted across content it never asked about.

The part that actually undermines the gate

Absolute similarity scores are not comparable across queries. Score distributions are query-dependent, and embedding spaces are anisotropic enough that 0.8 does not mean the same thing from one question to the next. So the fixed cutoff is not merely mis-tuned for vague questions; it is measuring something that has no stable meaning across the queries you are applying it to. It only looked reliable because the test set was homogeneous.

Three things to do instead

Repair the question before you embed it. This is the cheapest and it is almost always available. You usually know things the customer did not bother to type: which product they own, what they contacted you about last week, what page they are on. Turning "its getting hot" into a specific query using context you already hold costs the customer nothing and asks them nothing. That is exactly what having a canonical view available at the point of retrieval is worth.

Ask, when you genuinely cannot tell. If the context does not disambiguate, one short clarifying question converts a poor query into a good one and puts you back in the range the gate was calibrated for. Ask it only when you have to. A system that asks what product someone owns when it already knows reads as an organisation that does not.

Gate on the shape of the results, not on an absolute number. The useful signal is not "did the top hit clear 0.8", it is "does the top hit stand clear of the rest". A decisive gap between first and second means the retrieval found something, whatever the raw numbers are. A flat field of near-identical scores means it did not, even when they are all above your threshold. That comparison normalises itself per query, which is precisely the property a fixed constant lacks.

And calibrate on real questions

Whichever of those you adopt, the underlying discipline is the same: tune the system against questions customers actually asked, pulled from live transcripts, including the two-word ones and the ones with a typo in the product name. It is uncomfortable, because your numbers will get worse the moment you do it, and the ones you had were describing a population you never serve.

That is the useful outcome, not the discouraging one. A threshold that looks excellent against questions written by the team is not evidence of anything, and a measure that cannot move for the wrong reason cannot steer you. Better to know the number is 0.61 in the real world and design for it.