Add configurable minimum query length for lookups - #279
Merged
Conversation
Single-character lookups make Solr work hard while never returning a sensible result. Reject queries shorter than a configurable minimum (NAMERES_MINIMUM_QUERY_LENGTH, default 2) before they reach Solr. - Consolidate env-var config into a central `Config` dataclass; expose a public subset (currently just `minimum_query_length`) via `/status`. - `/lookup` returns HTTP 422 for too-short queries (documented in the OpenAPI schema); `/bulk-lookup` maps each too-short string to `[]` so one short string does not fail the whole batch. Note: empty-string `/lookup` now returns 422 instead of an empty list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Note the minimum-length requirement and 422 on /lookup, the graceful []-per-key behavior on /bulk-lookup, and the new config block in /status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # api/server.py
The setting was described in API.md's /status section but missing from the two places that enumerate every environment variable, so a deployer reading the configuration guide had no way to discover it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n for Code review on #279 found three problems with the minimum-length check and one with the test that covers it. The empty-string rejection it replaced was unconditional; the length check is not, so NAMERES_MINIMUM_QUERY_LENGTH=0 -- the obvious way to turn the minimum off, and what SOLR_TIMEOUT_SECONDS=0 means one setting above it -- let an empty query through to Solr as `"" OR ()`. That is a parse error, so an empty search box came back as an HTTP 500. The check now floors at 1 independently of the setting. The minimum is justified by the cost of a short query in the default tokenized search, where a one-character string matches a prefix of half the index. Exact mode has no such cost -- it is a single filter query against an untokenized field -- and single-character labels are real (the gene T, the element symbols), so it is now held to nothing but non-emptiness. A too-short query was reported with HTTPException(422), whose body is {"detail": "<string>"} where FastAPI's own validation returns {"detail": [...]}, so a client iterating `detail` as a list of errors broke on this one rejection. It now raises RequestValidationError instead. The hand-written `responses={422: ...}` on both /lookup operations went with it: FastAPI only generates the HTTPValidationError body for a 422 the operation has not declared itself, so declaring one cost the endpoint its schema and left client generators with an untyped error. Finally, the /status assertion pinned both the default and the exact key set of the config block, so it failed for anyone setting the very override this PR adds. It now compares against the running config. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Log analysis showed that single-character lookups make Solr work very hard while never returning a sensible result: in the default tokenized search a one-character query matches a prefix of a large fraction of the index, and nothing useful comes back. This adds a configurable minimum query length that rejects those queries before they reach Solr, and consolidates the service's scattered environment reads into one place so the new setting has somewhere to live and something to report itself through.
Related to #107, which is about short words inside a longer query (
Apolipoprotein A-I binding protein) rather than a short query as a whole; this does not close it.What's here
A configurable minimum, applied only where it is justified.
NAMERES_MINIMUM_QUERY_LENGTH(default 2 — Translator expects useful results at length 2) sets the shortest string the default tokenized search will accept, measured after leading and trailing whitespace is stripped. It deliberately does not apply toexactmode, which is a single filter query against an untokenized field: it has none of the cost that motivates the minimum, and single-character labels are real targets for it (the geneT, the element symbols). Independently of the setting, an empty query is always rejected — it would otherwise reach Solr as"" OR (), which is a parse error and so an HTTP 500 for what is really an empty search box. That is why the check floors at 1 rather than deriving its bound from the setting alone;NAMERES_MINIMUM_QUERY_LENGTH=0disables the minimum without reopening that hole.Rejections that look like every other rejection.
/lookupreports a too-short query by raising FastAPI's ownRequestValidationError, so the body is{"detail": [{...}]}— the same shape as any parameter that fails validation, rather than the{"detail": "<string>"}anHTTPExceptionwould produce. Correspondingly, the endpoints declare no custom422response: FastAPI only generates theHTTPValidationErrorschema for a422an operation has not declared itself, so a hand-written one would have stripped the schema from the OpenAPI document and left client generators with an untyped error./bulk-lookupdegrades instead of failing. A too-short or empty string maps to[]for its own key, so one bad string in a batch does not fail the whole request.Configuration consolidated into a
Configdataclass.SOLR_HOST,SOLR_PORT,SOLR_CORE,SOLR_MAX_CONCURRENT_LOOKUPSandSOLR_TIMEOUT_SECONDSwere module-level globals read at import; they are now fields on a frozenConfiginstantiated once, alongside the newminimum_query_length. Every environment variable name and default is unchanged, so no deployment needs touching. The dataclass also gives/statusaconfigblock, populated by an explicitpublic()allowlist so that connection details do not leak into a public endpoint by accident.Behaviour changes for callers
/lookupnow returns 422 rather than an empty list. Callers relying on the old empty-list response need to handle the status code./lookupqueries now return 422 rather than a (slow, useless) result set — the point of the PR./statusgains aconfigobject, currently{"minimum_query_length": 2}.What it deliberately does not do
/bulk-lookupgives no signal distinguishing "this string was too short" from "this string matched nothing" — both are[]. Keeping the batch alive was judged more valuable than the diagnostic; reporting it as well is Let/bulk-lookupcallers tell "not searched" apart from "no matches" #298.Testing
python -m pytest tests/→ 49 passed, against a live Solr 9.10 loaded withtests/data/test-synonyms.json. Beyond the minimum itself, the new tests pin the parts that are easy to break silently: that the422body is a list of errors and matches the shape of an ordinary validation failure on the same endpoint, that both/lookupoperations still document422asHTTPValidationErrorin the generated OpenAPI schema, that an empty query is rejected even with the minimum set to0, and thatexactmode resolves a two-character synonym with the minimum patched to5while the same query withoutexactis rejected.Before merging
Nothing outstanding — no blocking work is known.
History — a merge from
mainand one review round. Kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code comments,CLAUDE.mdand the docs above.Merge from
main. #273 (exact-match mode, parallel bulk lookups) landed while this was open and addedSOLR_CORE,SOLR_MAX_CONCURRENT_LOOKUPSandSOLR_TIMEOUT_SECONDSas module-level globals, conflicting with theConfigdataclass here. Rather than keep two configuration mechanisms side by side, the merge folded those three intoConfig— which is why this PR's diff touches settings unrelated to query length.Review round. Four findings, all fixed in
b8b5349:if string_lc == "": return []), but the length check that replaced it is not, soNAMERES_MINIMUM_QUERY_LENGTH=0let an empty query through to Solr and returned a 500. Now floored at 1.exactmode, where its rationale does not hold and where it put legitimate short labels out of reach. Now exempt.HTTPException(422), whose body shape differs from FastAPI's own validation errors, and the accompanying hand-writtenresponses={422: ...}suppressed theHTTPValidationErrorschema in the OpenAPI document. Both replaced as described above./statustest assertedconfig == {'minimum_query_length': 2}, pinning both the default and the exact key set, so it failed for anyone setting the override this PR adds. It now compares against the running config.Findings 1 and 3 are traps worth not falling into twice, so they are recorded in
CLAUDE.md's gotchas as well as in comments at the code that invites them.