Skip to content

Add configurable minimum query length for lookups - #279

Merged
gaurav merged 5 commits into
mainfrom
set-min-size
Sep 1, 2026
Merged

Add configurable minimum query length for lookups#279
gaurav merged 5 commits into
mainfrom
set-min-size

Conversation

@gaurav

@gaurav gaurav commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Log analysis showed that single-character lookups make Solr work very hard while never returning a sensible result: in the default tokenized search a one-character query matches a prefix of a large fraction of the index, and nothing useful comes back. This adds a configurable minimum query length that rejects those queries before they reach Solr, and consolidates the service's scattered environment reads into one place so the new setting has somewhere to live and something to report itself through.

Related to #107, which is about short words inside a longer query (Apolipoprotein A-I binding protein) rather than a short query as a whole; this does not close it.

What's here

A configurable minimum, applied only where it is justified. NAMERES_MINIMUM_QUERY_LENGTH (default 2 — Translator expects useful results at length 2) sets the shortest string the default tokenized search will accept, measured after leading and trailing whitespace is stripped. It deliberately does not apply to exact mode, which is a single filter query against an untokenized field: it has none of the cost that motivates the minimum, and single-character labels are real targets for it (the gene T, the element symbols). Independently of the setting, an empty query is always rejected — it would otherwise reach Solr as "" OR (), which is a parse error and so an HTTP 500 for what is really an empty search box. That is why the check floors at 1 rather than deriving its bound from the setting alone; NAMERES_MINIMUM_QUERY_LENGTH=0 disables the minimum without reopening that hole.

Rejections that look like every other rejection. /lookup reports a too-short query by raising FastAPI's own RequestValidationError, so the body is {"detail": [{...}]} — the same shape as any parameter that fails validation, rather than the {"detail": "<string>"} an HTTPException would produce. Correspondingly, the endpoints declare no custom 422 response: FastAPI only generates the HTTPValidationError schema for a 422 an operation has not declared itself, so a hand-written one would have stripped the schema from the OpenAPI document and left client generators with an untyped error.

/bulk-lookup degrades instead of failing. A too-short or empty string maps to [] for its own key, so one bad string in a batch does not fail the whole request.

Configuration consolidated into a Config dataclass. SOLR_HOST, SOLR_PORT, SOLR_CORE, SOLR_MAX_CONCURRENT_LOOKUPS and SOLR_TIMEOUT_SECONDS were module-level globals read at import; they are now fields on a frozen Config instantiated once, alongside the new minimum_query_length. Every environment variable name and default is unchanged, so no deployment needs touching. The dataclass also gives /status a config block, populated by an explicit public() allowlist so that connection details do not leak into a public endpoint by accident.

Behaviour changes for callers

  • Empty-string /lookup now returns 422 rather than an empty list. Callers relying on the old empty-list response need to handle the status code.
  • Single-character /lookup queries now return 422 rather than a (slow, useless) result set — the point of the PR.
  • /status gains a config object, currently {"minimum_query_length": 2}.

What it deliberately does not do

Testing

python -m pytest tests/49 passed, against a live Solr 9.10 loaded with tests/data/test-synonyms.json. Beyond the minimum itself, the new tests pin the parts that are easy to break silently: that the 422 body is a list of errors and matches the shape of an ordinary validation failure on the same endpoint, that both /lookup operations still document 422 as HTTPValidationError in the generated OpenAPI schema, that an empty query is rejected even with the minimum set to 0, and that exact mode resolves a two-character synonym with the minimum patched to 5 while the same query without exact is rejected.

Before merging

Nothing outstanding — no blocking work is known.

History — a merge from main and one review round. Kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code comments, CLAUDE.md and the docs above.

Merge from main. #273 (exact-match mode, parallel bulk lookups) landed while this was open and added SOLR_CORE, SOLR_MAX_CONCURRENT_LOOKUPS and SOLR_TIMEOUT_SECONDS as module-level globals, conflicting with the Config dataclass here. Rather than keep two configuration mechanisms side by side, the merge folded those three into Config — which is why this PR's diff touches settings unrelated to query length.

Review round. Four findings, all fixed in b8b5349:

  1. The empty-string guard this PR replaced was unconditional (if string_lc == "": return []), but the length check that replaced it is not, so NAMERES_MINIMUM_QUERY_LENGTH=0 let an empty query through to Solr and returned a 500. Now floored at 1.
  2. The minimum was applied to exact mode, where its rationale does not hold and where it put legitimate short labels out of reach. Now exempt.
  3. The too-short rejection used HTTPException(422), whose body shape differs from FastAPI's own validation errors, and the accompanying hand-written responses={422: ...} suppressed the HTTPValidationError schema in the OpenAPI document. Both replaced as described above.
  4. The /status test asserted config == {'minimum_query_length': 2}, pinning both the default and the exact key set, so it failed for anyone setting the override this PR adds. It now compares against the running config.

Findings 1 and 3 are traps worth not falling into twice, so they are recorded in CLAUDE.md's gotchas as well as in comments at the code that invites them.

gaurav and others added 2 commits July 17, 2026 19:32
Single-character lookups make Solr work hard while never returning a
sensible result. Reject queries shorter than a configurable minimum
(NAMERES_MINIMUM_QUERY_LENGTH, default 2) before they reach Solr.

- Consolidate env-var config into a central `Config` dataclass; expose a
  public subset (currently just `minimum_query_length`) via `/status`.
- `/lookup` returns HTTP 422 for too-short queries (documented in the
  OpenAPI schema); `/bulk-lookup` maps each too-short string to `[]` so
  one short string does not fail the whole batch.

Note: empty-string `/lookup` now returns 422 instead of an empty list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Note the minimum-length requirement and 422 on /lookup, the graceful
[]-per-key behavior on /bulk-lookup, and the new config block in /status.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gaurav and others added 3 commits August 31, 2026 16:04
# Conflicts:
#	api/server.py
The setting was described in API.md's /status section but missing from the
two places that enumerate every environment variable, so a deployer reading
the configuration guide had no way to discover it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n for

Code review on #279 found three problems with the minimum-length check and one
with the test that covers it.

The empty-string rejection it replaced was unconditional; the length check is
not, so NAMERES_MINIMUM_QUERY_LENGTH=0 -- the obvious way to turn the minimum
off, and what SOLR_TIMEOUT_SECONDS=0 means one setting above it -- let an empty
query through to Solr as `"" OR ()`. That is a parse error, so an empty search
box came back as an HTTP 500. The check now floors at 1 independently of the
setting.

The minimum is justified by the cost of a short query in the default tokenized
search, where a one-character string matches a prefix of half the index. Exact
mode has no such cost -- it is a single filter query against an untokenized
field -- and single-character labels are real (the gene T, the element
symbols), so it is now held to nothing but non-emptiness.

A too-short query was reported with HTTPException(422), whose body is
{"detail": "<string>"} where FastAPI's own validation returns
{"detail": [...]}, so a client iterating `detail` as a list of errors broke on
this one rejection. It now raises RequestValidationError instead. The
hand-written `responses={422: ...}` on both /lookup operations went with it:
FastAPI only generates the HTTPValidationError body for a 422 the operation has
not declared itself, so declaring one cost the endpoint its schema and left
client generators with an untyped error.

Finally, the /status assertion pinned both the default and the exact key set of
the config block, so it failed for anyone setting the very override this PR
adds. It now compares against the running config.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gaurav
gaurav merged commit cbdf0dd into main Sep 1, 2026
1 check passed
@gaurav
gaurav deleted the set-min-size branch September 1, 2026 02:55
@github-project-automation github-project-automation Bot moved this from Backlog to Done in NameRes sprints Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant