fix(search): match non-contiguous underscore segments

[neylwalecki]

Pull Request

Description

Fixes #1290.

retrieve_tools can now use non-contiguous underscore-delimited segments as an additional search signal. For example, work_upload_attachment finds work_start_task_attachment_upload, and segment order is not significant.

The implementation:

• Requires every query segment to match a complete tool_name segment boundary.
• Keeps the existing keyword field and does not require an index migration.
• Bounds identifier queries to 16 segments to limit wildcard expansion.
• Preserves the legacy query and exact-name scores, only repeating the search with the segment signal when there is no canonical exact-name hit.
• Keeps canonical tool IDs unchanged and does not add aliases or duplicate tools.

Testing

• I have tested these changes locally
• I have added/updated tests that prove my fix is effective or my feature works
• All existing tests pass

Validated locally with:

go test -race ./internal/index -count=1
go test -short -race -count=1 -timeout 20m -skip "Binary|MCP|E2E|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint" ./internal/...
make build
go build ./cmd/mcpproxy ./cmd/mcpproxy-tray
git diff --check

All applicable GitHub Actions checks passed, including unit, shuffled unit, retrieval/security regression, integration, E2E, lint, CodeQL, server, Docker, and cross-platform binary/logging jobs. Stress tests were skipped by the workflow because this PR is still a draft.

The repository's scripts/run-all-tests.sh uses a 5-minute package timeout and timed out locally in internal/server after the preceding tests passed. The workflow-equivalent -short -race command above uses the documented 20-minute timeout and completed successfully; internal/server finished locally in 373.736 seconds.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 82.60870% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/index/bleve.go	82.60%	4 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

[Dumbris]

Thanks for the thorough writeup and repro, @neylwalecki — nice catch, and I appreciate you laying out the two implementation options up front.

On the approach: Option 2 is the right call for this codebase. mcpproxy doesn't have index-rebuild/migration infrastructure today, so Option 1's "safe rebuild path" would be real unbuilt work, not a shortcut — and Option 2 doesn't foreclose a normalized field later if we ever need one.

On the PR itself: the core fix is solid — verified the segment-AND logic against reorder, negative (missing-segment), and boundary-vs-substring cases (e.g. network_work_upload_attachment against a network_upload_attachment decoy correctly returns empty). Query-time cost is bounded (16-segment cap) and doesn't touch the persisted mapping.

Two things I'd want addressed before merge:

1. hasExactToolName only checks tool_name, not full_tool_name. A canonical server:tool exact query can still trigger the new boosted second pass — which is exactly the kind of reordering the tool-identity scoring (Spec 105) is meant to prevent. Worth checking both fields.
2. The segment boost is aggressive (up to ~20+ for a 4-segment match vs. a prior max clause boost of ~5.0/2.5) and I don't see it run against the frozen tool-surface golden tests (byte-identity/full-golden-set tests over the existing tool corpus). Could you run/attach those, or add a regression showing the boost doesn't reorder an existing better match?

No concerns with Spec 032 (quarantine hashing) or Spec 085 (describe_tool/compact) — neither is touched by this change.

[Dumbris]

@neylwalecki could you take a pass at the two items above before we take this out of draft — check full_tool_name too in the exact-match gate, and re-run (or attach) the frozen golden tests against the new score boost? Happy to merge once those are green.

[neylwalecki]

Addressed both review items in 29e07229:

• The exact-match gate now checks both tool_name and full_tool_name before adding the segment-aware second pass.
• Added a ranking regression showing an existing legacy prefix match remains ahead of the new non-contiguous segment match.
• Added focused coverage for exact matches through either stored name field.

Fresh local validation:

go test -race ./internal/index -count=1
ok github.com/smart-mcp-proxy/mcpproxy-go/internal/index

go test ./bench -run 'TestArmIndex_GoldenSetCoversCorpusV2|TestArmIndex_BaselineGoldenParity' -count=1 -v
47 queries: Recall@5=0.6809, MRR=0.5680, nDCG@10=0.6090, MAP=0.5426

The focused full_tool_name test was also checked red/green by temporarily removing that field from the gate: the dedicated case failed, then passed after restoring the fix.

I'll keep the PR in draft until the refreshed CI completes.

[Dumbris]

Merged — thanks for the clean fix and the fast turnaround on the review feedback, @neylwalecki! 🎉