fix(#56): hide documents via query_enabled instead of Marqo purge

[nexus69420]

Summary

• Closes Chunk soft-disable, done properly: add query_enabled filter field before more indexes are created #56. Hide/show is a query_enabled flag flip, not a Marqo purge (except chunk hard Delete). Search ANDs query_enabled:true. Excluded chunks are still ingested so Include on does not need reingest.
• Doc Include off and Doc Delete are the same on search: both set Marqo query_enabled:false and exclude all SQLite chunks. Records stay in the index. They differ on the list: Delete also sets is_disabled, hides the card, and can cancel Temporal. Restore only unhides the card; Include on brings search back. Chunk Delete remains the only hard content delete.
• Cutover: create a new index with the query_enabled field and full reingest. Chat/voice (outside this repo) must filter query_enabled:true. Intentionally dropped Chunk soft-disable, done properly: add query_enabled filter field before more indexes are created #56's 409-on-missing-field (recreate instead). Privilege split (Review chunk vs Admin doc) left as-is. purge_artifacts from P1 Data lifecycle: prevent orphan artifacts/rows on soft-delete and hard-delete #135 is unchanged.

Test plan

• pytest tests/test_document_lifecycle.py tests/test_marqo_doc_scope.py tests/test_search_service.py tests/test_vector_store.py tests/test_tenant_isolation.py tests/test_db.py tests/test_activities.py (182 passed locally; the remaining failure is untracked scripts/_h100_reconcile_v2_marqo.py leaking a Marqo client, not this PR)
• Include off: card stays, search misses the doc, Marqo records remain with query_enabled:false
• Include on: search returns again without reingest
• Delete: card hidden, same search hide as Include off; Restore unhides card only; Include on restores search
• Chunk Include flips one record; chunk Delete still hard-deletes that Marqo/SQLite row
• Re-chunk carries is_excluded by unchanged original_text; reingest uses include_excluded=True
• 502 on a recorded index blocks the SQLite flip; historical indexes stay indexed (not removed)

[KDwevedi]

Please address these blockers before merge:

1. MarqoStore.set_query_enabled() discards the response from update_documents() and always reports every requested ID as updated. Marqo can return a successful HTTP response with errors: true and per-item failures; in that case the route proceeds to flip SQLite even though search visibility did not change. Please inspect the response, count only confirmed successes, and surface any failed items so the SQLite mutation is not performed.

2. apply_document_query_enabled() mutates recorded indexes sequentially. If an early index succeeds and a later index fails, the helper raises before SQLite changes, but the earlier index remains flipped. This is especially unsafe on re-enable because a historical index can become searchable while the document remains disabled in SQLite. Please make failure handling restore already-mutated indexes (or otherwise preserve a consistent per-index/SQLite state) and add a regression where the later sorted index fails.

There is also an acceptance/scope mismatch to resolve explicitly: this PR says it closes #56 while intentionally dropping the required 409 for indexes without query_enabled and leaving the duplicate REVIEW/ADMIN privilege surface unchanged. Please either implement those criteria or update #56 and get agreement on the revised contract. The coordinated index recreation and chat/voice query_enabled:true rollout should remain a merge/deploy prerequisite.

[nexus69420]

Addressed the review blockers.

1. MarqoStore.set_query_enabled() now unwraps the update_documents payload. Only status-200 items count as updated; errors: true with per-item failures (or no items) is reported in failed and does not flip SQLite.
2. apply_document_query_enabled() records indexes already flipped and restores them (opposite flag) if a later sorted index fails. Regression: test_later_index_failure_restores_earlier_index.
3. 409 when a target index schema has no query_enabled (never falls back to delete). Partial flips in the same call are restored first.

Privilege: chunk is_excluded stays review (PATCH .../chunks/{n}). Document Include / Delete stay admin lifecycle, same as before this PR — not a second chunk-toggle route. Commented on #56 with that contract.

Chat/voice query_enabled:true filter and coordinated index recreate remain a merge/deploy prerequisite.

[KDwevedi]

The substantive review blockers are addressed: per-item Marqo failures now fail closed, earlier indexes are compensated on a later failure, the missing-schema path returns 409, and the privilege contract is documented.

One branch-specific suite regression remains. The full suite now fails test_disable_document_hides_in_own_tenant_index_not_legacy: its fake tenant index does not advertise query_enabled, so the route correctly returns the new 409 instead of exercising the tenant-index isolation assertion. Please update that non-legacy fake/schema fixture to include the new filterable field while keeping the legacy fake without it, then rerun the full suite.

Current full result: 879 passed, 2 failed; the other failure is the existing taxonomy-console assertion that also fails on main.

[nexus69420]

The isolation fake now advertises query_enabled on non-legacy indexes (alongside instance). Indexes in _LEGACY_INDEXES still omit both fields, so hide/include 409s there instead of deleting.

test_disable_document_hides_in_own_tenant_index_not_legacy passes again. Full local suite: isolation + lifecycle + marqo-doc-scope green; remaining failures are the taxonomy-console assertions that also fail on main, plus a local untracked ops script tripping the Marqo-boundary scan (not in this PR).

Also merged upstream/main (includes #140).