[LightRAG][Step 3] Add asynchronous KDN indexing and canonical KID lifecycle mapping

[rickisba]

Parent Epic: #162
Depends on: #164
Related infrastructure contracts: #139, #140, #142

Goal

Make KDN the authoritative source for knowledge content and artifact identity while asynchronously maintaining a rebuildable LightRAG index.

A successful KDN registration must not wait for LightRAG entity extraction, graph construction, embedding, or remote model calls.

Required Workflow

KDN register/update/delete
        |
        v
Authoritative KDN Text/Artifact state and canonical KID
        |
        v
Durable or recoverable index task
        |
        v
LightRAG document/chunk index
        |
        v
Index status and canonical KID mapping

Scope

• Preserve the current canonical KID generation and KDN text persistence semantics.
• Define a mapping from one canonical KID to the LightRAG document ID and zero or more internal chunk IDs.
• Ensure LightRAG internal IDs never become data-plane Knowledge IDs.
• Add asynchronous indexing tasks with bounded concurrency and backpressure.
• Define index status vocabulary compatible with the repository state-model direction, covering at least queued, indexing, indexed, failed, stale, deleting, and deleted states.
• Persist or reconstruct enough task/mapping state to recover after KDN restart.
• Propagate knowledge updates and deletes to LightRAG with eventual consistency.
• Provide a full rescan/rebuild operation from KDN authoritative state.
• Track content/index version, content fingerprint, embedding/index fingerprint where available, retry count, timestamps, and structured errors.
• Add KDN health/heartbeat summary fields for indexed, pending, failed items and index version; do not place graph or chunk payloads in heartbeat messages.
• Add a final KDN-existence validation path so stale LightRAG results cannot inject deleted knowledge.

Failure and Consistency Rules

• KDN registration succeeds when authoritative KDN persistence succeeds, even if LightRAG is unavailable.
• Index failures are observable and retryable but never mark the KDN text artifact as absent.
• Delete becomes authoritative in KDN first; LightRAG deletion follows asynchronously.
• Repeated index/delete requests must be idempotent.
• A stale LightRAG candidate must be filtered before Scheduler routing and data-plane injection.
• Queue limits must reject or defer indexing work explicitly rather than grow without bound.

Repository Constraints

• Reuse the v1 state, operation, and protocol vocabulary where available; do not create an incompatible parallel state machine.
• Do not add a new functional root directory.
• Do not move existing KDN directories in this PR unless a separate architecture Issue approves it.
• Keep LightRAG SDK dependencies out of KDN by using the REST client from Step 2.

Non-Goals

• Do not make LightRAG the source of truth.
• Do not store KV bytes or LMCache-private objects in LightRAG.
• Do not alter Proxy Instance selection.
• Do not enable Scheduler production retrieval from LightRAG yet.
• Do not implement knowledge-compute joint scheduling.

Acceptance Criteria

• Registering knowledge returns its canonical KID without waiting for LightRAG.
• One KID can map to multiple LightRAG chunks and all query results can be aggregated back to that KID.
• Update/delete/retry/rebuild operations are idempotent and tested.
• Restart recovery does not silently lose queued or failed indexing state.
• KDN remains usable when LightRAG is down.
• Heartbeat reports bounded index summaries only.
• CPU-only tests use mocked LightRAG calls and require no CUDA or external service.
• An integration smoke test demonstrates KDN registration, asynchronous indexing, canonical mapping, deletion, and stale-result filtering.

Validation

Run focused state/mapping/queue tests, existing KDN registration tests, and relevant v1 compatibility tests. Include actual results in the PR and use Closes #165.