Skip to content

geaflow-ai: embedding vectors are never invalidated when an entity's value changes #844

Description

@E2ern1ty

Describe the bug

EmbeddingIndexStore keys each persisted vector by ModelUtils.getGraphEntityKey, which is a prefix, the id and the label, with nothing derived from the value that was embedded:

return Constants.PREFIX_V + ((GraphVertex) entity).getVertex().getId() + entity.getLabel();

initStore treats the presence of that key in the index file as "this entity is indexed" and skips it. So when an entity keeps its id but its value changes, the vector for the old value stays and is never refreshed. Nothing detects it: no request is issued, no warning is logged, and the record on disk is unchanged.

The consequence is not only a stale vector, it is a swap in both directions. Retrieval for the old value still reaches the entity, while the entity's current value has no vector anywhere and cannot be reached at all. Since the subgraph handed to the model is verbalized from the current graph, a query about the old value returns a context containing the new value. This is the "old and new facts switched" case raised in #832 review: the reason a chunk was retrieved and the content the model is then shown disagree, which is exactly the situation a fact-checking or provenance layer cannot detect from the outside.

Opened at the reviewer's suggestion on #832. This predates and is independent of that PR, which only changed which values are considered embeddable.

Expected behavior

A vector should be invalidated when the text it was produced from changes. An entity whose value changed should be re-embedded on the next initStore, and retrieval should reflect the current value rather than a previous one.

Additional context

Reproduction on master (3f73eb55), no key and no network needed, using a local OpenAI-compatible /v1/embeddings endpoint that records what it is asked to embed. One vertex v1, value changed between two runs, same index file:

requests issued index file lines vectors held by v1
run 1, value the master said learning without thought is labour lost 1 1 1
run 2, same id, value the stable burned down and nobody asked about the horse 0 1 1

The vector v1 holds after run 2 is the one produced from the value that is no longer there:

cosine(stored vector, embedding of the old value) = 1.0000
cosine(stored vector, embedding of the current value) = 0.2357

and recall through GraphMemoryServer, with an embedding vector only and no other index store registered, goes the wrong way on both queries:

query entity returned
the old value, which is no longer in the graph yes
the current value no

The first row is a false positive whose verbalized context shows the current value; the second is the current value being unreachable.

Secondary observation from the same run. A record whose entity has been deleted from the graph is never pruned. initStore drops it while rebuilding, since key2EntityMap has no such key, but the line stays in the file:

requests issued index file lines
run 3, v1 removed from the graph 0 1

So the file only ever grows, and it keeps vectors for values that no longer exist. Harmless for recall, since orphan records are not loaded, but it means the file size does not track the graph and a later entity reusing the same id would silently adopt the old vector.

Possible directions, not a proposal yet. The minimal change is to put a hash of the embedded text into the record and compare it during the rebuild, treating a mismatch as not indexed, which turns both symptoms into a re-embed. That leaves the index file format changed, so it needs a decision about existing files: records without a hash could be treated as unknown and re-embedded once, or as valid until the entity is next touched. Pruning orphan records wants a rewrite of the file rather than an append, which is a separate piece of work from invalidation and may be better done as compaction.

Happy to work on it once there is agreement on the record format and on how existing index files should be treated. I have the reproduction above as a runnable harness and can turn it into a test either way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions