diff --git a/.github/workflows/trace-adapters-tests.yml b/.github/workflows/trace-adapters-tests.yml index 9da82bf..c7acad5 100644 --- a/.github/workflows/trace-adapters-tests.yml +++ b/.github/workflows/trace-adapters-tests.yml @@ -153,6 +153,20 @@ jobs: pip install "langchain-core==1.6.0" "langgraph==1.2.11" python -m pytest integrations/langchain/test_langgraph_interop.py -q + llamaindex-workflow-adapter: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + - name: Test released FunctionAgent workflow observation + run: | + pip install -r integrations/llamaindex/requirements-interop.txt + python -m pytest integrations/llamaindex -q + openai-agents-adapter: # Two passes, on purpose. The first proves record construction and the # honesty rules work without the SDK installed; the second runs a real diff --git a/README.md b/README.md index ca321b1..afbba22 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ TRACE only works as a standard if it is genuinely neutral. Integrations are list | Google ADK | [Google ADK](integrations/google-adk/) | First-party `BasePlugin` lifecycle | Yes - Google ADK 2.7.1 `InMemoryRunner` | Callback-visible invocation, model, and available tool identity; no payloads, retries, agent graph, function-body execution, or policy enforcement | | LangChain | [LangChain](integrations/langchain/) | First-party `BaseCallbackHandler` callbacks | Yes - LangChain Core 1.6.0 callback contract | Tool identity and outcome plus model identity; no chain topology or runnable state | | LangGraph | [LangChain](integrations/langchain/) | First-party LangChain callbacks propagated by the graph | Yes - LangGraph 1.2.11 `StateGraph` with a nested tool call | Propagated tool callbacks; no nodes, edges, state transitions, checkpoints, or rollback decisions | -| LlamaIndex | [LlamaIndex](integrations/llamaindex/) | First-party `BaseEventHandler` events | No - current tests use representative event objects | Allow-listed tool and model fields; released-framework interoperability remains unverified | +| LlamaIndex | [LlamaIndex](integrations/llamaindex/) | First-party instrumentation or per-run workflow stream | Yes - LlamaIndex Core 0.14.24 `FunctionAgent`, Workflows 2.23.3 | Workflow tool-request names and call-id fingerprints; explicit model identity; no arguments/results, completion claims, graph state, or policy enforcement | | OpenAI Agents SDK | [OpenAI Agents SDK](integrations/openai-agents/) | First-party `TracingProcessor` spans | Yes - OpenAI Agents SDK 0.22.0 scripted model and tool run | Tool, handoff, agent, and MCP identity and order; no payloads, reasoning traces, guardrail outcomes, session state, or retries | | Pydantic AI | [OpenTelemetry GenAI](integrations/otel-genai/) | OpenTelemetry GenAI transcription | Yes - Pydantic AI 2.35.1 `TestModel` with a tool call | Telemetry-reported model and tool identity; no payloads; absent `gen_ai.tool.type` is not inferred | diff --git a/integrations/llamaindex/README.md b/integrations/llamaindex/README.md index e40fa94..47c2649 100644 --- a/integrations/llamaindex/README.md +++ b/integrations/llamaindex/README.md @@ -1,31 +1,98 @@ # LlamaIndex → TRACE -Emits a TRACE v0.2 Trust Record from LlamaIndex instrumentation events. +Builds a first-party TRACE Trust Record from LlamaIndex tool observations. +There are two distinct event routes: legacy instrumentation and the per-run +workflow stream used by modern `FunctionAgent`. A global instrumentation +handler alone does **not** observe `FunctionAgent` tool requests. + +## Modern FunctionAgent workflow + +The released-framework tests exercise `llama-index-core==0.14.24`, +`llama-index-workflows==2.23.3`, and `llama-index-instrumentation==0.6.0`, with +`agentrust-trace==0.9.0` and `agentrust-trace-tests==0.5.1`. Exact test pins live +in [requirements-interop.txt](requirements-interop.txt). + +Dependency-audit limitation: the tested environment resolves LlamaIndex's +transitive NLTK dependency to 3.10.3, affected by +[GHSA-8mgp-746c-j5xp](https://github.com/advisories/GHSA-8mgp-746c-j5xp), +with no patched release listed at verification time. These tests do not use +NLTK's affected model-file APIs. A passing interoperability run is not a clean +dependency-security audit; reassess dependencies for deployment. + +Use a fresh tracker for each run and pass events from that run's stream to +`observe_workflow`. No global dispatcher registration is needed. Supply the +model's identity explicitly from your configured agent; a process-global model +observer could mix identities from concurrent runs. +Once any event is passed to `observe_workflow`, record construction requires +both explicit model fields, even for a run with no tool requests. -## First-party, like the LangChain adapter - -A `BaseEventHandler` runs in the agent's own process, so what it observes is the operator's own agent. The record carries **no `origin` block** — absence means `self`. It does not use [`agentrust-trace-adapters`](../../packages/agentrust-trace-adapters), which exists for *somebody else's* evidence and would mislabel this as a third-party transcription. - -## The risk here is different from LangChain's - -LangChain has a dozen typed callbacks. LlamaIndex has **one method** — `handle(event)` — and several event types carry payloads: - -| Event | Payload it carries | -|---|---| -| `AgentToolCallEvent` | `arguments` | -| `LLMChatStartEvent` | the whole `messages` list | -| `LLMCompletionEndEvent` | `prompt` and `response` | - -One entry point is easier to consume and harder to consume *safely*. So this handler reads an **explicit allow-list of fields** rather than the event object: tool name, event id, span id, and `model_dict` for identity. Nothing else is read, which means a payload-bearing field added upstream in a future version is ignored by default rather than captured. - -Four tests hold that line: an IBAN in tool `arguments`, an IBAN in chat `messages`, an IBAN in a field a later version might add, and an entire unrelated event type carrying prompt and response. +```python +from agentrust_trace.sign import sign_record +from llamaindex_to_trace import TraceEventHandler -## Run it -```bash -pip install agentrust-trace llama-index-core +async def run_with_record( + agent, user_msg, *, subject, policy_bundle, workload_digest, + model_provider, model_id, signing_key, +): + tracker = TraceEventHandler() + handler = agent.run(user_msg=user_msg) + try: + async for event in handler.stream_events(): + tracker.observe_workflow(event) + result = await handler + finally: + if not handler.is_done(): + await handler.cancel_run() + + unsigned = tracker.build_record( + subject=subject, + policy_bundle=policy_bundle, # bytes of the declared policy + workload_digest=workload_digest, # digest of your artifact + model_provider=model_provider, + model_id=model_id, + data_class="internal", + ) + return result, sign_record(unsigned, signing_key) ``` +The caller supplies its own configured agent, identity, policy bytes, artifact +digest, and signing key. The offline tests supply a scripted local model and +real local tools; they need no provider account, API key, or network requests. +The stream has one consumer: if your application already processes it, call +`observe_workflow` in that existing loop rather than starting a second consumer. +The caller owns run cancellation and persistence. The adapter registers no +hooks or background tasks, and does not label partial/cancelled observations +as a successful run. + +### What enters the workflow transcript + +Each `ToolCall` contributes its `tool_name`, a SHA-256 fingerprint of `tool_id` +in the existing `event_id` field, and `span_id: null`. Workflow events do not +supply an instrumentation span. The call ID may be model-supplied, so its raw +text is not retained. Fingerprints support correlation, not authentication or +secrecy against guessing. Tool names remain visible metadata; do not put +sensitive content in names. + +`ToolCallResult` and other events are ignored without reading their payloads. +Arguments, results, prompts, responses, and arbitrary future fields never enter +this transcript. One request plus one result counts once. Distinct requests +with a repeated call ID still count separately: model IDs are not guaranteed +unique. Replaying the stream is outside this observer's contract. + +In the tested framework, `ToolCall` is emitted **before lookup and execution**. +The transcript therefore counts observed requests, including requests whose +tool is unavailable or fails. It does not prove function-body execution, +completion, business success, retry history, graph state, or exhaustive activity. +Only events the caller actually passes to this tracker are observed. + +## Legacy instrumentation + +Existing `AgentToolCallEvent` handling is preserved. `LLMChatStartEvent` and +`LLMCompletionStartEvent` can supply model identity through `model_dict`. +The framework-free tests continue to check this route's explicit field +allow-list. They do not establish support for every legacy LlamaIndex agent. + ```python from llama_index.core.instrumentation import get_dispatcher from llama_index.core.instrumentation.event_handlers import BaseEventHandler @@ -33,32 +100,62 @@ from llamaindex_to_trace import TraceEventHandler tracker = TraceEventHandler() + class Bridge(BaseEventHandler): def handle(self, event, **kwargs): tracker.observe(event) -get_dispatcher().add_event_handler(Bridge()) -# ... run your agent ... -record = tracker.build_record( - subject="spiffe://example.org/agent/index-bot", - policy_bundle=open("policy.cedar", "rb").read(), - # enforcement_mode defaults to "declared"; see below - workload_digest="sha256:...", - data_class="internal", -) +bridge = Bridge() +dispatcher = get_dispatcher() +dispatcher.add_event_handler(bridge) +try: + # Run one legacy agent here, with no unrelated concurrent runs. + ... +finally: + dispatcher.event_handlers.remove(bridge) ``` -## `enforcement_mode` defaults to `declared` +Global instrumentation is not per-run isolation. Do not feed legacy tool events +and workflow tool requests into the same tracker: their identifiers cannot be +reliably deduplicated. The first tool event selects a source; an event from the +other source raises `MissingEvidence` before appending. Unknown event types +are not recorded. Workflow requests with missing/non-string identity fields are +also refused without retaining their values. + +## Evidence boundary and conformance + +These are in-process observations of the operator's own agent. Records have +no `origin` block (self), default to `runtime.platform: software-only`, and +retain `appraisal.status: none`. Signing binds the record to its signing key; +it does not attest the observer, authenticate model-supplied tool identity, +prove safe behavior, or establish hardware provenance or runtime integrity. + +`policy.enforcement_mode` defaults to `declared`: the caller's policy is named +and hashed, but LlamaIndex has not evaluated or enforced it. Supplying another +mode requires an actual external policy layer. Supplied attestation fields are +passed through by the existing record builder; this adapter does not verify +them or independently establish Level 1 assurance. -**LlamaIndex enforces no policy.** `enforce`, `advisory` and `silent` all presuppose that something *evaluated* the policy; for a bare run, nothing did. TRACE 0.9.0 added `declared` for that case, so the default is now truthful rather than the closest available overstatement. Needs `agentrust-trace>=0.9`. +The real `FunctionAgent` tests sign and verify records and pass the released +TRACE Level 0 conformance checks with the honest `declared` mode. Level 0 +conformance does not raise this software-only evidence boundary. -## Conformance +## Reproduce -**Level 0** without an attestation, **Level 1** with one. +From the repository root in a fresh virtual environment: ```bash -python -m pytest test_llamaindex_to_trace.py -q +pip install pytest==9.1.1 agentrust-trace==0.9.0 +python -m pytest integrations/llamaindex/test_llamaindex_to_trace.py -q +pip install -r integrations/llamaindex/requirements-interop.txt +python -m pytest integrations/llamaindex -q ``` -20 tests. +Or run `nox -s framework_adapters`, which preserves the framework-free pass +and then installs the pinned released frameworks. CI runs both routes too. +The real workflow regression is [test_llamaindex_interop.py](test_llamaindex_interop.py); +it uses the released runner and a local scripted `MockFunctionCallingLLM`, not +hand-constructed stand-ins for workflow delivery. Tests cover streaming and +non-streaming model responses, request order, error and no-tool paths, +concurrent run isolation, payload exclusion, and signed record validation. diff --git a/integrations/llamaindex/integration.yaml b/integrations/llamaindex/integration.yaml index 25e09bb..cd18567 100644 --- a/integrations/llamaindex/integration.yaml +++ b/integrations/llamaindex/integration.yaml @@ -2,7 +2,7 @@ name: LlamaIndex vendor: agentrust-io integrates_with: - trace -description: Emits a TRACE Trust Record from LlamaIndex instrumentation events, reading an allow-list so payloads cannot leak. +description: Emits a TRACE Trust Record from LlamaIndex instrumentation or per-run workflow tool requests, excluding argument and result fields. maintainer: github: imran-siddique repository: https://github.com/agentrust-io/integrations @@ -16,3 +16,6 @@ marketplace: trace_roles: - record-producer trace_conformance_level: 0 +tested_against: + agentrust-trace: "0.9.0" + agentrust-trace-tests: "0.5.1" diff --git a/integrations/llamaindex/llamaindex_to_trace.py b/integrations/llamaindex/llamaindex_to_trace.py index cf276f6..e710a33 100644 --- a/integrations/llamaindex/llamaindex_to_trace.py +++ b/integrations/llamaindex/llamaindex_to_trace.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""LlamaIndex instrumentation events -> TRACE v0.2 Trust Record. +"""LlamaIndex instrumentation or workflow events -> TRACE v0.2 Trust Record. Same shape as the LangChain adapter and for the same reason: a ``BaseEventHandler`` runs in the agent's own process, so what it sees is @@ -22,6 +22,11 @@ AgentToolCallEvent -> tool identity LLMChatStartEvent -> model identity from ``model_dict`` LLMCompletionStartEvent -> same + +Modern ``FunctionAgent`` tool requests arrive on its per-run workflow stream, +not as ``AgentToolCallEvent`` instrumentation. Pass those events explicitly to +``observe_workflow``. Only ``ToolCall`` name and a fingerprint of its call id +are retained; observing a request does not establish execution or success. """ from __future__ import annotations @@ -44,7 +49,14 @@ #: ignored rather than filtered, so a new payload-bearing field cannot leak by #: being added upstream. TOOL_FIELDS = ("id_", "span_id") -PAYLOAD_FIELDS_NOT_READ = ("arguments", "messages", "prompt", "response", "output", "template_args") +PAYLOAD_FIELDS_NOT_READ = ( + "arguments", + "messages", + "prompt", + "response", + "output", + "template_args", +) class MissingEvidence(ValueError): @@ -85,7 +97,16 @@ def _model_from_dict(model_dict: Any) -> tuple[str | None, str | None]: model = model_dict.get("model") or model_dict.get("model_name") cls = str(model_dict.get("class_name", "")).lower() provider = None - for known in ("anthropic", "openai", "bedrock", "vertex", "azure", "ollama", "mistral", "gemini"): + for known in ( + "anthropic", + "openai", + "bedrock", + "vertex", + "azure", + "ollama", + "mistral", + "gemini", + ): if known in cls: provider = known break @@ -102,6 +123,8 @@ class TraceEventHandler: def __init__(self) -> None: self._observed = _Observed() + self._tool_source: str | None = None + self._workflow_observed = False def handle(self, event: Any, **kwargs: Any) -> None: """The ``BaseEventHandler`` surface.""" @@ -110,6 +133,7 @@ def handle(self, event: Any, **kwargs: Any) -> None: def observe(self, event: Any) -> None: name = getattr(type(event), "class_name", lambda: type(event).__name__)() if name == "AgentToolCallEvent": + self._select_tool_source("instrumentation") self._observed.tools.append( ToolCall( name=_tool_name(event), @@ -122,14 +146,60 @@ def observe(self, event: Any) -> None: self._observed.provider = self._observed.provider or provider self._observed.model_id = self._observed.model_id or model + def observe_workflow(self, event: Any) -> None: + """Observe one run's workflow stream without registering global hooks. + + ``ToolCall`` precedes tool lookup/execution. ``ToolCallResult`` and all + other events are ignored, so results cannot double-count requests or + leak outputs. Model identity must be supplied separately by the caller. + The model-supplied tool id is fingerprinted rather than copied into the + transcript. Neither that fingerprint nor the name is authenticated. + + Use one fresh tracker per stream. Do not mix workflow and legacy tool + events: their identifiers do not support reliable cross-source dedup. + """ + # Even a no-tool workflow must not inherit identity from process-global + # LLM instrumentation. Calling this API opts into explicit run identity. + self._workflow_observed = True + if type(event).__name__ != "ToolCall": + return + name = getattr(event, "tool_name", None) + tool_id = getattr(event, "tool_id", None) + if ( + not isinstance(name, str) + or not name + or not isinstance(tool_id, str) + or not tool_id + ): + raise MissingEvidence( + "workflow ToolCall requires non-empty string tool_name and tool_id" + ) + call = ToolCall(name=name, event_id=_digest(tool_id.encode()), span_id=None) + self._select_tool_source("workflow") + self._observed.tools.append(call) + + def _select_tool_source(self, source: str) -> None: + if self._tool_source is not None and self._tool_source != source: + raise MissingEvidence( + "use separate trackers for workflow and instrumentation tool events" + ) + self._tool_source = source + @property def tool_calls(self) -> list[ToolCall]: return list(self._observed.tools) def transcript_bytes(self) -> bytes: - """Tool identity only: name, event id, span id. No arguments, ever.""" + """Tool identity only; workflow event ids are call-id fingerprints. + + Workflow entries describe requests, not successful handler invocations. + No arguments or results are read into this transcript. + """ return json.dumps( - [{"tool": c.name, "event_id": c.event_id, "span_id": c.span_id} for c in self._observed.tools], + [ + {"tool": c.name, "event_id": c.event_id, "span_id": c.span_id} + for c in self._observed.tools + ], sort_keys=True, separators=(",", ":"), ).encode() @@ -147,6 +217,10 @@ def build_record( attestation: dict[str, str] | None = None, iat: int | None = None, ) -> dict[str, Any]: + if self._workflow_observed and (not model_provider or not model_id): + raise MissingEvidence( + "workflow records require explicit model_provider and model_id" + ) return build_record( subject=subject, policy_bundle=policy_bundle, @@ -233,7 +307,9 @@ def build_record( else: runtime = { "platform": "software-only", - "measurement": _digest(workload_digest.encode() + b"\n" + _digest(policy_bundle).encode()), + "measurement": _digest( + workload_digest.encode() + b"\n" + _digest(policy_bundle).encode() + ), } record: dict[str, Any] = { @@ -242,11 +318,17 @@ def build_record( "subject": subject, "model": {"provider": model_provider, "model_id": model_id}, "runtime": runtime, - "policy": {"bundle_hash": _digest(policy_bundle), "enforcement_mode": enforcement_mode}, + "policy": { + "bundle_hash": _digest(policy_bundle), + "enforcement_mode": enforcement_mode, + }, "data_class": data_class, "build_provenance": {"slsa_level": 0, "digest": workload_digest}, "appraisal": {"status": "none", "verifier": "llamaindex-adapter"}, } if tool_count: - record["tool_transcript"] = {"hash": _digest(transcript), "call_count": tool_count} + record["tool_transcript"] = { + "hash": _digest(transcript), + "call_count": tool_count, + } return record diff --git a/integrations/llamaindex/requirements-interop.txt b/integrations/llamaindex/requirements-interop.txt new file mode 100644 index 0000000..2d75581 --- /dev/null +++ b/integrations/llamaindex/requirements-interop.txt @@ -0,0 +1,8 @@ +# Exact released surfaces exercised by test_llamaindex_interop.py. +# Keep the framework-free evidence tests separate from this install. +pytest==9.1.1 +agentrust-trace==0.9.0 +agentrust-trace-tests==0.5.1 +llama-index-core==0.14.24 +llama-index-workflows==2.23.3 +llama-index-instrumentation==0.6.0 diff --git a/integrations/llamaindex/test_llamaindex_interop.py b/integrations/llamaindex/test_llamaindex_interop.py new file mode 100644 index 0000000..4a01f92 --- /dev/null +++ b/integrations/llamaindex/test_llamaindex_interop.py @@ -0,0 +1,531 @@ +"""Released LlamaIndex FunctionAgent workflow interoperability. + +The real workflow executes local tools using the SDK's shipped scripted model. +The observer consumes the real per-run stream, never fabricated events. CI pins +llama-index-core, workflows and instrumentation; no provider API is required. +ToolCall is a request emitted before dispatch, not a proof of successful work. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import pathlib +import socket +import sys +from contextvars import ContextVar + +import pytest +from agentrust_trace.models import TrustRecord +from agentrust_trace.sign import generate_key, sign_record, verify_record +from llama_index.core.agent.workflow import FunctionAgent +from llama_index.core.instrumentation import get_dispatcher +from llama_index.core.instrumentation.event_handlers import BaseEventHandler +from llama_index.core.llms import ChatMessage +from llama_index.core.llms.mock import MockFunctionCallingLLM +from llama_index.core.tools import ToolSelection +from trace_tests import runner as conformance_runner +from trace_tests.result import Status +from workflows.errors import WorkflowCancelledByUser + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) + +from llamaindex_to_trace import TraceEventHandler # noqa: E402 + +PAYLOAD = "private-customer-reference-must-not-enter-trace" +MODEL_OUTPUT = "private-model-output-must-not-enter-trace" +SUBJECT = "spiffe://example.org/agent/llamaindex-interop" + + +@pytest.fixture(autouse=True) +def no_network(monkeypatch): + """Model/provider calls and accidental telemetry are outside this test.""" + + def refuse(*args, **kwargs): + raise AssertionError("the released-framework test must remain offline") + + # Windows builds asyncio's internal socket pair using loopback TCP. + # Exempt only that synchronous construction, in this execution context. + # Ordinary loopback connections and connections on other threads stay blocked. + creating_pair = ContextVar("creating_socketpair", default=False) + original_pair = socket.socketpair + original_connect = socket.socket.connect + + def local_pair(*args, **kwargs): + token = creating_pair.set(True) + try: + return original_pair(*args, **kwargs) + finally: + creating_pair.reset(token) + + def guarded_connect(sock, address): + if ( + creating_pair.get() + and isinstance(address, tuple) + and address[0] in ("127.0.0.1", "::1") + ): + return original_connect(sock, address) + refuse() + + monkeypatch.setattr(socket, "socketpair", local_pair) + monkeypatch.setattr(socket.socket, "connect", guarded_connect) + monkeypatch.setattr(socket, "getaddrinfo", refuse) + + +def test_offline_guard_allows_socketpair_but_blocks_connections(): + left, right = socket.socketpair() + with left, right: + right.settimeout(1) + left.sendall(b"local") + assert right.recv(5) == b"local" + for host in ("127.0.0.1", "192.0.2.1"): + with socket.socket() as sock: + with pytest.raises(AssertionError, match="must remain offline"): + sock.connect((host, 9)) + with pytest.raises(AssertionError, match="must remain offline"): + socket.getaddrinfo("example.invalid", 443) + + +def model_for_calls(calls): + responses = [ + ChatMessage( + role="assistant", + content=MODEL_OUTPUT, + additional_kwargs={ + "tool_calls": [ + ToolSelection( + tool_id=call_id, + tool_name=name, + tool_kwargs=arguments, + ) + ] + }, + ) + for name, call_id, arguments in calls + ] + responses.append(ChatMessage(role="assistant", content=MODEL_OUTPUT)) + sequence = iter(responses) + return MockFunctionCallingLLM( + response_generator=lambda messages, **kwargs: next(sequence), + ) + + +async def observe_run(agent, tracker): + handler = agent.run(user_msg=PAYLOAD) + events = [] + try: + async for event in handler.stream_events(): + events.append(event) + tracker.observe_workflow(event) + return await handler, events + finally: + if not handler.is_done(): + await handler.cancel_run() + + +def signed_record(tracker, *, subject=SUBJECT): + key = generate_key() + signed = sign_record( + tracker.build_record( + subject=subject, + policy_bundle=b'{"declared-policy":true}', + workload_digest="sha256:" + "a" * 64, + data_class="internal", + model_provider="local-test-double", + model_id="MockFunctionCallingLLM", + ), + key, + ) + verify_record(signed, key.public_key()) + TrustRecord.model_validate(signed) + return signed + + +@pytest.mark.parametrize("streaming", [False, True]) +def test_real_function_agent_tool_stream_produces_signed_record(streaming): + invocations = [] + + def measure_payload(payload: str) -> str: + """Measure one input in a local tool.""" + invocations.append(payload) + return f"received {payload}" + + tracker = TraceEventHandler() + agent = FunctionAgent( + tools=[measure_payload], + llm=model_for_calls([("measure_payload", "call-1", {"payload": PAYLOAD})]), + streaming=streaming, + timeout=15, + ) + result, events = asyncio.run(observe_run(agent, tracker)) + + assert invocations == [PAYLOAD] + assert result.response.content == MODEL_OUTPUT + assert [type(event).__name__ for event in events].count("ToolCall") == 1 + assert [type(event).__name__ for event in events].count("ToolCallResult") == 1 + assert len(tracker.tool_calls) == 1 + call = tracker.tool_calls[0] + assert call.name == "measure_payload" + assert call.event_id == "sha256:" + hashlib.sha256(b"call-1").hexdigest() + assert call.span_id is None + + transcript = tracker.transcript_bytes() + signed = signed_record(tracker) + assert signed["tool_transcript"]["call_count"] == 1 + assert ( + signed["tool_transcript"]["hash"] + == "sha256:" + hashlib.sha256(transcript).hexdigest() + ) + assert signed["runtime"]["platform"] == "software-only" + assert signed["policy"]["enforcement_mode"] == "declared" + assert signed["appraisal"]["status"] == "none" + assert "origin" not in signed + assert "transparency" not in signed + assert PAYLOAD not in transcript.decode() + assert MODEL_OUTPUT not in transcript.decode() + assert PAYLOAD not in json.dumps(signed) + assert MODEL_OUTPUT not in json.dumps(signed) + # The framework really did carry both request and result payloads. + request = next(event for event in events if type(event).__name__ == "ToolCall") + outcome = next( + event for event in events if type(event).__name__ == "ToolCallResult" + ) + assert request.tool_kwargs["payload"] == PAYLOAD + assert PAYLOAD in outcome.tool_output.content + + +def test_real_function_agent_records_sequential_requests_once_in_order(): + invocations = [] + + def visit(label: str) -> str: + """Record a local visit.""" + invocations.append(label) + return label + + tracker = TraceEventHandler() + agent = FunctionAgent( + tools=[visit], + llm=model_for_calls( + [ + ("visit", "first", {"label": "one"}), + ("visit", "second", {"label": "two"}), + ] + ), + streaming=False, + timeout=15, + ) + asyncio.run(observe_run(agent, tracker)) + assert invocations == ["one", "two"] + assert [call.event_id for call in tracker.tool_calls] == [ + "sha256:" + hashlib.sha256(value.encode()).hexdigest() + for value in ["first", "second"] + ] + assert signed_record(tracker)["tool_transcript"]["call_count"] == 2 + + +def test_real_function_agent_model_only_run_omits_transcript(): + tracker = TraceEventHandler() + agent = FunctionAgent(llm=model_for_calls([]), streaming=False, timeout=15) + result, events = asyncio.run(observe_run(agent, tracker)) + assert result.response.content == MODEL_OUTPUT + assert not any(type(event).__name__ == "ToolCall" for event in events) + assert tracker.tool_calls == [] + assert "tool_transcript" not in signed_record(tracker) + + +def test_real_declared_workflow_passes_released_level_zero_conformance(): + """The released 0.5.1 suite accepts declared without claiming enforcement.""" + + def noop() -> str: + """Return a local response.""" + return "done" + + tracker = TraceEventHandler() + agent = FunctionAgent( + tools=[noop], + llm=model_for_calls([("noop", "call-1", {})]), + streaming=False, + timeout=15, + ) + asyncio.run(observe_run(agent, tracker)) + signed = signed_record(tracker) + findings = [ + finding + for module_findings in conformance_runner.run(signed, "trace", 0).values() + for finding in module_findings + ] + assert signed["policy"]["enforcement_mode"] == "declared" + assert signed["tool_transcript"]["call_count"] == 1 + assert findings + assert all(finding.status == Status.PASS for finding in findings) + assert any(finding.code == "TR-POL-002" for finding in findings) + assert any(finding.code == "TR-SIG-005" for finding in findings) + + +def test_real_workflow_call_identifier_is_fingerprinted_before_retention(): + def noop() -> str: + """Return a local response.""" + return "done" + + tracker = TraceEventHandler() + agent = FunctionAgent( + tools=[noop], + llm=model_for_calls([("noop", PAYLOAD, {})]), + streaming=False, + timeout=15, + ) + asyncio.run(observe_run(agent, tracker)) + assert ( + tracker.tool_calls[0].event_id + == "sha256:" + hashlib.sha256(PAYLOAD.encode()).hexdigest() + ) + assert PAYLOAD not in tracker.transcript_bytes().decode() + assert PAYLOAD not in json.dumps(signed_record(tracker)) + + +def test_concurrent_real_workflows_keep_separate_per_run_transcripts(): + async def scenario(): + started = set() + both_started = asyncio.Event() + invocations = [] + + async def rendezvous(label: str) -> str: + """Overlap two local tool invocations without timing assumptions.""" + started.add(label) + if len(started) == 2: + both_started.set() + await asyncio.wait_for(both_started.wait(), timeout=5) + invocations.append(label) + return label + + trackers = [TraceEventHandler(), TraceEventHandler()] + agents = [ + FunctionAgent( + tools=[rendezvous], + llm=model_for_calls( + [("rendezvous", f"call-{label}", {"label": label})] + ), + streaming=False, + timeout=15, + ) + for label in ["left", "right"] + ] + await asyncio.gather( + *(observe_run(agent, tracker) for agent, tracker in zip(agents, trackers)) + ) + return trackers, invocations + + trackers, invocations = asyncio.run(scenario()) + assert sorted(invocations) == ["left", "right"] + for label, tracker in zip(["left", "right"], trackers): + assert len(tracker.tool_calls) == 1 + assert ( + tracker.tool_calls[0].event_id + == "sha256:" + hashlib.sha256(f"call-{label}".encode()).hexdigest() + ) + signed = signed_record(tracker, subject=f"spiffe://example.org/agent/{label}") + assert signed["tool_transcript"]["call_count"] == 1 + assert trackers[0].transcript_bytes() != trackers[1].transcript_bytes() + + +def test_real_failed_tool_request_does_not_gain_a_success_claim(): + invocations = [] + + def fail_locally(payload: str) -> str: + """Raise a normal local tool error.""" + invocations.append(payload) + raise RuntimeError(f"failed {payload}") + + tracker = TraceEventHandler() + agent = FunctionAgent( + tools=[fail_locally], + llm=model_for_calls([("fail_locally", "failed-call", {"payload": PAYLOAD})]), + streaming=False, + timeout=15, + ) + _, events = asyncio.run(observe_run(agent, tracker)) + outcome = next( + event for event in events if type(event).__name__ == "ToolCallResult" + ) + assert outcome.tool_output.is_error is True + assert invocations == [PAYLOAD] + transcript = json.loads(tracker.transcript_bytes()) + assert len(transcript) == 1 + assert set(transcript[0]) == {"tool", "event_id", "span_id"} + assert PAYLOAD not in json.dumps(signed_record(tracker)) + + +def test_unknown_tool_request_is_observed_without_proving_execution(): + invocations = [] + + def available_tool() -> str: + """Record entry into an available local tool.""" + invocations.append("available_tool") + return "done" + + tracker = TraceEventHandler() + agent = FunctionAgent( + tools=[available_tool], + llm=model_for_calls([("unknown_tool", "unknown-call", {})]), + streaming=False, + timeout=15, + ) + _, events = asyncio.run(observe_run(agent, tracker)) + outcome = next( + event for event in events if type(event).__name__ == "ToolCallResult" + ) + assert outcome.tool_output.is_error is True + assert invocations == [] + assert [call.name for call in tracker.tool_calls] == ["unknown_tool"] + assert signed_record(tracker)["tool_transcript"]["call_count"] == 1 + + +def test_documented_legacy_bridge_does_not_observe_modern_workflow_tool_requests(): + """Keep the original normal-run mismatch visible beside the corrected path.""" + legacy = TraceEventHandler() + workflow = TraceEventHandler() + observed_classes = [] + + class Bridge(BaseEventHandler): + def handle(self, event, **kwargs): + observed_classes.append(event.class_name()) + legacy.observe(event) + + def noop() -> str: + """Return a local response.""" + return "done" + + dispatcher = get_dispatcher() + bridge = Bridge() + dispatcher.add_event_handler(bridge) + try: + agent = FunctionAgent( + tools=[noop], + llm=model_for_calls([("noop", "call-1", {})]), + streaming=False, + timeout=15, + ) + asyncio.run(observe_run(agent, workflow)) + finally: + dispatcher.event_handlers.remove(bridge) + assert "LLMChatStartEvent" in observed_classes + assert "AgentToolCallEvent" not in observed_classes + assert legacy.tool_calls == [] + assert len(workflow.tool_calls) == 1 + + +def test_cancelled_real_workflow_keeps_only_observed_request_then_fresh_run_works(): + async def scenario(): + entered = asyncio.Event() + exited = asyncio.Event() + completed = [] + + async def wait_locally() -> str: + """Wait until this local workflow is cancelled.""" + entered.set() + try: + await asyncio.Event().wait() + completed.append("wait_locally") + return "completed" + finally: + exited.set() + + partial = TraceEventHandler() + agent = FunctionAgent( + tools=[wait_locally], + llm=model_for_calls([("wait_locally", "cancelled-call", {})]), + streaming=False, + timeout=15, + ) + handler = agent.run(user_msg=PAYLOAD) + stream = handler.stream_events() + events = [] + try: + async for event in stream: + events.append(event) + partial.observe_workflow(event) + if type(event).__name__ == "ToolCall": + await asyncio.wait_for(entered.wait(), timeout=5) + await handler.cancel_run() + break + with pytest.raises(WorkflowCancelledByUser): + await handler + assert handler.is_done() + await asyncio.wait_for(exited.wait(), timeout=5) + finally: + await stream.aclose() + if not handler.is_done(): + await handler.cancel_run() + + assert completed == [] + assert not any(type(event).__name__ == "ToolCallResult" for event in events) + assert [call.name for call in partial.tool_calls] == ["wait_locally"] + assert set(json.loads(partial.transcript_bytes())[0]) == { + "tool", + "event_id", + "span_id", + } + + later_invocations = [] + + def finish_locally() -> str: + """Complete a fresh local run after cancellation.""" + later_invocations.append("finish_locally") + return "done" + + fresh = TraceEventHandler() + subsequent = FunctionAgent( + tools=[finish_locally], + llm=model_for_calls([("finish_locally", "fresh-call", {})]), + streaming=False, + timeout=15, + ) + await observe_run(subsequent, fresh) + assert later_invocations == ["finish_locally"] + assert [call.name for call in fresh.tool_calls] == ["finish_locally"] + assert [call.name for call in partial.tool_calls] == ["wait_locally"] + assert signed_record(fresh)["tool_transcript"]["call_count"] == 1 + + asyncio.run(scenario()) + + +def test_readme_workflow_example_runs_verbatim_with_released_agent(): + readme = pathlib.Path(__file__).with_name("README.md").read_text() + snippet = readme.split("```python\n", 1)[1].split("\n```", 1)[0] + namespace = {} + exec(compile(snippet, "README.md:first-python-example", "exec"), namespace) + invocations = [] + + def count_payload(payload: str) -> int: + """Count one local payload.""" + invocations.append(payload) + return len(payload) + + agent = FunctionAgent( + tools=[count_payload], + llm=model_for_calls([("count_payload", "readme-call", {"payload": PAYLOAD})]), + streaming=False, + timeout=15, + ) + key = generate_key() + result, signed = asyncio.run( + namespace["run_with_record"]( + agent, + PAYLOAD, + subject=SUBJECT, + policy_bundle=b'{"declared-policy":true}', + workload_digest="sha256:" + "a" * 64, + model_provider="local-test-double", + model_id="MockFunctionCallingLLM", + signing_key=key, + ) + ) + assert invocations == [PAYLOAD] + assert result.response.content == MODEL_OUTPUT + verify_record(signed, key.public_key()) + parsed = TrustRecord.model_validate(signed) + assert parsed.tool_transcript.call_count == 1 + assert parsed.policy.enforcement_mode == "declared" + assert parsed.runtime.platform == "software-only" + assert PAYLOAD not in json.dumps(signed) diff --git a/integrations/llamaindex/test_llamaindex_to_trace.py b/integrations/llamaindex/test_llamaindex_to_trace.py index 4608b00..d2a22ea 100644 --- a/integrations/llamaindex/test_llamaindex_to_trace.py +++ b/integrations/llamaindex/test_llamaindex_to_trace.py @@ -10,6 +10,7 @@ from __future__ import annotations +import hashlib import pathlib import sys @@ -119,7 +120,11 @@ def test_an_unknown_future_field_is_ignored_not_captured() -> None: def test_unrelated_event_types_are_ignored() -> None: h = TraceEventHandler() - h.handle(_Event("LLMCompletionEndEvent", prompt=f"pay {IBAN}", response="done", id_="e-9")) + h.handle( + _Event( + "LLMCompletionEndEvent", prompt=f"pay {IBAN}", response="done", id_="e-9" + ) + ) assert h.tool_calls == [] assert IBAN not in h.transcript_bytes().decode() @@ -137,13 +142,17 @@ def test_model_is_read_from_model_dict() -> None: def test_caller_overrides_a_guessed_provider() -> None: - record = _handler().build_record(**_kwargs(), model_provider="bedrock", model_id="x") + record = _handler().build_record( + **_kwargs(), model_provider="bedrock", model_id="x" + ) assert record["model"]["provider"] == "bedrock" def test_unnamed_tool_is_labelled_not_dropped() -> None: h = TraceEventHandler() - h.handle(_Event("AgentToolCallEvent", tool=None, arguments="{}", id_="e-1", span_id=None)) + h.handle( + _Event("AgentToolCallEvent", tool=None, arguments="{}", id_="e-1", span_id=None) + ) assert h.tool_calls[0].name == "" @@ -195,7 +204,8 @@ def test_unidentified_model_is_refused() -> None: def test_attestation_may_not_claim_software_only() -> None: with pytest.raises(MissingEvidence, match="attests nothing"): _handler().build_record( - **_kwargs(), attestation={"platform": "software-only", "measurement": DIGEST} + **_kwargs(), + attestation={"platform": "software-only", "measurement": DIGEST}, ) @@ -241,3 +251,128 @@ def test_build_record_is_usable_without_the_handler() -> None: tool_count=0, ) assert record["appraisal"]["status"] == "none" + + +# Workflow unit tests protect the field allow-list. The separate interop suite +# proves these events are actually delivered by a released FunctionAgent. +class ToolCall: + def __init__(self, tool_name="search", tool_id="request-1"): + self.tool_name = tool_name + self.tool_id = tool_id + + @property + def tool_kwargs(self): + raise AssertionError("workflow arguments must not be read") + + @property + def span_id(self): + raise AssertionError("a workflow event does not supply an instrumentation span") + + @property + def future_payload(self): + raise AssertionError("unknown fields must not be read") + + +class ToolCallResult: + def __getattr__(self, name): + raise AssertionError("result fields must not be read") + + +def test_workflow_records_only_request_name_and_fingerprinted_id() -> None: + h = TraceEventHandler() + h.observe_workflow(ToolCall(tool_id=IBAN)) + h.observe_workflow(ToolCallResult()) + assert len(h.tool_calls) == 1 + assert h.tool_calls[0].name == "search" + assert ( + h.tool_calls[0].event_id + == "sha256:" + hashlib.sha256(IBAN.encode()).hexdigest() + ) + assert h.tool_calls[0].span_id is None + assert IBAN not in h.transcript_bytes().decode() + + +def test_workflow_preserves_repeated_requests_even_with_same_id() -> None: + # Model ids are not guaranteed unique. Do not erase a second request by + # treating the call-id fingerprint as an exactly-once execution identifier. + h = TraceEventHandler() + h.observe_workflow(ToolCall()) + h.observe_workflow(ToolCall()) + assert len(h.tool_calls) == 2 + + +@pytest.mark.parametrize("field", ["tool_name", "tool_id"]) +@pytest.mark.parametrize("value", [None, "", 42, [], {"payload": IBAN}]) +def test_workflow_missing_identity_is_refused_without_mutation(field, value) -> None: + h = TraceEventHandler() + event = ToolCall() + setattr(event, field, value) + with pytest.raises(MissingEvidence, match="non-empty string") as exc: + h.observe_workflow(event) + assert IBAN not in str(exc.value) + assert h.tool_calls == [] + # Invalid evidence must not select a source either. + h.observe(_tool_event("search", "legacy-1")) + assert len(h.tool_calls) == 1 + + +@pytest.mark.parametrize("workflow_first", [True, False]) +def test_mixing_tool_sources_is_refused_before_append(workflow_first) -> None: + h = TraceEventHandler() + + def workflow(): + h.observe_workflow(ToolCall()) + + def legacy(): + h.observe(_tool_event("search", "legacy-1")) + + first, second = (workflow, legacy) if workflow_first else (legacy, workflow) + first() + before = h.transcript_bytes() + with pytest.raises(MissingEvidence, match="separate trackers"): + second() + assert h.transcript_bytes() == before + + +def test_irrelevant_workflow_events_do_not_read_fields_or_select_source() -> None: + h = TraceEventHandler() + h.observe_workflow(ToolCallResult()) + h.observe_workflow(_chat_event({"model": IBAN})) + assert h.tool_calls == [] + h.observe(_tool_event("legacy", "e-1")) + assert h.tool_calls[0].name == "legacy" + + +def test_workflow_requests_preserve_order_and_require_model_identity() -> None: + h = TraceEventHandler() + h.observe_workflow(ToolCall("first", "one")) + h.observe_workflow(ToolCall("second", "two")) + assert [c.name for c in h.tool_calls] == ["first", "second"] + with pytest.raises(MissingEvidence, match="explicit model_provider and model_id"): + h.build_record(**_kwargs()) + record = h.build_record(**_kwargs(), model_provider="local", model_id="test-model") + assert record["tool_transcript"]["call_count"] == 2 + + +@pytest.mark.parametrize("model_first", [True, False]) +@pytest.mark.parametrize("has_tool_request", [True, False]) +@pytest.mark.parametrize( + "explicit", [{}, {"model_provider": "local"}, {"model_id": "local"}] +) +def test_workflow_never_inherits_global_model_identity( + model_first, has_tool_request, explicit +): + h = TraceEventHandler() + model_event = _chat_event({"class_name": "OpenAI", "model": "unrelated-model"}) + event = ToolCall() if has_tool_request else ToolCallResult() + if model_first: + h.observe(model_event) + h.observe_workflow(event) + if not model_first: + h.observe(model_event) + with pytest.raises(MissingEvidence, match="explicit model_provider and model_id"): + h.build_record(**_kwargs(), **explicit) + record = h.build_record( + **_kwargs(), model_provider="run-provider", model_id="run-model" + ) + assert record["model"] == {"provider": "run-provider", "model_id": "run-model"} diff --git a/marketplace/catalog.json b/marketplace/catalog.json index bbb874d..482b5cc 100644 --- a/marketplace/catalog.json +++ b/marketplace/catalog.json @@ -74,7 +74,7 @@ "name": "LlamaIndex", "package_name": "LlamaIndex", "vendor": "agentrust-io", - "description": "Emits a TRACE Trust Record from LlamaIndex instrumentation events, reading an allow-list so payloads cannot leak.", + "description": "Emits a TRACE Trust Record from LlamaIndex instrumentation or per-run workflow tool requests, excluding argument and result fields.", "path": "integrations/llamaindex", "url": "https://github.com/agentrust-io/integrations/tree/main/integrations/llamaindex", "homepage": null, diff --git a/noxfile.py b/noxfile.py index ee16eef..cdfbe3d 100644 --- a/noxfile.py +++ b/noxfile.py @@ -46,6 +46,11 @@ def framework_adapters(session: nox.Session) -> None: session.install("langchain-core==1.6.0", "langgraph==1.2.11") pytest(session, "integrations/langchain/test_langgraph_interop.py") + # Modern LlamaIndex agents deliver tool requests on a per-run workflow + # stream, not through the legacy instrumentation event tested above. + session.install("-r", "integrations/llamaindex/requirements-interop.txt") + pytest(session, "integrations/llamaindex/test_llamaindex_interop.py") + # Pydantic AI needs no adapter: it instruments through OpenTelemetry and # emits the GenAI conventions otel-genai already maps. Verified against the # released package rather than asserted, the same treatment LangGraph got.