Skip to content

fix(llamaindex): observe per-run FunctionAgent tool requests - #160

Open
noah-ing wants to merge 1 commit into
agentrust-io:mainfrom
noah-ing:fix/llamaindex-workflow-interop
Open

fix(llamaindex): observe per-run FunctionAgent tool requests#160
noah-ing wants to merge 1 commit into
agentrust-io:mainfrom
noah-ing:fix/llamaindex-workflow-interop

Conversation

@noah-ing

@noah-ing noah-ing commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

What

  • Add explicit per-run TraceEventHandler.observe_workflow() for modern LlamaIndex FunctionAgent tool requests, preserving legacy instrumentation support.
  • Record request names and SHA-256 call-ID fingerprints only; exclude arguments/results, reject mixed tool-event sources before appending, and require explicit per-run model identity even on no-tool workflows.
  • Add pinned released-framework CI/nox coverage, execute the README example in a test, and update the framework matrix and generated Marketplace description.

Why

On main (1b6cc102), a normal released FunctionAgent executing a local add(2, 3) returned 5, but the documented global instrumentation Bridge recorded zero tool calls and the record omitted tool_transcript. The runner emits ToolCall / ToolCallResult on its workflow stream, whereas the adapter consumed only legacy instrumentation AgentToolCallEvent.

The original mismatch remains an executable regression beside the corrected path. This extends the real-framework testing approach from #124; it does not reopen or claim to close the broader adoption issue.

Evidence boundary

ToolCall is emitted before tool lookup/execution. The transcript counts observed requests, not successful invocations. Tests explicitly demonstrate unknown tools (one request, no invocation), tool exceptions, cancellation, and separate concurrent runs. Result events do not double-count requests. Model-supplied IDs are fingerprinted, not authenticated or assumed unique; tool names remain visible metadata.

Records remain first-party, software-only, enforcement_mode: declared, and appraisal.status: none. The real signed workflow record verifies against the supplied public key and passes released TRACE Level 0 conformance without changing that ceiling. No claims of attested execution, graph-native provenance, policy enforcement, exhaustive capture, or exactly-once execution.

Verification

Repeated from a clean checkout of 70d3bfa889c07e546c48dc6f15ff8629b323fb07 with fresh nox environments on Python 3.12.14:

  • nox -s framework_adapters: 107 passed (86 framework-free cases, 1 LangGraph, 12 real LlamaIndex, 8 Pydantic AI), zero skips/failures.
  • python -m pytest tests -q: 24 passed. Separate focused LlamaIndex run: 61 passed, including the 12 real-framework cases.
  • Repository Ruff gate; focused Ruff/format; strict mypy on the changed adapter; Bandit; workflow YAML/action-SHA checks; local README links; git diff --check: pass.
  • Manifest and compatibility validators: 35 integrations, zero failures. Generated integration index and Marketplace checks: pass.
  • Gitleaks on the staged patch and clean integration directory; detect-secrets on the changed public content: zero findings.
  • Separate AI-assisted read-only source review: resolved the global-model-identity fallback finding; no remaining source findings.

The framework tests use released LlamaIndex Core 0.14.24, Workflows 2.23.3, Instrumentation 0.6.0, TRACE 0.9.0 and conformance suite 0.5.1. MockFunctionCallingLLM supplies scripted local responses; the workflow, event delivery, and local tools are real. Socket connections/DNS are blocked during tests. No credentials or provider calls are needed. Existing upstream Pydantic inspection deprecation warnings remain visible.

This adapter is a standalone integration file, not part of the agentrust-trace-adapters wheel; no wheel-packaging claim is made.

Dependency-audit limitation

pip-audit is not clean: the tested environment has one unresolved finding in LlamaIndex's transitive nltk==3.10.3, GHSA-8mgp-746c-j5xp. The advisory lists no patched release at verification time. The affected NLTK model-file APIs are not exercised by these tests. The finding is documented in the README, not suppressed. The test runner is pinned to patched pytest==9.1.1.

The commit includes the DCO sign-off.

Consume the released workflow stream explicitly while preserving legacy instrumentation. Require per-run model identity, fingerprint model-supplied call IDs, and reject mixed tool sources before transcript mutation.

Exercise real FunctionAgent runs, concurrent streams, failures, cancellation, payload exclusion, signed TRACE Level 0 records, and the README example in pinned CI. Preserve the request-only software evidence boundary and document the unresolved transitive dependency advisory.

Signed-off-by: Noah Ingwers <98993329+noah-ing@users.noreply.github.com>
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant