fix(llamaindex): observe per-run FunctionAgent tool requests - #160
Open
noah-ing wants to merge 1 commit into
Open
fix(llamaindex): observe per-run FunctionAgent tool requests#160noah-ing wants to merge 1 commit into
noah-ing wants to merge 1 commit into
Conversation
Consume the released workflow stream explicitly while preserving legacy instrumentation. Require per-run model identity, fingerprint model-supplied call IDs, and reject mixed tool sources before transcript mutation. Exercise real FunctionAgent runs, concurrent streams, failures, cancellation, payload exclusion, signed TRACE Level 0 records, and the README example in pinned CI. Preserve the request-only software evidence boundary and document the unresolved transitive dependency advisory. Signed-off-by: Noah Ingwers <98993329+noah-ing@users.noreply.github.com>
|
🟡 Contributor Check: MEDIUM
Automated check by AgenTrust Contributor Check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TraceEventHandler.observe_workflow()for modern LlamaIndexFunctionAgenttool requests, preserving legacy instrumentation support.Why
On main (
1b6cc102), a normal releasedFunctionAgentexecuting a localadd(2, 3)returned 5, but the documented global instrumentation Bridge recorded zero tool calls and the record omittedtool_transcript. The runner emitsToolCall/ToolCallResulton its workflow stream, whereas the adapter consumed only legacy instrumentationAgentToolCallEvent.The original mismatch remains an executable regression beside the corrected path. This extends the real-framework testing approach from #124; it does not reopen or claim to close the broader adoption issue.
Evidence boundary
ToolCallis emitted before tool lookup/execution. The transcript counts observed requests, not successful invocations. Tests explicitly demonstrate unknown tools (one request, no invocation), tool exceptions, cancellation, and separate concurrent runs. Result events do not double-count requests. Model-supplied IDs are fingerprinted, not authenticated or assumed unique; tool names remain visible metadata.Records remain first-party, software-only,
enforcement_mode: declared, andappraisal.status: none. The real signed workflow record verifies against the supplied public key and passes released TRACE Level 0 conformance without changing that ceiling. No claims of attested execution, graph-native provenance, policy enforcement, exhaustive capture, or exactly-once execution.Verification
Repeated from a clean checkout of
70d3bfa889c07e546c48dc6f15ff8629b323fb07with fresh nox environments on Python 3.12.14:nox -s framework_adapters: 107 passed (86 framework-free cases, 1 LangGraph, 12 real LlamaIndex, 8 Pydantic AI), zero skips/failures.python -m pytest tests -q: 24 passed. Separate focused LlamaIndex run: 61 passed, including the 12 real-framework cases.git diff --check: pass.The framework tests use released LlamaIndex Core 0.14.24, Workflows 2.23.3, Instrumentation 0.6.0, TRACE 0.9.0 and conformance suite 0.5.1.
MockFunctionCallingLLMsupplies scripted local responses; the workflow, event delivery, and local tools are real. Socket connections/DNS are blocked during tests. No credentials or provider calls are needed. Existing upstream Pydantic inspection deprecation warnings remain visible.This adapter is a standalone integration file, not part of the
agentrust-trace-adapterswheel; no wheel-packaging claim is made.Dependency-audit limitation
pip-auditis not clean: the tested environment has one unresolved finding in LlamaIndex's transitivenltk==3.10.3, GHSA-8mgp-746c-j5xp. The advisory lists no patched release at verification time. The affected NLTK model-file APIs are not exercised by these tests. The finding is documented in the README, not suppressed. The test runner is pinned to patchedpytest==9.1.1.The commit includes the DCO sign-off.