feat(workflow): JoyAI-VL-Interaction tool capability node - #19
Conversation
New tool node tool.realtime.vl.interaction bridging the workflow runtime to a self-hosted JoyAI-VL-Interaction deployment (jd-opensource, 8B realtime video-language interaction model, Apache-2.0, served OpenAI-compatible via vLLM-Omni). - backend/workflow/joyai_vl_executor.py: executor mode joyai_vl_interaction, one chat-completions interaction turn (prompt + videoUrl/imageUrls) against JOYAI_VL_URL, reply emitted as event.vl.interaction.v1 - tool_capabilities registry entry (event[] in / event[] out, runnable) - opencli_hda_tracer dispatch mirroring the okx executor precedent; failures (incl. unconfigured endpoint) degrade to event.vl.interaction.error.v1 instead of crashing the run - SUPPORTED_TOOL_EXECUTOR_MODES + WorkflowToolCapabilityExecutor.mode literal extended; .env.example documents JOYAI_VL_URL / JOYAI_VL_API_KEY - 4 unit tests: unconfigured guard, request shape + reply parse, empty choices, registry resolution Full-suite note: 6 order-pollution flakes pre-exist on clean main (baselined this session, logged in docs/BUG-TRIAGE-20260718.md P3-6); branch adds 4 passing tests, zero new failures.
|
✅ Health: 7.1 📋 At a glance 🚨 Change risk: 8.7/10 (high)
📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-18 12:05 UTC |
|
Warning Review limit reached
Next review available in: 7 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the JoyAI-VL-Interaction tool capability executor (tool.realtime.vl.interaction), enabling real-time video-language interaction by bridging the workflow runtime with a self-hosted JoyAI-VL-Interaction deployment via vLLM-Omni. The changes include adding environment variables, updating workflow schemas, registering the new tool capability, integrating the executor into the runtime, and adding unit tests. Feedback on the implementation points out that the HTTP request and response parsing logic in the executor could raise unhandled exceptions outside the current try-except block, potentially crashing the workflow run instead of gracefully degrading. A robust, unified try-except block with strict type-checking and HTTPError body extraction is suggested to address this vulnerability.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| try: | ||
| with urllib.request.urlopen(request, timeout=timeout_seconds) as response: | ||
| payload = json.loads(response.read().decode("utf-8")) | ||
| except JoyAIVLExecutionError: | ||
| raise | ||
| except Exception as exc: # pragma: no cover - exercised by live smoke failures. | ||
| raise JoyAIVLExecutionError(f"JoyAI-VL request failed: {exc}") from exc | ||
|
|
||
| choices = payload.get("choices") or [] | ||
| if not choices: | ||
| raise JoyAIVLExecutionError(f"JoyAI-VL returned no choices: {payload}") | ||
| message = choices[0].get("message") or {} | ||
| reply = message.get("content") | ||
| if not isinstance(reply, str) or not reply.strip(): | ||
| raise JoyAIVLExecutionError(f"JoyAI-VL returned an empty reply: {payload}") |
There was a problem hiding this comment.
The HTTP request and response parsing logic has several potential failure points that could raise unhandled exceptions (such as AttributeError, TypeError, or urllib.error.HTTPError) outside of the try...except block. If any of these occur, they will not be caught as JoyAIVLExecutionError, causing the workflow run to crash instead of gracefully degrading to an error event as intended.
Additionally, when an HTTPError occurs, the response body often contains detailed validation or error messages from the vLLM server (e.g., model not found, invalid inputs). We can capture and read this error body to provide much more actionable error messages.
We should wrap both the HTTP request and the entire response parsing logic in a single unified try...except block, perform strict type checking on the returned JSON payload, and extract the error body from HTTPError if available.
try:
with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
payload = json.loads(response.read().decode("utf-8"))
if not isinstance(payload, dict):
raise JoyAIVLExecutionError(f"JoyAI-VL returned unexpected JSON response: {payload}")
choices = payload.get("choices")
if not isinstance(choices, list) or not choices:
raise JoyAIVLExecutionError(f"JoyAI-VL returned no choices: {payload}")
first_choice = choices[0]
if not isinstance(first_choice, dict):
raise JoyAIVLExecutionError(f"JoyAI-VL returned invalid choice structure: {payload}")
message = first_choice.get("message")
if not isinstance(message, dict):
raise JoyAIVLExecutionError(f"JoyAI-VL returned invalid message structure: {payload}")
reply = message.get("content")
if not isinstance(reply, str) or not reply.strip():
raise JoyAIVLExecutionError(f"JoyAI-VL returned an empty reply: {payload}")
except JoyAIVLExecutionError:
raise
except Exception as exc:
if hasattr(exc, "read"):
try:
err_body = exc.read().decode("utf-8")
raise JoyAIVLExecutionError(f"JoyAI-VL request failed with status {getattr(exc, 'code', 'unknown')}: {err_body}") from exc
except Exception:
pass
raise JoyAIVLExecutionError(f"JoyAI-VL request failed or returned invalid response: {exc}") from exc
Direction from 2026-07-18 discussion: every pipeline capability (tool nodes, agents, channels, notifiers) becomes a Dify-style plugin. Records current friction (PR #19 touched 5 files to add one tool node), minimal first steps, and open questions for the workflow discussion.
1 participant
What
新工具节点: JoyAI-VL-Interaction (jd-opensource, 8B 实时视频-语言交互模型, Apache-2.0, vLLM-Omni 部署) 接入工作流工具能力体系。
tool.realtime.vl.interaction注册进 tool capability registry — 画布external.tool.capability节点可绑定, event[] 入 / event[] 出joyai_vl_interaction(backend/workflow/joyai_vl_executor.py): 对JOYAI_VL_URL指向的 vLLM-Omni 端点发一次 OpenAI 兼容 chat-completions 交互轮 (prompt + videoUrl / imageUrls), 回复落成event.vl.interaction.v1opencli_hda_tracer按 okx 先例分发; 执行失败 (含端点未配) 降级event.vl.interaction.error.v1错误事件, 不崩 runSUPPORTED_TOOL_EXECUTOR_MODES+ schema Literal 加 mode;.env.example增JOYAI_VL_URL/JOYAI_VL_API_KEY说明MVP 边界: 单轮交互探测。JoyAI 的常驻流式模式 (持续看流主动开口) 需要持久会话, 二期同 executor 上扩。
Test
tests/unit/test_joyai_vl_executor.py×4: 未配置报可操作错 / 请求体媒体分片与回复解析 / 空 choices / 注册表可解析