Skip to content

Migrate NLP model inputs to BatchEncode and add named ONNX input binding - #33

Draft
tjwald with Copilot wants to merge 5 commits into
developfrom
copilot/update-tokenization-output-format
Draft

tjwald with Copilot wants to merge 5 commits into
developfrom
copilot/update-tokenization-output-format

Conversation

Copilot AI commented Sep 16, 2026

Copy link
Copy Markdown

This PR replaces flat tensor-array tokenization outputs with a Hugging Face-style BatchEncode contract and updates ONNX execution to bind inputs by semantic names when available. It also preserves backward compatibility for legacy Tensor<long>[] model pipelines.

  • Input contract unification

    • Added BatchEncode in FAI.Core.Pipelines to represent input_ids, optional attention_mask, and optional token_type_ids.
    • Updated tokenizer/tensorization flow to emit BatchEncode directly instead of positional tensor arrays.
  • NLP pipeline type migration

    • TextTensorization and TextMultipleChoiceTensorization now output BatchEncode.
    • Downstream pipeline wiring in examples/integration paths was updated to compose ONNX stages with BatchEncode.
  • ONNX named-input support

    • ONNX executors now accept IPipeline<BatchEncode, TensorOutputs<float>>.
    • Input resolution maps BatchEncode fields to model-declared names in model order (input_ids, attention_mask, token_type_ids).
    • If the model exposes HF names, only declared inputs are passed; missing required declared inputs fail fast with clear exceptions.
    • Legacy positional Tensor<long>[] execution remains supported through the existing interface path.
  • Factory/executor alignment

    • ModelExecutorFactory.CreateModelPipeline(...) now returns IPipeline<BatchEncode, TensorOutputs<float>>.
    • Legacy tensor-array compatibility wrappers remain available for existing callers still using positional inputs.
  • Targeted ONNX coverage updates

    • Updated ONNX factory/executor tests to validate BatchEncode execution behavior, including optional-input handling and model-name resolution behavior.
services
    .AddPipeline<ReadOnlyMemory<string>>()
    .Then<ReadOnlyMemory<TokenizedText>, TextTokenization>()
    .Then<BatchEncode, TextTensorization>()
    .ThenOnnxModel();

Copilot AI and others added 2 commits September 16, 2026 06:14
Co-authored-by: tjwald <52830708+tjwald@users.noreply.github.com>
Co-authored-by: tjwald <52830708+tjwald@users.noreply.github.com>
Copilot AI changed the title Switch tokenization to BatchEncode and support named ONNX inputs Migrate NLP model inputs to BatchEncode and add named ONNX input binding Sep 16, 2026
Copilot AI requested a review from tjwald September 16, 2026 06:26
Co-authored-by: tjwald <52830708+tjwald@users.noreply.github.com>
Co-authored-by: tjwald <52830708+tjwald@users.noreply.github.com>
Co-authored-by: tjwald <52830708+tjwald@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants