Grade conductor questions via headless conductor runtime - #387
Draft
RichDom2185 wants to merge 1 commit into
Draft
Grade conductor questions via headless conductor runtime#387RichDom2185 wants to merge 1 commit into
RichDom2185 wants to merge 1 commit into
Conversation
Conductor programming questions carry a (language, evaluator) pair resolved via @sourceacademy/language-directory to an evaluator bundle. The bundle is run headless in a Node worker_threads worker driven by the @sourceacademy/ conductor host protocol, and each testcase's stdout is graded against its expected answer, returning the same Summary shape as the legacy js-slang path. Scope: pure-TypeScript python-3 evaluators (cse/PVML); Pyodide/WASM deferred.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds grading support for conductor (external-runtime) programming questions. Until now the backend refused to send them here; they carry a
(language, evaluator)pair instead of the legacychapter/external/globals.Pairs with backend PR source-academy/backend#1367, which sends conductor answers to this grader as
library: { format: "conductor", language, evaluator }.How it works
runAll(src/index.ts) branches onlibrary.format === 'conductor'torunAllConductor(src/conductor.ts) before any legacy field is touched.runAllConductorresolves the evaluator via@sourceacademy/language-directory, fetches + caches the published evaluator bundle, and runs it headless in a Nodeworker_threadsworker driven by the@sourceacademy/conductorhost protocol (aself/ILinkshim + aNodeHostPlugin).testcase.answer(python-3 CSE surfaces values via stdout, not the result channel). The response is the sameSummaryshape as the legacy path, so the backend parser is unchanged.Scope
cse/ PVML). Pyodide/WASM evaluators need a browser runtime and are deferred (they surface as a run error here).Verification
yarn test— 48/48 (10 new conductor unit tests + existing suites),yarn typecheck,yarn format:check,yarn buildall pass.scripts/smoke-conductor.cjs, not in CI) run against the realPyCseEvaluator3.js: 4 python-3 testcases graded correctly (pass/pass/fail/runtime-error).Deploy notes
source-academy.github.ioat runtime, so the Lambda needs outbound egress — or vendor the bundle at build time.python3/python3Default).Draft: stacked on
richard/2601-maintenance(depends on its Node 24 / build-script updates).