fix: reset proctored exam attempts reliably when the provider is unavailable - #1337
fix: reset proctored exam attempts reliably when the provider is unavailable#1337Anas12091101 wants to merge 13 commits into
Conversation
|
Thanks for the pull request, @Anas12091101! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
Move the provider `remove_exam_attempt` call out of the `pre_delete` signal into `api.remove_exam_attempt`, before the local delete. A provider outage now raises a descriptive `BackendProviderCannotRemoveAttempt` (HTTP 502) before anything is deleted, so the local attempt is preserved and the DB connection is not left in a needs-rollback state -- a partially-failed multi-attempt reset can be retried in the same request. A provider "not found"/already-removed response is treated as success so the local delete proceeds and retries converge instead of failing forever. Keeps the REST backend request timeout. Also notifies the provider from `clear_onboarding_errors`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRF 3.18 returns partial list-validation errors as a dict keyed by the
index of only the invalid items ({1: {...}}) instead of a list padded
with empty dicts for valid items ([{}, {...}]). This broke the
pre-existing test_register_exams_mixed_valid on the django52 jobs only,
because DRF 3.18 dropped Django 4.2 support so drflatest resolves to an
older DRF there. Assert against both formats.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add tests for the "no backend configured" and best-effort (raise_on_error=False) branches of _remove_exam_attempt_from_backend so the new guard paths reach full patch coverage. - Register a default grades runtime service in ProctoredExamApiTests setUp/tearDown. Rejected-transition tests need a grades service but did not register one, so they only passed when a sibling test leaked one through the process-global runtime registry -- making them flaky under pytest-xdist (-n 3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Improves reliability of proctored-attempt resets during provider outages.
Changes:
- Moves provider deletion before local deletion and adds typed 502 errors.
- Adds configurable REST request timeouts.
- Expands reset and compatibility tests and releases version 6.1.0.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
package.json |
Bumps package version. |
edx_proctoring/__init__.py |
Bumps Python version. |
CHANGELOG.rst |
Documents release behavior. |
edx_proctoring/api.py |
Coordinates provider-first removal. |
edx_proctoring/apps.py |
Allows timeout configuration. |
edx_proctoring/exceptions.py |
Adds removal failure exception. |
edx_proctoring/handlers.py |
Removes provider deletion signal. |
edx_proctoring/models.py |
Adds best-effort onboarding cleanup. |
edx_proctoring/backends/rest.py |
Adds attempt-request timeout. |
edx_proctoring/backends/tests/test_rest.py |
Tests timeout and connection failures. |
edx_proctoring/tests/test_api.py |
Tests reset failure behavior. |
edx_proctoring/tests/test_handlers.py |
Removes obsolete signal test. |
edx_proctoring/tests/test_views.py |
Normalizes DRF validation errors. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Addresses the Copilot review comments on the attempt-reset changes: - remove_exam_attempt now treats a provider 404 as success (the attempt is already gone upstream, so retries converge) but raises on any other HTTP error, so a transient 5xx keeps the local attempt for a later retry instead of silently deleting it as "already removed". - Apply the configurable request timeout to every outbound provider call, not just attempt requests, so the documented guarantee is accurate. - Route the reset_attempts management command through the backend-removal flow, since provider cleanup no longer happens in the pre_delete signal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
remove_exam_attempt no longer routes through _make_attempt_request, which left its `if not attempt` short-circuit untested. Exercise it via mark_erroneous_exam_attempt(None) so the guard stays covered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Suppressed comments (2)
edx_proctoring/api.py:1938
- The PR description says a provider-supplied error message is preserved when a response body exists, but every exception is replaced with this hard-coded generic text. Consequently, HTTP errors with useful provider details and errors without bodies produce the same client response, and the described with/without-body behavior is not implemented. Preserve an appropriate sanitized provider message when available, with this text as the fallback.
raise BackendProviderCannotRemoveAttempt(
# Translators: shown to an instructor when an exam attempt could not be reset
# because the external proctoring provider is temporarily unavailable.
'The proctoring provider is temporarily unavailable, so this attempt could not '
'be fully reset. Please try again in a few minutes.'
edx_proctoring/handlers.py:102
- Removing the model-wide
pre_deletereceiver means cascading deletes of aProctoredExamor user now remove registered attempts without notifying the provider; both foreign keys useon_delete=CASCADE(models.py:375-377). The API, onboarding cleanup, and management command cover their own paths, but cascades bypass all three and can leave provider-side attempts orphaned. Add a safe cleanup path for cascades before removing this receiver.
@receiver(pre_save, sender=models.ProctoredExamStudentAttempt)
def on_attempt_changed(sender, instance, signal, **kwargs):
Second Copilot review pass:
- Wrap the instructor-facing removal error in gettext (_()) so it is
extracted and localizable, not hard-coded English.
- Resolve the backend inside remove-from-backend's error-handling block, so a
stale/unconfigured backend (get_backend_provider raising NotImplementedError)
still yields the typed 502 and does not abort best-effort callers.
- In the REST backend, reserve removal success for the documented
{"status": "deleted"} payload and a 404; raise on any other HTTP error, an
undecodable body, or an unconfirmed 2xx, so the local attempt is kept for
retry instead of being silently dropped.
- Give the reset_attempts command a per-run circuit breaker: once a backend
fails a provider-removal call, stop calling it and delete locally only, so an
unavailable provider cannot stall the command by a request timeout per attempt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Third Copilot review pass: - Backends now only report removal status; the API helper is the single place that produces the instructor-facing error, so a backend's own message can no longer be masked by re-wrapping (the REST backend returns False on an unconfirmed/undecodable 2xx instead of raising). - A falsy removal result now keeps the local attempt for strict callers (raising the generic 502 so a retry can converge) while best-effort callers still proceed. The no-op SoftwareSecure backend returns explicit success so it is not mistaken for a failed removal. - Add tests for the default 30s request timeout and a PROCTORING_BACKENDS timeout override, replacing the tautological assertion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Suppressed comments (2)
edx_proctoring/handlers.py:102
- Removing the
pre_deletereceiver also removes provider cleanup from every deletion that does not go through the few explicitly updated call sites. In particular,ProctoredExamStudentAttempt.delete_exam_attempt()still directly callsdelete(), and deleting aProctoredExamor user cascades to attempts; all of these previously notified the provider but now leave upstream attempts orphaned. Please provide a safe best-effort fallback for direct/cascade deletes (while avoiding duplicate calls from the strict API path), or route all such deletion entry points through one cleanup service.
@receiver(pre_save, sender=models.ProctoredExamStudentAttempt)
def on_attempt_changed(sender, instance, signal, **kwargs):
edx_proctoring/management/commands/reset_attempts.py:85
- This circuit breaker treats every removal failure as proof that the whole backend is unavailable.
_remove_exam_attempt_from_backenduses this same exception for attempt-specific failures too—for example, a 400 HTTP response or one malformed/unconfirmed response—so one bad attempt causes all later valid attempts for that backend to be deleted locally without provider notification. Only suppress later calls for transport/availability failures; continue attempting cleanup after per-attempt errors.
try:
_remove_exam_attempt_from_backend(attempt)
except BackendProviderCannotRemoveAttempt:
Fourth Copilot review pass: - Factor the "remove many attempts, skipping a backend after it fails once" logic into a single _remove_exam_attempts_from_backend helper, used by both the reset_attempts command and clear_onboarding_errors, so an unavailable provider no longer costs a request timeout per attempt in the onboarding-cleanup path either. - Eager-load proctored_exam (select_related) in both bulk paths to avoid a query per attempt. - Drop the now-redundant raise_on_error flag: the single-attempt helper always raises, and the bulk helper is the one place that swallows failures for best-effort callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
edx_proctoring/backends/tests/test_rest.py:288
- This test no longer exercises
_make_attempt_request;remove_exam_attemptnow builds the request directly. Rename it so failures identify the behavior actually under test.
def test_make_attempt_request_passes_timeout(self):
edx_proctoring/handlers.py:101
- Removing the
pre_deletereceiver also removes provider cleanup from deletion paths not routed through the three new helpers, including ORM cascades through the attempt'suserandproctored_examforeign keys (models.py:375-377) and direct model/queryset deletes. Those paths previously notified the provider and will now leave upstream attempts orphaned. Preserve a best-effort signal fallback that is skipped after strict API cleanup, or route every deletion entry point through a centralized cleanup service.
@receiver(pre_save, sender=models.ProctoredExamStudentAttempt)
edx_proctoring/api.py:1948
- This makes a truthy return a new mandatory contract for third-party backends, but the public abstract method only promises to remove the attempt (
backends/backend.py:61-65), and custom implementations are supported (docs/backends.rst:350-352). A valid backend that successfully deletes and returnsNonewill now block every reset after a minor-version upgrade. Introduce and document a versioned/capability-based confirmation contract before enforcing it, or release this as a breaking API change.
if not result:
# The provider responded but did not confirm removal, so keep the local attempt: a
# retry can converge once the provider is healthy instead of silently dropping it.
log.warning('Backend %s did not confirm removal of attempt_id=%s.', exam.backend, attempt.id)
raise BackendProviderCannotRemoveAttempt(unavailable_message)
If session.delete raises before a response is received (e.g. a timeout, now that a timeout is applied), `response` was never assigned, so the except block's `response.content` raised UnboundLocalError and masked the intended BackendProviderCannotRetireUser. Initialize response and guard the fallback, matching the pattern already used in on_exam_saved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
edx_proctoring/api.py:1923
- The new 502 detail is not surfaced by the existing instructor reset UI.
proctored_exam_attempt_view.js:227-239registers only a success callback for this DELETE request, so on provider failure the instructor sees no “temporarily unavailable” message and the cursor remains stuck onwait, contrary to the PR’s stated behavior. Add an AJAX error handler that displaysresponseJSON.detail(with a safe fallback), restores the cursor, and cover the failure path in the view spec.
unavailable_message = _(
'The proctoring provider is temporarily unavailable, so this attempt could not '
'be fully reset. Please try again in a few minutes.'
)
Description
When an instructor resets a student's proctored exam, edX also tells the
proctoring provider to delete its copy of the attempt. If the provider was down,
this used to fail with a confusing server error, and a reset could get stuck.
This PR makes the reset ask the provider to delete first, then delete locally,
and handles the provider's reply safely:
temporarily unavailable, please try again" message and the attempt stays put
(nothing is half-deleted).
trying again works.
can be retried instead of quietly disappearing.
backend), so a slow provider can't hang the request.
The
reset_attemptsadmin command was also updated to tell the provider beforedeleting, and to stop calling a provider that's down so it doesn't stall.
Testing
Automated:
pytest edx_proctoring/tests/test_api.py -k remove_exam_attemptManual:
with the provider).
its delete call fail.
message and the attempt stays in the list.
disappears.
Pre-Merge Checklist:
edx_proctoring/__init__.pyandpackage.jsonCHANGELOG.rstPost-Merge: