Skip to content

fix: reset proctored exam attempts reliably when the provider is unavailable - #1337

Open
Anas12091101 wants to merge 13 commits into
openedx:masterfrom
mitodl:anas/proctoring-reset-descriptive-error
Open

fix: reset proctored exam attempts reliably when the provider is unavailable#1337
Anas12091101 wants to merge 13 commits into
openedx:masterfrom
mitodl:anas/proctoring-reset-descriptive-error

Conversation

@Anas12091101

@Anas12091101 Anas12091101 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Description

When an instructor resets a student's proctored exam, edX also tells the
proctoring provider to delete its copy of the attempt. If the provider was down,
this used to fail with a confusing server error, and a reset could get stuck.

This PR makes the reset ask the provider to delete first, then delete locally,
and handles the provider's reply safely:

  • If the provider is down or errors out, the instructor sees a clear "provider is
    temporarily unavailable, please try again" message and the attempt stays put
    (nothing is half-deleted).
  • If the provider says the attempt is already gone, that counts as success, so
    trying again works.
  • If the provider replies but doesn't confirm the delete, the attempt is kept so it
    can be retried instead of quietly disappearing.
  • Every call to the provider now has a timeout (default 30s, configurable per
    backend), so a slow provider can't hang the request.

The reset_attempts admin command was also updated to tell the provider before
deleting, and to stop calling a provider that's down so it doesn't stall.

Testing

Automated: pytest edx_proctoring/tests/test_api.py -k remove_exam_attempt

Manual:

  1. Set up a proctored exam and have a learner start an attempt (so it registers
    with the provider).
  2. Make the provider unavailable — point its URL at an unreachable host, or make
    its delete call fail.
  3. As an instructor, reset that learner's attempt → you see the "try again"
    message and the attempt stays in the list.
  4. Bring the provider back and reset again → the reset works and the attempt
    disappears.

Pre-Merge Checklist:

  • Updated the version number in edx_proctoring/__init__.py and package.json
  • Described your changes in CHANGELOG.rst
  • Confirmed Github reports all automated tests/checks are passing
  • Approved by at least one additional reviewer

Post-Merge:

  • Create a tag matching the new version number

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 24, 2026
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @Anas12091101!

This repository is currently maintained by @openedx/openedx-unmaintained.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@mphilbrick211 mphilbrick211 moved this from Needs Triage to Ready for Review in Contributions Jul 27, 2026
@mphilbrick211 mphilbrick211 added the needs reviewer assigned PR needs to be (re-)assigned a new reviewer label Jul 27, 2026
@mphilbrick211
mphilbrick211 requested a review from feanil July 27, 2026 21:39
@mphilbrick211 mphilbrick211 removed the needs reviewer assigned PR needs to be (re-)assigned a new reviewer label Aug 19, 2026
Anas12091101 and others added 3 commits September 1, 2026 15:35
Move the provider `remove_exam_attempt` call out of the `pre_delete` signal into
`api.remove_exam_attempt`, before the local delete. A provider outage now raises a
descriptive `BackendProviderCannotRemoveAttempt` (HTTP 502) before anything is deleted,
so the local attempt is preserved and the DB connection is not left in a needs-rollback
state -- a partially-failed multi-attempt reset can be retried in the same request. A
provider "not found"/already-removed response is treated as success so the local delete
proceeds and retries converge instead of failing forever. Keeps the REST backend request
timeout. Also notifies the provider from `clear_onboarding_errors`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRF 3.18 returns partial list-validation errors as a dict keyed by the
index of only the invalid items ({1: {...}}) instead of a list padded
with empty dicts for valid items ([{}, {...}]). This broke the
pre-existing test_register_exams_mixed_valid on the django52 jobs only,
because DRF 3.18 dropped Django 4.2 support so drflatest resolves to an
older DRF there. Assert against both formats.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add tests for the "no backend configured" and best-effort
  (raise_on_error=False) branches of _remove_exam_attempt_from_backend so
  the new guard paths reach full patch coverage.
- Register a default grades runtime service in ProctoredExamApiTests
  setUp/tearDown. Rejected-transition tests need a grades service but did
  not register one, so they only passed when a sibling test leaked one
  through the process-global runtime registry -- making them flaky under
  pytest-xdist (-n 3).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Anas12091101 Anas12091101 changed the title feat: raise descriptive error when proctoring provider cannot remove an attempt fix: reset proctored exam attempts reliably when the provider is unavailable Sep 1, 2026
@Anas12091101
Anas12091101 requested a balanced review from Copilot September 1, 2026 11:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves reliability of proctored-attempt resets during provider outages.

Changes:

  • Moves provider deletion before local deletion and adds typed 502 errors.
  • Adds configurable REST request timeouts.
  • Expands reset and compatibility tests and releases version 6.1.0.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
package.json Bumps package version.
edx_proctoring/__init__.py Bumps Python version.
CHANGELOG.rst Documents release behavior.
edx_proctoring/api.py Coordinates provider-first removal.
edx_proctoring/apps.py Allows timeout configuration.
edx_proctoring/exceptions.py Adds removal failure exception.
edx_proctoring/handlers.py Removes provider deletion signal.
edx_proctoring/models.py Adds best-effort onboarding cleanup.
edx_proctoring/backends/rest.py Adds attempt-request timeout.
edx_proctoring/backends/tests/test_rest.py Tests timeout and connection failures.
edx_proctoring/tests/test_api.py Tests reset failure behavior.
edx_proctoring/tests/test_handlers.py Removes obsolete signal test.
edx_proctoring/tests/test_views.py Normalizes DRF validation errors.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread edx_proctoring/handlers.py
Comment thread edx_proctoring/backends/rest.py
Comment thread edx_proctoring/api.py Outdated
Comment thread edx_proctoring/backends/rest.py
Anas12091101 and others added 2 commits September 1, 2026 17:27
Addresses the Copilot review comments on the attempt-reset changes:

- remove_exam_attempt now treats a provider 404 as success (the attempt is
  already gone upstream, so retries converge) but raises on any other HTTP
  error, so a transient 5xx keeps the local attempt for a later retry instead
  of silently deleting it as "already removed".
- Apply the configurable request timeout to every outbound provider call, not
  just attempt requests, so the documented guarantee is accurate.
- Route the reset_attempts management command through the backend-removal flow,
  since provider cleanup no longer happens in the pre_delete signal.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
remove_exam_attempt no longer routes through _make_attempt_request, which left
its `if not attempt` short-circuit untested. Exercise it via
mark_erroneous_exam_attempt(None) so the guard stays covered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Suppressed comments (2)

edx_proctoring/api.py:1938

  • The PR description says a provider-supplied error message is preserved when a response body exists, but every exception is replaced with this hard-coded generic text. Consequently, HTTP errors with useful provider details and errors without bodies produce the same client response, and the described with/without-body behavior is not implemented. Preserve an appropriate sanitized provider message when available, with this text as the fallback.
            raise BackendProviderCannotRemoveAttempt(
                # Translators: shown to an instructor when an exam attempt could not be reset
                # because the external proctoring provider is temporarily unavailable.
                'The proctoring provider is temporarily unavailable, so this attempt could not '
                'be fully reset. Please try again in a few minutes.'

edx_proctoring/handlers.py:102

  • Removing the model-wide pre_delete receiver means cascading deletes of a ProctoredExam or user now remove registered attempts without notifying the provider; both foreign keys use on_delete=CASCADE (models.py:375-377). The API, onboarding cleanup, and management command cover their own paths, but cascades bypass all three and can leave provider-side attempts orphaned. Add a safe cleanup path for cascades before removing this receiver.
@receiver(pre_save, sender=models.ProctoredExamStudentAttempt)
def on_attempt_changed(sender, instance, signal, **kwargs):

Comment thread edx_proctoring/api.py Outdated
Comment thread edx_proctoring/backends/rest.py
Comment thread edx_proctoring/management/commands/reset_attempts.py Outdated
Comment thread edx_proctoring/api.py Outdated
Second Copilot review pass:

- Wrap the instructor-facing removal error in gettext (_()) so it is
  extracted and localizable, not hard-coded English.
- Resolve the backend inside remove-from-backend's error-handling block, so a
  stale/unconfigured backend (get_backend_provider raising NotImplementedError)
  still yields the typed 502 and does not abort best-effort callers.
- In the REST backend, reserve removal success for the documented
  {"status": "deleted"} payload and a 404; raise on any other HTTP error, an
  undecodable body, or an unconfirmed 2xx, so the local attempt is kept for
  retry instead of being silently dropped.
- Give the reset_attempts command a per-run circuit breaker: once a backend
  fails a provider-removal call, stop calling it and delete locally only, so an
  unavailable provider cannot stall the command by a request timeout per attempt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Comment thread edx_proctoring/api.py Outdated
Comment thread edx_proctoring/api.py Outdated
Comment thread edx_proctoring/backends/tests/test_rest.py
Third Copilot review pass:

- Backends now only report removal status; the API helper is the single place that
  produces the instructor-facing error, so a backend's own message can no longer be
  masked by re-wrapping (the REST backend returns False on an unconfirmed/undecodable
  2xx instead of raising).
- A falsy removal result now keeps the local attempt for strict callers (raising the
  generic 502 so a retry can converge) while best-effort callers still proceed. The
  no-op SoftwareSecure backend returns explicit success so it is not mistaken for a
  failed removal.
- Add tests for the default 30s request timeout and a PROCTORING_BACKENDS timeout
  override, replacing the tautological assertion.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Suppressed comments (2)

edx_proctoring/handlers.py:102

  • Removing the pre_delete receiver also removes provider cleanup from every deletion that does not go through the few explicitly updated call sites. In particular, ProctoredExamStudentAttempt.delete_exam_attempt() still directly calls delete(), and deleting a ProctoredExam or user cascades to attempts; all of these previously notified the provider but now leave upstream attempts orphaned. Please provide a safe best-effort fallback for direct/cascade deletes (while avoiding duplicate calls from the strict API path), or route all such deletion entry points through one cleanup service.
@receiver(pre_save, sender=models.ProctoredExamStudentAttempt)
def on_attempt_changed(sender, instance, signal, **kwargs):

edx_proctoring/management/commands/reset_attempts.py:85

  • This circuit breaker treats every removal failure as proof that the whole backend is unavailable. _remove_exam_attempt_from_backend uses this same exception for attempt-specific failures too—for example, a 400 HTTP response or one malformed/unconfirmed response—so one bad attempt causes all later valid attempts for that backend to be deleted locally without provider notification. Only suppress later calls for transport/availability failures; continue attempting cleanup after per-attempt errors.
                try:
                    _remove_exam_attempt_from_backend(attempt)
                except BackendProviderCannotRemoveAttempt:

Comment thread edx_proctoring/api.py Outdated
Comment thread edx_proctoring/management/commands/reset_attempts.py
Comment thread edx_proctoring/models.py Outdated
Fourth Copilot review pass:

- Factor the "remove many attempts, skipping a backend after it fails once" logic
  into a single _remove_exam_attempts_from_backend helper, used by both the
  reset_attempts command and clear_onboarding_errors, so an unavailable provider no
  longer costs a request timeout per attempt in the onboarding-cleanup path either.
- Eager-load proctored_exam (select_related) in both bulk paths to avoid a query per
  attempt.
- Drop the now-redundant raise_on_error flag: the single-attempt helper always raises,
  and the bulk helper is the one place that swallows failures for best-effort callers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

edx_proctoring/backends/tests/test_rest.py:288

  • This test no longer exercises _make_attempt_request; remove_exam_attempt now builds the request directly. Rename it so failures identify the behavior actually under test.
    def test_make_attempt_request_passes_timeout(self):

edx_proctoring/handlers.py:101

  • Removing the pre_delete receiver also removes provider cleanup from deletion paths not routed through the three new helpers, including ORM cascades through the attempt's user and proctored_exam foreign keys (models.py:375-377) and direct model/queryset deletes. Those paths previously notified the provider and will now leave upstream attempts orphaned. Preserve a best-effort signal fallback that is skipped after strict API cleanup, or route every deletion entry point through a centralized cleanup service.
@receiver(pre_save, sender=models.ProctoredExamStudentAttempt)

edx_proctoring/api.py:1948

  • This makes a truthy return a new mandatory contract for third-party backends, but the public abstract method only promises to remove the attempt (backends/backend.py:61-65), and custom implementations are supported (docs/backends.rst:350-352). A valid backend that successfully deletes and returns None will now block every reset after a minor-version upgrade. Introduce and document a versioned/capability-based confirmation contract before enforcing it, or release this as a breaking API change.
    if not result:
        # The provider responded but did not confirm removal, so keep the local attempt: a
        # retry can converge once the provider is healthy instead of silently dropping it.
        log.warning('Backend %s did not confirm removal of attempt_id=%s.', exam.backend, attempt.id)
        raise BackendProviderCannotRemoveAttempt(unavailable_message)

Comment thread edx_proctoring/backends/rest.py
If session.delete raises before a response is received (e.g. a timeout, now that a
timeout is applied), `response` was never assigned, so the except block's
`response.content` raised UnboundLocalError and masked the intended
BackendProviderCannotRetireUser. Initialize response and guard the fallback, matching
the pattern already used in on_exam_saved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

edx_proctoring/api.py:1923

  • The new 502 detail is not surfaced by the existing instructor reset UI. proctored_exam_attempt_view.js:227-239 registers only a success callback for this DELETE request, so on provider failure the instructor sees no “temporarily unavailable” message and the cursor remains stuck on wait, contrary to the PR’s stated behavior. Add an AJAX error handler that displays responseJSON.detail (with a safe fallback), restores the cursor, and cover the failure path in the view spec.
    unavailable_message = _(
        'The proctoring provider is temporarily unavailable, so this attempt could not '
        'be fully reset. Please try again in a few minutes.'
    )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

open-source-contribution PR author is not from Axim or 2U

Projects

Status: Ready for Review

Development

Successfully merging this pull request may close these issues.

4 participants