Skip to content

Loader checkpoint restoration restarts before a None item #1551

Description

@aswanth-07

Bug

torchdata.nodes.Loader cannot restore a checkpoint whose next item is None when restart_on_stop_iteration=True (the default). The look-ahead in LoaderIterator.has_next() treats None as meaning that no item was cached, decides the restored iterator is exhausted, and restarts it from the beginning.

Reproduction

from torchdata.nodes import IterableWrapper, Loader

loader = Loader(IterableWrapper([1, None, 2]))
iterator = iter(loader)
assert next(iterator) == 1
state = loader.state_dict()

restored = Loader(IterableWrapper([1, None, 2]))
restored.load_state_dict(state)
print(list(restored))

On current main (e640e6f), this prints:

[1, None, 2]

The expected continuation is:

[None, 2]

The restart duplicates data that was consumed before the checkpoint. Calling LoaderIterator.has_next() directly also returns False and consumes the value when the next item is None.

Root cause and proposed fix

LoaderIterator._cached_item currently uses None both as a valid data value and as its empty-cache sentinel. Tracking cache occupancy separately avoids that collision without changing the public API or the look-ahead behavior for other values.

I have a focused regression test and a small fix ready.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions