Bug
torchdata.nodes.Loader cannot restore a checkpoint whose next item is None when restart_on_stop_iteration=True (the default). The look-ahead in LoaderIterator.has_next() treats None as meaning that no item was cached, decides the restored iterator is exhausted, and restarts it from the beginning.
Reproduction
from torchdata.nodes import IterableWrapper, Loader
loader = Loader(IterableWrapper([1, None, 2]))
iterator = iter(loader)
assert next(iterator) == 1
state = loader.state_dict()
restored = Loader(IterableWrapper([1, None, 2]))
restored.load_state_dict(state)
print(list(restored))
On current main (e640e6f), this prints:
The expected continuation is:
The restart duplicates data that was consumed before the checkpoint. Calling LoaderIterator.has_next() directly also returns False and consumes the value when the next item is None.
Root cause and proposed fix
LoaderIterator._cached_item currently uses None both as a valid data value and as its empty-cache sentinel. Tracking cache occupancy separately avoids that collision without changing the public API or the look-ahead behavior for other values.
I have a focused regression test and a small fix ready.
Bug
torchdata.nodes.Loadercannot restore a checkpoint whose next item isNonewhenrestart_on_stop_iteration=True(the default). The look-ahead inLoaderIterator.has_next()treatsNoneas meaning that no item was cached, decides the restored iterator is exhausted, and restarts it from the beginning.Reproduction
On current
main(e640e6f), this prints:The expected continuation is:
The restart duplicates data that was consumed before the checkpoint. Calling
LoaderIterator.has_next()directly also returnsFalseand consumes the value when the next item isNone.Root cause and proposed fix
LoaderIterator._cached_itemcurrently usesNoneboth as a valid data value and as its empty-cache sentinel. Tracking cache occupancy separately avoids that collision without changing the public API or the look-ahead behavior for other values.I have a focused regression test and a small fix ready.