Add retry logic for transient network failures and failure tracking - #9
Merged
Merged
Conversation
A backfill of thirty videos rarely gets through without YouTube hanging up
part-way through one of them:
[download] Got error: ('Connection aborted.', ConnectionResetError(10054,
'远程主机强迫关闭了一个现有的连接。', None, 10054, None))
That is the network, not the video, and it was costing the whole video: the
run reported a failure and moved on. Two changes, one for each timescale.
During a run, a request that fails on something that looks like network
trouble is made again after retry_backoff seconds, doubling for each of
download_retries extra attempts. yt-dlp resumes from the .part file, so a
drop at 63% costs the pause rather than the 63%. Transient failures are told
apart from refusals by walking the exception chain for connection and timeout
types and their messages, so the check survives a localised OS string; a
members-only or deleted video still fails on the first try, as before.
Between runs, a video that still fails goes into the state file under
"failures" with the error, an attempt count and enough metadata to fetch it
again. This is what a backfill needed: a plain run only looks at the newest
check_limit videos, so a video that failed during the initial thirty was out
of the window by the next run and would never have been seen again.
`run --retry-failed` puts that list back in front of the queue whatever its
age, `run --only-failed` does those and nothing else without listing the
channel, and retry_failed = true makes every run do it. A video that keeps
failing is picked up retry_max_attempts times and then left alone, which
--only-failed overrides and `ytscript failures --clear` resets.
`ytscript failures` shows the list; a video that succeeds drops off it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116BFtHjqyoP9CQoe7mVcym
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds comprehensive retry handling for transient network failures during YouTube downloads and channel listings, along with persistent failure tracking across runs.
Summary
When downloading videos from YouTube, network interruptions (connection resets, timeouts, etc.) are common and often recoverable with a retry. This change implements automatic retry logic with exponential backoff for transient failures, while distinguishing them from permanent refusals (members-only, deleted videos). Failed videos are now recorded in the state file so they can be retried in later runs, even after they've scrolled out of the normal check window.
Key Changes
Network Failure Detection & Retry Logic
is_transient()function to distinguish network troubles from permanent refusals by examining exception chains and error messages_attempt()wrapper inYouTubeClientthat retries transient failures with exponential backoff (configurable viaretry_backoff)YouTubeError.transientflag to mark errors that should be retriedretries,fragment_retries,extractor_retries,socket_timeout,continuedl)Failure Tracking & Recovery
Stateclass withfailuresdict to persistently record failed videos with attempt counts, timestamps, and metadatarecord_failure(),failed_videos(),forget_failures(), andattempts()methods toStatePipeline.run()to acceptretry_failedandonly_failedflags for retrying previously failed videosConfiguration & CLI
download_retries,retry_backoff,socket_timeout,retry_failed,retry_max_attempts--retry-failed,--no-retry-failed,--only-failed,--retries Nytscript failurescommand to list and clear failure recordsTesting
Implementation Details
__cause__and__context__) forConnectionError,TimeoutError, specificOSErrorerrno values, and known error message patternsretry_backoff * 2^(attempt-1), so with default 5s it waits 5s, 10s, 20s, etc..partfile resumption means a dropped download at 63% only costs the retry pause, not re-downloading the 63%failuresfield; version 1 files load without failureshttps://claude.ai/code/session_0116BFtHjqyoP9CQoe7mVcym