Make .protocol report SIP content, not source-folder totals - #6
Open
hyeeyoungkim wants to merge 1 commit into
Open
Make .protocol report SIP content, not source-folder totals#6hyeeyoungkim wants to merge 1 commit into
hyeeyoungkim wants to merge 1 commit into
Conversation
- data_stats() gains list_to_exclude=None and filters via include_files() - create_protocol() gains list_to_exclude=None and passes it through - main() passes getattr(args, "excludedFileNames", None), tolerating hand-built Namespaces that omit the field - .protocol <size>/<files> and the sip_report dict now describe what is packaged instead of the raw input folder; keys and types are unchanged - excluded files such as Thumbs.db, .DS_Store and desktop.ini were counted because create_protocol() ran before create_xip() applied include_files() Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.protocolreports the file count and byte size of the source folder, including files that-excludedFileNameskeeps out of the SIP. This makesdata_stats()apply the same exclusion listthe packer already uses, so
<files>and<size>describe the SIP. Three functions, about 5 lines,nothing reordered.
The problem
main()writes.protocolbefore anything is packed:data_stats()walks the source folder with an unfilteredrglob("*"), so excluded files are stillcounted. The same return value feeds
reporting_std_out(), so thesip_reportdict handed tocallers like
yesc_bulk.pycarries the same numbers — readingsip_reportinstead of.protocoldoes not help.
Example, on a folder of 7 files / 400 bytes, of which 3 files / 300 bytes are content:
Longstanding, and not specific to one branch:
data_stats()is identical on all six branches ofthis repo carrying
yesc.py, andgit log -Sshows the call order has never differed.The change
data_stats()list_to_exclude; filters inside the existing comprehensioncreate_protocol()list_to_exclude; passes it todata_stats()main()create_protocol(data_in_path, getattr(args, 'excludedFileNames', None))The same folder then reports
<size>300</size><files>3</files>, matching the SIP.getattr()rather thanargs.excludedFileNamesbecause callers that import yesc build their ownNamespace, and a plain lookup would raiseAttributeErrorfor any that omit the field.What this affects
-ef: numbers unchanged, sinceinclude_files()returnsTruefor an empty ormissing list. Covered by
test_data_stats_counts_everything_when_no_exclusions. Costs one extracall per file.
-ef:.protocoland the printed stats report SIP totals. Anyone reading the old numbersas source-folder totals will get different values; on a 300-item production batch the gap was in
the file count only, bytes negligible, well under 1%.
sip_reportkeeps its keys and value types.-ef:include_files()prints whenever it has a list to check, so thoselines now appear while counting as well as while packing. Happy to guard them or add a quiet flag;
I left them alone to keep the diff small.
What this does not do
include_files(). It matches withif file_name in list_to_exclude, asubstring test against the unsplit string, and its
exclude_files = list_to_exclude.split(",")isnever used — so
"s.d"is excluded by a list containing"Thumbs.db". This PR only calls it andinherits that. Separate PR:
pr/fix-include-files-fnmatch..protocolwrite to after packing. Worth doing on its own, since.protocolis the watch-folder ingest trigger and a crash mid-packaging can point it at anincomplete SIP. But that is crash-safety rather than counting, and
localAIPstris created insidecreate_protocol()and used later bycreate_xip().Only standard export mode is verified.
data_stats()walks the whole tree, so modes that pack asubset of it (
-assetonly,-singleasset,--representations) may still over-count — no worse thantoday, but not fixed here.
Tests
Six tests exist on my fork but are not in this PR:
yesc.pyends with a bareargs = parser.parse_args(); main(args), soimport yescruns argparse and callsmain(), andpytest cannot load the module.
pr/fix-import-guardfixes that, and these can be added once it lands.test_data_stats_excludes_listed_namestest_data_stats_counts_everything_when_no_exclusionsNoneand""behave as beforetest_data_stats_recurses_nested_subfolderstest_protocol_matches_sip_when_exclusions_fire.protocolmatches SIP contenttest_main_tolerates_namespace_without_excludedfilenamesgetattr()pathtest_sip_report_keys_unchangedsip_reportkeys and types are untouched