Skip to content

Make .protocol report SIP content, not source-folder totals - #6

Open
hyeeyoungkim wants to merge 1 commit into
decirella:mainfrom
hyeeyoungkim:pr/protocol-sip-content
Open

Make .protocol report SIP content, not source-folder totals#6
hyeeyoungkim wants to merge 1 commit into
decirella:mainfrom
hyeeyoungkim:pr/protocol-sip-content

Conversation

@hyeeyoungkim

Copy link
Copy Markdown

Summary

.protocol reports the file count and byte size of the source folder, including files that
-excludedFileNames keeps out of the SIP. This makes data_stats() apply the same exclusion list
the packer already uses, so <files> and <size> describe the SIP. Three functions, about 5 lines,
nothing reordered.

The problem

main() writes .protocol before anything is packed:

create_protocol(data_in_path)   # calls data_stats()
create_xip(args)                # only during packing does include_files() apply -ef

data_stats() walks the source folder with an unfiltered rglob("*"), so excluded files are still
counted. The same return value feeds reporting_std_out(), so the sip_report dict handed to
callers like yesc_bulk.py carries the same numbers — reading sip_report instead of .protocol
does not help.

Example, on a folder of 7 files / 400 bytes, of which 3 files / 300 bytes are content:

$ yesc.py -i src/item_ao1234 -o out -export -sha512 \
          -excludedFileNames "Thumbs.db,.DS_Store,desktop.ini,.BridgeSort"

.protocol:        <size>400</size>  <files>7</files>     <- everything on disk
actually in SIP:  3 files, 300 bytes

Longstanding, and not specific to one branch: data_stats() is identical on all six branches of
this repo carrying yesc.py, and git log -S shows the call order has never differed.

The change

def data_stats(data_path, list_to_exclude=None):
    # only count files that will actually be packed into the SIP
    file_list = [name for name in Path(data_path).rglob("*")
                 if Path(name).is_file() and include_files(Path(name).name, list_to_exclude)]
Function Change
data_stats() new optional list_to_exclude; filters inside the existing comprehension
create_protocol() new optional list_to_exclude; passes it to data_stats()
main() create_protocol(data_in_path, getattr(args, 'excludedFileNames', None))

The same folder then reports <size>300</size><files>3</files>, matching the SIP.

getattr() rather than args.excludedFileNames because callers that import yesc build their own
Namespace, and a plain lookup would raise AttributeError for any that omit the field.

What this affects

  • Without -ef: numbers unchanged, since include_files() returns True for an empty or
    missing list. Covered by test_data_stats_counts_everything_when_no_exclusions. Costs one extra
    call per file.
  • With -ef: .protocol and the printed stats report SIP totals. Anyone reading the old numbers
    as source-folder totals will get different values; on a 300-item production batch the gap was in
    the file count only, bytes negligible, well under 1%.
  • Callers importing yesc: both parameters are optional, so existing calls work unchanged, and
    sip_report keeps its keys and value types.
  • More output with -ef: include_files() prints whenever it has a list to check, so those
    lines now appear while counting as well as while packing. Happy to guard them or add a quiet flag;
    I left them alone to keep the diff small.

What this does not do

  1. Does not touch include_files(). It matches with if file_name in list_to_exclude, a
    substring test against the unsplit string, and its exclude_files = list_to_exclude.split(",") is
    never used — so "s.d" is excluded by a list containing "Thumbs.db". This PR only calls it and
    inherits that. Separate PR: pr/fix-include-files-fnmatch.
  2. Does not move the .protocol write to after packing. Worth doing on its own, since
    .protocol is the watch-folder ingest trigger and a crash mid-packaging can point it at an
    incomplete SIP. But that is crash-safety rather than counting, and localAIPstr is created inside
    create_protocol() and used later by create_xip().

Only standard export mode is verified. data_stats() walks the whole tree, so modes that pack a
subset of it (-assetonly, -singleasset, --representations) may still over-count — no worse than
today, but not fixed here.

Tests

Six tests exist on my fork but are not in this PR: yesc.py ends with a bare
args = parser.parse_args(); main(args), so import yesc runs argparse and calls main(), and
pytest cannot load the module. pr/fix-import-guard fixes that, and these can be added once it lands.

Test Covers
test_data_stats_excludes_listed_names listed names are not counted
test_data_stats_counts_everything_when_no_exclusions omitted, None and "" behave as before
test_data_stats_recurses_nested_subfolders exclusions apply at every depth
test_protocol_matches_sip_when_exclusions_fire end to end: .protocol matches SIP content
test_main_tolerates_namespace_without_excludedfilenames the getattr() path
test_sip_report_keys_unchanged sip_report keys and types are untouched

- data_stats() gains list_to_exclude=None and filters via include_files()
- create_protocol() gains list_to_exclude=None and passes it through
- main() passes getattr(args, "excludedFileNames", None), tolerating
  hand-built Namespaces that omit the field
- .protocol <size>/<files> and the sip_report dict now describe what is
  packaged instead of the raw input folder; keys and types are unchanged
- excluded files such as Thumbs.db, .DS_Store and desktop.ini were counted
  because create_protocol() ran before create_xip() applied include_files()

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant