Skip to content

feat: add schema and entity helper utilities - #107

Open
kaitj wants to merge 4 commits into
mainfrom
feat/generalize-ents/phase1
Open

feat: add schema and entity helper utilities#107
kaitj wants to merge 4 commits into
mainfrom
feat/generalize-ents/phase1

Conversation

@kaitj

@kaitj kaitj commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

This PR

Phase 1 of the generalized entities refactor (supersedes #86). Adds a schema-driven helper function for querying BIDS entities in order to replace hard coded entity assumptions in downstream phases.

_schema.py — 3 new functions:

  • get_dataset_types(adapter) — dataset types from schema rules
  • get_json_data_suffixes(adapter) — JSON data file suffixes (replaces hardcoded {"coordsystem"})
  • get_entity_directory_order(adapter) — deque of entity prefixes by directory depth

_entities.py — 5 new functions:

  • get_entity_name() — maps entity type to short prefix ("subject" → "sub")
  • get_entity_regex() — compiled regex for prefix-value matching
  • get_entity_glob_pattern() — glob pattern for entity discovery ("sub-*")
  • get_root_entity_types() — root-level directory entity prefixes
  • get_file_entity_prefixes() — non-directory, non-special entity prefixes

All three entity lookup functions gracefully handle unknown entities by returning None and logging a warning.

Type of Change

  • Bug fix
  • Enhancement / feature
  • Documentation
  • Other

Related Issue(s)


Stack created with GitHub Stacks CLIGive Feedback 💬

@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown

Coverage

Coverage Report
FileStmtsMissCoverMissing
__init__.py100100% 
__main__.py69888%106, 129, 131–132, 137, 165, 169, 173
_entities.py128199%301
_indexing.py209598%150, 159–160, 393, 435
_logging.py31487%31, 38, 40–41
_metadata.py49492%40–41, 67, 72
_pathlib.py21386%19, 21, 23
_schema.py120199%298
_version.py110100% 
pybids
   __init__.py40100% 
   _bidsfile.py381366%67–69, 73–75, 79–81, 85–87, 91
   _layout.py1664573%68, 77, 86, 109, 119–120, 123, 143–144, 158–159, 173–174, 177–181, 185, 187–188, 191–192, 219, 224, 238, 315–317, 379–384, 386, 389–394, 396, 450, 470
   _utils.py14564%47–51
TOTAL8708990% 

Tests Skipped Failures Errors Time
175 0 💤 0 ❌ 0 🔥 11.060s ⏱️

Comment thread bids2table/_entities.py Outdated
Comment on lines +223 to +227
_FORMAT_RE: dict[str, str] = {
"label": "[a-zA-Z0-9]+",
"index": "[0-9]+",
"special": ".+",
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that label and index are found in schema.objects.formats. Label should support plus (+) characters.

https://github.com/bids-standard/bids-specification/blob/bb1df92/src/schema/objects/formats.yaml#L21

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! These are now derived from the schema with the "special" fallback and stored as part of the adapter in 219b02f instead of the hardcoded regex pattern mapping.

@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown

Benchmark Results

Local index (1 workers) Remote index (4 workers) Bold query Metadata query Subject query
feat/generalize-ents/phase1 128.154 (128.202 ± 0.231) ms 249.990 (249.951 ± 1.752) s 341.321 (348.839 ± 42.954) µs 400.548 (394.446 ± 32.615) µs 53.590 (61.219 ± 17.102) µs
main 130.342 (130.336 ± 1.012) ms 250.475 (250.636 ± 1.021) s 435.788 (444.651 ± 31.698) µs 437.217 (447.213 ± 40.336) µs 52.945 (55.293 ± 8.119) µs
v2.3.1 129.409 (129.279 ± 0.470) ms 250.255 (250.540 ± 1.190) s 416.882 (416.725 ± 35.158) µs 400.756 (413.598 ± 29.493) µs 56.176 (60.140 ± 9.453) µs
feat/generalize-ents/phase1 vs main ratio ⚪ 0.983 ⚪ 0.998 🟢 0.783 🟢 0.916 ⚪ 1.012
feat/generalize-ents/phase1 vs v2.3.1 ratio ⚪ 0.990 ⚪ 0.999 🟢 0.819 ⚪ 0.999 ⚪ 0.954

median (mean ± std)

🔴 Slower   ⚪ No change (<5 %)   🟢 Faster

@kaitj
kaitj force-pushed the feat/generalize-ents/phase1 branch from 219b02f to 6011261 Compare August 11, 2026 20:51
@kaitj
kaitj marked this pull request as ready for review August 20, 2026 18:14
@kaitj
kaitj force-pushed the feat/generalize-ents/phase1 branch 2 times, most recently from 6d659e2 to b418662 Compare August 27, 2026 20:30
Add `rules` field to BIDSSchemaAdapter storing `schema["rules"].to_dict()`
so downstream helpers can query directories, files, and tabular data
without re-loading the schema. The `Namespace` cannot be pickled
(circular references), so the dict representation is used with
`hash=False` to keep adapter hashing stable (similar to entities).

Add three schema helper functions:
- get_dataset_types(adapter) — returns dataset type keys ("study", "raw", "derivative")
- get_json_data_suffixes(adapter) — derives JSON data file suffixes from
  rules.files/tabular_data, replacing the previously hardcoded {"coordsystem"}
- get_entity_directory_order(adapter) — BFS over rules.directories to return
  depth-ordered deque of entity prefixes (e.g. sub, tpl, ses, cohort)

All helpers accept the adapter (not spec) to ensure the queried rules
correspond to the same schema spec used to build the adapter, avoiding
repeated load_schema calls.
Add schema-driven helper functions for querying BIDS entity metadata,
replacing hardcoded entity assumptions.

New functions in _entities.py:
- get_entity_name() — map entity type to short prefix (e.g. "subject" -> "sub")
- get_entity_regex() — compiled regex for prefix-value matching
- get_entity_glob_pattern() — glob pattern for entity discovery
- get_root_entity_types() — root-level directory entity prefixes
- get_file_entity_prefixes() — non-directory, non-special entity prefixes

Unknown entity types return None and log a warning instead of raising,
matching the graceful degradation pattern used throughout bids2table.

get_file_entity_prefixes excludes all directory entities via
get_entity_directory_order() (not just root-level ones), so ses, cohort,
and future directory entities are also filtered out.

Adds a module-scoped adapter fixture to conftest.py so tests load the
BIDS schema once instead of repeatedly.
Extract label and index format patterns from schema.objects.formats
instead of hardcoding them. This ensures regex patterns stay in sync
with the BIDS schema version.

The adapter now carries a `format_patterns` dict with only the
entity-relevant formats ("label", "index"). "special" is added as a
fallback since it is not defined in the BIDS schema.
@kaitj
kaitj force-pushed the feat/generalize-ents/phase1 branch from b418662 to 6d980c1 Compare September 2, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generalize entities [Phase 1]

2 participants