Conversation
Add Google-style Args: to HuggingFaceSource, Flux, JointFlux, FilterOp, WrappedOp, Tee, StandardizeOp, ThresholdOp, ConnectedComponentsOp, ToTensorOp, resolve_expression. These surface as FluxStudio widget tooltips and navigaitor form-spec field descriptions via confluid.parse_param_docs. New tests/test_node_docs.py pins full coverage; AGENTS mandate added.
…ojection
- Flux.from_ops_yaml(path, source): attach a FluxStudio-exported ops-only Confluid YAML ({ops: [!class:...()]}) to any source. Routes the op-list through confluid.materialize before attaching — confluid.load leaves !class: markers nested under a mapping key deferred, which Flux rejects at iteration by design.
- Discovery categories split by ROLE: Flux/JointFlux/DatasetSplit -> 'engine', HuggingFaceSource -> 'source' (replaces the inverted 'dataset'=engine scheme); test_categories + AGENTS updated.
- Field projection surface exported from package root: SupportsProjection / project / iter_inputs / iter_targets / num_classes.
… engine
Every Sample->Sample op (ops/{copy,numpy,parallel,stash,swap,tee,torch}) now carries
category=op so FluxStudio's positive {op,source,dataset} allowlist surfaces it;
FilterOp/WrappedOp move op->engine (raw-callable wrappers, not GUI nodes).
Updates test_categories + AGENTS Discovery Categories.
…ications - Added tests for target movers and encoders in `tests/test_target_ops.py`, covering `MetadataToTargetOp`, `EncodeTargetOp`, and `DecodeTargetOp`. - Improved type specification tests in `tests/test_typespec.py` to validate closed literals for `Framework`, `ImageLayout`, and `DtypeFamily`. - Updated projection tests in `tests/test_projection.py` to utilize new constants for field names. - Enhanced source tests in `tests/test_sources.py` to validate three-way splits and property API for `DatasetSplit`. - Added round-trip tests for array metadata in HDF5 and Zarr storage in `tests/test_storage.py`. - Refactored tests to ensure clarity and maintainability, including checks for metadata handling in various storage formats.
…espec/storage refactor - Lazy/zero-arg convention applied to every @configurable (29 classes): default all ctor params, defer validation/materialization to the use site (__call__ / cached @Property / .open()); HuggingFaceSource is the reference (no network in __init__). tests/test_lazy_construction.py pins it. - Pre-existing: generic image ops (ops/image), target ops (ops/target), AnnotationJoinSource rename, typespec/projection closed Literals, DatasetSplit/RangeSource/ConcatSource, ThresholdOp two-bound; intake + hf_core + node-manifest removal.
…colormap/PIL Standalone quantization stage (dB spectrogram / logit map → 8-bit grid) with optional fixed vmin/vmax; value_to_image reuses NormalizeToUint8Op.normalize_to_uint8 for its 2-D-map and float-array paths. README + tests updated.
Move the modality-neutral Enable (toggle an op-list via one named CLI flag) and SampleSinkOp (adapt a DataSink as a pass-through op) from waivefront into core dataflux.ops — they thread any Sample through any ops and have no signal dependency. Both are zero-arg constructible with lazy validation. Register entry points (dataflux-ops-enable/-sink), pin categories/groups in test_categories, and move test_enable here.
- dataflux.labels.LabelMap: bidirectional name<->id map, the fittable companion to EncodeTargetOp/DecodeTargetOp. fit() derives a deterministic mapping from a target stream (sklearn LabelEncoder), save/load round-trips marainer's class_names.json, encode_op/decode_op hand back the dataflux ops. - Zero-arg constructible, side-effect-free __init__; non-empty enforced lazily. - Exported from dataflux.__init__; scikit-learn dep (lazy-imported in fit).
… accessors Sample.metadata is now Union[Dict, List[Dict]] so a Sample distinguishes a single item (one dict) from a batch (a list of per-item dicts, as the collate functions produce). Adds Sample.is_batched plus the narrowing accessors .meta (the dict; raises on a batch) and .batch_meta (the list; raises on a single); describe()/with_type() guard the batch form. Rewrites all per-sample .metadata[...] dict-access through .meta.
- CocoToTorchVisionDetectionOp (HF/COCO objects -> {boxes xyxy, labels})
- MasksToDetectionBoxesOp (segmentation mask -> boxes; per-instance or CC)
- factor connected_component_bboxes out of ConnectedComponentsOp (shared)
- ToTensorOp(mode=...) coerces PIL via Image.convert before arraying (RGBA fix)
Wraps any Sample→Sample callable behind a coin flip (probability=0.5). Confluid Fluid ops are resolved lazily on first call and then cached. Registered as @configurable(category="op", group="compose", random=True) and wired into the confluid.configurables entry-point group. 8 unit tests covering zero-arg construction, deterministic extremes (probability=0/1), no-op passthrough, Fluid lazy resolution, and confluid registry membership.
- _resolve() raises ValueError if called with None (guard against misuse; __call__ already filters None via self.low_level/high_level is not None, but the signature now reflects it) - __call__ also strips blank STRING values (empty widget) to None before the None check, so leaving a FluxStudio STRING widget empty silently disables that bound
…tions - Introduced TransformChain class to group a sequence of ops into a single unit. - Each op in the chain is applied in order, and if any op returns None, the chain stops early. - Added SqueezeOp and UnsqueezeOp for tensor dimension manipulation in PyTorch. - Implemented StashTargetOp and UnstashTargetOp for managing sample targets in metadata. - Enhanced RandomApply to support reproducibility with a random_state parameter. - Updated tests to cover new functionality and ensure compatibility with existing ops.
…nion-attr ConfigureOp (per-sample parameter injection) and FormulaOp (restricted math formula over sample.input) are now exported from dataflux.ops and covered by tests/test_ops.py + test_categories.py. ConfigureOp.__call__ now tracks current: Sample (not Optional) so mypy no longer flags union-attr on .input after the ops loop.
…storage-sink discovery categories - 1-D FFT family (numpy+torch FourierOp/InverseFourierOp/FftShiftOp/IfftShiftOp) + WindowOp/SpectrumScalingOp, backed by the dataflux.windows calibration library. - ops/image: draw_text + array_histogram/select_channel/channel_count helpers (back FluxStudio Draw Text / Array Histogram nodes). - CaptureOutputOp (records an op @output value into metadata). - Storage sinks (HDF5Sink/ZarrGroupSink/ZarrBatchSink/DirectorySink) carry category="sink" + dataflux-storage-* entry points, so FluxStudio surfaces them as DatasetProcessor sink nodes.
…m; fix ThresholdOp numeric bound handling - New `dataflux.ops.metadata.DropMetadataOp`: strips metadata keys matching fnmatch globs (include-wins model like rsync); entry-pointed as `dataflux-ops-metadata` - New `dataflux.ops.debug.PrintSampleOp`: pass-through probe logging/printing per-sample summary (shape+dtype+metadata); level restricted to trace/debug; entry-pointed as `dataflux-ops-debug` - `UnstashInputOp`/`UnstashTargetOp` gain `remove: bool = True` — deletes the stash key after restoring so snapshots never leak into downstream sinks; only the final unstash of a fan-out key removes it - `ThresholdOp._resolve` now accepts any float()-able value (NumPy scalars, 0-d arrays) not just Python int/float, enabling the ConfigureOp per-sample value-chain path - AGENTS.md and pyproject.toml updated to document new ops and entry points
…nclude normalization and extraction utilities
The logflow distribution was renamed log-flow -> logflow-ml (PyPI ultranormalization similarity check rejects log-flow vs the existing abandoned logflow project). Import package unchanged (import logflow).
The logging library was renamed and republished as loggair (github.com/Gearlux/loggair, dist == import == loggair, v0.1.0): imports `logflow` -> `loggair`, dependency logflow-ml>=0.2.0 -> loggair>=0.1.0, LOGFLOW_* env vars -> LOGGAIR_*, docs/CI references updated.
…code extraPaths with generated pyrightconfig.json
…, 1->N ops, SigMF Complete identity switch from dataflux (taken on PyPI): package dir, dist name, entry points, docs. On top of the rename, the FlowGraph plan lands: - context.py: per-sample named-cell Context (the graph data plane, never sample.metadata) activated via a ContextVar around the plain op loop. - ops/context.py: Save/Use/Drop/Apply/Capture/Mix — the flat-list building blocks that let the serial Flux engine execute fan-out/fan-in graphs. - flow.py: the readable named-step flow: document, the FlowGraph engine, and BIDIRECTIONAL converters to_ops/from_ops with pinned execution parity both ways (Flux.from_flow_yaml / FlowGraph.from_yaml/from_ops_yaml). - kinds.py: op-contract introspection (sample/pair/value/any taxonomy, EXPANDS detection, class-attr overrides); Flux(native=True) carries metadata-free pairs/values with per-op adaptation. - collate.py: pluggable collate registry (sample/pair/value defaults, additive task aliases). - 1->N expanding ops: generator-returning ops flatten depth-first across all routes; expanding pipelines are iterable-only (len/getitem raise). - storage/sigmf.py: SigMFSink<->SigMFSource recording pair (hand-rolled JSON, dtype<->core:datatype, meta_encoder/decoder hooks). - storage/query.py: SupportsMetadataScan protocol (HDF5/Zarr/SigMF scan metadata without array loads) + MetadataFilterSource (where expressions). 951 tests; examples/flow_graph.py runs the full flow<->ops round-trip.
Regenerated via 'aisland jenkins scaffold sampleflux --force' after the rename — picks up the directory-examples runner (examples/*/run.py) the templates gained on 2026-07-16.
…; transform-taxonomy grid; op consolidation; docs split
Transform taxonomy (kinds.py): field-scope x call-style GRID with combinable
per-parameter bindings — Pair/InputMeta/TargetMeta NamedTuple views, INPUT/TARGET
marks, packed/unpacked call styles, metadata-only scope, single-view-return guard.
Relocations (no back-compat; new homes in waivefront):
- storage/sigmf.py -> waivefront.sigmf (query layer stays; protocol is structural)
- windows.py + the 1-D FFT/windowing/scaling op family (numpy+torch) ->
waivefront.{windows,fourier,fourier_torch}
- paired.py (AnnotationJoinSource/AnnotationStore) -> waivefront.paired
Op consolidation (one wiring plane):
- DELETE Tee (executionally identical to TransformChain) and CaptureOutputOp
(context Capture is the op; Apply(source=cell) replaces ConfigureOp+Unstash idiom)
- stash family narrowed to Parallel-boundary crossing + sink persistence;
graph wiring is exclusively the context ops
Docs: README split into slim landing page + 8 per-topic docs/*.md; all
FluxStudio references genericized (UI/engine separation); AGENTS mandates
updated (modality-neutral rule, consolidation, narrowed stash charter).
…ted per-transform op families - AlbumentationsOp / TorchvisionTransformOp adapters (joint input+mask/boxes draws, target Literal knob) - auto-generated Alb*/Tv* op families via ops/_augment_bridge (one @configurable op per library transform) - palette groups augment/albumentations + augment/torchvision; [vision] extra for torchvision - docs/augmentation.md + two runnable examples; RandomApply/TransformChain/Enable test updates
- _augment_bridge: targeted type-ignores on dynamic base __init__/property access - albumentations_transforms / torchvision_transforms: module __getattr__ fallback so mypy accepts the generated names; raises a descriptive AttributeError for missing transforms
… discovery docstring - graph.md/kinds.md link the context-wiring and collate-registry rationale to docs/architecture.md - projection.md describes the LabelMap save format generically (docs never name dependents) - discovery.py module docstring expanded (schema surface + MCP end-goal wording)
…ge, engines (migration stages 1-3)
The typed bag is THE sampleflux data model (legacy Sample survives only until every
consumer migrates; staged plan in the root TASKS.md — purge stage renames TypedSample
back to Sample).
Stage 1 — frozen typed API + core primitives:
- sampleflux.bag: TypedSample (named bag of typed items, role tags input/target/aux/pred),
hybrid items (Image/Mask NDArrayItem subclasses w/ attr-preserving __array_finalize__;
Regions/Label dataclass wrappers), kernel-registry type dispatch (MRO-aware), Transform/
Pipeline with once-per-sample params + only= key filter
- bare library transforms drop into Pipeline via the adapter coercion registry
(register_adapter/coerce_transform; torchvision-v2 + albumentations matchers self-register
by MRO module name — no eager imports); native HorizontalFlip DELETED (libraries cover
augmentation; kernels survive as the tests/_bag_fixtures.py FixtureFlip parity fixture)
- full typed surface frozen at the package top level (from sampleflux import TypedSample, ...)
- primary(sample, role) + TypedSample.merge (ordered field/role union, last-wins) + rename
- bag/io.py item codec registry (EncodedItem/encode_item/decode_item/register_io) extracted
from interop (the Sample bridge half stays only until the purge stage)
- ops/structure.py: SetRole/RenameField/DropField/CopyField/SelectFields (entry-pointed)
- collate 'typed': batched TypedSample (per-field stacked payloads, attrs as lists) —
the ONE batch convention replacing list-form and per_sample dict-nest
- typespec.infer_field_types (per-field specs beside the legacy SampleType)
Stage 2 — typed storage (one field-group layout, three backends):
- HDF5/ZarrGroup/Directory write typedsample-v1 (per-field group: item type + role +
native scalar attrs + payload dataset + array attrs; JSON-tagged wire format in
storage/base.py split_attrs/restore_attrs — tuples survive); one carrier per store
(cross-carrier append raises); backends serialize ONLY through the bag.io codec so
externally-registered item types round-trip with zero storage edits
- DirectorySink gains its missing matching DirectorySource; ZarrBatchSink typed path
(primary payload rows + one-time uniform item template)
- typed metadata scans yield nested {field: {attr: value}}; MetadataFilterSource.where
addresses '<field>.<attr>' (query._AttrView); TypedDataSource/TypedDataSink protocols
Stage 3 — typed engines:
- TypedSample passes VERBATIM on every Flux route (core._as_carrier — no native=True
needed) and through _apply_op/_apply_op_native (bag applied verbatim; kinds binding +
stored-type refresh are legacy-only); Use/Apply/Capture/_cell_field typed-aware
(Apply gains key=; _cell_field on a bag = named item or primary())
- FlowGraph merge_from: typed fan-in (union via TypedSample.merge, slot-order last-wins;
mutually exclusive with legacy target_from/metadata_from) lowered to the new MergeFields
context op, lifted back by from_ops; bind grammar gains step[key] (bare step = primary)
- branch idiom: produce -> SelectFields([new_field]) -> merge_from
Docs: typed-model.md rewritten as THE model (engines/storage/query sections),
architecture.md record updated, AGENTS mandates rewritten for the migration.
1126 tests green (zero legacy regressions); mypy clean from the workspace root.
…edComponents Additive typed-bag twins of three generic ops so a typed pipeline can go array -> Image -> Mask -> Regions with no legacy Sample (legacy *Op untouched). 1160 passed, mypy clean (108 files), flake8 clean. - ops/image.py: ConvertToImage(Transform) — array item -> Image field (reuses value_to_image) - ops/numpy.py: Threshold(Transform) -> Mask; ConnectedComponents(Transform) -> Regions with (row_min,row_max,col_min,col_max) bin-box tuples (the connected-components contract) - bag/items.py: Regions.extras dict (per-box parallel arrays / region-set measurements) - docs/architecture.md: record for native typed transforms that change a field's TYPE - tests/test_typed_generic_ops.py: parity vs legacy + the array->Image->Mask->Regions chain
…MetadataToTarget Additive typed-bag twins for the classification input/target op path (legacy *Op untouched). 1199 passed (+39), mypy clean (109 files), flake8 clean. - ops/torch.py: ToTensor(Transform) — array item -> CHW float32 Image(layout="CHW") payload byte-identical to ToTensorOp; numpy->tensor happens at the collate/model boundary (a live-Tensor item is the flagged Tensor-subclass follow-up). - ops/target.py: EncodeTarget/DecodeTarget (delegate to the legacy pinned-map ops for byte-parity) + MetadataToTarget (largely redundant in the typed model — the source emits a Label target directly — kept for parity/config-compat). - docs/architecture.md: decision record; tests/test_typed_target_ops.py (39 tests).
…tion + MasksToDetectionBoxes Additive typed-bag twins for the detection target path (legacy *Op untouched). 1286 passed (+22), mypy clean, flake8 clean. - ops/target.py: CocoToTorchVisionDetection (Label objects -> Regions target) and MasksToDetectionBoxes (Mask -> Regions target), both role "target". Target is a Regions item (boxes xyxy + labels) — typed_collate gathers it into per-sample lists (variable-N detection-target convention). Delegate to the legacy ops on a shim Sample for byte-parity (pinned via torch.equal). - docs/architecture.md: consequence note; tests/test_typed_detection_target_ops.py (+23).
"GPU-aware batch processing engine" and "S3 storage backend support" each restated a detailed entry further down the file. Kept the detailed halves; 21 open -> 19.
The tracking package is now matrainer — a portmanteau of MARINER (the one who keeps the log) and TRAINER — so every import, class path, YAML tag, extra name and doc reference here moves with it. There is no back-compat alias upstream, so a missed reference fails loudly rather than silently resolving. Mechanical apart from that: every changed line is the token swap.
…ead anchor Adapted from confluid: the absolute-URL rule is dropped because this project's README links to docs RELATIVELY, which is correct for a project not published to PyPI — that rule exists because a PyPI landing page cannot resolve a relative link. Those relative links are still covered, since the README is in the scanned set. It found a real one on the first run. `docs/kinds.md` pointed at `architecture.md#10-the-frameworks-batching-half-lives-beside-the-collate-2026-07-30`, but the heading gained a parenthetical since that link was written, so the real anchor ends `...-recordstreamkeras-2026-07-30`. That is the failure mode this check exists for: a wrong anchor does not 404, it silently lands the reader at the top of the page. Nothing would ever have reported it. 34 cross-doc links, all resolving now.
…storation contract
The collate becomes a CHOICE rather than an accident: `collate_list` ("list")
stacks nothing and keeps items as items, `RecordSequence` takes `collate=`, and
every read-back helper (`batch_values` / `batch_regions` / `batch_metadata`)
accepts both shapes. A stack failure now names the key, the differing shapes and
the way out, where numpy's raw ValueError named none of the three.
A bare albumentations transform receives only albumentations' own key vocabulary,
so a `Regions` detection target sits out the call and does not move — measured, a
Resize takes a 200x200 image to 64x64 and leaves the boxes where they were, and a
HorizontalFlip changes no shape at all. `_apply_op` now warns, matching on the
library's OWN taxonomy (a DualTransform is by definition one that applies to
boxes) so there is no name list to drift, and once per transform TYPE.
Every op that makes or re-frames a Regions now fills in its `canvas`, including
for an empty target — the frame was previously known only where it was least
needed. `image_frame` is the shared read; the lookup stays narrow so a Regions'
own [N, 4] box array is never mistaken for an N x 4 raster.
Plus `RestorationOutput` + `restoration_output()` — one key (`image`), because an
image-to-image model emits the answer rather than something to interpret.
…ord_dataset, loader_slots, RunnableTask The data half of the workspace trainer-base consolidation: - batch.py: per_record_predictions (the batched-output -> per-record slices ladder, lifted from three byte-identical consumer copies) - core/stream.py: prepare_record_dataset = ensure_record_dataset + ensure_materialized (the fork-safety normalize+warm pair, documented once) - loaders.py (new, torch-extra-gated): loader_slots -> LoaderSlots NamedTuple of Lazy[DataLoader] train/val/test slots, with collate_fn + **loader_kw passthrough and a shuffle-refusal guard; deliberately NOT root-exported so 'import recordstream' stays torch-free - runnable.py: RunnableTask Literal (the standard four-verb vocabulary) - collate.py docstrings: the task-collate-stays-engine-internal decision
…e examples Current huggingface_hub rejects namespace-less ids (HfUriError); worse, a stale local cache makes a bare id appear to work on one machine and fail on a fresh one. docs/sources.md now states the rule beside the first example.
Every run artifact (tracker metrics, checkpoints, exports, framework logs) now writes under runs/, so a project needs exactly one ignored output directory. See the workspace AGENTS.md mandate 'Run Output Lands In A Gitignored runs/'.
The structured box item carried two coordinate systems in one field (pixel [x0,y0,x1,y1] or signal [f0,f1,t0,t1]) with no discriminator. It is now Boxes and PIXEL-ONLY by contract; the signal-domain region item moved to the signal package, registered through the same open item registry. No back-compat alias — a stored typedrecord-v1 'Regions' tag fails loudly. - connected_component_bboxes -> connected_component_boxes, renamed WITH its contract: half-open xyxy (x=col, y=row) instead of inclusive row/col tuples; ConnectedComponents emits Boxes with canvas set (empty masks included); masks_to_detection's reconciling transpose is gone - batch_regions -> batch_boxes, _REGION_FIELDS -> _BOX_FIELDS - geometry-desync guards renamed; they now correctly stay silent for a domain package's raster-independent region item - docs: record-model example is a clean pixel one; architecture record #14 captures the split + the ConnectedComponents normalization
…s actionable guidance A parens-less !class:X in a source: slot is a CONFIG error (the fix is !class:X() ), and the view sources (RangeSource/ConcatSource/DatasetSplit via sources/base.py::_guard_live_source) now raise the same slot-naming TypeError Stream does instead of a cryptic 'got Class' — deliberately message-only, never flowed (user decision 2026-08-10; distinct from the flow-first free functions). _fluid_source_guidance takes the owning slot name. Docs note in docs/sources.md; pins in tests/test_view_sources_deferred.py.
…resolution An op with a field=-style knob resolves its entry through ONE helper: an explicit field must exist and hold the expected item type (ValueError naming owner, param and the record's keys), a blank/None field falls back to the first value of that type. fallback=False forbids the blank-field scan (mandatory-key ops), required=False turns every miss into None (the probe form); resolve_entry returns (key, value) for ops that write back to the resolved key. Typed overloads keep required=True call sites Optional-free. Extracted from twelve byte-parallel _find_* copies in the signal package — the item_value/first_value story again.
The HDF5 row decoder HDF5Source iterates through is now the public
read_record_group(group, slices=...): the slices map ({record key ->
slice of that field's data dataset}) is an h5py PARTIAL read, so a
consumer windowing large stored rows decodes one window without
materializing the row. Sliced decode pinned identical to
full-read-then-slice; attrs are never sliced; slicing a payload-free
field raises. Documented in docs/storage.md + the storage mandate;
the source-windowing TASKS item refreshed to current reality.
ResizeDetection's target ride-along probe is the shared resolver's probe form (required=False, fallback=False) instead of a hand-rolled record.get + isinstance, and the EncodeTarget/DecodeTarget _find_label twins (18 byte-parallel lines each) collapse into one module-level _find_label_key with the pinned messages parameterized by op name. The six payload-kind finders (Threshold/ToTensor/ConvertToImage/ ConvertToMask and the two two-tier mask finders) deliberately do NOT fold: they search by what the payload IS (ndarray/PIL/2-3-D), not by item type, and routing them through resolve_entry would grow it the predicate knob the workspace bans; _find_label_key's docstring states the same boundary for the type-tuple + TypeError contract.
`!class:X` and `!class:X()` are the same target since confluid merged its eager and deferred markers, so "write it with parens" was advice that changes nothing — the user follows it and hits the identical error. The only ways to reach these messages now are asking for deferral explicitly (`_partial_: true`) or wiring a hand-built `PartialClass(...)`, so they say to drop the deferral instead. A source or op slot needs a live object and nothing downstream will flow it.
Phase 5a. The names became aliases when confluid merged its two construction
modes (Class/Instance -> Target) and renamed Lazy -> Partial; this moves every
consumer onto the canonical spelling so the aliases can be deleted.
lazy_param_names -> partial_param_names
LazyClass -> PartialClass
Lazy -> Partial
Instance -> Target
Class -> Target
Mechanical, by codemod. `Class` and `Instance` are ordinary English words, so
they were renamed ONLY in a file that demonstrably takes them from confluid —
an import or a `confluid.`-qualified reference. The other three are
confluid-specific and safe everywhere. The lowercase `lazy=True` registration
mark is untouched: it is still spelled that way.
…ute reference confluid removed attribute references (record 19, phase 2): `!ref:my_split.train` is refused. A view is a DatasetSplit marker with `split:` set — the recipe anchored on the train view and <<:-merged into the others; every view !ref:s the same upstream source, which therefore still loads once. No code change (the split= selector already existed): the class docstring, docs/sources.md, docs/runnable.md and AGENTS.md now show that spelling.
…folded (cli, stream, flow.graph, tests, docs)
Regenerated after `aisland source set loggair local`: loggair joins confluid and liquifai in the internal-dependency block, installed with --no-deps from git+https://github.com/Gearlux/loggair.git@main before `-e .[dev]` so the extra resolves it pre-satisfied instead of reaching for PyPI. Generated artefact — the template is aisland's, not this file.
…s` override `persistent_workers` still DERIVES from `num_workers != 0` when the new keyword is left at `None`, which is what nearly every run wants. An explicit value covers the case the derivation cannot see — a HOST fact: macOS terminates persistent workers slowly enough that a short run spends longer stopping than training, so a config there sets `false` while keeping its workers. `True` with `num_workers=0` is refused HERE with a message naming the fix; torch otherwise raises for that pairing when the loader is first iterated, minutes into a run. That invariant was the one thing the derived-only value protected by construction.
The generated slow-test pipeline writes `slow-test-report.xml` (aisland/aisland/services/jenkins.py:864 for the Jenkins stage, :1068 for the GH Actions job), which the exact-name `test-report.xml` entry never matched — so a local slow run dropped an untracked artifact one non-ignored file away from being committable. The glob covers both names.
A deferred marker is materialized only when its target class declares the identity protocol (dataset_uri / dataset_url) or a source slot the walk would follow; anything else answers None without being constructed — a run provenance capture used to build a full pl.Trainer and attempt a dataset-less DataLoader per walk just to learn each names no dataset. An unresolvable target keeps the old build path, so nothing that answered before answers differently.
A pipeline can carry its own inference: ModelPredict runs any callable model wrapper on each record and stamps the prediction back as a record field — a Label (classification), Boxes (detection), an int class mask (segmentation, <output>_mask), or the restored image (restoration). The wrapper's heavy work (build the network, load checkpoint_path) happens in its solidify(), called lazily on the first record; the op imports no ML framework (torch duck-typed). Registered as category="op" (new confluid.configurables entry point), so a visual editor's palette and registry pickers list it automatically.
Regenerated by 'aisland source set loggair pypi' (loggair 0.2.0 released): the internal-dependency pre-step installing loggair from git@main is gone; .[dev] now resolves the published wheel.
Regenerated by 'aisland source set confluid pypi' (confluid 0.3.0 released): the internal-dependency pre-step installing confluid from git@main is gone; .[dev] now resolves the published wheel.
Regenerated by 'aisland source set liquifai pypi' (liquifai 0.2.0 released).
…rs, absolute README links The package built fine without any of this, which is why all of it was missing at once: `readme` / `license` / `license-files` / `classifiers` / `[project.urls]` are optional to the build but ARE the PyPI project page. Add them, plus a LICENSE and a CHANGELOG, and set the version to the pre-release it actually is (0.1.0a1, matching the `Development Status :: 3 - Alpha` classifier). - Floors now state what the suite runs against: confluid>=0.3.0, loggair>=0.2.0, liquifai>=0.2.0. The old `confluid>=0.1.0` described a combination nobody could install — liquifai itself requires confluid>=0.2.0. - README links to repo files by absolute GitHub URL: PyPI resolves a relative link against pypi.org, so `docs/storage.md` 404s there while working on GitHub. - tests/test_packaging.py pins the metadata, the classifier/version agreement, the declared readme+license files, and that no requirement is a direct URL (PyPI refuses those); test_docs_links.py gains the absolute-link rules. - setuptools floor raised to >=77 for PEP 639 `license`/`license-files`. - Two follow-ups recorded in TASKS.md (torch-less discovery warning, the `materialize_runnable` shim liquifai 0.2.0 makes removable).
Adds the hand-written release pipeline this project was missing: a `v*` tag now builds the sdist + wheel, guards that the tag matches the pyproject version, runs `twine check --strict`, smoke-tests the built wheel in a clean venv from /tmp (the repo directory shares its name with the package, so a cwd-local import would silently test the source tree), and publishes through PyPI Trusted Publishing — no stored token. Also completes the distribution itself: - MANIFEST.in ships CHANGELOG.md in the sdist. setuptools builds an sdist from what is declared — `readme` pulls README.md, `license-files` pulls LICENSE, `package-data` pulls the package — and nothing references a changelog, so it was silently absent. - build + twine join the dev extra, matching the other published projects. - The changelog entry is dated.
…ports PR #7's Quality Gates failed on three pre-existing errors that no local check can see: keras ships no py.typed, scipy.ndimage stubs are absent, and matplotlib is not installed at all. None is a defect. All three are optional by design — `[keras]` and `[vision]` extras, plus the lazy matplotlib import inside ops.image._apply_colormap that only a non-gray colormap needs. What was missing is that this project's own override table never learned about them. The gap is invisible locally because the project is type-checked twice with different configs: `mypy recordstream` from the workspace root reads the shared mypy.ini, which already carries scipy and matplotlib, and reports Success on the same tree that CI fails. CI runs `mypy .` from the project directory and reads only pyproject's table — the config a published package must satisfy. Both invocations are now clean. AGENTS.md records why the project's own config has to be self-sufficient rather than leaning on the workspace one.
`ops.image._apply_colormap` lazily imports matplotlib for every non-gray colormap, and CI installs `.[dev,torch,keras]` — which had no matplotlib — so tests/test_typed_generic_ops.py::TestConvertToImage::test_parity_with_render_ helper_default_sizing died with ModuleNotFoundError. It passes locally only because the workspace venv has matplotlib installed for other reasons. Added to `dev` rather than skipped, matching the torchvision precedent two lines above: skipping would leave the colormap half of ConvertToImage and value_to_image untested in CI. It stays out of `dependencies` deliberately — the gray path is matplotlib-free by design, and a consumer rendering only grayscale must not be made to install a plotting library. Verified in a venv built to match CI exactly (`-e .[dev,torch,keras]` on 3.12): 841 passed, and all five examples run.
`notebooks/01 - cat_exploration.ipynb` raised RuntimeError when DATA_ROOT was unset, so Verify Notebooks failed on every runner — CI sets it nowhere and no .env is committed. It passed locally only because the notebook walks up to the workspace .env and loads it. The variable only chooses WHERE the HF cache lives. The notebook then loads huggingface/cats-image — one image — from the Hub, which needs no configured volume. So an unset DATA_ROOT now falls through to huggingface_hub's default cache, while a DATA_ROOT that is set but missing still raises: that is a misconfigured machine, not an absent preference. Verified both paths. In a checkout with no .env in any parent, matching a CI runner: "DATA_ROOT = (unset — using the default HF cache)" then "Dataset loaded with 1 examples." On this workstation, unchanged: HF_HOME = /Volumes/Store/ huggingface. Note that `env -u DATA_ROOT` alone reports a FALSE PASS — the notebook re-reads the .env itself — so the reproduction has to run outside the workspace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First public pre-release of RecordStream, plus the packaging work that makes it installable.
The release
0.1.0a1, published to PyPI by a tag-triggered workflow. Verified end to end before opening this PR: the sdist builds,twine checkpasses on both artifacts, and installing from the sdist into a clean venv resolves every dependency from PyPI (confluid==0.3.0,liquifai==0.2.0,loggair==0.2.0), imports, runs a pipeline, and exposes therecordstreamconsole script.What was missing
The package was sound; the distribution around it was not.
LICENSEdespite the README claiming MIT, and noCHANGELOG.md.readme,license,classifiersor[project.urls]inpyproject.toml. All four are optional to the build — the wheel builds byte-identically without them — so the gap was invisible until the project page would have rendered blank.twine checkreported onlylong_description missing.confluid>=0.1.0described a combination nobody can install:uv pip install recordstream confluid==0.1.0fails with "Because liquifai<=0.1.0 depends on confluid>=0.2.0 … your requirements are unsatisfiable". Floors now state what the suite actually runs against.pypi.org, not the repository — everydocs/*.mdlink would have 404'd while working perfectly on GitHub. Converted to absolute URLs, matching the other published projects.MANIFEST.in, so the sdist shipped no changelog.v*tag did nothing.The release pipeline
Hand-written (
ci.ymlis generated; this one is separate so regeneration cannot clobber it). On av*tag it guards that the tag matches the pyproject version, builds, runstwine check --strict, smoke-tests the built wheel in a clean venv from/tmp— the repo directory shares its name with the package, so a cwd-local import would silently test the source tree instead of the wheel — and publishes via Trusted Publishing (OIDC, no stored token).Tests
Five new packaging rules in
tests/test_packaging.py, each written against a measured failure rather than a checklist, plus two README rules intests/test_docs_links.py. Every one failed before the corresponding fix. The direct-URL rule is the sharpest: PyPI refuses anyRequires-Distcarrying a URL (warehouse/forklift/metadata.py, "Can't have direct dependency"), and the rejection arrives only after the artifact is built, checked and uploaded.Quality
isort/black/flake8clean,mypyclean across 107 source files, 841 tests passing, and all fiveexamples/*.pyrun.