Daylily Omics Analysis

Daylily execution requires the broader Daylily ecosystem. daylily-omics-analysis is the Snakemake 7 execution-plane repository. It consumes manifests and staged data prepared by daylily-ephemeral-cluster, runs analysis on prepared AWS ParallelCluster headnodes, and emits evidence. Ursa and Bloom are also key Daylily services: Bloom manages upstream sequencing/run objects, and Ursa brokers operator worksets and DayOA launches. DayOA does not create AWS infrastructure, interpret QC, release results, or own canonical artifact registration.

DayOA produces local evidence and workflow provenance. External orchestration handles artifact registration and downstream evidence import after export.

What This Repo Owns

Surface	Responsibility
workflow/Snakefile and workflow/rules/	Snakemake 7 workflow DAG and target aliases.
workflow/scripts/	Deterministic helpers for staging, validation, MultiQC shaping, and local evidence manifests.
config/samples.tsv	One row per biological sample or control. Supplied by workset staging.
config/units.tsv	One row per sequencing unit, lane, read pair, CRAM/BAM, or downsampled unit. Supplied by workset staging.
results/day/<build>/	Analysis evidence generated by the DAG.
results/runs/<RUNID>/	Run-context evidence, including run QC and BCL Convert outputs. See docs/ops/results_directory_structure.md.
docs/	Current operator, workflow, evidence, and tool documentation.
quarantine/legacy-docs/	Historical documentation preserved for provenance, not active guidance.

DayOA expects references under /fsx/references on a configured headnode. Cluster lifecycle, FSx mounts, reference staging, data staging, and production manifest creation belong to daylily-ephemeral-cluster and the dyec CLI.

Design Philosophy

DayOA is intentionally narrow. It owns reproducible workflow execution and local evidence generation; it does not own cluster provisioning, sample identity authority, registry writes, QC disposition, release decisions, or long-lived service state. Inputs are explicit manifests and configured references. Outputs are deterministic files under the analysis workdir. Missing config, missing credentials, missing references, missing licenses, malformed manifests, or absent runtime assets are hard failures.

Snakemake is the execution engine here because it gives DayOA a transparent DAG, rule-level benchmarks, provenance-friendly file targets, and direct operator dry-runs. DayOA is not meant to become a universal platform monolith. New work should enter as focused pipeline modules with explicit config, tests, and evidence contracts. If a pipeline is better maintained in another workflow system, DYEC can provision the cluster and FSx/export contract while that external workflow runs in its own repository.

Ecosystem Planes

%%{init: {"theme":"base","themeVariables":{"primaryColor":"#101827","primaryTextColor":"#ffffff","primaryBorderColor":"#ef4444","lineColor":"#ef4444","secondaryColor":"#0f766e","tertiaryColor":"#facc15","fontFamily":"Inter,Arial,sans-serif"}}}%%
flowchart LR
  Bloom["Bloom<br/>run and lab objects"]:::svc
  Ursa["Ursa<br/>worksets and launch UX"]:::svc
  EC["daylily-ephemeral-cluster<br/>clusters, FSx, staging"]:::critical
  DayOA["daylily-omics-analysis<br/>execution plane"]:::exec
  Registry["external registry<br/>artifact registration"]:::registry
  Observe["downstream evidence import"]:::observe
  R2["R2<br/>QC, disposition, release"]:::release

  Bloom --> Ursa --> EC --> DayOA --> Registry --> Observe --> R2

  classDef critical fill:#7f1d1d,stroke:#ef4444,color:#fff,stroke-width:3px
  classDef exec fill:#1e293b,stroke:#38bdf8,color:#fff,stroke-width:3px
  classDef registry fill:#14532d,stroke:#22c55e,color:#fff,stroke-width:3px
  classDef observe fill:#312e81,stroke:#a78bfa,color:#fff,stroke-width:3px
  classDef release fill:#713f12,stroke:#facc15,color:#fff,stroke-width:3px
  classDef svc fill:#0f172a,stroke:#94a3b8,color:#fff,stroke-width:2px

Loading

Local Smoke Test

On a Mac, activate the DAY-EC environment before using this repo:

eval "$(conda shell.zsh hook)" && conda activate DAY-EC

Use a scratch checkout or preserve any existing manifests before copying fixtures:

mkdir -p config
cp .test_data/data/0.01xwgs_HG002_hg38.samples.tsv config/samples.tsv
cp .test_data/data/0.01xwgs_HG002_hg38.units.tsv config/units.tsv
source dyoainit
dy-a local hg38
dy-r produce_alignstats -n -p -j 1

The Mac smoke path validates command wiring and small fixtures. Routine workflows run on a prepared ParallelCluster headnode through SSM and dyec.

Headnode Launch Pattern

Do not use direct SSH, PEM files, or pcluster ssh for DayOA headnodes. Use dyec/SSM and a login bash shell.

cd /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster
source ./activate
dyec headnode connect --profile <profile> --region <region> --cluster <cluster>
exec bash -l
id -un
command -v day-clone
command -v tmux
command -v squeue
dyec analysis --help

Launch long workflow work from an explicitly pinned DayOA checkout. Use day-clone -t <dayoa_version> -d <analysis_id> manually, or pass --git-tag <dayoa_version> through DYEC launch commands. Do not rely on default refs. Send source dyoainit, dy-a, and dy-r as separate commands in one persistent tmux session.

Sentieon License Configuration

Sentieon workflows require SENTIEON_LICENSE to point at a valid license file on the headnode. The preferred configuration is explicit in ~/.config/daylily/daylily_cli_global.yaml:

daylily:
  sentieon_lic_path: /fsx/references/runtime_assets/sentieon/license.lic

After initialization and activation, verify the resolved license before launching Sentieon targets:

source dyoainit
dy-a slurm hg38
test -f "$SENTIEON_LICENSE" && echo "$SENTIEON_LICENSE"

DayOA does not scan runtime-asset directories for license files and does not choose a license automatically. If SENTIEON_LICENSE is set to a missing file, if daylily.sentieon_lic_path is absent, or if the configured file does not exist, initialization fails before Sentieon targets can launch. Do not commit license files to this repository.

Common Targets

Target	Purpose
produce_alignstats	Alignment statistics and alignstats_combo_mqc.tsv.
produce_snv_concordances	Small-variant concordance evidence.
produce_manta_sv_vcf, produce_tiddit_sv_vcf, produce_dysgu_sv_vcf	Structural variant evidence.
produce_multiqc_input_data	Input sequence-data MultiQC.
produce_multiqc_cram	CRAM/alignment MultiQC.
produce_multiqc_snv, produce_multiqc_sv	Variant-scope MultiQC.
produce_multiqc_all	Canonical final routine MultiQC aggregation.
produce_multiqc_generic	Generic existing-output MultiQC scan over the current results/day/<build>/ tree.
produce_dayoa_evidence_manifest	Deterministic post-MultiQC local evidence manifest.
produce_ont_run_qc	Mounted ONT run QC plus demux FASTQ QC and focused MultiQC when demux FASTQs are present under RUN_DIR.
produce_ultima_run_qc	Mounted Ultima run QC plus demux FastQC/SeqKit and focused MultiQC when demux FASTQs are present under RUN_DIR.
produce_bclconvert_fastqs_and_metrics	Lane-split Illumina BCL Convert demultiplexing plus generated units and demux metrics.
produce_illumina_run_qc_and_bclconvert	Mounted Illumina run QC plus lane-split BCL Convert in one run-context workflow.

Workflow Families And Tool Inventory

DayOA workflows are grouped around input/demux, FASTQ QC, alignment, deduplication, SNV/indel calling, SV/CNV/STR evidence, contamination/identity, metagenomics screening, concordance benchmarking, annotation, and MultiQC/evidence manifest generation. The authoritative inventory for tools, targets, rule files, outputs, environment/version evidence, tests, and public tool references is docs/catalog_of_tools.md. Keep that catalog repo-grounded: add a row only when the tool exists in rules, scripts, config, fixtures, or tests.

The routine HG003 short-read SNV evidence path uses configured HG003 FASTQs, Sentieon alignment/markdup/DNAscope targets, RTG/GIAB concordance, alignstats, and final MultiQC/evidence manifest targets. nf-core/Nextflow comparison runs are intentionally not added to DayOA; DYEC can stage the same data and run an nf-core repository beside DayOA when the pipeline honors the same FSx analysis-root and export contract.

Mounted Run Directories And BCL Convert

Run-directory workflows are designed to read from mounted, read-only FSx DRA paths supplied by daylily-ephemeral-cluster. DayOA must not copy full staged run directories into results/ before analysis. For BCL Convert, the native rule reads the mounted run directory directly and launches one bcl-convert job per detected Data/Intensities/BaseCalls/L### lane. Each lane job writes lane-local FASTQs and BCL reports under results/.../bclconvert/lane_fastqs/. By default bclconvert.merge_lane_fastqs: false, so DayOA does not spend time stitching lane FASTQs into one final directory; generated units, metrics, demux FastQC, and MultiQC consume the lane-level fastq_list.csv and report files directly. Set merge_lane_fastqs: true only when a downstream consumer requires the merged legacy FASTQ/report tree.

For tile-level demux benchmarking, set bclconvert.tile_shard_level to an integer shard count such as 2, 4, 10, 20, or 40, and optionally restrict execution with bclconvert.tile_shard_lanes: "L003". DayOA discovers exact tile names from *.filter files, writes zero-padded shard directories such as 0001_tiles0001-0020, passes those exact tiles to BCL Convert with --tiles, then concatenates shard FASTQs and aggregates shard reports back into the lane-level lane_fastqs/L###/Reports/ tree consumed by the existing downstream steps.

After demultiplexing, DayOA prepares collision-safe FastQC input links for every demux FASTQ using identifiers that include run, lane, sample, read group, and read number. The BCL Convert terminal target runs FastQC on those links, fails on identifier collisions, and builds a focused MultiQC report that includes native BCL Convert metrics, custom demux tables, and demux FastQC sections.

The Slurm profile is tuned for solo 192-vCPU BCL runs on i192: 24 parallel tiles, 4 conversion threads per tile, 64 compression threads, 32 decompression threads, gzip level 1, shared O_DIRECT output threads enabled, and legacy stats output enabled. Tile-shard jobs default to 48 vCPU, 180 GB memory, 8 parallel tiles, 2 conversion threads, 24 compression threads, and 8 decompression threads. The lane sample sheet is generated from the normalized sample sheet and injects BarcodeMismatchesIndex1,0 and BarcodeMismatchesIndex2,0 by default. Other BCL Convert sample-sheet settings are wired through config but remain unset unless explicitly configured and tested.

For live validation of the zero-mismatch BCL path, use a day-clone -t <dayoa_version> -d bclconvert_0_mm workset name so the analysis directory itself records the matching policy and the checkout is version-pinned.

Mounted ONT And Ultima Run QC

produce_ont_run_qc and produce_ultima_run_qc are mounted run-context targets. They require an explicit config/runs.tsv row for the platform with RUN_DIR, and they fail if demultiplexed FASTQs are not present when the demux QC report is requested. DayOA reads the mounted run directory directly and writes QC outputs under results/runs/<RUNID>/run_qc/<platform>/; it does not copy the run folder into results/.

ONT demux QC groups FASTQs by containing directory and runs SeqKit, nanoq, NanoStat, NanoPlot, and focused MultiQC. Ultima demux QC groups FASTQs the same way, creates collision-checked symlink inputs, runs FastQC and SeqKit, and builds ultima_demux_fastq.multiqc.html.

Hybrid Ultima And ONT

The modular Ultima+ONT Sentieon hybrid path (sentdhuomr) hard-fails on Stage1 driver errors and no longer uses process substitution for HAP and INS streams. Stage1 now runs the Sentieon drivers sequentially into temp SAM files, validates the target haplotype BAM and realigned BAM before downstream rules consume them, and indexes the haplotype BAM only after integrity checks. Stage2, Stage3, and Pass2 also tolerate genuinely empty target/refined-region shards by emitting explicit empty outputs with log messages rather than feeding empty target-hap state into Sentieon.

MultiQC And Evidence Boundaries

MultiQC HTML is a report, not canonical data. The parser-relevant evidence is in DAY_final_multiqc_data/, staged source manifests, custom _mqc.tsv files, logs, benchmarks, and the local DayOA evidence manifest.

%%{init: {"theme":"base","themeVariables":{"primaryColor":"#0f172a","primaryTextColor":"#ffffff","primaryBorderColor":"#38bdf8","lineColor":"#22c55e","secondaryColor":"#064e3b","tertiaryColor":"#7f1d1d","fontFamily":"Inter,Arial,sans-serif"}}}%%
flowchart TD
  A["analysis rules"] --> B["stage_multiqc_inputs"]
  B --> C["multiqc_final_wgs<br/>artifact generation only"]
  C --> D["write_dayoa_evidence_manifest<br/>local manifest only"]

  D -. "no registry calls here" .-> D

Loading

See docs/ops/multiqc_qc_targets.md.

Documentation Map

Document	Use
docs/README.md	Active documentation index.
docs/catalog_of_tools.md	Tool, rule, output, test, and public-link inventory.
docs/ops/multiqc_qc_targets.md	MultiQC staging, runtime gating, and local evidence boundaries.
docs/ops/results_directory_structure.md	Analysis, results/day, results/runs, run QC, and BCL Convert output layout.
docs/examples/multiqc/README.md	Example MultiQC report handling and cluster-dependent example guidance.

Contributing

Contributions should keep the execution contract explicit and testable:

• Add focused rules or scripts instead of expanding a single all-purpose DayOA path.
• Put required inputs in manifests or named config keys; do not infer buckets, references, licenses, sample identities, or runtime assets.
• Make new targets callable through dy-r <target> and document expected outputs under results/day/<build>/ or results/runs/<RUNID>/.
• Add or update catalog rows in docs/catalog_of_tools.md for every enabled tool.
• Add focused pytest or shell coverage for parser-visible docs, command rendering, MultiQC/evidence outputs, and hard-failure behavior.
• Keep HTML reports as navigational artifacts; parser-relevant data belongs in source manifests, _mqc.tsv files, logs, benchmarks, and dayoa_evidence_manifest.json.

Validation

Focused local checks:

eval "$(conda shell.zsh hook)" && conda activate DAY-EC
bash tests/test_cli_commands.sh
bash tests/test_bclconvert_bootstrap.sh
python -m pytest -q tests/test_multiqc_qc_targets.py tests/test_multiqc_staging_contracts.py tests/test_multiqc_sample_identifiers.py tests/test_evidence_manifest.py
python -m coverage run -m pytest -q tests
python -m coverage report

The shell checks are developer contract tests for wrappers and BCL Convert bootstrap logic. They do not replace pinned headnode workflow validation through dyec, source dyoainit, dy-a, and dy-r.

Cluster examples are valid only after a working headnode is available through dyec/SSM with an explicit non-default AWS profile.