Buoy 2d wind

[ncatolico]

No description provided.

[Copilot]

Pull request overview

Adds a new “windBuoy” (2D buoy wind) processing DAG plus supporting ingest/calibration/location plumbing and R/combined-module code needed to generate Level 1 + publication outputs.

Changes:

• Introduces a full set of Pachyderm pipeline specs for windBuoy (thresholds, SRF, group path, compass correction + padding, QA/QC, stats/QM, consolidation, pub).
• Adds/updates rmyoung_hmr3300 ingest + metadata pipelines (Trino/GCS loaders, calibration + location asset/active-dates loaders, gap fill) and removes older per-source-type pipe lists/specs.
• Adds a new nominal calibration conversion function (def.cal.conv.nmnl) with unit tests and supporting fixtures; adds combined-module Dockerfiles and new flow scripts for compass correction and direction stats.

Reviewed changes

Copilot reviewed 66 out of 71 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
utilities/vignettes/stand_up_product_dag_example.sh	Updates end-to-end “stand up DAG” example for rmyoung → windBuoy.
utilities/vignettes/delete_product_dag_example.sh	Updates end-to-end “delete DAG” example for rmyoung → windBuoy.
utilities/R_coding/flow.renv.init.rstr.R	Points dirWork at wind buoy compass correction flow for renv/init work.
utilities/R_coding/flow.pack.is.proc.R	Adjusts package selection and test-running toggles for local packaging workflow.
pipe/windBuoy/windBuoy_threshold.yaml	Adds threshold loader pipeline for wind buoy terms/contexts.
pipe/windBuoy/windBuoy_threshold_select.yaml	Adds threshold selection pipeline for buoy-wind term/context pairs.
pipe/windBuoy/windBuoy_srf_loader.yaml	Adds SRF loader pipeline for wind buoy group prefix.
pipe/windBuoy/windBuoy_srf_assignment.yaml	Adds SRF assignment pipeline.
pipe/windBuoy/windBuoy_speed_stats_group_and_compute.yaml	Adds speed stats grouping + computation pipeline (filter-joiner + stats).
pipe/windBuoy/windBuoy_qm_group_and_compute.yaml	Adds quality metrics grouping + computation pipeline (filter-joiner + qm).
pipe/windBuoy/windBuoy_pub_group.yaml	Adds publication grouping pipeline (pub_grouper).
pipe/windBuoy/windBuoy_pub_format_and_package.yaml	Adds pub transformer + packager + pub_files + maintenance-table load pipeline.
pipe/windBuoy/windBuoy_pub_egress_and_publish.yaml	Adds pub egress to GCS + publish/upload/sync pipeline.
pipe/windBuoy/windBuoy_level1_group_consolidate_srf.yaml	Adds L1 grouping/consolidation + SRF apply + L1 bucket export pipeline.
pipe/windBuoy/windBuoy_group_path.yaml	Adds group_path pipeline, collapsing multiple source-type location paths.
pipe/windBuoy/windBuoy_group_loader.yaml	Adds group loader pipeline for wind buoy group prefix.
pipe/windBuoy/windBuoy_group_assignment.yaml	Adds group assignment pipeline.
pipe/windBuoy/windBuoy_direction_stats_ucrt.yaml	Adds buoy-wind direction stats + uncertainty pipeline (combined module).
pipe/windBuoy/windBuoy_cron_monthly_and_pub_control.yaml	Adds monthly cron/pub-control pipeline for publication timing.
pipe/windBuoy/windBuoy_compass_correct_and_pad.yaml	Adds compass correction + timeseries padding pipeline (combined module).
pipe/windBuoy/windBuoy_analyze_pad_and_qaqc_plau.yaml	Adds analyzer + plausibility QA/QC pipeline after padding.
pipe/windBuoy/pipe_list_windBuoy.txt	Adds pipe list for bringing up/down the windBuoy DAG.
pipe/rmyoung/site-list.json	Removes old rmyoung standalone site list (migrated to combined source-type).
pipe/rmyoung/rmyoung_data_source_trino.yaml	Removes old rmyoung standalone Trino loader pipeline spec.
pipe/rmyoung/rmyoung_data_source_kafka.yaml	Removes old rmyoung standalone Kafka loader pipeline spec.
pipe/rmyoung/pipe_list_rmyoung.txt	Removes old rmyoung standalone pipe list.
pipe/rmyoung/pipe_list_rmyoung_development.txt	Removes old rmyoung standalone dev pipe list.
pipe/rmyoung_hmr3300/site-list.json	Adds combined site list for rmyoung_hmr3300.
pipe/rmyoung_hmr3300/rmyoung_ingest_days.yaml	Adds ingest-days pipeline for rmyoung staging/manifest distribution.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_location_loader.yaml	Adds combined location loader (runs for both source types).
pipe/rmyoung_hmr3300/rmyoung_hmr3300_location_group_and_restructure.yaml	Adds location grouping/restructure pipeline.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_location_asset.yaml	Adds location-asset loader pipeline (both source types).
pipe/rmyoung_hmr3300/rmyoung_hmr3300_location_asset_assignment.yaml	Adds location-asset assignment pipeline.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_location_active_dates_assignment.yaml	Adds namedLocation/active-dates assignment pipeline.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_fill_date_gaps.yaml	Adds date-gap filling (incl non-regularized fill) pipeline.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_data_source_trino.yaml	Updates Trino loader to support multiple source types via path parsing + config vars.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_data_source_gcs.yaml	Adds GCS loader for backfill outside Kafka retention period.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_cron_daily_and_date_control.yaml	Updates cron/date-control to generate dates for both rmyoung and hmr3300.
pipe/rmyoung_hmr3300/rmyoung_hmr3300_calibration_group_and_convert.yaml	Adds calibration grouping + nominal conversion pipeline (combined source types).
pipe/rmyoung_hmr3300/rmyoung_cron_daily_and_date_control.yaml	Updates rmyoung cron date range and site-list repo reference.
pipe/rmyoung_hmr3300/rmyoung_calibration_loader.yaml	Adds calval loader pipeline for rmyoung calibration files.
pipe/rmyoung_hmr3300/rmyoung_calibration_list_files.yaml	Adds calibration file listing pipeline (driven by cron tick).
pipe/rmyoung_hmr3300/rmyoung_calibration_assignment.yaml	Adds calibration assignment pipeline by year.
pipe/rmyoung_hmr3300/pipe_list_rmyoung_hmr3300.txt	Adds pipe list for combined rmyoung_hmr3300 DAG.
pipe/rmyoung_hmr3300/pipe_list_rmyoung_hmr3300_development.txt	Adds dev pipe list for combined rmyoung_hmr3300 DAG.
pipe/rmyoung_hmr3300/hmr3300_ingest_days.yaml	Adds ingest-days pipeline for hmr3300 staging/manifest distribution.
pipe/hmr3300/site-list.json	Removes old hmr3300 standalone site list (migrated to combined source-type).
pipe/hmr3300/pipe_list_hmr3300.txt	Removes old hmr3300 standalone pipe list.
pipe/hmr3300/hmr3300_data_source_kafka.yaml	Removes old hmr3300 standalone Kafka loader pipeline spec.
pack/NEONprocIS.cal/tests/testthat/test-cal-conv-nmnl.R	Adds unit tests for nominal calibration conversion.
pack/NEONprocIS.cal/tests/testthat/calibrations/nominal/calibration_nominal.xml	Adds nominal calibration XML fixture for tests.
pack/NEONprocIS.cal/R/def.cal.conv.nmnl.R	Adds def.cal.conv.nmnl() implementation.
pack/NEONprocIS.cal/NAMESPACE	Exports def.cal.conv.nmnl.
pack/NEONprocIS.cal/man/def.cal.conv.nmnl.Rd	Adds generated Rd docs for def.cal.conv.nmnl.
pack/NEONprocIS.cal/DESCRIPTION	Updates roxygen configuration metadata.
pack/NEONprocIS.cal/data-raw/DATASET.R	Adds data-raw placeholder script.
modules/pub_transformer/pub_transformer.py	Adjusts integer formatting to handle NaNs as empty strings.
modules_combined/windBuoy_compass_correct_and_pad/Dockerfile	Adds combined module image for compass correction + padder.
modules_combined/wind_buoy_direction_stats_ucrt/Dockerfile	Adds combined module image for direction stats + filter-joiner.
modules_combined/calibration_group_and_convert/Dockerfile	Pins base image to a specific digest/tag for calibration group/convert.
flow/flow.wind.buoy.direction.stats.ucrt/wrap.wind.buoy.direction.stats.ucrt.R	Adds wrapper for direction stats + uncertainty.
flow/flow.wind.buoy.direction.stats.ucrt/flow.wind.buoy.direction.stats.ucrt.R	Adds workflow driver for direction stats + uncertainty.
flow/flow.wind.buoy.direction.stats.ucrt/dockerfile_in_combined_module.txt	Records combined-module dockerfile mapping for direction stats.
flow/flow.wind.buoy.compass.correction/wrap.wind.buoy.compass.correction.R	Adds wrapper for compass correction + buoy-specific flags.
flow/flow.wind.buoy.compass.correction/renv.lock	Adds renv lockfile for compass correction flow.
flow/flow.wind.buoy.compass.correction/flow.wind.buoy.compass.correction.R	Adds workflow driver for compass correction + buoy-specific flags.
flow/flow.wind.buoy.compass.correction/Dockerfile	Adds Dockerfile for compass correction flow image.
flow/flow.qaqc.qm/wrap.qaqc.qm.R	Changes parquet write to not swallow errors (silent=FALSE).
flow/flow.gap.fill.nonrglr/wrap.gap.fill.nonrglr.R	Formatting-only change (trailing whitespace/newline).
flow/flow.gap.fill.nonrglr/flow.gap.fill.nonrglr.R	Formatting-only change (trailing whitespace/newline).

Files not reviewed (1)

• pack/NEONprocIS.cal/man/def.cal.conv.nmnl.Rd: Generated file

Suppressed comments (2)

pipe/windBuoy/windBuoy_qm_group_and_compute.yaml:74

• The Pachyderm glob patterns are malformed (unbalanced parentheses) and will fail to parse/match. Use proper capture groups for the three path components you later reference in joinOn: $1/$2/$3.
  pipe/windBuoy/windBuoy_qm_group_and_compute.yaml:80
• The Pachyderm glob pattern is malformed (unbalanced parentheses) and will fail to parse/match. Use proper capture groups for the three path components you later reference in joinOn: $1/$2/$3.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[covesturtevant]

Looking good. Please see comments for specific changes to R modules. I haven't made it through pipeline specs yet, but I wanted to let you get started on the R code mods. I'll work through the pipelines and integrate Argo ingest by the end of the week.

[covesturtevant]

Please delete

[covesturtevant]

As always, please reduce to just enough data for testing. This comment applies to all data files included in this PR.

[covesturtevant]

This test fails. Since AI makes it easy to do, please also ask Copilot to review and adjust the test for completeness and robustness. Here's what it returned when I asked for that. Not everything needs to be implemented, but copilot can really help us improve our testing. Please do this for the other test as well.

1. Column name mismatches — schema vs. wrapper output

The wrapper writes columns speed_calibrated, compass_direction_raw, direction_corrected, direction_corrected_rad, but the data schema names them speed, vectorAverageHeading, direction, direction_rad. The flags schema uses buoyWindDirDeadZoneQF / buoyWindDirCalmWindQF, but the wrapper produces buoyWindDirDeadZone / buoyWindDirCalmWind. These mismatches likely cause the wrong column names in the parquet output. The test assertions for direction, direction_rad, vectorAverageHeading assume the schema renames columns on write; the assertions for buoyWindDirDeadZone/buoyWindDirCalmWind assume it doesn't. Both can't be right.

2. Vacuous direction range check

validDir <- windData$direction[!is.na(windData$direction)] testthat::expect_true(all(validDir >= 0 & validDir < 360))

If direction doesn't exist in the output, windData$direction is NULL, validDir is logical(0), and all(logical(0)) is TRUE. This check always passes regardless of actual data.

3. file.exists(..., recursive = TRUE) is silently wrong

file.exists() has no recursive argument — the extra argument is silently dropped. Use dir.exists(testDirOutPath) instead.

4. length() on a data.frame returns column count, not row count

testthat::expect_true(length(windData) != 0) # counts columns, not rows

Use nrow(windData) > 0.

Missing coverage
5. Flag values never validated — the test confirms the flag columns exist but never checks that their values are within the expected set {-1, 0, 1}.

6. Dead band logic not verified — no assertion that direction_calibrated is replaced with 357.5 for rows where it was ≥ 355°.

7. Calm wind logic not verified — no assertion that direction (corrected) is 0 when speed_calibrated ≤ 0.5, or that buoyWindDirCalmWind == 1 for those rows.

8. No row count consistency check — data and flags output should have the same number of rows and identical readout_time values.

9. DirSubCopy parameter is never exercised — the symbolic-link copy path has zero test coverage.

10. No error-path tests — the wrapper has explicit stop() calls for missing files, multiple files, row-count mismatches, and missing thresholds terms. None are tested.

11. Test 1 is nearly empty — it only checks dir.exists() on the output path. It should at minimum verify one data file and one flags file were written.

Minor
12. try() + manual check is fragile — replacing with testthat::expect_no_error(...) (or wrapping in expect_error(..., NA)) gives clearer failure messages.

13. rmyoung is hardcoded in CFGLOC discovery path — file.path(testDirOutPath, 'rmyoung') works today but would silently skip the check if the instrument folder name ever changes. Discovering directly under testDirOutPath would be more resilient.

[covesturtevant]

This change is not needed. Please revert. The issue was that two fields in the pub workbook, buoyWindDirCalmWindQAQCRpt and buoyWindDirDeadZoneQAQCRpt, were trying to be converted from string to integer. (The change to pub_transformer solved the issue only because there was no data for these columns.) This was fixed by changing the pubFormat column in the pub workbook for these columns to "asIs" to match the other string columns. I tested the change on INT. The pub workbooks on PROD and CERT need to be updated.

[covesturtevant]

Please update to reflect that new calibrated columns are output and the originals are retained.