Add real-time multitaper spectrum diagnostics - #3
Draft
galenlynch wants to merge 48 commits into
Draft
Conversation
Track active acquisition callbacks before they access FIFO storage, reject starts that would replace a live worker, and retain processing state when a cooperative stop times out. Join the configuration builder cooperatively so JUCE never force-kills it while it owns planning state.
Snapshot the stream and global channel mapping at acquisition start, reject invalid mappings as whole callback blocks, and guard stream, channel, and frequency-selection lookups. Adapted from the defensive validation in Anjal Doshi's refactor-optimization branch while retaining the atomic planar FIFO transport. Co-authored-by: Anjal Doshi <anjaldoshi09@gmail.com>
Remove obsolete OpenEphysNumerics setup from all CI jobs now that the selected eigensolver and private OpenBLAS backend live in Spectrum Viewer. Enable MSVC's standard C++ exception model, adapted from Anjal Doshi's build modernization. Co-authored-by: Anjal Doshi <anjaldoshi09@gmail.com>
Delete the unused CumulativeTFR implementation and its custom atomic synchronization primitive. The tested planar input FIFO, worker-owned analysis pipeline, and latest-frame output FIFO now provide the production path.
Keep the active analysis and input route live while a replacement is prepared. Publish stream, channel, and generation metadata together so the callback cannot mix routes across a configuration boundary.
Explain the runtime data flow, source layers, hot-swap boundary, and real-time invariants. Link the guide from the main README and clarify variable-channel input FIFO metadata.
FetchContent previously added OpenBLAS as a normal subdirectory, so the plugin install target inherited its rules and attempted to write the private static dependency under /usr/local. Add the populated source with EXCLUDE_FROM_ALL while preserving support for the repository's CMake 3.15 minimum. Plugin installs now deploy only the plugin artifact.
The display reducer previously created columns narrower than one FFT bin when a selected band contained fewer bins than horizontal pixels. Repeated values then rendered as artificial stair steps. Publish each native bin once when no downsampling is required. Continue using area-weighted means and peak envelopes when the spectrum has more bins than the display can represent.
A requested route can update editor state while the previous analysis remains live. The canvas previously refreshed channel labels only when its generation changed, so the legend could disagree with the frame being drawn. Refresh the bounded stream and channel metadata from every consumed frame. The legend now follows the data through channel and stream hot-swaps.
The growing control set was laid out as one vertical column and extended far below the host's fixed-height processor editor. Arrange input, display, amplitude, capture, and comparison controls in four compact columns. All controls now remain inside the signal-chain strip without removing diagnostic features.
Add optional aperiodic-background display and removal, stable automatic dB ranges, compact canvas controls, and a persistent peak-envelope toggle. Keep hidden peak data in the range fit so narrow spectral lines remain visible. Size transport slots for actual stream callback payloads instead of GenericProcessor's nominal 128-sample quantum, allowing analysis to start on the first acquisition. Cover the DSP, transport, lifecycle, and rendering behavior with regressions.
The acquisition, analysis, configuration, and message threads exchange several generation-tagged objects. Document their ownership, mailbox and FIFO boundaries, configuration replacement sequence, overload behavior, and shutdown ordering so future changes preserve the real-time contract.
Building OpenBLAS during plugin configuration made setup slow and tied the numerical backend to each local toolchain. Fetch pinned conda-forge packages into a build-local, content-addressed stage, with a prepared-root override for offline builds. Preserve package provenance and notices, hide the static symbols on Unix, and namespace the Windows runtime. Limit OpenBLAS to one thread before the first eigensolve so asynchronous DPSS preparation cannot oversubscribe the host.
Spectrum Viewer CI repeatedly failed because it built against the main FFTW branch, which lacks the required batched API. Windows also selected a runner without the requested Visual Studio 2022 generator. Build the stacked FFTW branch on stable native runners, test the DSP suite, and validate private OpenBLAS linkage. Upload short-lived review artifacts, including a self-contained Windows rig-test bundle, while retaining JFrog deployment for main only.
The managed FFTW bundle uses lib-prefixed filenames on Unix, but Spectrum Viewer clears CMake's default shared-library prefix. Its tests consequently failed to find either precision in the new staging directories. Search the managed layouts explicitly and pass a platform-neutral relative checkout path. This also prevents Windows from interpreting the backslashes in github.workspace as part of the CMake cache value.
galenlynch
changed the title
Collaborator
Author
|
Depends on open-ephys-plugins/OpenEphysFFTW#4 |
The managed FFTW link libraries name versioned runtime files that live in the bundle's bin directory. The test process could therefore link successfully but failed before main when the system did not provide the same FFTW version. Resolve both staged runtimes for every supported platform, copy them beside the test executable, and use an executable-relative build rpath on Unix. Tests now exercise the dependency bundle instead of an incidental system installation.
The optimized trace renderer inherited from XYLine and called its methods directly. The supported Windows host import library does not export those symbols, so the plugin could not link even though InteractivePlot itself is part of the plugin API. Store line data in FrequencyPlot and draw each trace as one JUCE path from paintOverChildren. This preserves the measured batching improvement without adding an undeclared host ABI requirement.
JUCE imports Cocoa from its core implementation on macOS, so compile that translation unit as Objective-C++ and link its declared system frameworks. Use runtime-sized pointer storage in the callback-partition test because MSVC does not accept the enclosing function's constant as a template argument inside the lambda.
The Windows test objects imported JUCE symbols even though JuceCore.cpp compiled those symbols into the test executable. This mismatch left AbstractFifo and other core methods unresolved at link time.\n\nForce a test-only JUCE configuration header into every Windows translation unit so the standalone executable consistently owns JUCE core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
not ready to be merged: need to test with the GUI
Summary
FIFO and worker-owned circular history. Callbacks now copy each selected
channel once and publish or drop the complete multichannel block.
multitaper PSD estimation using runtime-generated DPSS tapers and persistent
float FFTW plans.
PSD/ASD units, linear or logarithmic frequency mapping, cursor values, and a
worker-side mean-plus-peak pixel reduction.
stable fixed/automatic dB ranges, low-variance Fine capture, and session
reference overlay/delta comparison.
message thread. Remove the unused legacy spectrum pipeline.
Why
The released implementation can discard callback tails, join samples across
gaps, and publish channels independently. It also plots an unnormalized squared
FFT magnitude after several undocumented smoothing stages. The result depends
on callback partitioning, window length, sample rate, and display range, which
makes comparisons unreliable during rig debugging.
This change gives every accepted block and spectrum an explicit sample range,
configuration generation, channel map, estimator description, and physical
unit. Expensive allocation, DPSS generation, and FFT planning occur on a
dedicated configuration thread while the previous valid runtime remains live.
Validation
detrending, leakage, noise density, channel isolation, and odd/even lengths.
overflow, backlog shedding, asynchronous replacement, failed preparation,
stop/restart races, capture, references, and display metadata.
0.45 ms Fast, 1.04 ms Balanced, and 4.67 ms Fine at the median. Software
repaint medians were 2.75–5.54 ms. These are local measurements, not target-rig
acceptance results.
Dependencies and remaining validation
OpenEphysFFTW and corresponding cross-platform
fftw3fbinaries.OpenBLAS backend; hosted Windows and universal-macOS builds remain required.
latency measurements remain before requesting final review.
scientific work; this PR keeps the raw calibrated spectrum visible.
Attribution
Defensive stream/range checks and the MSVC exception setting were adapted from
Anjal Doshi's
refactor-optimizationbranch. The relevant commits preserveco-authorship.