Fix test criteria to fix paqa_latency.c by philburk · Pull Request #1111 · PortAudio/portaudio

philburk · 2025-12-26T21:36:38Z

It was failing because the latency was too low.

Now we allow lower latency.

Fixes #742

philburk · 2025-12-26T23:49:48Z

After discussion with Ross, the plan is to:

Start checking at 0.0
expect the finalLatency to be >= suggested, UNTIL it clamps and then it should not increase.

philburk · 2026-01-03T01:26:46Z

expect the finalLatency to be >= suggested, UNTIL it clamps and then it should not increase.

I added that test. But we are seeing failures where the latency hits the maximum and then keeps going up!

Using device #4: 'Maono PD200W Mic USB' (Core Audio)
------------------------ paqaCheckMultipleSuggested - INPUT
 lowLatency  = 0.00441667
 highLatency = 0.01375
 numChannels = 1
 sampleRate  = 48000
   suggestedLatency[ 0] = 0.000000, finalLatency = 0.003396
   suggestedLatency[ 1] = 0.001375, finalLatency = 0.003396
   suggestedLatency[ 2] = 0.002750, finalLatency = 0.003396
   suggestedLatency[ 3] = 0.004125, finalLatency = 0.004125
   suggestedLatency[ 4] = 0.005500, finalLatency = 0.005500
   suggestedLatency[ 5] = 0.006875, finalLatency = 0.006875
     maximumLatency = 0.006875
   suggestedLatency[ 6] = 0.008250, finalLatency = 0.008250
/Users/phil/Work/portaudio/pagit/qa/paqa_latency.c:310 - ERROR - Latency should be == maximumLatency
INPUT CHECK FAILED !!! #4: 'Maono PD200W Mic USB'

also

Using device #7: 'ZoomAudioDevice' (Core Audio)
------------------------ paqaCheckMultipleSuggested - OUTPUT
 lowLatency  = 0.0333333
 highLatency = 0.0426667
 numChannels = 2
 sampleRate  = 48000
   suggestedLatency[ 0] = 0.000000, finalLatency = 0.032313
   suggestedLatency[ 1] = 0.004267, finalLatency = 0.032313
   suggestedLatency[ 2] = 0.008533, finalLatency = 0.032313
   suggestedLatency[ 3] = 0.012800, finalLatency = 0.032313
   suggestedLatency[ 4] = 0.017067, finalLatency = 0.032313
   suggestedLatency[ 5] = 0.021333, finalLatency = 0.032313
   suggestedLatency[ 6] = 0.025600, finalLatency = 0.032313
   suggestedLatency[ 7] = 0.029867, finalLatency = 0.032313
   suggestedLatency[ 8] = 0.034133, finalLatency = 0.034125
     maximumLatency = 0.034125
   suggestedLatency[ 9] = 0.038400, finalLatency = 0.038396
/Users/phil/Work/portaudio/pagit/qa/paqa_latency.c:310 - ERROR - Latency should be == maximumLatency
OUTPUT CHECK FAILED !!! #7: 'ZoomAudioDevice'

RossBencina · 2026-02-06T00:31:07Z

NEXT STEP: confirm that rounding error is the problem. investigate source of rounding error. consider revising the spec

I think that the test is valid with respect to the current spec.

I think the failing tests reflect a bug in the code in the sense that the test results indicate a violation of the current spec for suggestedLatency in portaudio.h here:

portaudio/include/portaudio.h

Lines 570 to 580 in 5d01510

    
               /** The desired latency in seconds. Where practical, implementations should 
        
                configure their latency based on these parameters. Implementations should 
        
                round the actual latency up to the next viable value, except when suggested 
        
                latency exceeds the upper limit for the device. 
        
                Actual latency values for an open stream may be retrieved using the 
        
                inputLatency and outputLatency fields of the PaStreamInfo structure 
        
                returned by Pa_GetStreamInfo(). 
        
                @see default*Latency in PaDeviceInfo, *Latency in PaStreamInfo 
        
               */ 
        
               PaTime suggestedLatency;

The USB mic result is difficult to interpret because the displayed floating point precision doesn't show that actual latency < suggested latency, even though it must be by the logic of the test.

I suspect the issue is that there is rounding down when the spec says to always round up. This could easily be a bug in PA/CoreAudio. It would be good to know whether the test also fails with other host APIs.

We are undecided whether rounding down within some epsilon should be in specification . It would be user friendly to allow for +/- half (or 1/4 or 0.1) sample period of slop to allow for incorrect rounding in floating point calculations.

I definitely think that the actual latency should be reported as accurately as possible, so the code should not "pretend" to have honoured the suggested latency when it actually rounded down, even a little bit.

RossBencina · 2026-02-06T00:35:28Z

Suggestions for improving the test:

Rename maximumLatency -> detectedMaximumLatency and reflect this in the test output/printfs
Add a third column "final - suggested" where we expect that column to be >= 0. A negative value indicates that the value was rounded down and will indicate by how much.
Could print the sample period as a reference value. eg display "sample period: 0.000020" under "sampleRate = 44100" where sample period is 1.0/sampleRate

philburk · 2026-02-19T23:23:05Z

Print the delta value between final and suggested latency with scientific notation.

RossBencina

Phil and I agreed to merge this as a failing test once the following two improvements are made:

Print in seconds (next to sample rate) 1/sampleRate
Print suggested, final delta

Just expect monotonic increases, and not much higher latency than suggested. Fixes #742

philburk · 2026-07-16T23:01:13Z

I made the suggested changes. Now I get this:

Using device #4: 'Maono PD200W Mic USB' (Core Audio)
------------------------ paqaCheckMultipleSuggested - INPUT
 lowLatency   = 0.00441667
 highLatency  = 0.01375
 numChannels  = 1
 sampleRate   = 48000 Hz
 samplePeriod = 2.083333e-05 seconds
   suggestedLatency[ 0] = 0.000000, finalLatency = 0.003396, (final - suggested) = 3.395833e-03
   suggestedLatency[ 1] = 0.001375, finalLatency = 0.003396, (final - suggested) = 2.020833e-03
   suggestedLatency[ 2] = 0.002750, finalLatency = 0.003396, (final - suggested) = 6.458333e-04
   suggestedLatency[ 3] = 0.004125, finalLatency = 0.004125, (final - suggested) = 0.000000e+00
   suggestedLatency[ 4] = 0.005500, finalLatency = 0.005500, (final - suggested) = 0.000000e+00
   suggestedLatency[ 5] = 0.006875, finalLatency = 0.006875, (final - suggested) = -8.673617e-19
     detectedMaximumLatency = 0.006875
   suggestedLatency[ 6] = 0.008250, finalLatency = 0.008250, (final - suggested) = 0.000000e+00
/Users/phil/Work/portaudio/pagit/qa/paqa_latency.c:312 - ERROR - Latency should be == detectedMaximumLatency
INPUT CHECK FAILED !!! #4: 'Maono PD200W Mic USB'

See attachment for the full log: qa_latency_PR1111_20160716.txt

philburk · 2026-08-08T00:41:03Z

TODO
Show the level of nesting for the tests. Indentation.
Show delta in number of sample frames.

Could add a test for more serious failures, "smoke".
Separate out the tests so one known failure cannot mask new or more serious errors.

Call this monotonic test paqa_latency_monotonic.c

Suggest 0.0, get >= defaultMinLatency.
Check final >= 0.0.

RossBencina · 2026-08-08T00:48:18Z

I think test could be merged now, but I agree that making the output easier to interpret would be helpful.

We also discussed splitting out a smoke test:

suggested: 0 -> final > 0 && <= default-min
suggested: default min -> final >= default min
suggested: default max -> final >= default max

This allows the rest of paqa_latency.c to pass.

philburk · 2026-08-14T00:44:25Z

I pulled out the monotonic test that was failing.
Now it runs more tests that came after the failing test.
I also improved the appearance of the test output.

RossBencina

Looks good. I've made some suggestions to improve readability.

RossBencina

Approved modulo two comments and also, please add to qa/CMakeLists.txt

RossBencina · 2026-08-21T05:59:49Z

+    if( err != paNoError ) goto error;

    err = Pa_CloseStream( stream );
    Pa_Sleep( 1 * 1000 );

-
-    printf("-------------------------------------\n");
+    printf("-------------------------------------\n\n");
    return err;


This code is identical to the error path. Consider whether to remove lines 180-186 and fall through to label error:.

RossBencina · 2026-08-21T06:48:34Z

+                printf("     detectedMaximumLatency = %8.6f\n", detectedMaximumLatency );
+            }
+        }
+        /* If we are not at maximum then we should be rounding up. */


The comment is not well worded and is redundant with existing comments below. Suggest:

Suggested change

/* If we are not at maximum then we should be rounding up. */

/* Now check that reported latency behaves correctly: */

philburk added this to the V19.8 milestone Dec 26, 2025

philburk self-assigned this Dec 26, 2025

philburk requested a review from RossBencina December 26, 2025 21:36

philburk assigned RossBencina Dec 26, 2025

RossBencina reviewed Dec 26, 2025

View reviewed changes

Comment thread qa/paqa_latency.c Outdated

philburk force-pushed the latency_qa_742 branch from deb402c to fbecb61 Compare January 3, 2026 01:19

RossBencina requested changes Feb 19, 2026

View reviewed changes

RossBencina added the test-qa Test code in /qa (automated test program) label Mar 27, 2026

philburk added 3 commits July 16, 2026 15:17

Change test criteria so that lower latency is OK

688418f

Just expect monotonic increases, and not much higher latency than suggested. Fixes #742

Fix alignment for continuation lines

faeb2a4

Check to make sure latency does not drop below maximumLatency

3093947

print final-suggested and samplePeriod

fe6d39e

philburk force-pushed the latency_qa_742 branch from fbecb61 to fe6d39e Compare July 16, 2026 23:03

separate out paqa_latency_monotonic.c, which is failing

b4cd84c

This allows the rest of paqa_latency.c to pass.

philburk commented Aug 14, 2026

View reviewed changes

Comment thread qa/pa_latency_monotonic.c Outdated

Comment thread qa/pa_latency_monotonic.c Outdated

Comment thread qa/pa_latency_monotonic.c Outdated

RossBencina reviewed Aug 14, 2026

View reviewed changes

fix file header

1d8cadb

RossBencina reviewed Aug 21, 2026

View reviewed changes

Comment thread qa/paqa_latency_monotonic.c

philburk added 2 commits August 20, 2026 18:37

rename to paqa_

6387f77

address comments from Ross, scan bufferSizes

6c8356a

philburk mentioned this pull request Aug 21, 2026

paqa_latency_monotonic failing on Mac OS due to small deltas #1171

Open

RossBencina approved these changes Aug 21, 2026

View reviewed changes

	/* If we are not at maximum then we should be rounding up. */
	/* Now check that reported latency behaves correctly: */

Uh oh!

Conversation

philburk commented Dec 26, 2025

Uh oh!

Uh oh!

philburk commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philburk commented Jan 3, 2026

Uh oh!

RossBencina commented Feb 6, 2026

Uh oh!

RossBencina commented Feb 6, 2026

Uh oh!

philburk commented Feb 19, 2026

Uh oh!

RossBencina left a comment

Choose a reason for hiding this comment

Uh oh!

philburk commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philburk commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RossBencina commented Aug 8, 2026

Uh oh!

philburk commented Aug 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RossBencina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RossBencina left a comment

Choose a reason for hiding this comment

Uh oh!

RossBencina Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

RossBencina Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

philburk commented Dec 26, 2025 •

edited

Loading

philburk commented Jul 16, 2026 •

edited

Loading

philburk commented Aug 8, 2026 •

edited

Loading