PDP11: Allow draining backlogged receive characters faster than one per call

[ddrown]

DZ's receive silo (16 characters, shared across all lines on one controller) was designed to be serviced in batches, but dz_update_rcvi() only ever promoted one character per call regardless of how many were already due, and tmxr_getc_ln() reset its pacing clock (rxnexttime) relative to "now" on every successful extraction rather than advancing from its own previous value -- so a caller could never catch up on a backlog, capping DZ receive throughput on fast/bursty transports (e.g. a TCP-attached line) far below the configured baud rate.

• sim_tmxr.h: add TMLN.rxmaxbacklog, an opt-in cap (0 = disabled, preserving prior behavior for every other tmxr client) on how many already-due characters tmxr_getc_ln() will treat as legitimately waiting.
• sim_tmxr.c: tmxr_getc_ln() now clamps rxnexttime to at most rxmaxbacklog character-times behind "now" (only when rxdeltausecs is nonzero, to avoid pinning rxnexttime at exactly "now" forever). Whether the line can actually deliver right now is left to the existing extraction gate just below and to _tmxr_activate_delay(), so the clamp doesn't need to duplicate that check. It also advances rxnexttime from its previous value instead of resetting relative to "now" on success, which is what lets repeated calls drain a genuine backlog at the correct pace instead of being capped at one character per call.
• sim_tmxr.c: _tmxr_activate_delay() now requires a line be able to actually deliver (connected/buffered and receive-enabled) before treating its raw buffered byte count as "due immediately," and gains a circuit breaker: if it's ever asked to compute a near-zero delay more than 50 times within the same real millisecond -- a rate with no legitimate cause -- it falls back to the caller's safe interval instead, so a scheduling edge case can't livelock the simulator.
• pdp11_dz.c: dz_reset() sets rxmaxbacklog to the silo depth for every DZ line; dz_update_rcvi() drains as many already-due characters per line as fit in the shared silo, instead of just one.

Measured throughput on a TCP-attached DZ line went from ~60-170 cps to ~970-1024 cps against a 9600 baud (960 cps theoretical) configuration.

Fixes #571

[markpizz]

Hi there Dan (@ddrown),

It seems that you reported the issue #571 and then immediately had some sort of fix in your #572 PR.

I'm guessing that you'd been dealing with this for a long time, possibly on a PiDP11.

As it turns out, the version of simh that Oscar picked up many years ago had numerous problems with various aspects of behaviors in DZ devices that affected consistent delivery at selected speeds and numerous other things.

These were reported in github.com/simh/simh 6 or 7 years ago well before the simh/simh vs open-simh/simh schism and I had fixed things in both the DZ device and the TMXR support. Others have also provided feedback to clarify of some function of in the TMXR support.

It would have been useful for you to raise the issue either in github.com/simh/simh or github.com/open-simh/simh and discuss your thoughts before changing code.

Had we discussed your issues and ideas before you created the changes you have submitted, the potential subtleties that you concluded and described in the PR above, would have been hashed out and the details explained.

You had said:

Digging into what may be causing this, my theory is the DZ driver only adds 1 byte to the queue at every driver interrupt (dz_update_rcvi) and 2.11 BSD is only polling at ~100Hz. I have never used the actual hardware, but my assumption is that the actual DZ hardware would be placing bytes into the silo as they arrived, asynchronously until the silo was full. I am interested to hear if my assumption is flawed.

Your theory about what this is doing is somewhat flawed.

You are precisely right about real hardware (serial ports), data doesn't arrive faster than the bit rate that the UART in the device is programmed at. If "2.11 BSD is only polling at ~100Hz" really only processes incoming data from the DZ device at most 100 times a second, that could be a factor in what you are seeing, but I would be very surprised if it did that. Real operating systems use interrupts to react to arriving data and I would presume that 2.11 BSD is such an operating system. The simh TMXR devices may poll for 'initial' input at 100hz, but once any data arrives, polling for the port it arrives on will happen at the programmed bit rate. Since the sending side will almost certainly be sending a number of bytes in the TCP send, those bytes will be available, and already read, along with the previously mentioned 'initial' input and thus will be extracted from TMXR's internal buffer at precisely the rate the port is programmed at. The only time the internal DZ silo ever ends up holding more than one character is: if, for some reason, the OS driver doesn't actually respond to the generated interrupt and doesn't extract data from the DZ device before the next character would arrive. The silo really exists and likely gets use when multiple ports on the multiplexor are in use with multiple ones driving input simultaneously.

My measurements with heavy incoming DZ traffic is precisely correct at lower port speeds and pretty good at higher ones, as you observed outbound speeds are already pretty good.

I tested on a VAX running VMS with a VMS specific program I wrote to read data from a specified port and display the data rate of the data that has been read. On the sending side I've got a program which does a simple TCP connect and writes varying data as fast as possible stalling only when the TCP flow is stalled by being faster than the data being read. The VMS device driver for terminal devices has a kernel typeahead buffer which holds incoming data before some program running in the OS reads it. If the sending side starts first, that typeahead buffer fills and the first read activity returns an overrun indicator status since too much data arrived before being read had to drop some. After that initial data loss indication data is read at what should be reasonable rates. If the reading side starts first all data being sent arrives cleanly.

I tried the same testing activity with the receiving simulator running on a Raspberry Pi-5 and got precisely the same results.

These observations removes the DZ device simulator (and the TMXR logic it uses), from the faulty slow input behavior you are seeing.

Meanwhile, your setup has a number of steps along the data path your input must travel and something in that path can be part of what is going on with your unexpected behavior.

Additionally, when real serial port hardware was used for network traffic by VMS and other systems, the operating systems doing these things needed to increase the size of their type ahead buffers in the kernel so that data overruns didn't happen. I also had to do this in the VMS system being used in the above mentioned tests. I have no idea how such an adjustment would be made on the 2.11 BSD OS you're working with, but that is where you might get better results doing the inbound file transfers without making the simulated hardware do things that never happened. A more modern approach to getting such files into a 2.11 BSD systems would seem to be using the provided network stack which would require rebuilding the kernel with the INET option enabled.

[ddrown]

Hi Mark,

Thanks for taking the time to look at this, and in terms of the PR, I don't mind if my effort was wasted and I have to go back to the drawing board for an accepted implementation. For what it's worth, the PR was meant to make the proposed mechanism concrete rather than abstract, not to skip discussing it first. My main goal is to make sure the simulator matches the hardware and I was hoping that the patch would better describe what I believe to be the problem and my proposed solution. I am open to changing the code in whatever way best emulates the true hardware.

I don't think this is something in my setup. I've been measuring this multiple ways, and I think the numbers make a pretty clean case, so let me lay out what I found.

There's a few different modes that 2.11BSD uses to receive data from the DZ peripheral. One of those is delivering per-character interrupts. The driver (https://github.com/retroprom/bsd2-src/blob/master/sys/pdpuba/dz.c) doesn't stay in that mode under load - it adaptively switches away from it:

• dzscan() runs once a second and tracks characters received per mux (dzchars[i]). If that exceeds dzhighrate (100 cps), it sets CSR_SAE (silo alarm enable) on that mux (dz.c:547-556). At 9600 baud this trips almost immediately on any real bulk transfer.
• With CSR_SAE set, the DZ stops raising a per-character interrupt. It only interrupts once the (shared, 16-slot) silo completely fills (CSR_SA). Until then, the driver relies entirely on a software poll timer it arms itself: dztimer(), firing every dztimerintvl = FASTTIMER = 2 clock ticks (dz.c:35,112,564-578) - at a 60Hz guest clock, that's ~33ms, ~30 times a second.

So under real load, 2.11BSD deliberately trades interrupt-per-byte precision for a ~30Hz software poll, specifically to avoid interrupt overhead. That's a sensible design choice on the driver's part - but it means the simulator has to keep the silo adequately stocked between those ~30Hz polls to sustain anything close to programmed baud rate. The pre-fix dz_update_rcvi() didn't: it only ever pulled one already-arrived character out of TMXR's buffer per invocation - but that's not the same as per guest poll. It's called on every RBUF register read (not just once per dztimer tick - 2.11BSD's dzrint() rereads RBUF in a loop until empty, and each read triggers a fresh one-character refill attempt as a side effect), plus separately by the simulator's own background polling between guest polls. That nets out to well under two characters resident by the time a ~33ms guest poll happens - measured below at ~1.93 avg - not a hard one-per-poll ceiling, but still nowhere near the 16-slot silo the hardware should be able to achieve.

I traced this live against the pre-fix binary (commit a1f57fa3) with set debug -r -t dz_trace.log / set dz debug=REG;RCV;TRC during the same zmodem upload that showed the original ~60-170 cps ceiling. A few things came out of it:

1. CSR_SAE does flip on almost immediately and stays on for the bulk of the transfer (observed oscillating ~14.5s on / ~1s off, consistent with the received rate hovering right around the dzhighrate/dzlowrate (100/75 cps) thresholds).
2. The RBUF-read cadence during an alarm-mode window measures ~32.4ms avg (449 poll cycles sampled) - matching the derived ~33.3ms (2 ticks @ 60Hz) prediction closely.
3. ~1.93 characters get drained per poll cycle on average.
4. Multiplying those out: 865 characters / 14.54s = 59.5 cps - lands almost exactly on the low end of the originally reported range, derived purely from the poll cadence and per-poll drain size, no baud-rate math involved.

I'm glad to share the full trace log if useful.

For the typeahead buffer, I don't think that's it either - and I'm not sure the VMS comparison carries over directly. I'm assuming you tested on one of the Unibus VAX targets (vax780, vax750, vax730, microvax1/microvax2, vax8200/8600) rather than one of the VAXstation/MicroVAX 3100-family targets, which use a separate vax4xx_dz.c I haven't looked at - if that assumption's right, the simulated DZ11 hardware itself is identical either way: those builds compile the exact same pdp11_dz.c, same silo/alarm logic, no VAX-specific behavior in it. The difference is on the guest OS side: your VMS test sounds like it's exercising a driver that keeps taking per-character interrupts and reacting to them, without the alarm-mode throttling I traced above in 2.11BSD's dz.c. An overrun-then-recover pattern on VMS doesn't tell us much about whether 2.11BSD's driver-level poll-rate ceiling is the cause here, since 2.11BSD is in that alarm-mode for essentially the whole transfer, regardless of what VMS's driver does.

Setting the OS difference aside, this also isn't a burst-and-catch-up pattern to begin with, and it isn't new evidence either - the netstat trace in the original report already showed it: the receive queue started at 18,635 bytes sitting in the kernel's TCP receive buffer, and across that capture 42 of 63 one-second samples show zero bytes drained even though the backlog was sitting right there, with every nonzero drain landing in ~241-byte increments (matching TMXR's per-call read size). I re-confirmed the same signature with strace on the DZ line's socket - same ~241-byte buffer-maxed reads, same multi-second gaps between them. Once bytes are in the kernel's Recv-Q, socat/minicom have already done their job; what's left to explain is purely how fast simh reads out of a queue that already has the data waiting.

The fix (dz_update_rcvi() draining everything already-due per invocation instead of one character, plus tmxr_getc_ln() advancing its pacing clock from its own previous value instead of resetting relative to "now" each
time) lets the silo fill at the true baud rate instead of ~1-2 characters per ~33ms guest poll. At 9600 baud (~960 cps, ~1.04ms/char), the 16-slot silo now fills in ~16.7ms on its own, which trips CSR_SA - a real hardware interrupt, not the 30ms software backstop - so the driver ends up draining a full 16-character silo roughly every ~16.7ms: ~60 times/sec x 16 = ~960 cps, matching the theoretical max. The 30Hz dztimer() backstop is what kept pre-fix throughput from collapsing to zero; it's not what limits post-fix throughput, since the silo is now hitting its own alarm threshold faster than that backstop would even fire. Happy to trace the post-fix binary the same way if that mechanism is worth confirming further.

I have never used the real PDP11 or VAX hardware, so I am aware that my understanding may be flawed. I am open to learning or running other measurements that you may see fit.

• Dan

[markpizz]

Hi Dan,

First, to assure you that you and I are looking at the same DZ implementation, I tested with the VAX (aka MicroVAX3900) simulator which leverages the pdp11_dz.c code to implement the DZ devices just like the PDP11. Unibus or Qbus doesn't matter for this programmed I/O (Non DMA) device.

I'm glad you explained the structural logic in the 2.11BSD driver behaviors, which convert to polling under load. This is certainly a different use of the DZ device's logic, so the details of those activities do indeed need to be examined closely. I have confirmed that the VMS DZ driver does not ever enable CSR_SAE on this device, so we're only working with interrupt per character and the drivers from 2.11BSD exercise the DZ hardware differently.

However, before responding to each detail you mentioned in your response, you need to realize that although, as you say the dz_update_rcvi() routine only calls tmxr_getc_ln() once is ABSOLUTELY deliberate, and it is ONLY called when a character is due based on the programmed port speed. When that first character has arrived another CAN NOT arrive any sooner than a whole character time. The system is designed to arrange the next call to dz_update_rcvi() precisely when the next character can arrive.

Looking at your change to sim_tmxr.c where you change:

lp->rxnexttime = floor (sim_gtime_now + ((lp->rxdeltausecs * sim_timer_inst_per_sec ()) / USECS_PER_SECOND));

to

lp->rxnexttime += floor (((lp->rxdeltausecs * sim_timer_inst_per_sec ()) / USECS_PER_SECOND));

This looks promising, however the net result is precisely the same when lp->rxnexttime == sim_gtime_now, so this isn't an improvement since the design of TMXR is to make sure that this routine is called when lp->rxnexttime == sim_gtime_now. Meanwhile, this change is actually completely broken since lp->rxnexttime starts at 0 and not the current instruction count, so everything that has arrived will automatically be ready to read and thus you can accumulate a backlog which really shouldn't be there...

Rather than debugging your changes it makes the most sense for me to walk through the details of how this hardware is actually used by the 2.11 BSD driver. Doing so directly rather than at arms length a little at a time through your activities.
As it turns out, I happen to have a VAX disk image that contains the latest NetBSD for vax and looking that the DZ driver for NetBSD as compared to the 2.11BSD and the structure is more or less the same. A google search also suggested that NetBSD used silo based I/O for this device. I set up the test situation and indeed the CSR_SAE is enabled.

I'll be glad to look into what's happening but I'll need to see the details directly rather than at arms length.

I'll let you know what I find.

• Mark

[ddrown]

Hi Mark,

In regards to multiple calls to tmxr_getc_ln, I was having trouble getting dz_update_rcvi to run often enough when BSD was in software polling mode (details are in my previous reply). That's why I built a catch-up mechanism to insert multiple bytes to simulate as if dz_update_rcvi had been called at 960Hz.

As for rxnexttime starting at 0 (or falling behind), the code for dealing with that is in tmxr_getc_ln. It checks to see if rxnexttime is before the max backlog (16 byte-time worth) and moves it forward if so.

I understand that you want to see this behavior for yourself, and that takes time. I patiently look forward to your results from testing NetBSD.

• Dan