DosForceDup (INT 21h/AH=46h) does not validate NewHandle: out-of-bounds JFT access + dup2(h,h) closes the handle

[geraldo-netto]

DosForceDup (INT 21h/AH=46h) does not validate NewHandle: out-of-bounds JFT access + dup2(h,h) closes the handle

Summary

DosForceDup() in kernel/dosfns.c validates OldHandle but never validates
NewHandle against ps_maxfiles, and it has no OldHandle == NewHandle
short-circuit. Because NewHandle comes straight from the caller's CX in
INT 21h/AH=46h (the DOS "force duplicate handle" / dup2 primitive), any
unprivileged program can:

1. force an out-of-bounds read, and conditionally an out-of-bounds write,
   into memory adjacent to the Job File Table (JFT); and
2. make dup2(h, h) close an open handle instead of being a no-op, which is
   a fully deterministic, always-reproducible failure.

Both stem from the same missing validation and are fixed by the same small patch.

Affected code

kernel/dosfns.c (line numbers from current master):

COUNT DosForceDup(unsigned OldHandle, unsigned NewHandle)
{
  psp FAR *p = MK_FP(cu_psp, 0);
  sft FAR *Sftp;

  /* Get the SFT block that contains the SFT                      */
  if ((Sftp = get_sft(OldHandle)) == (sft FAR *) - 1)     /* OldHandle validated */
    return DE_INVLDHNDL;

  /* now close the new handle if it's open                        */
  if ((UBYTE) p->ps_filetab[NewHandle] != 0xff)           /* <-- OOB read: NewHandle unchecked */
  {
    COUNT ret;

    if ((ret = DosClose(NewHandle)) != SUCCESS)
      return ret;
  }

  /* If everything looks ok, bump it up.                          */
  p->ps_filetab[NewHandle] = p->ps_filetab[OldHandle];    /* <-- OOB write: NewHandle unchecked */
  /* possible hazard: integer overflow ska*/
  Sftp->sft_count += 1;
  return SUCCESS;
}

Reached from the INT 21h dispatcher in kernel/inthndlr.c:

    /* Force Duplicate File Handle */
    case 0x46:
      rc = DosForceDup(lr.BX, lr.CX);   /* lr.CX = NewHandle, fully caller-controlled */
      goto short_check;

Contrast with the correct guard used everywhere else in the kernel,
get_sft_idx() in kernel/dosfns.c:

int get_sft_idx(unsigned hndl)
{
  psp FAR *p = MK_FP(cu_psp, 0);
  int idx;

  if (hndl >= p->ps_maxfiles)      /* the check DosForceDup is missing */
    return DE_INVLDHNDL;
  ...
}

Why it is wrong

The JFT is ps_maxfiles bytes long, pointed to by ps_filetab (see
hdr/process.h: default ps_files[20] at PSP offset 0x18, ps_maxfiles at
0x32, ps_filetab at 0x34). Valid handle indices are 0 .. ps_maxfiles-1.

DosForceDup indexes ps_filetab[NewHandle] for read (the "is it open?"
test) and, when the byte read is 0xFF, for write, with NewHandle taken
verbatim from CX (0x0000 .. 0xFFFF).

• The read at ps_filetab[NewHandle] is unconditional and out of bounds for
  any NewHandle >= ps_maxfiles. ps_filetab is a FAR pointer, so the effective
  address is FP_SEG(ps_filetab):(FP_OFF(ps_filetab) + NewHandle) with the
  offset wrapping modulo 64 KiB inside the segment. With the default in-PSP JFT
  at offset 0x18, a large CX reads elsewhere in the PSP segment (environment
  pointer, PSP header, etc.).
• The write ps_filetab[NewHandle] = ps_filetab[OldHandle] corrupts that
  same out-of-bounds byte whenever the pre-existing byte reads as 0xFF
  ("unused slot"), which is common in adjacent DOS memory. The written value is
  OldHandle's SFT index — attacker-influenced.
• MS-DOS returns error 06h (invalid handle) when CX is out of range for
  AH=46h. FreeDOS instead performs the access and can return SUCCESS.

Separately, when OldHandle == NewHandle and the handle is open, the
function first calls DosClose(NewHandle) — which sets ps_filetab[NewHandle] = 0xFF and decrements the SFT count — and then executes
ps_filetab[NewHandle] = ps_filetab[OldHandle], i.e. ps_filetab[h] = ps_filetab[h], copying back the now-closed 0xFF. The handle is left closed
even though the call reports SUCCESS. MS-DOS defines dup2(h, h) as a
successful no-op that leaves the handle open.

Reproduction

Case A — deterministic: dup2(h, h) closes an open handle

100% reproducible on real hardware or any emulator (DOSBox, 86Box, QEMU), no
memory-layout assumptions.

Assemble as a .COM (NASM: nasm -f bin dup2same.asm -o dup2same.com):

; dup2same.asm - demonstrates that DosForceDup mishandles OldHandle == NewHandle
        org 100h

        ; open this program's own file read-only to obtain a real handle
        mov ax, 3D00h           ; INT 21h/AH=3Dh open, AL=0 read-only
        mov dx, fname
        int 21h
        jc  fail_open
        mov [handle], ax        ; save handle h (e.g. 5)

        ; dup2(h, h): AH=46h, BX = OldHandle = h, CX = NewHandle = h
        mov ah, 46h
        mov bx, [handle]
        mov cx, [handle]
        int 21h                 ; MS-DOS: success, handle still open
                                ; FreeDOS: returns success but CLOSES the handle

        ; probe the handle: seek to current position (AH=42h, AL=1, offset 0)
        mov ax, 4201h
        mov bx, [handle]
        xor cx, cx
        xor dx, dx
        int 21h
        jc  handle_closed       ; FreeDOS: CF set, AX=6 (invalid handle) -> BUG
        ; MS-DOS: CF clear -> handle survived, correct

        mov dx, msg_ok
        jmp print_exit
handle_closed:
        mov dx, msg_bug
        jmp print_exit
fail_open:
        mov dx, msg_openfail
print_exit:
        mov ah, 09h
        int 21h
        mov ax, 4C00h
        int 21h

handle      dw 0
fname       db "DUP2SAME.COM",0
msg_ok      db "OK: handle survived dup2(h,h)",13,10,'$'
msg_bug     db "BUG: dup2(h,h) closed the handle (AX=6)",13,10,'$'
msg_openfail db "could not open own file",13,10,'$'

Expected on MS-DOS / PC-DOS / DR-DOS: OK: handle survived dup2(h,h).
Observed on FreeDOS: BUG: dup2(h,h) closed the handle (AX=6).

Equivalent under DEBUG.COM (open handle 5 first, then):

-A
xxxx:0100 MOV AH,46
xxxx:0102 MOV BX,0005
xxxx:0105 MOV CX,0005
xxxx:0108 INT 21
xxxx:010A INT 20
xxxx:010C
-G =100 10A

After this, JFT entry 5 reads FF (closed) on FreeDOS; on MS-DOS it is unchanged.

Case B — out-of-bounds JFT access (security impact)

The missing bounds check lets CX address up to 64 KiB from the JFT. Minimal
trigger (any .COM):

        mov ah, 46h
        mov bx, 0001h           ; OldHandle = STDOUT (open, valid)
        mov cx, 0FFFFh          ; NewHandle = 65535, far out of range
        int 21h
        ; A conforming DOS returns CF=1, AX=6.
        ; FreeDOS performs ps_filetab[0xFFFF] read, and if that byte == 0xFF,
        ; writes ps_filetab[OldHandle] there -> out-of-bounds write, CF=0.

Whether a given CX yields a crash, silent corruption, or an accidental error
depends on the byte currently at the target address, which is exactly why an
unchecked, caller-controlled index here is dangerous. Sweeping CX from
ps_maxfiles upward and watching for a CF=0 (success) return reliably
exhibits the out-of-bounds write in an emulator with a debugger attached.

Suggested fix

Two guards are added, and their order matters:

1. Bounds-check NewHandle first, before it is ever used as a JFT index.
   This closes both the out-of-bounds read and the out-of-bounds write for any
   caller-supplied CX.
2. Short-circuit OldHandle == NewHandle to SUCCESS, but only after
   get_sft(OldHandle) has confirmed the source handle is actually open. Placing
   the no-op check after the get_sft validation is deliberate: dup2(h, h)
   where h is in range but not open must still fail with DE_INVLDHNDL
   (matching MS-DOS), so the short-circuit must not precede that check.

Explicit diff against current master:

--- a/kernel/dosfns.c
+++ b/kernel/dosfns.c
@@ -683,24 +683,34 @@ COUNT DosForceDup(unsigned OldHandle, unsigned NewHandle)
 COUNT DosForceDup(unsigned OldHandle, unsigned NewHandle)
 {
   psp FAR *p = MK_FP(cu_psp, 0);
   sft FAR *Sftp;
 
+  /* Validate NewHandle before it is ever used as a JFT index. NewHandle
+     is the caller's CX from INT 21h/AH=46h; without this check an
+     out-of-range value indexes ps_filetab[] outside the ps_maxfiles-entry
+     Job File Table, causing an out-of-bounds read and (when the byte read
+     is 0xFF) an out-of-bounds write. A conforming DOS returns error 06h. */
+  if (NewHandle >= p->ps_maxfiles)
+    return DE_INVLDHNDL;
+
   /* Get the SFT block that contains the SFT                      */
   if ((Sftp = get_sft(OldHandle)) == (sft FAR *) - 1)
     return DE_INVLDHNDL;
 
+  /* dup2(h, h) is a successful no-op in MS-DOS. This must come AFTER the
+     get_sft(OldHandle) check above, so that duplicating an unopened handle
+     onto itself still fails. Without it, the code below would DosClose()
+     the handle and then copy the just-cleared 0xFF back over it, leaving a
+     handle that the caller believes is open but is actually closed. */
+  if (OldHandle == NewHandle)
+    return SUCCESS;
+
   /* now close the new handle if it's open                        */
   if ((UBYTE) p->ps_filetab[NewHandle] != 0xff)
   {
     COUNT ret;
 
     if ((ret = DosClose(NewHandle)) != SUCCESS)
       return ret;
   }
 
   /* If everything looks ok, bump it up.                          */
   p->ps_filetab[NewHandle] = p->ps_filetab[OldHandle];
   /* possible hazard: integer overflow ska*/
   Sftp->sft_count += 1;
   return SUCCESS;
 }

Post-patch behavior:

• Case A prints OK: handle survived dup2(h,h) — the handle is left open.
• Case B returns CF=1, AX=06h (invalid handle) with no memory access beyond
  the JFT.
• dup2(closed_handle, closed_handle) still returns CF=1, AX=06h, because the
  no-op short-circuit sits behind the get_sft(OldHandle) validation.

Notes

• Still present on origin/UNSTABLE (the NewHandle index there, lines 704/716,
  remains unchecked; only an SFT-status guard was added around the write).
• This defect was found using adversarial AI agents performing a static
  review of the 16-bit resident kernel — multiple independent agents auditing the
  same code and cross-checking each other's findings against the source. The
  result was then verified by hand against the kernel sources. It has not yet
  been build-tested; the reproducers above are the recommended validation.