Tracking Issue for directory handles

[the8472]

View all comments

Feature gate: #![feature(dirfd)]

This is a tracking issue for directory handles. Such handles provide a stable reference to an underlying filesystem object (typically directories) that are less vulnerable to TOCTOU attacks and similar races. These security properties will be platform-dependent. Platforms that don't provide the necessary primitives will fall back to operations on absolute paths.

Additionally they may also provide performance benefits by avoiding repeated path lookups when performing many operations on a directory.

Sandboxing is a non-goal. If a platform supports upwards path traversal via .. or symlinks then directory handles will not prevent that. Providing O_BENEATH-style traversal is left to 3rd-party crates or future extensions.

Public API

impl Dir {
    pub fn open<P: AsRef<Path>>(path: P) -> Result<Self>;
    /// This could be put on OpenOptions instead
    pub fn open_with<P: AsRef<Path>>(path: P, opts: &OpenOptions) -> io::Result<Self>;
    pub fn open_for_traversal<P: AsRef<Path>>(path: P) -> io::Result<Self>;

    pub fn try_clone(&self) -> io::Result<Self>;
    pub fn self_metadata(&self) -> Result<Metadata>;

    pub fn open_file<P: AsRef<Path>>(&self, path: P) -> io::Result<File>;
    /// This could be put on OpenOptions instead
    pub fn open_file_with<P: AsRef<Path>>(&self, path: P, opts: &OpenOptions) -> io::Result<File>;
    pub fn open_dir<P: AsRef<Path>>(&self, path: P) -> io::Result<Self>;
    /// This could be put on OpenOptions instead
    pub fn open_dir_with<P: AsRef<Path>>(&self, path: P, opts: &OpenOptions) -> io::Result<Self>
    pub fn create_dir<P: AsRef<Path>>(&self, path: P) -> Result<()>;
    pub fn rename<P: AsRef<Path>, Q: AsRef<Path>>(&self, from: P, to_dir: &Self, to: Q) -> Result<()>;
    pub fn remove_file<P: AsRef<Path>>(&self, path: P) -> Result<()>;
    pub fn remove_dir<P: AsRef<Path>>(&self, path: P) -> Result<()>;
    pub fn symlink<P: AsRef<Path>, Q: AsRef<Path>>(&self, original: P, link: Q) -> Result<()>;
    pub fn metadata(&self, path: impl AsRef<Path>) -> Result<Metadata>;
    pub fn symlink_metadata(&self, path: impl AsRef<Path>) -> Result<Metadata>;
}

// Various trait impls: AsFd, From<OwnedFd>, ...

impl DirEntry {
    pub fn open(&self) -> Result<File>
    /// This could be put on OpenOptions instead
    pub fn open_with(&self, options: &OpenOptions) -> Result<File>
    pub fn remove_file(&self) -> Result<()>
    pub fn remove_dir(&self) -> Result<()>
}

Steps / History

• ACP: Add openat/unlinkat/etc. abstractions to ReadDir/DirEntry/OpenOptions libs-team#259
• minimal dirfd implementation (1/4) #146341
• dirfd file operations (2/4) #150679
• directory handles: add Dir::metadata #157192
• getdents to get free conversion between dirfds and ReadDir
• add more *at calls
• Final comment period (FCP)¹
• Stabilization PR

Unresolved Questions

• AdRawFd can only be implemented on platforms that use actual directory handles, not all fd platforms. Is this fine?
• Should the Dir operations that take paths accept absolute paths (where self is effectively irrelevant)?
• This has been added in piecemeal with a number of fallbacks involved. Prior to stabilization, there should be an implementation review verifying that we're actually using the TOCTOU-resistant operations when available, and that we refuse to fallback to racy versions on tier 1 platforms (Linux, macOS, Windows).
• What should the Dir method for getting the metadata of the directory itself be called (if we have such an operation at all)? Dir::metadata already corresponds to fs::metadata. For now, it is called self_metadata.
• Dir::open_file_with (and likely more Dir methods) on Windows does not support / #163032

Footnotes

1. https://std-dev-guide.rust-lang.org/feature-lifecycle/stabilization.html ↩

[the8472]

During the ACP a survey of platform support was requested to make sure that fallbacks won't be needed on Tier-1 platforms.

libc/syscall survey

For restricted conversion between *DIR and a file descriptor we need fdopendir + dirfd. For free conversion we need an alternative to readdir that operates on file descriptors.

Presence of *at functions was surveyed by looking at the libc crate, so this may be incomplete.

Tier 1

Windows

• openat: NtCreateFile
• readdir: GetFileInformationByHandleEx + FILE_FULL_DIR_INFO.
• renameat:SetFileInformationByHandle + FILE_RENAME_INFORMATION
• unlinkat: SetFileInformationByHandle + FILE_DISPOSITION_INFO_EX
• mkdirat: NtCreateFile
• symlinkat: DeviceIoControl + FSCTL_SET_REPARSE_POINT
• fstatat: either NtQueryInformationByName (if available) or GetFileInformationByHandle and GetFileInformationByHandleEx + FILE_ATTRIBUTE_TAG_INFO

Linux

• readdir: getdents
• has fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

macos

• readdir: getattrlistbulk
• has fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

Tier 2

freebsd, illumos, netbsd, solaris

• readdir: getdents
• have fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

fuchsia

• readdir: ReadDirents
• has fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

ios

• readdir: getattrlistbulk
• has fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

wasm-emscripten

• readdir: getdents
• has fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

wasm-wasi

• readdir: fd_readdir
• has fdopendir, dirfd, openat, renameat, unlinkat, mkdirat, symlinkat, fstatat

redox

libc only lists renameat, unlinkat, symlinkat, fstatat

The redox syscall crate lists no *at calls at all?

uefi

Listed as no_std in the platform support documentation. Somehow has some pieces in std anyway. No libc support either.

Not checked

• tier 3 targets
• fortanix

[jackpot51]

Regarding redox, relibc is the right place to look for support. We should be able to support the same list of functions as Linux and the BSDs.

[zopsicle]

The ACP mentions “it may be possible to add the Dir methods directly to ReadDir instead”. This is problematic because ReadDir: !From<OwnedFd> (#56777). Being able to open a file with File::open, then check its file type using Metadata::file_type and convert it to a Dir object if it is a directory would be very useful IMO, for instance to hash a tree of files. That said it may be better to just address #56777 than to have the design of #120426 hinge on this.

[RalfJung]

From a Miri perspective I'd love to see this, it would let us implement support for emulation of openat and similar functions in a cross-platform way. :)

portable, insecure openat emulation based on Paths

I am somewhat surprised to see this TODO item. In other areas Rust chose to not provide lower-quality fallback implementations, e.g. famously Rust atomics are always hardware atomics and never lock emulated. I can't find discussion of this point in the ACP either. What is the reasoning here for preferring an insecure emulation over a security guarantee stated in the API?

[the8472]

Operating on a directory is a fairly basic operation, like opening a File. If we returned an error that basically says "we could do that, just the way you're opening it is unsupported on this system" then it would become less portable and likely only interesting for tools like sudo which won't support such platforms anyway.

Dir isn't just about security. It provides a handle to a directory that should keep working even if the directory gets renamed (granted, fallbacks don't provide this). It helps with performance. It allows replacing global process state (the current working directory) with local state. It can prevent non-malicious races.
So applications that aren't security-sensitive would still want to use Dir in their filesystem code.

Additionally, the platforms where openat isn't available are more likely to be single-user systems without privilege separation, on those fallbacks wouldn't be an issue anyway. Afaik UEFI has none of that.

That said, maybe we can add a fail_closed flag, impl Dir { const IS_RACEFREE: bool } or something like that for applications that need some extra assurances.

Also, this wouldn't be a runtime fallback but a platform-based fallback. All tier-1 platforms will use real handles and fail if they're not available. E.g. if something blocks openat via seccomp on linux I do not intend to try open.

[addisoncrump]

Also, this wouldn't be a runtime fallback but a platform-based fallback.

Can this be completed with a compiler warning for these platforms? Since this will be something included in the target type, I would imagine so.

[the8472]

@rustbot ping fuchsia

Since it appears that functions being available in the libc crate is not a reliable indicatior that they actually work, is there a better overview of the available posix APIs? I'm interested in the file descriptor APIs listed in #120426 (comment)

[rustbot]

Hey friends of Fuchsia! This issue could use some guidance on how this should be
resolved/implemented on Fuchsia. Could one of you weigh in?

cc @djkoloski @erickt @P1n3appl3 @tmandry

[erickt]

@the8472 how odd, we do support it at the Fuchsia layer, maybe we're not exposing it through fdio. We'll do some investigation and report back.

[erickt]

@the8472 got confirmation from our team that we do support dirfd on fuchsia, so this is just a bug we'll get fixed.

[cgwalters]

Sandboxing is a non-goal. If a platform supports upwards path traversal via .. or symlinks then directory handles will not prevent that. Providing O_BENEATH-style traversal is left to 3rd-party crates or future extensions.

I guess that's a reasonable position, but I would also say that for a lot of the code that would be intentionally switching to these APIs, it often makes sense (when rewriting) to just go to such a sandboxed approach anyways.

I'm a bit surprised this tracking issue doesn't reference cap-std as prior art - I'm sure it's known but we definitely should think about it intentionally.

Especially because when considering things like platform support, there's a lot of prior art that landed there for some raw functionality.

A very important choice to make here is whether the directory is opened as O_PATH (on Linux) or not - cap-std does it this way and I think that's right, but it does have some implications.