From b06d0a54b44cd2935389b7d8a5a44f46c8c8c448 Mon Sep 17 00:00:00 2001 From: Hawkynt Date: Sun, 6 Sep 2026 16:31:37 +0200 Subject: [PATCH] # UDF padded its directories and could read only one block size MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The writer and the reader agreed with each other and with nothing else. Every fault below was found by putting foreign software on one side: mkudffs and the kernel's udf driver built and filled the volumes we read, udfinfo and that same driver read the volumes we wrote. Writing: - A File Identifier Descriptor may span a logical block boundary — ECMA-167 4/14.4, and mkudffs does it: in a volume it built, the record at byte 2020 of a directory runs into the next block. We padded the boundary instead, so the driver hit a zero tag where the next record belongs: "udf_verify_fi: directory (ino 259) has entry at pos 2020 with incorrect tag 0", repeated without end, and the walk never returned. Every directory over 2048 bytes was unreadable. - Volume identifiers were written as bare ASCII. ECMA-167 1/7.2.12 spells a dstring as a compression byte, the characters, and the used length in the field's last byte, so udfinfo reported the label, the logical volume, the volume set and the file set as empty. - The volume carried one anchor, one descriptor sequence, and neither an unallocated space, implementation use nor integrity descriptor. udfinfo printed seven warnings and called the logical volume inconsistent. It now records both sequences, both anchors, and a closed integrity descriptor whose file and directory counts match the tree. - File names went out as UTF-8 under compression identifier 8, which means one byte per character. Anything above U+007F was unreadable to everything else. - An empty file was still given a block, and a file's extent was rounded up to one, so the allocation said more than the size did. Extents now stop at the logical length and a zero-length file records none. - File version number, tag locations, blocks recorded, checkpoint and unique identifiers were all left at zero. Files past 1 GiB now take several extents; past what one File Entry can hold, CanAccept declines rather than the writer producing a volume nothing can address. Reading: - The anchor's address, logical block 256, is counted in blocks, and we assumed 2048. Every volume mkudffs formats at 512, 1024 or 4096 threw "invalid AVDP tag" before a byte was read. The block size is now probed from the anchor and confirmed against the logical volume descriptor. - Compression identifier 8 was decoded as UTF-8; it is one byte per character. A file the kernel called cafe.bin came back with a replacement character. - Allocation descriptors stopped at the first continuation (extent type 3), so an object whose extents outgrow its File Entry lost everything past it. At 512 bytes a block that is a directory of 200 entries: 300 of 332 files vanished. Continuations are now followed, with a visited set bounding the walk. In place editing keeps up with the rest: both partition descriptors grow, the second anchor follows the partition, and the integrity descriptor's size, free space and file count are maintained. Verified: a volume with the size sweep, long and Unicode names, nested directories and a 500-entry directory mounts read-only and all 577 files come back byte for byte, with nothing in the driver log; udfinfo exits zero with an empty stderr and reports the right label, revision, counts and closed integrity; 7z tests it clean. In the other direction, volumes mkudffs built at 512, 1024, 2048 and 4096 bytes a block and the kernel filled read back byte for byte, as does a 64 MB volume of 579 files. Each fix was reverted in turn and the matching test fails without it. --- .../FileSystems/FileSizeBoundaryTests.cs | 2 +- .../UdfFilesystemDriverTests.cs | 11 +- Compression.Tests/Udf/UdfModifierTests.cs | 7 +- Compression.Tests/Udf/UdfNativeToolTests.cs | 294 +++++++ .../Udf/UdfOnDiskStructureTests.cs | 253 ++++++ Compression.Tests/Udf/UdfTests.cs | 27 +- Compression.Tests/Udf/UdfWipeEmptyTests.cs | 5 +- .../FileSystem.Udf/OstaCompressedUnicode.cs | 108 +++ FileSystems/FileSystem.Udf/UdfBlockMover.cs | 15 +- FileSystems/FileSystem.Udf/UdfDescriptors.cs | 157 ++++ FileSystems/FileSystem.Udf/UdfExtentMap.cs | 49 +- .../FileSystem.Udf/UdfFormatDescriptor.cs | 26 +- FileSystems/FileSystem.Udf/UdfModifier.cs | 272 +++++-- FileSystems/FileSystem.Udf/UdfReader.cs | 278 +++++-- FileSystems/FileSystem.Udf/UdfWriter.cs | 734 ++++++++++-------- Hawkynt.FileFormats.FileSystems/REFERENCE.md | 8 +- 16 files changed, 1748 insertions(+), 498 deletions(-) create mode 100644 Compression.Tests/Udf/UdfNativeToolTests.cs create mode 100644 Compression.Tests/Udf/UdfOnDiskStructureTests.cs create mode 100644 FileSystems/FileSystem.Udf/OstaCompressedUnicode.cs create mode 100644 FileSystems/FileSystem.Udf/UdfDescriptors.cs diff --git a/Compression.Tests/FileSystems/FileSizeBoundaryTests.cs b/Compression.Tests/FileSystems/FileSizeBoundaryTests.cs index 6c13e928c..6de3e0978 100644 --- a/Compression.Tests/FileSystems/FileSizeBoundaryTests.cs +++ b/Compression.Tests/FileSystems/FileSizeBoundaryTests.cs @@ -47,7 +47,7 @@ public sealed class FileSizeBoundaryTests { /// private static readonly string[] Filesystems = [ "Fat", "ExFat", "Jfs", "HfsPlus", "Btrfs", "Ext", - "CramFs", "Erofs", "MinixFs", "SquashFs", "Iso", "ReiserFs", "Jffs2", + "CramFs", "Erofs", "MinixFs", "SquashFs", "Iso", "ReiserFs", "Jffs2", "Udf", ]; /// diff --git a/Compression.Tests/FilesystemDrivers/UdfFilesystemDriverTests.cs b/Compression.Tests/FilesystemDrivers/UdfFilesystemDriverTests.cs index 57dd6d4e1..48596f599 100644 --- a/Compression.Tests/FilesystemDrivers/UdfFilesystemDriverTests.cs +++ b/Compression.Tests/FilesystemDrivers/UdfFilesystemDriverTests.cs @@ -124,8 +124,14 @@ public void ExistingExtractorAlsoFollowsMultipleAllocationDescriptors() { Assert.That(reader.Extract(entry), Is.EqualTo(payload)); } + /// + /// A continuation descriptor points at a block that must open with an + /// Allocation Extent Descriptor (ECMA-167 §4/14.5). Aiming one at the file's + /// own data instead leaves the object's extents unreachable, and the mount + /// has to close rather than serve a file it cannot address. + /// [Test, Category("Driver"), Category("Corruption")] - public void ContinuationAllocationDescriptorFailsMountClosed() { + public void BrokenContinuationAllocationDescriptorFailsMountClosed() { var payload = Enumerable.Repeat((byte)0xA5, 6000).ToArray(); var bytes = BuildImage(("continued.bin", payload)); var fe = FindRegularFileEntry(bytes, payload.Length); @@ -139,7 +145,8 @@ public void ContinuationAllocationDescriptorFailsMountClosed() { Assert.Multiple(() => { Assert.That(profile.CanMount, Is.False); - Assert.That(profile.Limitations.Any(text => text.Contains("continuation", StringComparison.OrdinalIgnoreCase)), Is.True); + Assert.That(profile.Limitations.Any(text => text.Contains("logical file bytes", StringComparison.OrdinalIgnoreCase)), Is.True, + string.Join("; ", profile.Limitations)); }); } diff --git a/Compression.Tests/Udf/UdfModifierTests.cs b/Compression.Tests/Udf/UdfModifierTests.cs index f75d90fe3..c7ceb7cb9 100644 --- a/Compression.Tests/Udf/UdfModifierTests.cs +++ b/Compression.Tests/Udf/UdfModifierTests.cs @@ -210,10 +210,11 @@ public void AfterAdd_PartitionDescriptorTagStillValid() { UdfModifier.AddFile(img, "extra.bin", new byte[200]); var bytes = img.ToArray(); - // PD is at sector 33 in writer layout. - var off = 33 * SectorSize; + // The main volume descriptor sequence starts at sector 32 with the primary + // volume descriptor, then the logical volume descriptor, then the partition. + var off = 34 * SectorSize; Assert.That(BinaryPrimitives.ReadUInt16LittleEndian(bytes.AsSpan(off)), Is.EqualTo((ushort)5), - "expected Partition Descriptor at sector 33"); + "expected Partition Descriptor at sector 34"); var crcLen = BinaryPrimitives.ReadUInt16LittleEndian(bytes.AsSpan(off + 10)); var storedCrc = BinaryPrimitives.ReadUInt16LittleEndian(bytes.AsSpan(off + 8)); diff --git a/Compression.Tests/Udf/UdfNativeToolTests.cs b/Compression.Tests/Udf/UdfNativeToolTests.cs new file mode 100644 index 000000000..02a40827d --- /dev/null +++ b/Compression.Tests/Udf/UdfNativeToolTests.cs @@ -0,0 +1,294 @@ +#pragma warning disable CS1591 +using System.Diagnostics; +using System.Text; +using Compression.Tests.Support; +using FileSystem.Udf; + +namespace Compression.Tests.Udf; + +/// +/// Checks both directions of the UDF implementation against the software that +/// owns the format on this host: udfinfo and the kernel's udf +/// driver read what we write, and what mkudffs plus that same driver +/// wrote is read back by us. +/// +/// +/// +/// A writer and a reader that only ever meet each other agree on their own +/// mistakes. Every check here therefore has foreign software on one side of it: +/// nothing in this fixture passes because our reader liked our writer. +/// +/// +/// Each test says which tool it needs and stands down when the host does not +/// have it. A tool that is present and rejects the volume is a failure, never +/// a skip. +/// +/// +[TestFixture] +[Category("ExternalFsInterop")] +public sealed class UdfNativeToolTests { + + private const int BlockSize = 2048; + + private string _temp = null!; + + [SetUp] + public void Setup() { + this._temp = Path.Combine(Path.GetTempPath(), "cwb_udf_" + Guid.NewGuid().ToString("N")[..12]); + Directory.CreateDirectory(this._temp); + } + + [TearDown] + public void Teardown() { + try { Directory.Delete(this._temp, recursive: true); } catch { /* best effort */ } + } + + // ── the volume we write ─────────────────────────────────────────────────── + + /// + /// Content for a file of bytes, every byte a + /// function of its own offset so a misplaced block shows up as wrong bytes + /// rather than merely a wrong length. + /// + private static byte[] Pattern(int size) { + var body = new byte[size]; + for (var i = 0; i < size; ++i) + body[i] = (byte)((i * 2_654_435_761L) >> 13); + return body; + } + + /// + /// The shapes that break UDF writers: the sizes either side of a logical + /// block, an empty file, names that need both OSTA compressions, a name at + /// the identifier's length limit, nesting, and a directory with enough + /// entries that its File Identifier Descriptors run past one block — which + /// is where a record has to be allowed to span the boundary. + /// + private static IReadOnlyList<(string Name, byte[] Data)> Corpus() { + var files = new List<(string, byte[])>(); + foreach (var size in new[] { 0, 1, 2047, 2048, 2049, 4095, 4096, 4097, 65_537 }) + files.Add(($"sizes/size_{size:D8}.bin", Pattern(size))); + + files.Add(("names/" + new string('n', 180) + ".txt", "long identifier\n"u8.ToArray())); + foreach (var name in new[] { "café.bin", "äöü.txt", "日本語.txt", "русский.dat" }) + files.Add(("names/" + name, Encoding.UTF8.GetBytes(name + "\n"))); + + var nested = ""; + for (var depth = 0; depth < 5; ++depth) { + nested += $"deep{depth}/"; + files.Add((nested + "leaf.bin", Encoding.ASCII.GetBytes($"depth {depth}\n"))); + } + + for (var i = 0; i < 200; ++i) + files.Add(($"many/entry_{i:D4}.txt", Encoding.ASCII.GetBytes($"entry number {i}\n"))); + + return files; + } + + private string WriteOurVolume(string fileName = "ours.img") { + var writer = new UdfWriter { VolumeIdentifier = "CWBUDF" }; + foreach (var (name, data) in Corpus()) + writer.AddFile(name, data); + + var path = Path.Combine(this._temp, fileName); + using (var output = File.Create(path)) + writer.WriteTo(output); + return path; + } + + // ── write direction ─────────────────────────────────────────────────────── + + /// + /// udfinfo is the format's own tool, and it reports every structure a + /// UDF volume is supposed to carry that is missing. It exits zero either way, + /// so the volume is only accepted when it also had nothing to say: before the + /// reserve descriptor sequence, the second anchor, the unallocated space and + /// implementation use descriptors and the integrity descriptor were written, + /// it printed seven warnings and called the logical volume inconsistent. + /// + [Test] + public void UdfinfoAcceptsOurVolumeWithoutWarnings() { + var udfinfo = Which("udfinfo"); + if (udfinfo == null) + Assert.Ignore("udfinfo is not installed (udftools)."); + + var image = this.WriteOurVolume(); + var (stdout, stderr, exit) = Run(udfinfo!, Quote(image)); + + Assert.Multiple(() => { + Assert.That(exit, Is.Zero, $"udfinfo exit {exit}\n{stdout}\n{stderr}"); + Assert.That(stderr.Trim(), Is.Empty, $"udfinfo complained about our volume:\n{stderr}"); + }); + + var fields = ParseFields(stdout); + Assert.Multiple(() => { + Assert.That(fields.GetValueOrDefault("label"), Is.EqualTo("CWBUDF"), + "the volume identifier is a dstring: a compression byte, the characters, and the used length in the field's last byte"); + Assert.That(fields.GetValueOrDefault("lvid"), Is.EqualTo("CWBUDF")); + Assert.That(fields.GetValueOrDefault("fsid"), Is.EqualTo("CWBUDF")); + Assert.That(fields.GetValueOrDefault("integrity"), Is.EqualTo("closed"), + "without a logical volume integrity descriptor the volume reads as inconsistent"); + Assert.That(fields.GetValueOrDefault("udfrev"), Is.EqualTo("2.01")); + Assert.That(fields.GetValueOrDefault("blocksize"), Is.EqualTo("2048")); + Assert.That(fields.GetValueOrDefault("numfiles"), Is.EqualTo(Corpus().Count.ToString())); + }); + } + + /// + /// The kernel's udf driver reads a directory as one uninterrupted run of File + /// Identifier Descriptors and lets a record span a block boundary. Padding + /// the boundary instead made it stop at the first pad byte — "entry at pos + /// 2020 with incorrect tag 0" — and lose every entry past it, which is most + /// of a directory of any size. + /// + [Test] + public void TheKernelDriverReadsEveryFileBackFromOurVolume() { + var image = this.WriteOurVolume(); + var expected = Corpus().Select(static entry => entry.Data).ToList(); + + var result = ThirdPartyFsCheck.ReadBack("Udf", image, expected); + if (!result.Ran) + Assert.Ignore(result.Detail); + + Assert.That(result.Ok, Is.True, $"{result.Tool}: {result.Detail}"); + } + + // ── read direction ──────────────────────────────────────────────────────── + + /// + /// A volume neither our writer nor our reader had a hand in: mkudffs + /// formats it, the kernel fills it, and only then does our reader see it. + /// + /// + /// Run for each block size mkudffs will format, because the anchor sits at + /// logical block 256 and that address is counted in blocks: assuming 2048 + /// looked for it in the wrong place on every other volume and the reader + /// threw before it read anything at all. At 512 bytes a block, a directory of + /// two hundred entries also outgrows the descriptors its File Entry has room + /// for, so the rest of them live in a continuation extent the reader has to + /// follow. + /// + [TestCase(512)] + [TestCase(1024)] + [TestCase(2048)] + [TestCase(4096)] + public void OurReaderReadsAVolumeTheNativeToolsBuilt(int blockSize) { + var mkudffs = Which("mkudffs"); + if (mkudffs == null) + Assert.Ignore("mkudffs is not installed (udftools)."); + if (!CanMount) + Assert.Ignore("filling a volume needs the kernel's udf driver and passwordless sudo."); + + var image = Path.Combine(this._temp, $"native_{blockSize}.img"); + using (var file = File.Create(image)) + file.SetLength(32L * 1024 * 1024); + + var (mkStdout, mkStderr, mkExit) = Run(mkudffs!, + $"--media-type=hd --blocksize={blockSize} --udfrev=2.01 --label=NATIVE {Quote(image)}"); + Assert.That(mkExit, Is.Zero, $"mkudffs exit {mkExit}\n{mkStdout}\n{mkStderr}"); + + var written = FillThroughTheKernel(image); + if (written == null) + Assert.Ignore("the kernel refused to mount a volume mkudffs had just made writable."); + + using var stream = File.OpenRead(image); + using var reader = new UdfReader(stream, leaveOpen: true); + + var got = new Dictionary(StringComparer.Ordinal); + foreach (var entry in reader.Entries) { + if (entry.IsDirectory) continue; + using var buffer = new MemoryStream(); + reader.ExtractTo(entry, buffer); + got[entry.Name] = buffer.ToArray(); + } + + Assert.Multiple(() => { + foreach (var (name, data) in written!) { + Assert.That(got.ContainsKey(name), Is.True, $"our reader lost {name}"); + if (got.TryGetValue(name, out var actual)) + Assert.That(actual, Is.EqualTo(data), $"{name} came back with different bytes"); + } + + Assert.That(got, Has.Count.EqualTo(written.Count), + "our reader invented entries the kernel did not put there"); + }); + } + + /// + /// Mounts writable, writes the corpus into it with + /// the kernel's own driver, and returns what was written keyed by the path + /// our reader will report. Null when the volume would not mount. + /// + private static IReadOnlyDictionary? FillThroughTheKernel(string image) { + var mountPoint = Path.Combine(Path.GetTempPath(), "cwb_udfrw_" + Guid.NewGuid().ToString("N")[..8]); + Directory.CreateDirectory(mountPoint); + try { + var uid = Run("id", "-u").StdOut.Trim(); + var gid = Run("id", "-g").StdOut.Trim(); + var (_, stderr, exit) = Run("sudo", + $"-n mount -t udf -o loop,noatime,uid={uid},gid={gid} {Quote(image)} {Quote(mountPoint)}"); + if (exit != 0) { + TestContext.Out.WriteLine($"mount refused the mkudffs volume: {stderr}"); + return null; + } + + try { + var written = new Dictionary(StringComparer.Ordinal); + foreach (var (name, data) in Corpus()) { + var target = Path.Combine(mountPoint, name); + Directory.CreateDirectory(Path.GetDirectoryName(target)!); + File.WriteAllBytes(target, data); + written[name] = data; + } + + return written; + } finally { + Run("sudo", $"-n umount {Quote(mountPoint)}"); + } + } finally { + try { Directory.Delete(mountPoint, recursive: true); } catch { /* the mount point may be gone */ } + } + } + + // ── process plumbing ────────────────────────────────────────────────────── + + private static readonly bool CanMount = OperatingSystem.IsLinux() && Run("sudo", "-n true").Exit == 0; + + private static Dictionary ParseFields(string stdout) { + var fields = new Dictionary(StringComparer.Ordinal); + foreach (var line in stdout.Split('\n')) { + var split = line.IndexOf('='); + if (split > 0) + fields[line[..split].Trim()] = line[(split + 1)..].Trim(); + } + + return fields; + } + + private static string Quote(string path) => "\"" + path + "\""; + + private static string? Which(string tool) { + var (stdout, _, exit) = Run("which", tool); + var path = stdout.Trim(); + return exit == 0 && path.Length > 0 ? path : null; + } + + private static (string StdOut, string StdErr, int Exit) Run(string file, string arguments) { + try { + var start = new ProcessStartInfo(file, arguments) { + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + using var process = Process.Start(start); + if (process == null) return ("", "could not start " + file, -1); + var stdout = process.StandardOutput.ReadToEnd(); + var stderr = process.StandardError.ReadToEnd(); + process.WaitForExit(120_000); + return (stdout, stderr, process.ExitCode); + } catch (Exception ex) { + return ("", ex.Message, -1); + } + } +} diff --git a/Compression.Tests/Udf/UdfOnDiskStructureTests.cs b/Compression.Tests/Udf/UdfOnDiskStructureTests.cs new file mode 100644 index 000000000..bf3591dfd --- /dev/null +++ b/Compression.Tests/Udf/UdfOnDiskStructureTests.cs @@ -0,0 +1,253 @@ +#pragma warning disable CS1591 +using System.Buffers.Binary; +using System.Text; +using FileSystem.Udf; + +namespace Compression.Tests.Udf; + +/// +/// Reads the bytes our writer produces and checks the few structural rules that +/// other implementations depend on. These need no tool installed, so they hold +/// the line on a host where the native checks stand down. +/// +[TestFixture] +public sealed class UdfOnDiskStructureTests { + + private const int BlockSize = 2048; + private const int PartitionStart = 257; + + private static byte[] Build(params (string Name, byte[] Data)[] files) { + var writer = new UdfWriter(); + foreach (var (name, data) in files) + writer.AddFile(name, data); + using var image = new MemoryStream(); + writer.WriteTo(image); + return image.ToArray(); + } + + private static ReadOnlySpan Block(byte[] image, int block) + => image.AsSpan(block * BlockSize, BlockSize); + + /// The root directory's File Entry, at partition block 1. + private static ReadOnlySpan RootFileEntry(byte[] image) => Block(image, PartitionStart + 1); + + /// + /// A name whose characters all fit in a byte is recorded with compression 8 + /// and one byte per character. Recording UTF-8 under that identifier makes + /// every accented name unreadable to anything but the writer that produced + /// it — OSTA UDF §2.1.1. + /// + [Test] + public void ALatin1NameIsRecordedOneBytePerCharacter() { + var image = Build(("café.bin", "x"u8.ToArray())); + var fids = DirectoryBytes(image, RootFileEntry(image)); + + // The parent record comes first and carries no identifier. + var identifier = FirstNamedIdentifier(fids); + Assert.Multiple(() => { + Assert.That(identifier[0], Is.EqualTo(8), "characters below U+0100 take the single-byte compression"); + Assert.That(identifier.Length, Is.EqualTo(1 + "café.bin".Length)); + Assert.That(identifier[^5], Is.EqualTo(0xE9), "é is one byte, its own code point, not two UTF-8 bytes"); + }); + } + + /// A name that needs more than a byte per character takes the wide compression. + [Test] + public void AWideNameIsRecordedTwoBytesPerCharacterBigEndian() { + var image = Build(("日本.bin", "x"u8.ToArray())); + var identifier = FirstNamedIdentifier(DirectoryBytes(image, RootFileEntry(image))); + + Assert.Multiple(() => { + Assert.That(identifier[0], Is.EqualTo(16)); + Assert.That(identifier.Length, Is.EqualTo(1 + 2 * "日本.bin".Length)); + Assert.That(identifier[1], Is.EqualTo(0x65), "big-endian: the high byte of U+65E5 comes first"); + Assert.That(identifier[2], Is.EqualTo(0xE5)); + }); + } + + /// + /// A directory is a dense run of File Identifier Descriptors. ECMA-167 + /// §4/14.4 lets one span a logical block boundary and the kernel's udf driver + /// relies on that; padding the boundary instead leaves a zero tag where the + /// next record should be, and the driver stops there. + /// + [Test] + public void DirectoryRecordsRunBackToBackAcrossBlockBoundaries() { + // Enough entries that the directory needs several blocks, so at least one + // record has to straddle a boundary. + var files = Enumerable.Range(0, 200) + .Select(i => ($"entry_{i:D4}.txt", "x"u8.ToArray())) + .ToArray(); + var image = Build(files); + var fids = DirectoryBytes(image, RootFileEntry(image)); + + var straddling = 0; + var position = 0; + var records = 0; + while (position + 38 <= fids.Length) { + var tag = BinaryPrimitives.ReadUInt16LittleEndian(fids.AsSpan(position)); + Assert.That(tag, Is.EqualTo(257), + $"byte {position} of the directory is not the start of a File Identifier Descriptor"); + var implementationUse = BinaryPrimitives.ReadUInt16LittleEndian(fids.AsSpan(position + 36)); + var length = (38 + implementationUse + fids[position + 19] + 3) & ~3; + + Assert.That(BinaryPrimitives.ReadUInt16LittleEndian(fids.AsSpan(position + 16)), Is.EqualTo(1), + "OSTA UDF §2.3.4.1: the file version number of every record is one"); + Assert.That(BinaryPrimitives.ReadUInt32LittleEndian(fids.AsSpan(position + 12)), + Is.EqualTo((uint)(RootDirectoryFirstBlock(image) + position / BlockSize)), + "a record's tag names the block it starts in"); + + if (position / BlockSize != (position + length - 1) / BlockSize) + ++straddling; + position += length; + ++records; + } + + Assert.Multiple(() => { + Assert.That(records, Is.EqualTo(files.Length + 1), "one record per entry, plus the parent"); + Assert.That(position, Is.EqualTo(fids.Length), "the records fill the directory exactly"); + Assert.That(straddling, Is.GreaterThan(0), + "with this many entries at least one record has to cross a block boundary"); + }); + } + + /// + /// A zero-length file is given no extent. An allocation descriptor naming a + /// block it does not own is a chain longer than the size says, which is what + /// every filesystem's own checker calls corruption. + /// + [Test] + public void AnEmptyFileIsGivenNoExtent() { + var image = Build(("empty.bin", []), ("full.bin", new byte[3000])); + + var empty = FileEntryFor(image, "empty.bin"); + var full = FileEntryFor(image, "full.bin"); + + Assert.Multiple(() => { + Assert.That(BinaryPrimitives.ReadUInt64LittleEndian(image.AsSpan(empty + 56)), Is.Zero, + "information length"); + Assert.That(BinaryPrimitives.ReadUInt32LittleEndian(image.AsSpan(empty + 172)), Is.Zero, + "an empty file records no allocation descriptors at all"); + Assert.That(BinaryPrimitives.ReadUInt64LittleEndian(image.AsSpan(empty + 64)), Is.Zero, + "and no blocks recorded"); + + Assert.That(BinaryPrimitives.ReadUInt32LittleEndian(image.AsSpan(full + 172)), Is.EqualTo(8u), + "a file with content records one descriptor"); + Assert.That(BinaryPrimitives.ReadUInt32LittleEndian(image.AsSpan(full + 176)) & 0x3FFFFFFF, + Is.EqualTo(3000u), "whose length is the file's, not the block it was rounded up to"); + }); + } + + /// + /// ECMA-167 §3/8.4 wants an anchor at logical block 256 and at the volume's + /// last block, and each says which block it is. A volume with only the first + /// has one copy of the only descriptor found by address rather than by being + /// pointed at. + /// + [Test] + public void BothAnchorsAreRecordedAndNameTheirOwnBlock() { + var image = Build(("a.bin", "a"u8.ToArray())); + var lastBlock = image.Length / BlockSize - 1; + + Assert.Multiple(() => { + foreach (var block in new[] { 256, lastBlock }) { + var tag = Block(image, block); + Assert.That(BinaryPrimitives.ReadUInt16LittleEndian(tag), Is.EqualTo(2), + $"no anchor at block {block}"); + Assert.That(BinaryPrimitives.ReadUInt32LittleEndian(tag[12..]), Is.EqualTo((uint)block), + $"the anchor at block {block} names a different block"); + byte sum = 0; + for (var i = 0; i < 16; ++i) + if (i != 4) sum = (byte)(sum + tag[i]); + Assert.That(sum, Is.EqualTo(tag[4]), $"the anchor at block {block} has a broken tag checksum"); + } + }); + } + + /// + /// Both volume descriptor sequences the anchor names are recorded, each + /// carrying the descriptors a UDF volume is required to have. + /// + [Test] + public void BothVolumeDescriptorSequencesAreRecorded() { + var image = Build(("a.bin", "a"u8.ToArray())); + var anchor = Block(image, 256); + var main = (int)BinaryPrimitives.ReadUInt32LittleEndian(anchor[20..]); + var reserve = (int)BinaryPrimitives.ReadUInt32LittleEndian(anchor[28..]); + + Assert.That(reserve, Is.Not.Zero, "the anchor records no reserve sequence"); + Assert.Multiple(() => { + foreach (var start in new[] { main, reserve }) { + var tags = new List(); + for (var i = 0; i < 16; ++i) { + var tag = BinaryPrimitives.ReadUInt16LittleEndian(Block(image, start + i)); + if (tag == 0) break; + tags.Add(tag); + if (tag == 8) break; + } + + // Primary volume, logical volume, partition, implementation use, + // unallocated space, terminator. + Assert.That(tags, Is.EquivalentTo(new ushort[] { 1, 6, 5, 4, 7, 8 }), + $"the sequence at block {start} is missing descriptors"); + } + }); + } + + // ── walking helpers ─────────────────────────────────────────────────────── + + private static int RootDirectoryFirstBlock(byte[] image) + => (int)BinaryPrimitives.ReadUInt32LittleEndian(RootFileEntry(image)[180..]); + + /// Concatenates the blocks a directory File Entry's descriptors name. + private static byte[] DirectoryBytes(byte[] image, ReadOnlySpan fileEntry) { + var informationLength = (int)BinaryPrimitives.ReadUInt64LittleEndian(fileEntry[56..]); + var lengthOfDescriptors = (int)BinaryPrimitives.ReadUInt32LittleEndian(fileEntry[172..]); + + var bytes = new List(); + for (var at = 176; at + 8 <= 176 + lengthOfDescriptors; at += 8) { + var length = (int)(BinaryPrimitives.ReadUInt32LittleEndian(fileEntry[at..]) & 0x3FFFFFFF); + var block = (int)BinaryPrimitives.ReadUInt32LittleEndian(fileEntry[(at + 4)..]); + for (var i = 0; i < length; ++i) + bytes.Add(image[(PartitionStart + block) * BlockSize + i]); + } + + return [.. bytes.Take(informationLength)]; + } + + /// The identifier bytes of the first record that names something. + private static byte[] FirstNamedIdentifier(byte[] directory) { + var position = 0; + while (position + 38 <= directory.Length) { + var implementationUse = BinaryPrimitives.ReadUInt16LittleEndian(directory.AsSpan(position + 36)); + var identifierLength = directory[position + 19]; + if (identifierLength > 0) + return directory[(position + 38 + implementationUse)..(position + 38 + implementationUse + identifierLength)]; + position += (38 + implementationUse + identifierLength + 3) & ~3; + } + + throw new InvalidOperationException("the directory holds no named record"); + } + + /// Byte offset of the File Entry the root directory names . + private static int FileEntryFor(byte[] image, string name) { + var directory = DirectoryBytes(image, RootFileEntry(image)); + var position = 0; + while (position + 38 <= directory.Length) { + var implementationUse = BinaryPrimitives.ReadUInt16LittleEndian(directory.AsSpan(position + 36)); + var identifierLength = directory[position + 19]; + if (identifierLength > 1) { + var text = Encoding.Latin1.GetString( + directory, position + 39 + implementationUse, identifierLength - 1); + if (text == name) { + var block = BinaryPrimitives.ReadUInt32LittleEndian(directory.AsSpan(position + 24)); + return (int)((PartitionStart + block) * BlockSize); + } + } + + position += (38 + implementationUse + identifierLength + 3) & ~3; + } + + throw new InvalidOperationException($"the root directory holds no entry named {name}"); + } +} diff --git a/Compression.Tests/Udf/UdfTests.cs b/Compression.Tests/Udf/UdfTests.cs index 2bee50c59..86aaf065e 100644 --- a/Compression.Tests/Udf/UdfTests.cs +++ b/Compression.Tests/Udf/UdfTests.cs @@ -56,6 +56,12 @@ private static byte[] BuildMinimalUdf(params (string Name, byte[] Data)[] files) // Main VDS extent: length at offset 16, location at offset 20 BinaryPrimitives.WriteUInt32LittleEndian(avdp[16..], 4 * (uint)sectorSize); // 4 sectors BinaryPrimitives.WriteUInt32LittleEndian(avdp[20..], 32); // starts at sector 32 + // The anchor is the one descriptor a reader finds by address rather than by + // being pointed at, so it has to say which block it is and carry a valid + // ECMA-167 §7.2 checksum — that pair is what separates a real anchor from + // file data that happens to start with the same two bytes. + BinaryPrimitives.WriteUInt32LittleEndian(avdp[12..], 256); + SealTagChecksum(img, 256 * sectorSize); // Partition Descriptor at sector 32 (tag 5) var pd = img.AsSpan(32 * sectorSize); @@ -155,6 +161,22 @@ private static byte[] BuildMinimalUdf(params (string Name, byte[] Data)[] files) return img; } + /// + /// Writes the ECMA-167 §7.2 TagChecksum of the descriptor tag at + /// : the sum modulo 256 of the tag's other fifteen + /// bytes. + /// + private static void SealTagChecksum(byte[] image, int offset) { + image[offset + 4] = 0; + byte sum = 0; + for (var i = 0; i < 16; ++i) { + if (i == 4) continue; + sum = (byte)(sum + image[offset + i]); + } + + image[offset + 4] = sum; + } + [Test, Category("HappyPath")] public void Read_SingleFile() { var content = "Hello UDF!"u8.ToArray(); @@ -286,8 +308,9 @@ public void Writer_HasNsrMagic() { using var ms = new MemoryStream(); w.WriteTo(ms); var bytes = ms.ToArray(); - // NSR02 at sector 17, offset 1 - Assert.That(Encoding.ASCII.GetString(bytes, 17 * 2048 + 1, 5), Is.EqualTo("NSR02")); + // NSR03 at sector 17, offset 1: the writer records UDF 2.01, whose volume + // recognition sequence names the third-generation structure. + Assert.That(Encoding.ASCII.GetString(bytes, 17 * 2048 + 1, 5), Is.EqualTo("NSR03")); } // ── Descriptor tag CRC-16 (ECMA-167 §7.2.1) validation ────────────────── diff --git a/Compression.Tests/Udf/UdfWipeEmptyTests.cs b/Compression.Tests/Udf/UdfWipeEmptyTests.cs index 1b38ac21c..a385c449c 100644 --- a/Compression.Tests/Udf/UdfWipeEmptyTests.cs +++ b/Compression.Tests/Udf/UdfWipeEmptyTests.cs @@ -40,8 +40,9 @@ public void WipeEmpty_ZerosClusterTip_AndFileRoundTrips() { var fileExtent = UdfExtentMap.Enumerate(ms) .First(e => e.Kind == DefragBlockKind.Used && e.FileName == "data.bin" && e.Classification != DefragBlockClass.Directory); - Assert.That(fileExtent.Length, Is.GreaterThan(content.Length), - "The data extent is sector-padded, so a tip exists beyond the logical size"); + Assert.That(fileExtent.Length, Is.EqualTo(content.Length), + "The allocation descriptor records the logical length, so the extent stops there " + + "and the rest of the sector is the tip"); // Dirty the cluster tip (immediately after the file's logical end). var tipOffset = fileExtent.Offset + content.Length; diff --git a/FileSystems/FileSystem.Udf/OstaCompressedUnicode.cs b/FileSystems/FileSystem.Udf/OstaCompressedUnicode.cs new file mode 100644 index 000000000..78b17ddf5 --- /dev/null +++ b/FileSystems/FileSystem.Udf/OstaCompressedUnicode.cs @@ -0,0 +1,108 @@ +using System.Text; + +namespace FileSystem.Udf; + +/// +/// OSTA Compressed Unicode (CS0), the character set every UDF identifier is +/// recorded in — file names in File Identifier Descriptors and the dstrings of +/// the volume, logical-volume and file-set descriptors. +/// +/// +/// +/// A CS0 byte string starts with a compression identifier and continues with +/// the characters themselves. Compression 8 means one byte per character, each +/// byte being the whole Unicode code point, so the representable range is +/// U+0000..U+00FF. Compression 16 means two big-endian bytes per character. +/// The encoder picks 8 when every character fits in a byte and 16 otherwise — +/// OSTA UDF §2.1.1. +/// +/// +/// Compression 8 is not UTF-8. Reading it as UTF-8 turns every accented Latin-1 +/// name a native tool wrote into replacement characters, and writing UTF-8 +/// under the identifier 8 produces names no other implementation can read. +/// +/// +internal static class OstaCompressedUnicode { + + /// Compression identifier for one byte per character. + public const byte SingleByte = 8; + + /// Compression identifier for two big-endian bytes per character. + public const byte DoubleByte = 16; + + /// + /// Decodes a CS0 byte string, compression identifier included. An empty span, + /// or one holding nothing but the identifier, decodes to the empty string. + /// + public static string Decode(ReadOnlySpan raw) { + if (raw.Length <= 1) + return string.Empty; + + var body = raw[1..]; + var text = raw[0] switch { + SingleByte => Encoding.Latin1.GetString(body), + DoubleByte => Encoding.BigEndianUnicode.GetString(body[..(body.Length & ~1)]), + // No other identifier is defined for UDF identifiers; treating the whole + // field as single-byte characters is the least destructive reading. + _ => Encoding.Latin1.GetString(raw), + }; + + return text.TrimEnd('\0'); + } + + /// + /// Encodes as a CS0 byte string, choosing the + /// narrower compression whenever every character fits in a single byte. + /// Returns an empty array for an empty string: a zero-length identifier is + /// recorded with no compression byte at all. + /// + public static byte[] Encode(string text) { + ArgumentNullException.ThrowIfNull(text); + if (text.Length == 0) + return []; + + var singleByte = true; + foreach (var c in text) + if (c > 0xFF) { + singleByte = false; + break; + } + + if (singleByte) { + var result = new byte[1 + text.Length]; + result[0] = SingleByte; + for (var i = 0; i < text.Length; ++i) + result[i + 1] = (byte)text[i]; + return result; + } + + var wide = Encoding.BigEndianUnicode.GetBytes(text); + var buffer = new byte[1 + wide.Length]; + buffer[0] = DoubleByte; + wide.CopyTo(buffer, 1); + return buffer; + } + + /// + /// Writes an ECMA-167 §1/7.2.12 dstring: a CS0 byte string left-aligned in a + /// fixed-width field whose final byte records how many bytes of it are used. + /// The text is truncated on a character boundary when it will not fit. + /// + public static void WriteDString(byte[] buffer, int offset, int fieldLength, string text) { + ArgumentNullException.ThrowIfNull(buffer); + Array.Clear(buffer, offset, fieldLength); + if (string.IsNullOrEmpty(text)) + return; + + var encoded = Encode(text); + // The length byte occupies the last position, so at most fieldLength-1 bytes + // of characters fit; drop whole characters until they do. + var characterBytes = encoded[0] == DoubleByte ? 2 : 1; + var usable = (fieldLength - 1 - 1) / characterBytes * characterBytes + 1; + if (encoded.Length > usable) + encoded = encoded[..usable]; + + encoded.CopyTo(buffer, offset); + buffer[offset + fieldLength - 1] = (byte)encoded.Length; + } +} diff --git a/FileSystems/FileSystem.Udf/UdfBlockMover.cs b/FileSystems/FileSystem.Udf/UdfBlockMover.cs index bf7056c98..f0ef5ec82 100644 --- a/FileSystems/FileSystem.Udf/UdfBlockMover.cs +++ b/FileSystems/FileSystem.Udf/UdfBlockMover.cs @@ -1,7 +1,6 @@ #pragma warning disable CS1591 using System.Buffers; using System.Buffers.Binary; -using System.Text; using Compression.Core.Checksums; using Compression.Core.Layout; using Compression.Registry; @@ -163,10 +162,7 @@ private void IndexDirectory(Stream image, SectorCache cache, int partitionStart, /// Decodes a file identifier, which names its own character set in its first byte. private static string ReadFileIdentifier(byte[] bytes, int at, int length) { if (at + length > bytes.Length) return string.Empty; - var name = length > 1 && bytes[at] == 8 ? Encoding.UTF8.GetString(bytes, at + 1, length - 1) - : length > 1 && bytes[at] == 16 ? Encoding.BigEndianUnicode.GetString(bytes, at + 1, length - 1) - : Encoding.ASCII.GetString(bytes, at, length); - return name.TrimEnd('\0'); + return OstaCompressedUnicode.Decode(bytes.AsSpan(at, length)); } /// @@ -264,14 +260,7 @@ private static int FindFileIcbStream(Stream image, SectorCache cache, int partSt if (!isParent && !isDeleted && idLen > 0) { var nameStart = pos + 38 + lIu; - string name; - if (idLen > 1 && dirBytes[nameStart] == 8) - name = Encoding.UTF8.GetString(dirBytes, nameStart + 1, idLen - 1); - else if (idLen > 1 && dirBytes[nameStart] == 16) - name = Encoding.BigEndianUnicode.GetString(dirBytes, nameStart + 1, idLen - 1); - else - name = Encoding.ASCII.GetString(dirBytes, nameStart, idLen); - name = name.TrimEnd('\0'); + var name = OstaCompressedUnicode.Decode(dirBytes.AsSpan(nameStart, idLen)); if (name.Equals(targetName, StringComparison.OrdinalIgnoreCase) || targetName.Equals("*", StringComparison.Ordinal)) diff --git a/FileSystems/FileSystem.Udf/UdfDescriptors.cs b/FileSystems/FileSystem.Udf/UdfDescriptors.cs new file mode 100644 index 000000000..403abe96c --- /dev/null +++ b/FileSystems/FileSystem.Udf/UdfDescriptors.cs @@ -0,0 +1,157 @@ +using System.Buffers.Binary; +using System.Text; +using Compression.Core.Checksums; + +namespace FileSystem.Udf; + +/// +/// The small ECMA-167 building blocks every UDF descriptor is assembled from: +/// the descriptor tag, the entity identifier, the character specification and +/// the timestamp. +/// +internal static class UdfDescriptors { + + /// + /// Descriptor version stamped into every tag. ECMA-167 defines 2; OSTA UDF + /// §2.2.1.2 requires 3 from revision 2.00 onwards, and the volumes this + /// writer produces declare 2.01. + /// + public const ushort DescriptorVersion = 3; + + /// Revision this writer records, little-endian BCD as UDF stores it. + public const ushort UdfRevision = 0x0201; + + /// + /// Suffix of the *OSTA UDF Compliant domain entity identifier + /// (OSTA UDF §2.1.5.3): UDF revision, domain flags, five reserved bytes. + /// + public static readonly byte[] DomainSuffix = + [UdfRevision & 0xFF, UdfRevision >> 8, 0, 0, 0, 0, 0, 0]; + + /// + /// Suffix of a UDF-defined entity identifier (OSTA UDF §2.1.5.3): UDF + /// revision, operating-system class, operating-system identifier, reserved. + /// Class 0 is "undefined", which is what a portable writer can honestly claim. + /// + public static readonly byte[] UdfEntitySuffix = + [UdfRevision & 0xFF, UdfRevision >> 8, 0, 0, 0, 0, 0, 0]; + + /// + /// Suffix of an implementation entity identifier (OSTA UDF §2.1.5.3): + /// operating-system class, operating-system identifier, six free bytes. + /// + public static readonly byte[] ImplementationSuffix = [0, 0, 0, 0, 0, 0, 0, 0]; + + /// + /// Names the UDF implementation rather than the product, so an image says + /// which filesystem code shaped it whichever tool drove that code. + /// + public const string ImplementationId = "*Linux UDFFS"; + + /// + /// Recording timestamp stamped into every descriptor. A fixed instant keeps + /// two runs over the same input byte-identical, which the streaming writer's + /// contract with the buffered one depends on. + /// + public static readonly DateTime RecordingTime = new(2000, 1, 1, 0, 0, 0, DateTimeKind.Utc); + + /// + /// Writes the fixed part of an ECMA-167 §7.2 descriptor tag. The checksum, + /// CRC and CRC length are filled in by once the + /// body exists. + /// + public static void WriteTag(byte[] buffer, int offset, ushort tagIdentifier, uint tagLocation) { + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(offset), tagIdentifier); + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(offset + 2), DescriptorVersion); + buffer[offset + 4] = 0; // TagChecksum + buffer[offset + 5] = 0; // Reserved + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(offset + 6), 1); // TagSerialNumber + BinaryPrimitives.WriteUInt32LittleEndian(buffer.AsSpan(offset + 12), tagLocation); + } + + /// + /// Completes an ECMA-167 §7.2 descriptor tag: the CRC-16/CCITT (init 0, + /// polynomial 0x1021, non-reflected) over + /// bytes following the tag, that length, and finally the byte-sum-mod-256 + /// checksum over the tag's other fifteen bytes. + /// + public static void FinalizeTag(byte[] buffer, int tagOffset, int bodyLength) { + var bodyStart = tagOffset + 16; + if (bodyStart + bodyLength > buffer.Length) + bodyLength = buffer.Length - bodyStart; + if (bodyLength < 0) + bodyLength = 0; + + var crc = Crc16Ccitt.Compute(buffer.AsSpan(bodyStart, bodyLength)); + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(tagOffset + 8), crc); + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(tagOffset + 10), (ushort)bodyLength); + + buffer[tagOffset + 4] = 0; + byte sum = 0; + for (var i = 0; i < 16; ++i) { + if (i == 4) continue; + sum = (byte)(sum + buffer[tagOffset + i]); + } + + buffer[tagOffset + 4] = sum; + } + + /// + /// Writes an ECMA-167 §1/7.4 entity identifier: one flags byte, a 23-byte + /// identifier and an 8-byte suffix whose shape depends on which kind of + /// identifier this is. + /// + public static void WriteEntityId(byte[] buffer, int offset, string identifier, ReadOnlySpan suffix) { + Array.Clear(buffer, offset, 32); + var bytes = Encoding.ASCII.GetBytes(identifier); + Array.Copy(bytes, 0, buffer, offset + 1, Math.Min(bytes.Length, 23)); + if (!suffix.IsEmpty) + suffix[..Math.Min(suffix.Length, 8)].CopyTo(buffer.AsSpan(offset + 24, 8)); + } + + /// + /// Writes an ECMA-167 §1/7.2.1 character specification naming OSTA + /// Compressed Unicode, the only character set UDF volumes use. + /// + public static void WriteCharacterSet(byte[] buffer, int offset) { + Array.Clear(buffer, offset, 64); + buffer[offset] = 0; + Encoding.ASCII.GetBytes("OSTA Compressed Unicode").CopyTo(buffer, offset + 1); + } + + /// Writes an ECMA-167 §1/7.3 timestamp, recorded as UTC. + public static void WriteTimestamp(byte[] buffer, int offset, DateTime utc) { + // Type 1 is "local time", and with a zero offset that is UTC. Type 0 would + // mean the interpretation is not specified at all. + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(offset), 1 << 12); + BinaryPrimitives.WriteInt16LittleEndian(buffer.AsSpan(offset + 2), (short)utc.Year); + buffer[offset + 4] = (byte)utc.Month; + buffer[offset + 5] = (byte)utc.Day; + buffer[offset + 6] = (byte)utc.Hour; + buffer[offset + 7] = (byte)utc.Minute; + buffer[offset + 8] = (byte)utc.Second; + buffer[offset + 9] = 0; + buffer[offset + 10] = 0; + buffer[offset + 11] = 0; + } + + /// Writes an ECMA-167 §1/7.1 extent descriptor: byte length, then block. + public static void WriteExtent(byte[] buffer, int offset, uint length, uint location) { + BinaryPrimitives.WriteUInt32LittleEndian(buffer.AsSpan(offset), length); + BinaryPrimitives.WriteUInt32LittleEndian(buffer.AsSpan(offset + 4), location); + } + + /// + /// Writes an ECMA-167 §4/14.14.2 long allocation descriptor addressing + /// bytes at of the first + /// partition, with in the UDF-defined part of + /// its implementation-use area (OSTA UDF §2.3.4.3). + /// + public static void WriteLongAd(byte[] buffer, int offset, uint length, uint block, uint uniqueId = 0) { + Array.Clear(buffer, offset, 16); + BinaryPrimitives.WriteUInt32LittleEndian(buffer.AsSpan(offset), length); + BinaryPrimitives.WriteUInt32LittleEndian(buffer.AsSpan(offset + 4), block); + BinaryPrimitives.WriteUInt16LittleEndian(buffer.AsSpan(offset + 8), 0); // partition reference + BinaryPrimitives.WriteUInt32LittleEndian(buffer.AsSpan(offset + 12), uniqueId); + } +} diff --git a/FileSystems/FileSystem.Udf/UdfExtentMap.cs b/FileSystems/FileSystem.Udf/UdfExtentMap.cs index 127c8fb53..2d0399885 100644 --- a/FileSystems/FileSystem.Udf/UdfExtentMap.cs +++ b/FileSystems/FileSystem.Udf/UdfExtentMap.cs @@ -70,10 +70,26 @@ public static IEnumerable Enumerate(Stream image) { var mainVdsLoc = BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(20)); var mainVdsLen = BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(16)); + var reserveVdsLoc = BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(28)); + var reserveVdsLen = BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(24)); + + // A second anchor sits in the volume's last block (ECMA-167 §3/8.4). Left + // uncovered it reads as free space and the wiper zeroes it, which costs the + // volume the redundancy the standard asks for. + var lastBlock = image.Length / SectorSize - 1; + if (lastBlock > AvdpSector) { + var tailOff = lastBlock * SectorSize; + cache.Read(tailOff, sectorBuf); + if (BinaryPrimitives.ReadUInt16LittleEndian(sectorBuf) == 2) + yield return new DefragBlockInfo(tailOff, SectorSize, DefragBlockKind.MetadataReserved, + FileName: "UDF AVDP"); + } // Walk VDS to find PD (5) and LVD (6). int partStart = 0; int fsdLbn = 0; + long lvidLoc = 0; + long lvidLen = 0; var vdsSectors = (int)(mainVdsLen / SectorSize); for (var i = 0; i < vdsSectors && i < 64; i++) { var off = (long)(mainVdsLoc + i) * SectorSize; @@ -87,9 +103,33 @@ public static IEnumerable Enumerate(Stream image) { partStart = (int)BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(188)); } else if (tagId == 6) { fsdLbn = (int)BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(252)); + lvidLen = BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(432)); + lvidLoc = BinaryPrimitives.ReadUInt32LittleEndian(sectorBuf.AsSpan(436)); } else if (tagId == 8) break; // terminator } + // The reserve sequence the anchor names, and the logical volume integrity + // sequence the logical volume descriptor names, are both real structures + // outside the partition. + var reserveSectors = (int)(reserveVdsLen / SectorSize); + for (var i = 0; i < reserveSectors && i < 64; i++) { + var off = (long)(reserveVdsLoc + i) * SectorSize; + if (off + SectorSize > image.Length) break; + cache.Read(off, sectorBuf); + var tagId = BinaryPrimitives.ReadUInt16LittleEndian(sectorBuf); + yield return new DefragBlockInfo(off, SectorSize, DefragBlockKind.MetadataReserved, + FileName: $"UDF reserve VDS tag={tagId}"); + if (tagId == 8) break; + } + + var lvidSectors = (int)(lvidLen / SectorSize); + for (var i = 0; i < lvidSectors && i < 64; i++) { + var off = (lvidLoc + i) * SectorSize; + if (off + SectorSize > image.Length) break; + yield return new DefragBlockInfo(off, SectorSize, DefragBlockKind.MetadataReserved, + FileName: "UDF LVID"); + } + // Read FSD. var fsdOffset = (long)(partStart + fsdLbn) * SectorSize; if (fsdOffset + SectorSize > image.Length) yield break; @@ -214,14 +254,7 @@ private static IEnumerable WalkDirectory(Stream image, SectorCa if (!isParent && !isDeleted && fidIdLen > 0) { var nameStart = pos + 38 + lIu; - string name; - if (fidIdLen > 1 && dirBytes[nameStart] == 8) - name = Encoding.UTF8.GetString(dirBytes, nameStart + 1, fidIdLen - 1); - else if (fidIdLen > 1 && dirBytes[nameStart] == 16) - name = Encoding.BigEndianUnicode.GetString(dirBytes, nameStart + 1, fidIdLen - 1); - else - name = Encoding.ASCII.GetString(dirBytes, nameStart, fidIdLen); - name = name.TrimEnd('\0'); + var name = OstaCompressedUnicode.Decode(dirBytes.AsSpan(nameStart, fidIdLen)); var fullPath = string.IsNullOrEmpty(basePath) ? name : $"{basePath}/{name}"; diff --git a/FileSystems/FileSystem.Udf/UdfFormatDescriptor.cs b/FileSystems/FileSystem.Udf/UdfFormatDescriptor.cs index c60ee2d7d..34df3e770 100644 --- a/FileSystems/FileSystem.Udf/UdfFormatDescriptor.cs +++ b/FileSystems/FileSystem.Udf/UdfFormatDescriptor.cs @@ -111,9 +111,31 @@ public void UpdateAllocationAfterMove(Stream image, string fileName, long oldOff /// public string AcceptedInputsDescription => "UDF 2.01 disc image; any files, flat directory."; /// - /// Performs the can accept operation. + /// Accepts any file the writer can address. Short allocation descriptors cap + /// an extent below 2^30 bytes (OSTA UDF §2.3.10.1) and only so many of them + /// fit in one File Entry, so a file past that ceiling is declined here rather + /// than written as a volume nothing can read. /// - public bool CanAccept(ArchiveInputInfo input, out string? reason) { reason = null; return true; } + public bool CanAccept(ArchiveInputInfo input, out string? reason) { + reason = null; + if (input is null || input.IsDirectory) + return true; + + long size; + if (input.InMemoryContent is { } content) + size = content.LongLength; + else if (File.Exists(input.FullPath)) + size = new FileInfo(input.FullPath).Length; + else + return true; + + if (size <= UdfWriter.MaxFileBytes) + return true; + + reason = $"UDF addresses at most {UdfWriter.MaxFileBytes:N0} bytes per file without " + + "allocation descriptor continuation, which this writer does not record."; + return false; + } /// /// Gets the id. diff --git a/FileSystems/FileSystem.Udf/UdfModifier.cs b/FileSystems/FileSystem.Udf/UdfModifier.cs index 546ee6ba1..321d6354b 100644 --- a/FileSystems/FileSystem.Udf/UdfModifier.cs +++ b/FileSystems/FileSystem.Udf/UdfModifier.cs @@ -1,6 +1,5 @@ #pragma warning disable CS1591 using System.Buffers.Binary; -using System.Text; using Compression.Core.Checksums; namespace FileSystem.Udf; @@ -18,7 +17,11 @@ namespace FileSystem.Udf; /// The root directory's FID extent — appended into trailing sector padding, /// or extended via a second short_ad on the root File Entry. /// The root File Entry sector — info length and L_AD updated, tag re-CRC'd. -/// The Partition Descriptor sector — partition length grown. +/// Every Partition Descriptor — the main sequence's and the reserve +/// sequence's copy — grown to the partition's new length. +/// The volume's second anchor, re-recorded in its new last block. +/// The Logical Volume Integrity Descriptor — size, free space and the +/// file count it advertises. /// /// /// Remove uses the FID Characteristics "deleted" flag (bit 2 = 0x04) per @@ -62,8 +65,10 @@ public static void AddFile(Stream image, string name, byte[] data) { var ctx = ReadContext(image); - // Compute new file's allocation: FE sector + data sectors, all at partition tail. - var dataSectors = data.Length == 0 ? 1 : (data.Length + SectorSize - 1) / SectorSize; + // Compute new file's allocation: FE sector + data sectors, all at partition + // tail. A zero-length file is given no data block: an allocation descriptor + // naming a block it does not own is a chain longer than the size says. + var dataSectors = (data.Length + SectorSize - 1) / SectorSize; var feLbn = ctx.HighWaterLbn; var dataLbn = feLbn + 1; var newHighWater = dataLbn + dataSectors; @@ -72,12 +77,14 @@ public static void AddFile(Stream image, string name, byte[] data) { var fid = BuildFid(flags: 0x00, icbLbn: feLbn, name); // Write file data (zero-padded to sector boundary). - var dataAbs = (ctx.PartitionStart + dataLbn) * (long)SectorSize; - EnsureLength(image, dataAbs + dataSectors * (long)SectorSize); - image.Position = dataAbs; - image.Write(data); - var dataPad = dataSectors * SectorSize - data.Length; - if (dataPad > 0) image.Write(new byte[dataPad]); + if (dataSectors > 0) { + var dataAbs = (ctx.PartitionStart + dataLbn) * (long)SectorSize; + EnsureLength(image, dataAbs + dataSectors * (long)SectorSize); + image.Position = dataAbs; + image.Write(data); + var dataPad = dataSectors * SectorSize - data.Length; + if (dataPad > 0) image.Write(new byte[dataPad]); + } // Write file File Entry sector. var feSector = BuildFileFe(feLbn, data.Length, dataLbn); @@ -88,6 +95,7 @@ public static void AddFile(Stream image, string name, byte[] data) { // Grow the partition descriptor and image to cover the new allocations. UpdatePartitionLength(image, ctx, newHighWater); + AdjustFileCount(image, ctx, +1); } /// @@ -134,6 +142,7 @@ public static bool RemoveFile(Stream image, string name, bool wipeData = true) { WipeFileEntry(image, ctx, hit.IcbLbn); } + AdjustFileCount(image, ctx, -1); return true; } @@ -142,10 +151,15 @@ public static bool RemoveFile(Stream image, string name, bool wipeData = true) { private sealed record Context( int PartitionStart, int PartitionLengthSectors, - int PdSectorLba, + IReadOnlyList PdSectorLbas, int RootFeLbn, int HighWaterLbn, - long ImageSize); + int LvidLba, + long ImageSize) { + + /// The main sequence's partition descriptor, which always exists. + public int PdSectorLba => this.PdSectorLbas[0]; + } private static Context ReadContext(Stream image) { if (image.Length < 257L * SectorSize) @@ -157,28 +171,46 @@ private static Context ReadContext(Stream image) { throw new InvalidDataException("UDF: invalid AVDP tag."); var mainVdsLoc = (int)BinaryPrimitives.ReadUInt32LittleEndian(avdp.AsSpan(20)); var mainVdsLen = (int)BinaryPrimitives.ReadUInt32LittleEndian(avdp.AsSpan(16)); + var reserveVdsLoc = (int)BinaryPrimitives.ReadUInt32LittleEndian(avdp.AsSpan(28)); + var reserveVdsLen = (int)BinaryPrimitives.ReadUInt32LittleEndian(avdp.AsSpan(24)); // Walk VDS for PD and LVD - int partStart = 0, partLen = 0, pdSectorLba = 0; - int fsdLbn = 0; - var vdsSectors = mainVdsLen / SectorSize; - for (var i = 0; i < vdsSectors && i < 64; i++) { - var sectorLba = mainVdsLoc + i; - if ((long)(sectorLba + 1) * SectorSize > image.Length) break; - var sec = ReadSector(image, sectorLba); - var tag = BinaryPrimitives.ReadUInt16LittleEndian(sec); - if (tag == PdTagId) { - partStart = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(188)); - partLen = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(192)); - pdSectorLba = sectorLba; - } else if (tag == LvdTagId) { - fsdLbn = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(252)); - } else if (tag == 8) { - break; + int partStart = 0, partLen = 0; + int fsdLbn = 0, lvidLba = 0; + var pdSectors = new List(); + + void Scan(int loc, int len, bool primary) { + var sectors = len / SectorSize; + for (var i = 0; i < sectors && i < 64; i++) { + var sectorLba = loc + i; + if ((long)(sectorLba + 1) * SectorSize > image.Length) break; + var sec = ReadSector(image, sectorLba); + var tag = BinaryPrimitives.ReadUInt16LittleEndian(sec); + if (tag == PdTagId) { + if (primary) { + partStart = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(188)); + partLen = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(192)); + } + pdSectors.Add(sectorLba); + } else if (tag == LvdTagId) { + if (primary) { + fsdLbn = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(252)); + lvidLba = (int)BinaryPrimitives.ReadUInt32LittleEndian(sec.AsSpan(436)); + } + } else if (tag == 8) { + break; + } } } - if (pdSectorLba == 0) + Scan(mainVdsLoc, mainVdsLen, primary: true); + // The reserve sequence carries the same partition descriptor. Growing the + // partition without growing its copy leaves a reader that falls back to the + // reserve seeing a volume that stops short of its own data. + if (reserveVdsLen > 0) + Scan(reserveVdsLoc, reserveVdsLen, primary: false); + + if (pdSectors.Count == 0) throw new InvalidDataException("UDF: partition descriptor not found."); // FSD → root ICB LBN @@ -192,9 +224,10 @@ private static Context ReadContext(Stream image) { return new Context( PartitionStart: partStart, PartitionLengthSectors: partLen, - PdSectorLba: pdSectorLba, + PdSectorLbas: pdSectors, RootFeLbn: rootLbn, HighWaterLbn: partLen, + LvidLba: lvidLba, ImageSize: image.Length); } @@ -339,15 +372,8 @@ private sealed record FidHit(int Offset, int Length, int IcbLbn); return null; } - private static string DecodeCs0(byte[] buf, int offset, int len) { - if (len <= 0) return ""; - var compressionId = buf[offset]; - if (compressionId == 8 && len > 1) - return Encoding.UTF8.GetString(buf, offset + 1, len - 1); - if (compressionId == 16 && len > 1) - return Encoding.BigEndianUnicode.GetString(buf, offset + 1, len - 1); - return Encoding.ASCII.GetString(buf, offset, len); - } + private static string DecodeCs0(byte[] buf, int offset, int len) + => len <= 0 ? "" : OstaCompressedUnicode.Decode(buf.AsSpan(offset, len)); // ── Root FID growth (Add path) ──────────────────────────────────────────── @@ -372,6 +398,10 @@ private static void AppendFidToRoot(Stream image, Context ctx, byte[] fid, ref i capacity += sectors * SectorSize; } + // The record starts wherever the directory's bytes currently end, and its + // tag has to name the logical block that position falls in. + StampFidLocation(fid, BlockAtDirectoryOffset(ext, oldLength)); + if (newLength <= capacity) { // Fits inside the trailing pad of the last extent's last sector. WriteFidIntoExistingExtent(image, ctx, ext, fid); @@ -382,6 +412,8 @@ private static void AppendFidToRoot(Stream image, Context ctx, byte[] fid, ref i // First, write any spillover into the existing tail (if some bytes still fit). var tailRoom = (int)(capacity - oldLength); var spilled = Math.Min(tailRoom, fid.Length); + if (spilled == 0) + StampFidLocation(fid, highWaterLbn); if (spilled > 0) { // Fill the tail of the last extent with the first `spilled` bytes. WriteFidPartialToExistingTail(image, ctx, ext, fid, 0, spilled); @@ -403,6 +435,22 @@ private static void AppendFidToRoot(Stream image, Context ctx, byte[] fid, ref i } } + /// + /// The logical block holding byte of a directory's + /// data, walking its extents in order. Returns -1 when the offset lies past + /// everything the extents cover. + /// + private static int BlockAtDirectoryOffset(FidExtent ext, long offset) { + foreach (var (lbn, len) in ext.Extents) { + var capacity = (long)((len + SectorSize - 1) / SectorSize) * SectorSize; + if (offset < capacity) + return lbn + (int)(offset / SectorSize); + offset -= capacity; + } + + return -1; + } + private static void WriteFidIntoExistingExtent(Stream image, Context ctx, FidExtent ext, byte[] fid) { // Last extent absorbs the new FID — bytes past its logical length but // within its sector capacity. @@ -518,24 +566,40 @@ private static byte[] BuildFileFe(int lbn, int fileSize, int dataLbn) { BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(44), (1u << 12) | (1u << 7) | (1u << 2)); BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(48), 1); BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(56), (ulong)fileSize); + var sectors = (fileSize + SectorSize - 1) / SectorSize; + BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(64), (ulong)sectors); // LogicalBlocksRecorded + UdfDescriptors.WriteTimestamp(buf, 72, UdfDescriptors.RecordingTime); // AccessTime + UdfDescriptors.WriteTimestamp(buf, 84, UdfDescriptors.RecordingTime); // ModificationTime + UdfDescriptors.WriteTimestamp(buf, 96, UdfDescriptors.RecordingTime); // AttributeTime + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(108), 1); // Checkpoint + UdfDescriptors.WriteEntityId(buf, 128, UdfDescriptors.ImplementationId, + UdfDescriptors.ImplementationSuffix); BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(168), 0); // L_EA - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(172), 8); // L_AD = 8 (one short_ad) - var allocLen = Math.Max(fileSize, SectorSize); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(176), (uint)allocLen); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(180), (uint)dataLbn); - FinalizeTag(buf, 0, (176 - 16) + 0 + 8); // body covers FE header + L_EA + L_AD + // A zero-length file records no extent at all. + var lAd = fileSize > 0 ? 8 : 0; + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(172), (uint)lAd); + if (lAd > 0) { + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(176), (uint)fileSize); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(180), (uint)dataLbn); + } + FinalizeTag(buf, 0, (176 - 16) + 0 + lAd); // body covers FE header + L_EA + L_AD return buf; } // ── FID construction ────────────────────────────────────────────────────── + /// + /// Builds a File Identifier Descriptor. Its tag location is stamped by + /// once the caller knows which block the + /// record will start in. + /// private static byte[] BuildFid(byte flags, int icbLbn, string name) { - var nameBytes = name.Length == 0 ? [] : EncodeCs0(name); + var nameBytes = OstaCompressedUnicode.Encode(name); var fidLen = 38 + nameBytes.Length; var padded = (fidLen + 3) & ~3; var buf = new byte[padded]; - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(0), FidTagId); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(2), 2); // descriptor version + WriteTag(buf, 0, FidTagId, 0); + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(16), 1); // FileVersionNumber buf[18] = flags; buf[19] = (byte)nameBytes.Length; BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(20), (uint)SectorSize); // ICB ext length @@ -545,12 +609,13 @@ private static byte[] BuildFid(byte flags, int icbLbn, string name) { return buf; } - private static byte[] EncodeCs0(string name) { - var utf8 = Encoding.UTF8.GetBytes(name); - var result = new byte[1 + utf8.Length]; - result[0] = 8; // CS0 = UTF-8 - utf8.CopyTo(result, 1); - return result; + /// + /// Records the logical block a File Identifier Descriptor starts in and + /// re-seals its tag. A record spanning two blocks names the first of them. + /// + private static void StampFidLocation(byte[] fid, int lbn) { + BinaryPrimitives.WriteUInt32LittleEndian(fid.AsSpan(12), (uint)lbn); + FinalizeTag(fid, 0, fid.Length - 16); } // ── Wipe (Remove path) ──────────────────────────────────────────────────── @@ -623,38 +688,93 @@ private static void WipeRange(Stream image, long start, long count) { private static void UpdatePartitionLength(Stream image, Context ctx, int newPartLenSectors) { if (newPartLenSectors <= ctx.PartitionLengthSectors) return; - var pd = ReadSector(image, ctx.PdSectorLba); - BinaryPrimitives.WriteUInt32LittleEndian(pd.AsSpan(192), (uint)newPartLenSectors); - FinalizeTag(pd, 0, 496); // PdBodySize - WriteSector(image, ctx.PdSectorLba, pd); + foreach (var lba in ctx.PdSectorLbas) { + var pd = ReadSector(image, lba); + if (BinaryPrimitives.ReadUInt16LittleEndian(pd) != PdTagId) continue; + BinaryPrimitives.WriteUInt32LittleEndian(pd.AsSpan(192), (uint)newPartLenSectors); + FinalizeTag(pd, 0, 496); // PdBodySize + WriteSector(image, lba, pd); + } - EnsureLength(image, (long)(ctx.PartitionStart + newPartLenSectors) * SectorSize); + // The volume keeps one block past the partition for its second anchor + // (ECMA-167 §3/8.4), which has to follow the partition as it grows. + var lastBlock = ctx.PartitionStart + newPartLenSectors; + EnsureLength(image, (long)(lastBlock + 1) * SectorSize); + MoveTailAnchor(image, lastBlock); + UpdateIntegrity(image, ctx, newPartLenSectors); } - // ── Tag helpers (mirror UdfWriter) ──────────────────────────────────────── + /// + /// Re-records the volume's second anchor in its new last block, copied from + /// the one at block 256 so both name the same descriptor sequences. + /// + private static void MoveTailAnchor(Stream image, int lastBlock) { + var anchor = ReadSector(image, AvdpLba); + if (BinaryPrimitives.ReadUInt16LittleEndian(anchor) != 2) return; + BinaryPrimitives.WriteUInt32LittleEndian(anchor.AsSpan(12), (uint)lastBlock); + FinalizeTag(anchor, 0, 496); + WriteSector(image, lastBlock, anchor); + } - private static void WriteTag(byte[] buf, int off, ushort tagId, uint tagLocation) { - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(off), tagId); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(off + 2), 2); // descriptor version - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(off + 12), tagLocation); + /// + /// Refreshes the logical volume integrity descriptor after the partition + /// grew: a stale size table describes a volume smaller than its own data, + /// and every tool reads the difference as missing blocks. + /// + private static void UpdateIntegrity(Stream image, Context ctx, int newPartLenSectors) { + if (ctx.LvidLba <= 0) return; + if ((long)(ctx.LvidLba + 1) * SectorSize > image.Length) return; + + var lvid = ReadSector(image, ctx.LvidLba); + if (BinaryPrimitives.ReadUInt16LittleEndian(lvid) != 9) return; + + var partitions = BinaryPrimitives.ReadUInt32LittleEndian(lvid.AsSpan(72)); + if (partitions < 1) return; + var implementationUse = (int)BinaryPrimitives.ReadUInt32LittleEndian(lvid.AsSpan(76)); + var tables = 80 + (int)partitions * 4; + + BinaryPrimitives.WriteUInt32LittleEndian(lvid.AsSpan(80), 0); // free space + BinaryPrimitives.WriteUInt32LittleEndian(lvid.AsSpan(tables), (uint)newPartLenSectors); // size + FinalizeTag(lvid, 0, (tables + (int)partitions * 4 - 16) + implementationUse); + WriteSector(image, ctx.LvidLba, lvid); } - private static void FinalizeTag(byte[] buf, int tagOffset, int bodyLength) { - var bodyStart = tagOffset + 16; - if (bodyStart + bodyLength > buf.Length) bodyLength = buf.Length - bodyStart; - if (bodyLength < 0) bodyLength = 0; - var crc = Crc16Ccitt.Compute(buf.AsSpan(bodyStart, bodyLength)); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(tagOffset + 8), crc); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(tagOffset + 10), (ushort)bodyLength); - buf[tagOffset + 4] = 0; - byte sum = 0; - for (var i = 0; i < 16; i++) { - if (i == 4) continue; - sum = (byte)(sum + buf[tagOffset + i]); + /// Adjusts the file count the integrity descriptor advertises. + private static void AdjustFileCount(Stream image, Context ctx, int delta) { + if (ctx.LvidLba <= 0 || delta == 0) return; + if ((long)(ctx.LvidLba + 1) * SectorSize > image.Length) return; + + var lvid = ReadSector(image, ctx.LvidLba); + if (BinaryPrimitives.ReadUInt16LittleEndian(lvid) != 9) return; + + var partitions = (int)BinaryPrimitives.ReadUInt32LittleEndian(lvid.AsSpan(72)); + if (partitions < 1) return; + var implementationUse = (int)BinaryPrimitives.ReadUInt32LittleEndian(lvid.AsSpan(76)); + // OSTA UDF §2.2.6.4 puts the counts after the entity identifier that opens + // the implementation use area. + var counts = 80 + partitions * 8 + 32; + if (implementationUse < 40 || counts + 8 > SectorSize) return; + + var files = (int)BinaryPrimitives.ReadUInt32LittleEndian(lvid.AsSpan(counts)); + BinaryPrimitives.WriteUInt32LittleEndian(lvid.AsSpan(counts), (uint)Math.Max(0, files + delta)); + // A new object also consumes the next unique identifier. + if (delta > 0) { + var next = BinaryPrimitives.ReadUInt64LittleEndian(lvid.AsSpan(40)); + BinaryPrimitives.WriteUInt64LittleEndian(lvid.AsSpan(40), next + (ulong)delta); } - buf[tagOffset + 4] = sum; + + FinalizeTag(lvid, 0, (80 + partitions * 8 - 16) + implementationUse); + WriteSector(image, ctx.LvidLba, lvid); } + // ── Tag helpers ─────────────────────────────────────────────────────────── + + private static void WriteTag(byte[] buf, int off, ushort tagId, uint tagLocation) + => UdfDescriptors.WriteTag(buf, off, tagId, tagLocation); + + private static void FinalizeTag(byte[] buf, int tagOffset, int bodyLength) + => UdfDescriptors.FinalizeTag(buf, tagOffset, bodyLength); + // ── Stream helpers ──────────────────────────────────────────────────────── private static byte[] ReadSector(Stream image, int lba) { diff --git a/FileSystems/FileSystem.Udf/UdfReader.cs b/FileSystems/FileSystem.Udf/UdfReader.cs index 3b448fd58..b1319c1f4 100644 --- a/FileSystems/FileSystem.Udf/UdfReader.cs +++ b/FileSystems/FileSystem.Udf/UdfReader.cs @@ -58,12 +58,102 @@ private ulong U64(long off) ? BinaryPrimitives.ReadUInt64LittleEndian(_img.Read(off, 8)) : 0ul; + /// + /// Logical block sizes ECMA-167 volumes are recorded with, most common first. + /// The Anchor Volume Descriptor Pointer is the only descriptor at a fixed + /// address (logical block 256), and that address is counted in logical blocks + /// — so until the anchor is found the block size is unknown and has to be + /// probed, exactly as udftools does. + /// + private static readonly int[] CandidateBlockSizes = [2048, 512, 1024, 4096, 8192, 16384, 32768]; + + /// + /// True when a descriptor tag sits at with the + /// given identifier and records as its + /// own address. The ECMA-167 §7.2 TagChecksum is verified too, so a run of + /// file data that happens to start with the anchor's tag identifier cannot be + /// mistaken for a descriptor. + /// + private bool IsTagAt(long offset, ushort identifier, uint expectedLocation) { + if (offset < 0 || offset + 16 > _len) + return false; + + var tag = _img.Read(offset, 16); + if (BinaryPrimitives.ReadUInt16LittleEndian(tag) != identifier) + return false; + if (BinaryPrimitives.ReadUInt32LittleEndian(tag.AsSpan(12)) != expectedLocation) + return false; + + byte sum = 0; + for (var i = 0; i < 16; ++i) + if (i != 4) + sum = (byte)(sum + tag[i]); + + return sum == tag[4]; + } + + /// + /// Locates the Anchor Volume Descriptor Pointer and, with it, the volume's + /// logical block size. ECMA-167 §3/8.4 puts an anchor at logical block 256 and + /// at the last block of the volume (and optionally 256 blocks before it); each + /// is tried for every plausible block size. + /// + private long FindAnchor() { + foreach (var blockSize in CandidateBlockSizes) { + var totalBlocks = _len / blockSize; + if (totalBlocks <= 256) + continue; + + foreach (var block in new[] { 256L, totalBlocks - 1, totalBlocks - 257 }) { + if (block < 256) + continue; + var offset = block * blockSize; + if (!this.IsTagAt(offset, 2, (uint)block)) + continue; + // The sequence the candidate names has to describe a volume of the same + // block size, so a run of file data that survives the tag checks cannot + // carry the read off to the wrong addresses in silence. + if (!this.SequenceDeclaresBlockSize(U32(offset + 20), U32(offset + 16), blockSize)) + continue; + + this._blockSize = blockSize; + return offset; + } + } + + throw new InvalidDataException("UDF: no Anchor Volume Descriptor Pointer found."); + } + + /// + /// True when the volume descriptor sequence at + /// holds a Logical Volume Descriptor declaring . + /// + private bool SequenceDeclaresBlockSize(uint location, uint byteLength, int blockSize) { + var descriptors = Math.Min(byteLength / (uint)blockSize, 64); + for (var i = 0u; i < descriptors; ++i) { + var offset = ((long)location + i) * blockSize; + if (offset < 0 || offset + 512 > _len) + return false; + + var tag = U16(offset); + if (tag == 6) + return U32(offset + 212) == (uint)blockSize; + if (tag == 8) + return false; + } + + return false; + } + private void Parse() { - if (_len < 257L * SectorSize) + if (_len < 257L * 512) throw new InvalidDataException("UDF: image too small."); + // ECMA-167 §2/9.1: the Volume Recognition Sequence starts at byte 32768 and + // occupies consecutive logical sectors, whose size is 2048 or the block size + // when that is larger. Scanning at the 2048 stride covers both. var foundNsr = false; - for (var sector = 16L; sector < 20 && sector * SectorSize + 6 < _len; ++sector) { + for (var sector = 16L; sector < 24 && sector * SectorSize + 6 < _len; ++sector) { var off = sector * SectorSize; var id = Encoding.ASCII.GetString(_img.Read(off + 1, 5)); if (id is "NSR02" or "NSR03") { @@ -74,18 +164,16 @@ private void Parse() { if (!foundNsr) throw new InvalidDataException("UDF: no NSR02/NSR03 descriptor found."); - var avdpOff = 256L * SectorSize; - if (U16(avdpOff) != 2) - throw new InvalidDataException("UDF: invalid AVDP tag."); + var avdpOff = this.FindAnchor(); var mainVdsLoc = U32(avdpOff + 20); var mainVdsLen = U32(avdpOff + 16); long partStart = 0; long fsdLbn = 0; - var vdsSectors = (int)(mainVdsLen / SectorSize); + var vdsSectors = (int)(mainVdsLen / (uint)_blockSize); for (var i = 0; i < vdsSectors && i < 64; ++i) { - var off = ((long)mainVdsLoc + i) * SectorSize; + var off = ((long)mainVdsLoc + i) * _blockSize; if (off + 512 > _len) break; @@ -93,9 +181,9 @@ private void Parse() { if (tagId == 5) { partStart = U32(off + 188); } else if (tagId == 6) { - _blockSize = checked((int)U32(off + 212)); - if (_blockSize == 0) - _blockSize = SectorSize; + var declared = checked((int)U32(off + 212)); + if (declared > 0) + _blockSize = declared; fsdLbn = U32(off + 252); } else if (tagId == 8) { break; @@ -115,8 +203,11 @@ private void Parse() { ReadDirectory(rootIcbLbn, checked((int)rootIcbLen), ""); } + // The Partition Starting Location (ECMA-167 §3/10.5.9) is counted in logical + // blocks, not in 2048-byte sectors: scaling it by a fixed 2048 addressed the + // wrong place on every volume whose block size is not 2048. private long PartitionOffset(long lbn) - => checked(_partitionStart * SectorSize + lbn * (long)_blockSize); + => checked((_partitionStart + lbn) * (long)_blockSize); private void ReadDirectory(long icbLbn, int icbLen, string basePath) { var feOffset = PartitionOffset(icbLbn); @@ -182,14 +273,7 @@ private void ReadDirectory(long icbLbn, int icbLen, string basePath) { if (nameStart > dirData.Length - fidIdLen) break; - string name; - if (fidIdLen > 1 && dirData[nameStart] == 8) - name = Encoding.UTF8.GetString(dirData, nameStart + 1, fidIdLen - 1); - else if (fidIdLen > 1 && dirData[nameStart] == 16) - name = Encoding.BigEndianUnicode.GetString(dirData, nameStart + 1, fidIdLen - 1); - else - name = Encoding.ASCII.GetString(dirData, nameStart, fidIdLen); - name = name.TrimEnd('\0'); + var name = OstaCompressedUnicode.Decode(dirData.AsSpan(nameStart, fidIdLen)); var fullPath = string.IsNullOrEmpty(basePath) ? name : $"{basePath}/{name}"; if (isDir) { @@ -227,6 +311,79 @@ private long GetFileSize(long icbLbn) { return (long)size; } + /// One decoded allocation descriptor. + private readonly record struct AllocationDescriptor(int ExtentType, long Length, uint Block, ushort Partition); + + /// + /// Walks an allocation descriptor list, following the continuations that + /// ECMA-167 §4/14.14.1.1 records as extent type 3. Once a File Entry's own + /// descriptor area is full the rest of the list lives in an Allocation Extent + /// Descriptor (tag 258) in a block of its own, and a reader that stops at the + /// type-3 entry sees only as much of the object as fitted in the entry — for + /// a directory that means most of its children vanish. + /// + private IEnumerable EnumerateAllocationDescriptors(long adStart, int lAd, int adType) { + if (adType is not (0 or 1)) + yield break; + + var stride = adType == 0 ? 8 : 16; + var visited = new HashSet(); + var pos = adStart; + var end = adStart + lAd; + + while (true) { + if (pos < 0 || end > _len || end < pos) + yield break; + + long? continuation = null; + while (pos + stride <= end) { + var raw = U32(pos); + var extentType = (int)(raw >> ExtentTypeShift); + var length = (long)(raw & ExtentLengthMask); + var block = U32(pos + 4); + var partition = adType == 1 ? U16(pos + 8) : (ushort)0; + pos += stride; + + if (length == 0) + continue; + + if (extentType == 3) { + // The continuation replaces the rest of this list; anything after it + // in the current block is not part of the object. + continuation = block; + break; + } + + yield return new(extentType, length, block, partition); + } + + if (continuation is not { } nextBlock) + yield break; + + long nextOffset; + try { + nextOffset = PartitionOffset(nextBlock); + } catch (OverflowException) { + yield break; + } + + // A continuation pointing back at a block already walked would loop for + // ever; refusing to revisit one bounds the walk. + if (!visited.Add(nextOffset)) + yield break; + if (nextOffset < 0 || nextOffset + 24 > _len) + yield break; + // ECMA-167 §4/14.5: the continuation block opens with an Allocation + // Extent Descriptor whose header says how many bytes of descriptors follow. + if (U16(nextOffset) != 258) + yield break; + + var nextLength = U32(nextOffset + 20); + pos = nextOffset + 24; + end = pos + nextLength; + } + } + private UdfFileDataLayout GetFileDataLayout(long icbLbn, long informationLength) { if (informationLength < 0) return new([], "UDF file has a negative logical length."); @@ -272,35 +429,21 @@ private UdfFileDataLayout GetFileDataLayout(long icbLbn, long informationLength) if (adType is not (0 or 1)) return new([], $"UDF allocation descriptor type {adType} is not yet supported for mounted reads."); - var stride = adType == 0 ? 8 : 16; var segments = new List(); long logicalOffset = 0; - var pos = adStart; - - while (pos + stride <= adEnd && logicalOffset < informationLength) { - var rawLength = U32(pos); - var extentType = (int)(rawLength >> ExtentTypeShift); - var extentLength = (long)(rawLength & ExtentLengthMask); - var extentLbn = U32(pos + 4); - if (extentLength == 0) { - pos += stride; - continue; - } - if (extentType == 3) - return new(segments, "UDF continuation allocation descriptors are not yet supported for mounted reads."); + foreach (var ad in this.EnumerateAllocationDescriptors(adStart, lAd, adType)) { + if (logicalOffset >= informationLength) + break; - if (adType == 1) { - var partitionReference = U16(pos + 8); - if (partitionReference != 0) - return new(segments, $"UDF long allocation descriptor references partition map {partitionReference}; only the decoded primary partition is supported."); - } + if (ad.Partition != 0) + return new(segments, $"UDF long allocation descriptor references partition map {ad.Partition}; only the decoded primary partition is supported."); - var logicalLength = Math.Min(extentLength, informationLength - logicalOffset); - if (extentType == 0) { + var logicalLength = Math.Min(ad.Length, informationLength - logicalOffset); + if (ad.ExtentType == 0) { long physicalOffset; try { - physicalOffset = PartitionOffset(extentLbn); + physicalOffset = PartitionOffset(ad.Block); } catch (OverflowException) { return new(segments, "UDF file extent address overflows the image address space."); } @@ -314,7 +457,6 @@ private UdfFileDataLayout GetFileDataLayout(long icbLbn, long informationLength) } logicalOffset += logicalLength; - pos += stride; } if (logicalOffset != informationLength) @@ -337,45 +479,33 @@ private UdfFileDataLayout GetFileDataLayout(long icbLbn, long informationLength) return null; using var ms = new MemoryStream(checked((int)infoLength)); - var pos = adStart; - long end; - try { - end = checked(adStart + lAd); - } catch (OverflowException) { - return null; - } - if (adStart < 0 || end > _len) + if (adStart < 0 || adStart + lAd > _len) return null; - var stride = adType == 0 ? 8 : 16; var zeroBuffer = new byte[8192]; - while (pos + stride <= end && ms.Length < infoLength) { - var rawLength = U32(pos); - var extentType = (int)(rawLength >> ExtentTypeShift); - var extentLength = (long)(rawLength & ExtentLengthMask); - var extentLbn = U32(pos + 4); - if (extentType == 3) - return null; - if (adType == 1 && U16(pos + 8) != 0) + foreach (var ad in this.EnumerateAllocationDescriptors(adStart, lAd, adType)) { + if (ms.Length >= infoLength) + break; + if (ad.Partition != 0) return null; - var take = Math.Min(extentLength, infoLength - ms.Length); - if (take > 0) { - if (extentType == 0) { - var physical = PartitionOffset(extentLbn); - if (physical < 0 || physical > _len - take) - return null; - _img.CopyTo(physical, ms, take); - } else { - var remaining = take; - while (remaining > 0) { - var chunk = checked((int)Math.Min(zeroBuffer.Length, remaining)); - ms.Write(zeroBuffer, 0, chunk); - remaining -= chunk; - } + var take = Math.Min(ad.Length, infoLength - ms.Length); + if (take <= 0) + continue; + + if (ad.ExtentType == 0) { + var physical = PartitionOffset(ad.Block); + if (physical < 0 || physical > _len - take) + return null; + _img.CopyTo(physical, ms, take); + } else { + var remaining = take; + while (remaining > 0) { + var chunk = checked((int)Math.Min(zeroBuffer.Length, remaining)); + ms.Write(zeroBuffer, 0, chunk); + remaining -= chunk; } } - pos += stride; } return ms.Length == infoLength ? ms.ToArray() : null; diff --git a/FileSystems/FileSystem.Udf/UdfWriter.cs b/FileSystems/FileSystem.Udf/UdfWriter.cs index 0d7fe6c96..348f3aff5 100644 --- a/FileSystems/FileSystem.Udf/UdfWriter.cs +++ b/FileSystems/FileSystem.Udf/UdfWriter.cs @@ -1,60 +1,97 @@ #pragma warning disable CS1591 using System.Buffers.Binary; -using System.Text; -using Compression.Core.Checksums; +using static FileSystem.Udf.UdfDescriptors; namespace FileSystem.Udf; /// -/// Writes a minimal UDF 1.02 filesystem image (ECMA-167). Builds a real -/// directory tree from slash-separated file paths, short allocation -/// descriptors. Computes ECMA-167 §7.2.1 DescriptorCRC -/// (CRC-16/CCITT, init=0, poly=0x1021, non-reflected) and TagChecksum for -/// every descriptor tag so that strict readers (xorriso, Linux udf.ko, -/// mkudffs fsck) accept the produced images. +/// Writes a UDF 2.01 volume image (ECMA-167 plus the OSTA UDF profile). Builds a +/// real directory tree from slash-separated file paths using short allocation +/// descriptors, and records every descriptor the standard's own tools look for: +/// both volume descriptor sequences, both anchors, the unallocated space and +/// implementation use descriptors, and a closed logical volume integrity +/// descriptor. /// -/// Layout: +/// Layout (2048-byte logical blocks): /// -/// Sectors 0-15: System area -/// Sector 16: VRS BEA01 -/// Sector 17: VRS NSR02 -/// Sector 18: VRS TEA01 -/// Sector 32-35: Main VDS (PVD + Partition + LVD + Terminator) -/// Sector 256: AVDP -/// Sector 257: Partition start: File Set Descriptor (FSD) at LBN 0 -/// Sector 258: Root directory File Entry at LBN 1 -/// Sector 259+: Per-node File Entries, directory FID data, file data +/// Block 0-15: System area +/// Block 16: VRS BEA01 +/// Block 17: VRS NSR03 +/// Block 18: VRS TEA01 +/// Block 32-47: Main volume descriptor sequence +/// Block 48-63: Reserve volume descriptor sequence +/// Block 64-65: Logical volume integrity sequence (LVID + terminator) +/// Block 256: Anchor volume descriptor pointer +/// Block 257: Partition start; File Set Descriptor at LBN 0 +/// Block 258: Root directory File Entry at LBN 1 +/// Block 259+: Per-node File Entries, directory FID data, file data +/// Last block: Second anchor volume descriptor pointer /// /// -/// A directory's data is a sequence of File Identifier Descriptors (FID, -/// tag 257). The first FID of every directory is the parent entry (Parent -/// flag 0x08, zero-length identifier, ICB pointing at the parent FE). Every -/// directory and file is a File Entry (FE, tag 261); directories carry file -/// type 4, regular files file type 5. A subdirectory FID carries the -/// Directory flag 0x02 and points at the child directory's FE. +/// A directory's data is a dense sequence of File Identifier Descriptors (FID, +/// tag 257) — dense because ECMA-167 lets a FID span a logical block boundary +/// and Linux's udf driver reads directory bytes as one uninterrupted run, so +/// padding a FID onto the next block makes the directory unreadable. The first +/// FID of every directory is the parent entry (Parent flag 0x08, zero-length +/// identifier). Every directory and file is a File Entry (FE, tag 261); +/// directories carry file type 4, regular files file type 5. /// public sealed class UdfWriter { private const int Sector = 2048; + + private const int VrsFirstSector = 16; + private const int MainVdsSector = 32; + private const int VdsSectors = 16; + private const int ReserveVdsSector = MainVdsSector + VdsSectors; + private const int LvidSector = ReserveVdsSector + VdsSectors; + private const int LvidSectors = 2; + private const int AnchorSector = 256; private const int PartitionStartSector = 257; - // Descriptor body sizes per ECMA-167 §10. The body starts at offset 16 - // (after the 16-byte descriptor tag) and DescriptorCRCLength covers - // exactly these many bytes. Using fixed structure sizes (rather than - // the full sector) keeps us compatible with real UDF implementations. - private const int PvdBodySize = 496; // AVDP/PVD/PD sector size 512 - 16 tag - private const int AvdpBodySize = 496; - private const int PdBodySize = 496; - private const int LvdBodySize = 440 - 16; // 440 header + zero partition maps - private const int TerminatorBodySize = 496; - private const int FsdBodySize = 496; - private const int FeBodyHeader = 160; // 176 - 16 (up to L_EA), plus L_EA + L_AD content + /// File Set Descriptor block, relative to the partition start. + private const int FsdLbn = 0; + + /// Root directory File Entry block, relative to the partition start. + private const int RootFeLbn = 1; + + // Descriptor body sizes per ECMA-167 §3/10. The body starts 16 bytes after the + // tag and DescriptorCRCLength covers exactly these many bytes. + private const int VolumeDescriptorBodySize = 496; // 512-byte descriptor minus its tag + private const int LvdBodySize = 446 - 16; // through the single Type-1 partition map + private const int UsdBodySize = 24 - 16; // no allocation descriptors follow + private const int LvidBodySize = (88 - 16) + LvidImplementationUseSize; + private const int LvidImplementationUseSize = 46; + private const int FeHeaderBodySize = 176 - 16; // File Entry header, up to L_EA + + /// + /// Bytes one short allocation descriptor may address. ECMA-167 caps an extent + /// length at 2^30-1 and OSTA UDF §2.3.10.1 requires every extent but the last + /// of a file to be a whole number of blocks, so the usable maximum is the + /// largest block multiple below 2^30. + /// + private const long MaxExtentBytes = (1L << 30) - Sector; + + /// + /// Short allocation descriptors that fit in one File Entry. They start after + /// the 176-byte header and the extended attributes, of which this writer + /// records none. + /// + private const int MaxAllocationDescriptors = (Sector - 176) / 8; + + /// Largest file this writer can address without descriptor continuation. + internal const long MaxFileBytes = MaxAllocationDescriptors * MaxExtentBytes; + + /// + /// First unique identifier handed to a file or directory. OSTA UDF §3.2.1 + /// reserves 0 for the root directory and 1..15 for the standard's own use. + /// + private const uint FirstUniqueId = 16; private readonly List<(string name, byte[] data)> _files = []; /// /// ECMA-167 PVD Volume Identifier (dstring at PVD offset 24, 32 bytes). /// Linux's udf driver surfaces this as the volume label. Default "UDF Volume". - /// Truncated to 31 bytes (ECMA-167 dstring length byte caps at 31). /// public string VolumeIdentifier { get; set; } = "UDF Volume"; @@ -117,7 +154,8 @@ private sealed class Node { public int FeLbn; // File Entry block public int DataLbn; // first data block (FID data for dirs, payload for files) public int DataSectors; // sectors occupied by data - public int DataLength; // exact byte length of the (directory or file) data + public long DataLength; // exact byte length of the (directory or file) data + public uint UniqueId; // OSTA UDF §3.2.1 unique identifier } /// @@ -126,46 +164,47 @@ private sealed class Node { public void WriteTo(Stream output) { var root = BuildTree(); - // LBN 0 is the FSD; the root File Entry lives at LBN 1. Everything else - // (per-node FEs, directory FID data, file payloads) is laid out after. - root.FeLbn = 1; - var nextLbn = 2; - AssignLayout(root, ref nextLbn); + root.FeLbn = RootFeLbn; + root.UniqueId = 0; + var nextLbn = RootFeLbn + 1; + var nextUniqueId = FirstUniqueId; + AssignLayout(root, ref nextLbn, ref nextUniqueId); var totalPartitionSectors = nextLbn; // LBNs 0..nextLbn-1 are all in use - var totalImageSectors = PartitionStartSector + totalPartitionSectors; + var lastBlock = PartitionStartSector + totalPartitionSectors; - // ── Write system area (sectors 0-15) ── - WritePadding(output, 16); + var (fileCount, directoryCount) = Count(root); - // ── Write VRS (sectors 16-18) ── + // ── System area (blocks 0-15) ── + WritePadding(output, VrsFirstSector); + + // ── Volume recognition sequence (blocks 16-18) ── WriteVrs(output, "BEA01"); - WriteVrs(output, "NSR02"); + WriteVrs(output, "NSR03"); WriteVrs(output, "TEA01"); - // ── Padding to sector 32 ── - WritePadding(output, 32 - 19); + WritePadding(output, MainVdsSector - (VrsFirstSector + 3)); + + // ── Both volume descriptor sequences ── + this.WriteVds(output, MainVdsSector, totalPartitionSectors); + this.WriteVds(output, ReserveVdsSector, totalPartitionSectors); - // ── Main VDS at sectors 32-35 ── - this.WritePvd(output, 32, totalImageSectors); - WritePartitionDescriptor(output, 33, PartitionStartSector, totalPartitionSectors); - WriteLvd(output, 34); - WriteTerminator(output, 35); + // ── Logical volume integrity sequence ── + this.WriteLvid(output, LvidSector, totalPartitionSectors, fileCount, directoryCount, nextUniqueId); + WriteTerminator(output, LvidSector + 1); - // ── Padding to sector 256 ── - WritePadding(output, 256 - 36); + WritePadding(output, AnchorSector - (LvidSector + LvidSectors)); - // ── AVDP at sector 256 ── - WriteAvdp(output, 256, mainVdsLoc: 32, mainVdsLen: 4 * Sector); + // ── First anchor ── + WriteAnchor(output, AnchorSector); - // ── Partition data (starting at sector 257 = LBN 0) ── - WriteFsd(output, lbn: 0, rootIcbLbn: root.FeLbn); + // ── Partition data (starting at block 257 = LBN 0) ── + this.WriteFsd(output, FsdLbn, RootFeLbn); // Emit blocks in LBN order so the stream stays sequential. Gather every // block-producing action keyed by its starting LBN, then drain in order. - blocksOutput = output; var blocks = new SortedDictionary(); - CollectBlocks(root, blocks); + CollectBlocks(root, blocks, output); var written = 1; // LBN 0 (FSD) already written foreach (var (lbn, emit) in blocks) { @@ -180,6 +219,11 @@ public void WriteTo(Stream output) { if (written != totalPartitionSectors) throw new InvalidOperationException( $"UDF layout mismatch: wrote {written} partition sectors, expected {totalPartitionSectors}."); + + // ── Second anchor, in the volume's last block ── + // ECMA-167 §3/8.4 wants an anchor at block 256 and at the last block of the + // volume; a volume carrying only the first is one udfinfo reports on. + WriteAnchor(output, lastBlock); } /// @@ -233,17 +277,34 @@ private Node BuildTree() { return root; } + /// Files and directories below (and including) . + private static (int files, int directories) Count(Node node) { + if (!node.IsDirectory) return (1, 0); + var files = 0; + var directories = 1; + foreach (var child in node.Children) { + var (f, d) = Count(child); + files += f; + directories += d; + } + + return (files, directories); + } + /// /// Assigns File Entry and data block numbers depth-first. The node's own FE /// LBN must already be set by the caller; this method assigns FE LBNs for /// all children first (so a directory's FID data can reference them), then /// the directory's own FID-data block(s), then recurses. /// - private void AssignLayout(Node node, ref int nextLbn) { + private static void AssignLayout(Node node, ref int nextLbn, ref uint nextUniqueId) { if (node.IsDirectory) { - // Reserve FE blocks for every child up front. - foreach (var child in node.Children) + // Reserve FE blocks and identifiers for every child up front: the parent's + // FIDs name both. + foreach (var child in node.Children) { child.FeLbn = nextLbn++; + child.UniqueId = nextUniqueId++; + } // Reserve this directory's FID-data block(s). var fidData = BuildFidData(node); @@ -252,13 +313,26 @@ private void AssignLayout(Node node, ref int nextLbn) { node.DataLbn = nextLbn; nextLbn += node.DataSectors; + if (node.DataSectors > MaxAllocationDescriptors) + throw new InvalidOperationException( + $"UDF directory '{node.Name}' needs {node.DataSectors} data blocks but only " + + $"{MaxAllocationDescriptors} short allocation descriptors fit in one File Entry; " + + "allocation descriptor continuation is not supported."); + // Recurse into children so their own subtree blocks are laid out. foreach (var child in node.Children) - AssignLayout(child, ref nextLbn); + AssignLayout(child, ref nextLbn, ref nextUniqueId); } else { - var len = node.EffectiveLength; - node.DataLength = (int)len; - node.DataSectors = Math.Max(1, (int)((len + Sector - 1) / Sector)); + var length = node.EffectiveLength; + if (length > MaxFileBytes) + throw new InvalidOperationException( + $"UDF file '{node.Name}' is {length:N0} bytes; this writer addresses at most " + + $"{MaxFileBytes:N0} without allocation descriptor continuation."); + + node.DataLength = length; + // An empty file gets no extent at all: an allocation descriptor naming a + // block a zero-length file does not own is a chain longer than the size. + node.DataSectors = (int)((length + Sector - 1) / Sector); node.DataLbn = nextLbn; nextLbn += node.DataSectors; } @@ -269,45 +343,42 @@ private void AssignLayout(Node node, ref int nextLbn) { /// the starting LBN of each block group, so the writer can drain them in /// strictly ascending order. /// - private void CollectBlocks(Node node, SortedDictionary blocks) { + private static void CollectBlocks(Node node, SortedDictionary blocks, Stream output) { if (node.IsDirectory) { - // This directory's File Entry. var dirNode = node; - blocks[node.FeLbn] = () => WriteDirectoryFe(blocksOutput, dirNode); + blocks[node.FeLbn] = () => WriteDirectoryFe(output, dirNode); - // This directory's FID data. blocks[node.DataLbn] = () => { var fidData = BuildFidData(dirNode); - blocksOutput.Write(fidData); + output.Write(fidData); var pad = dirNode.DataSectors * Sector - fidData.Length; - if (pad > 0) blocksOutput.Write(new byte[pad]); + if (pad > 0) output.Write(new byte[pad]); }; foreach (var child in node.Children) - CollectBlocks(child, blocks); + CollectBlocks(child, blocks, output); } else { var fileNode = node; - blocks[node.FeLbn] = () => WriteFileFe(blocksOutput, fileNode.FeLbn, fileNode.DataLength, fileNode.DataLbn); + blocks[node.FeLbn] = () => WriteFileFe(output, fileNode); + if (fileNode.DataSectors == 0) + return; + blocks[node.DataLbn] = () => { long produced; if (fileNode.StreamOpener != null) { // Stream the body straight into the sequential output in 64 KiB chunks // — never buffered as a byte[]. Exactly DataLength bytes are copied. - produced = StreamCopy(blocksOutput, fileNode.StreamOpener, fileNode.DataLength); + produced = StreamCopy(output, fileNode.StreamOpener, fileNode.DataLength); } else { - blocksOutput.Write(fileNode.Data); + output.Write(fileNode.Data); produced = fileNode.Data.Length; } var pad = (long)fileNode.DataSectors * Sector - produced; - if (pad > 0) blocksOutput.Write(new byte[pad]); + if (pad > 0) output.Write(new byte[pad]); }; } } - // The output stream is captured for the duration of WriteTo so the block - // actions stay simple closures; set before CollectBlocks runs. - private Stream blocksOutput = Stream.Null; - /// /// Copies up to bytes from a freshly opened source /// stream to in 64 KiB chunks and returns the number of @@ -335,103 +406,56 @@ private static long StreamCopy(Stream dst, Func opener, long size) { /// identifier, parent + directory flags) pointing at the parent's FE, /// followed by one FID per child referencing the child's FE. /// - /// ECMA-167 §4/14.4: a File Identifier Descriptor may not cross a logical - /// block boundary. When the next FID would straddle the current block, the - /// remainder of that block is zero-padded and the FID starts at the next - /// block. The returned buffer is therefore a multiple of the block size, so - /// the directory spans whole blocks regardless of entry count. + /// The records are written back to back. ECMA-167 §4/14.4 permits a File + /// Identifier Descriptor to span a logical block boundary, and both mkudffs + /// and Linux's udf driver rely on that: a directory padded onto block + /// boundaries makes the driver stop at the first pad byte with "entry at + /// pos N with incorrect tag 0". /// /// private static byte[] BuildFidData(Node dir) { using var ms = new MemoryStream(); - // Parent FID (flags=0x0A: parent + directory). The parent of the root is + // Parent FID (flags 0x0A: parent + directory). The parent of the root is // itself. - var parentFeLbn = (dir.Parent ?? dir).FeLbn; - WriteFidBlockAligned(ms, 0x0A, parentFeLbn, ""); + var parent = dir.Parent ?? dir; + WriteFid(ms, dir, 0x0A, parent.FeLbn, parent.UniqueId, ""); foreach (var child in dir.Children) { var flags = child.IsDirectory ? (byte)0x02 : (byte)0x00; - WriteFidBlockAligned(ms, flags, child.FeLbn, child.Name); + WriteFid(ms, dir, flags, child.FeLbn, child.UniqueId, child.Name); } return ms.ToArray(); } /// - /// Writes one FID, first padding to the next logical block boundary if the - /// FID would otherwise cross it (ECMA-167 §14.4 forbids that crossing). + /// Writes one File Identifier Descriptor into a directory's byte stream. The + /// tag records the logical block the record starts in, which for a record + /// spanning two blocks is the first of them. /// - private static void WriteFidBlockAligned(MemoryStream ms, byte flags, int icbLbn, string name) { - var fidLen = FidLength(name); - var posInBlock = (int)(ms.Length % Sector); - if (posInBlock + fidLen > Sector) { - var pad = Sector - posInBlock; - ms.Write(new byte[pad]); - } - WriteFid(ms, flags, icbLbn, name); - } - - /// Padded on-disk byte length of a FID for the given identifier. - private static int FidLength(string name) { - var nameLen = name.Length == 0 ? 0 : EncodeCs0(name).Length; - return (38 + nameLen + 3) & ~3; - } - - private static void WriteFid(Stream s, byte flags, int icbLbn, string name) { - var nameBytes = name.Length == 0 ? [] : EncodeCs0(name); - var fidLen = 38 + nameBytes.Length; - var padded = (fidLen + 3) & ~3; + private static void WriteFid(MemoryStream ms, Node dir, byte flags, int icbLbn, uint uniqueId, string name) { + var nameBytes = OstaCompressedUnicode.Encode(name); + var padded = (38 + nameBytes.Length + 3) & ~3; var buf = new byte[padded]; - // Tag: FID = 257 - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(0), 257); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(2), 2); // descriptor version + var startBlock = dir.DataLbn + (int)(ms.Length / Sector); + WriteTag(buf, 0, 257, (uint)startBlock); + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(16), 1); // FileVersionNumber, OSTA UDF §2.3.4.1 buf[18] = flags; - buf[19] = (byte)nameBytes.Length; // identifier length - // ICB at offset 20: long_ad (16 bytes) — length(4) + lbn(4) + partRef(2) + impl(6) - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(20), (uint)Sector); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(24), (uint)icbLbn); - // lIU at offset 36 = 0 - // Name at offset 38 + buf[19] = (byte)nameBytes.Length; + WriteLongAd(buf, 20, Sector, (uint)icbLbn, uniqueId); + // LengthOfImplementationUse at offset 36 stays zero. nameBytes.CopyTo(buf, 38); - // ECMA-167 §14.4: FID DescriptorCRCLength covers the entire padded FID - // minus the 16-byte tag. + // ECMA-167 §4/14.4.9: the CRC covers the whole record, padding included. FinalizeTag(buf, 0, padded - 16); - s.Write(buf); - } - - private static byte[] EncodeCs0(string name) { - var utf8 = Encoding.UTF8.GetBytes(name); - var result = new byte[1 + utf8.Length]; - result[0] = 8; // CS0 compression ID = UTF-8 - utf8.CopyTo(result, 1); - return result; + ms.Write(buf); } // ── Descriptor writers ──────────────────────────────────────────────────── - // UDF domain entity-identifier suffix (UDF 1.02): UDFRevision(2 LE)=0x0102, - // DomainFlags(1)=0, Reserved(5). Stamped after the "*OSTA UDF Compliant" id. - private static readonly byte[] DomainSuffix = [0x02, 0x01, 0, 0, 0, 0, 0, 0]; - - // ECMA-167 §7.4 EntityID (regid): Flags(1) + Identifier(23) + Suffix(8). - private static void WriteRegid(byte[] buf, int off, string id, ReadOnlySpan suffix) { - buf[off] = 0; - var idb = Encoding.ASCII.GetBytes(id); - Array.Copy(idb, 0, buf, off + 1, Math.Min(idb.Length, 23)); - if (!suffix.IsEmpty) - suffix[..Math.Min(suffix.Length, 8)].CopyTo(buf.AsSpan(off + 24, 8)); - } - - // ECMA-167 §7.2.1 charspec: CharacterSetType(1)=0 (CS0) + CharacterSetInfo(63). - private static void WriteCharspec(byte[] buf, int off) { - buf[off] = 0; - Encoding.ASCII.GetBytes("OSTA Compressed Unicode").CopyTo(buf, off + 1); - } - /// Owner/group/other read+execute, in ECMA-167 permission bits. private const uint DirectoryPermissions = (1u << 12) | (1u << 10) | (1u << 7) | (1u << 5) | (1u << 2) | (1u << 0); @@ -439,42 +463,6 @@ private static void WriteCharspec(byte[] buf, int off) { /// Owner/group/other read, in ECMA-167 permission bits. private const uint FilePermissions = (1u << 12) | (1u << 7) | (1u << 2); - private static void WriteTag(byte[] buf, int off, ushort tagId, uint tagLocation) { - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(off), tagId); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(off + 2), 2); // descriptor version - // DescriptorCRC (off+8..9), DescriptorCRCLength (off+10..11), TagChecksum (off+4) - // filled in by FinalizeTag after the descriptor body is populated. - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(off + 12), tagLocation); - } - - /// - /// Finalizes a UDF descriptor tag per ECMA-167 §7.2.1 by computing the - /// CRC-16/CCITT (init=0, poly=0x1021, non-reflected) over - /// bytes starting at tagOffset + 16, storing it in the tag at offsets 8..9, - /// writing the DescriptorCRCLength at offsets 10..11, and finally computing the - /// byte-sum-mod-256 TagChecksum at offset 4. - /// - private static void FinalizeTag(byte[] buf, int tagOffset, int bodyLength) { - var bodyStart = tagOffset + 16; - if (bodyStart + bodyLength > buf.Length) - bodyLength = buf.Length - bodyStart; - if (bodyLength < 0) bodyLength = 0; - - var crc = Crc16Ccitt.Compute(buf.AsSpan(bodyStart, bodyLength)); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(tagOffset + 8), crc); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(tagOffset + 10), (ushort)bodyLength); - - // TagChecksum = (sum of bytes [0..3, 5..15]) mod 256. Byte at offset 4 - // is excluded (it IS the checksum) and must be zero while computing. - buf[tagOffset + 4] = 0; - byte sum = 0; - for (var i = 0; i < 16; i++) { - if (i == 4) continue; - sum = (byte)(sum + buf[tagOffset + i]); - } - buf[tagOffset + 4] = sum; - } - private static void WritePadding(Stream output, int sectors) { for (var i = 0; i < sectors; i++) output.Write(new byte[Sector]); } @@ -482,166 +470,290 @@ private static void WritePadding(Stream output, int sectors) { private static void WriteVrs(Stream output, string id) { var buf = new byte[Sector]; buf[0] = 0; // structure type - Encoding.ASCII.GetBytes(id).CopyTo(buf, 1); + System.Text.Encoding.ASCII.GetBytes(id).CopyTo(buf, 1); buf[6] = 1; // structure version output.Write(buf); } - private static void WriteAvdp(Stream output, int sectorNum, int mainVdsLoc, int mainVdsLen) { + /// + /// ECMA-167 §3/10.2 anchor: it names both volume descriptor sequences and + /// records its own block, which is how a reader recognises it and, with it, + /// the volume's logical block size. + /// + private static void WriteAnchor(Stream output, int block) { var buf = new byte[Sector]; - WriteTag(buf, 0, 2, (uint)sectorNum); - // Main VDS extent: length(4) + location(4) at offset 16 - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), (uint)mainVdsLen); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(20), (uint)mainVdsLoc); - FinalizeTag(buf, 0, AvdpBodySize); + WriteTag(buf, 0, 2, (uint)block); + WriteExtent(buf, 16, VdsSectors * Sector, MainVdsSector); + WriteExtent(buf, 24, VdsSectors * Sector, ReserveVdsSector); + FinalizeTag(buf, 0, VolumeDescriptorBodySize); output.Write(buf); } - private void WritePvd(Stream output, int sectorNum, int totalSectors) { - var buf = new byte[Sector]; - WriteTag(buf, 0, 1, (uint)sectorNum); // Primary Volume Descriptor - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 1); // VDS number - // Volume Identifier — ECMA-167 dstring (max 31 ASCII bytes + length byte at offset+31). - var volId = string.IsNullOrEmpty(this.VolumeIdentifier) ? "UDF Volume" : this.VolumeIdentifier; - if (volId.Length > 31) volId = volId[..31]; - Encoding.ASCII.GetBytes(volId).CopyTo(buf, 24); - FinalizeTag(buf, 0, PvdBodySize); - output.Write(buf); + /// + /// Writes one complete volume descriptor sequence: primary volume, logical + /// volume, partition, implementation use and unallocated space descriptors, + /// then the terminator, then zero blocks out to the sequence's extent. Both + /// the main and the reserve sequence carry the same descriptors — only their + /// tag locations differ. + /// + private void WriteVds(Stream output, int firstBlock, int partitionSectors) { + this.WritePvd(output, firstBlock); + this.WriteLvd(output, firstBlock + 1); + WritePartitionDescriptor(output, firstBlock + 2, PartitionStartSector, partitionSectors); + this.WriteIuvd(output, firstBlock + 3); + WriteUsd(output, firstBlock + 4); + WriteTerminator(output, firstBlock + 5); + WritePadding(output, VdsSectors - 6); + } + + /// + /// Volume set identifier (ECMA-167 §3/10.1.10). OSTA UDF §2.2.2.5 requires + /// its first sixteen characters to be unique among volume sets, and to be + /// hexadecimal digits; deriving them from the volume identifier keeps two + /// runs over the same input byte-identical while still separating volumes + /// that are named differently. + /// + private string VolumeSetIdentifier { + get { + var hash = 0xCBF29CE484222325UL; + foreach (var c in this.EffectiveVolumeIdentifier) { + hash ^= c; + hash *= 0x100000001B3UL; + } + + return hash.ToString("x16") + this.EffectiveVolumeIdentifier; + } } - private static void WritePartitionDescriptor(Stream output, int sectorNum, int partStart, int partLen) { + private string EffectiveVolumeIdentifier + => string.IsNullOrEmpty(this.VolumeIdentifier) ? "UDF Volume" : this.VolumeIdentifier; + + private void WritePvd(Stream output, int block) { var buf = new byte[Sector]; - WriteTag(buf, 0, 5, (uint)sectorNum); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 1); // VDS number - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(20), 1); // partition flags = allocated - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(22), 0); // partition number 0 - WriteRegid(buf, 24, "+NSR02", default); // partition contents (ECMA-167 §4) - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(184), 1); // access type = read-only - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(188), (uint)partStart); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(192), (uint)partLen); - FinalizeTag(buf, 0, PdBodySize); + WriteTag(buf, 0, 1, (uint)block); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 1); // VolumeDescriptorSequenceNumber + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(20), 0); // PrimaryVolumeDescriptorNumber + OstaCompressedUnicode.WriteDString(buf, 24, 32, this.EffectiveVolumeIdentifier); + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(56), 1); // VolumeSequenceNumber + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(58), 1); // MaximumVolumeSequenceNumber + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(60), 2); // InterchangeLevel + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(62), 3); // MaximumInterchangeLevel + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(64), 1); // CharacterSetList + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(68), 1); // MaximumCharacterSetList + OstaCompressedUnicode.WriteDString(buf, 72, 128, this.VolumeSetIdentifier); + WriteCharacterSet(buf, 200); // DescriptorCharacterSet + WriteCharacterSet(buf, 264); // ExplanatoryCharacterSet + WriteTimestamp(buf, 376, RecordingTime); + WriteEntityId(buf, 388, ImplementationId, ImplementationSuffix); + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(488), 1); // Flags: volume set identification + FinalizeTag(buf, 0, VolumeDescriptorBodySize); output.Write(buf); } - private void WriteLvd(Stream output, int sectorNum) { + private void WriteLvd(Stream output, int block) { var buf = new byte[Sector]; - WriteTag(buf, 0, 6, (uint)sectorNum); - WriteCharspec(buf, 20); // descriptor character set (CS0) - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(212), (uint)Sector); // logical block size - WriteRegid(buf, 216, "*OSTA UDF Compliant", DomainSuffix); // domain identifier (kernel-mandatory) - // logical_volume_contents_use @248: FSD long_ad (length=4, lbn=4, partRef=2, impl=6) - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(248), (uint)Sector); // extent length - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(252), 0); // FSD LBN = 0 - // partRef at 256 = 0 (default) - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(264), 6); // map table length (one Type-1 map) - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(268), 1); // number of partition maps - // The implementation identifier names the UDF implementation, not the product: - // an image made on Linux says so, whichever tool wrote it. - WriteRegid(buf, 272, "*Linux UDFFS", default); // implementation identifier - // Type-1 partition map @440: type(1)=1, length(1)=6, vol_seq(2)=1, part_num(2)=0 + WriteTag(buf, 0, 6, (uint)block); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 2); // VolumeDescriptorSequenceNumber + WriteCharacterSet(buf, 20); // DescriptorCharacterSet + OstaCompressedUnicode.WriteDString(buf, 84, 128, this.EffectiveVolumeIdentifier); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(212), Sector); + WriteEntityId(buf, 216, "*OSTA UDF Compliant", DomainSuffix); + WriteLongAd(buf, 248, Sector, FsdLbn); // LogicalVolumeContentsUse: the FSD + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(264), 6); // MapTableLength + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(268), 1); // NumberOfPartitionMaps + WriteEntityId(buf, 272, ImplementationId, ImplementationSuffix); + WriteExtent(buf, 432, LvidSectors * Sector, LvidSector); // IntegritySequenceExtent + // Type-1 partition map: type(1), length(1), volume sequence(2), partition(2). buf[440] = 1; buf[441] = 6; BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(442), 1); BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(444), 0); - // CRC must cover through the partition map (offset 16..446). - FinalizeTag(buf, 0, 446 - 16); + FinalizeTag(buf, 0, LvdBodySize); + output.Write(buf); + } + + private static void WritePartitionDescriptor(Stream output, int block, int partStart, int partLen) { + var buf = new byte[Sector]; + WriteTag(buf, 0, 5, (uint)block); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 3); // VolumeDescriptorSequenceNumber + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(20), 1); // PartitionFlags: allocated + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(22), 0); // PartitionNumber + WriteEntityId(buf, 24, "+NSR03", default); // PartitionContents + // PartitionContentsUse holds the Partition Header Descriptor (OSTA UDF + // §2.2.3). All its space tables stay unrecorded, which a read-only + // partition is allowed to do since nothing will ever allocate in it. + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(184), 1); // AccessType: read-only + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(188), (uint)partStart); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(192), (uint)partLen); + WriteEntityId(buf, 196, ImplementationId, ImplementationSuffix); + FinalizeTag(buf, 0, VolumeDescriptorBodySize); + output.Write(buf); + } + + /// + /// ECMA-167 §3/10.4 implementation use volume descriptor carrying the OSTA + /// UDF §2.2.7 "*UDF LV Info" payload: the logical volume's name and the + /// identity of whoever recorded it. + /// + private void WriteIuvd(Stream output, int block) { + var buf = new byte[Sector]; + WriteTag(buf, 0, 4, (uint)block); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 4); // VolumeDescriptorSequenceNumber + WriteEntityId(buf, 20, "*UDF LV Info", UdfEntitySuffix); + WriteCharacterSet(buf, 52); // LVICharset + OstaCompressedUnicode.WriteDString(buf, 116, 128, this.EffectiveVolumeIdentifier); + // LVInfo1..3 at 244/280/316 name the owner, organisation and contact; this + // writer knows none of them and leaves all three empty. + WriteEntityId(buf, 352, ImplementationId, ImplementationSuffix); + FinalizeTag(buf, 0, VolumeDescriptorBodySize); + output.Write(buf); + } + + /// + /// ECMA-167 §3/10.8 unallocated space descriptor. The volume this writer + /// emits has no space outside the partition to hand out, so it records no + /// allocation descriptors — but the descriptor itself has to be there. + /// + private static void WriteUsd(Stream output, int block) { + var buf = new byte[Sector]; + WriteTag(buf, 0, 7, (uint)block); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(16), 5); // VolumeDescriptorSequenceNumber + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(20), 0); // NumberOfAllocationDescriptors + FinalizeTag(buf, 0, UsdBodySize); output.Write(buf); } - private static void WriteTerminator(Stream output, int sectorNum) { + private static void WriteTerminator(Stream output, int block) { var buf = new byte[Sector]; - WriteTag(buf, 0, 8, (uint)sectorNum); - FinalizeTag(buf, 0, TerminatorBodySize); + WriteTag(buf, 0, 8, (uint)block); + FinalizeTag(buf, 0, VolumeDescriptorBodySize); output.Write(buf); } - private static void WriteFsd(Stream output, int lbn, int rootIcbLbn) { + /// + /// ECMA-167 §3/10.10 logical volume integrity descriptor. Its integrity type + /// says whether the volume was left in a consistent state; without one, every + /// tool reports the logical volume as inconsistent and the free-space and + /// object counts as unknown. + /// + private void WriteLvid(Stream output, int block, int partitionSectors, + int fileCount, int directoryCount, uint nextUniqueId) { + var buf = new byte[Sector]; + WriteTag(buf, 0, 9, (uint)block); + WriteTimestamp(buf, 16, RecordingTime); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(28), 1); // IntegrityType: close + // NextIntegrityExtent at 32 stays empty: this is the only integrity + // descriptor the volume has. + BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(40), nextUniqueId); // next UniqueID + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(72), 1); // NumberOfPartitions + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(76), LvidImplementationUseSize); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(80), 0); // FreeSpaceTable: packed solid + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(84), (uint)partitionSectors); // SizeTable + WriteEntityId(buf, 88, ImplementationId, ImplementationSuffix); + // The root directory counts: a freshly made empty volume reports one. + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(120), (uint)fileCount); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(124), (uint)directoryCount); + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(128), UdfRevision); // MinimumUDFReadRevision + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(130), UdfRevision); // MinimumUDFWriteRevision + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(132), UdfRevision); // MaximumUDFWriteRevision + FinalizeTag(buf, 0, LvidBodySize); + output.Write(buf); + } + + private void WriteFsd(Stream output, int lbn, int rootIcbLbn) { var buf = new byte[Sector]; WriteTag(buf, 0, 256, (uint)lbn); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(28), 3); // interchange level - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(30), 3); // max interchange level - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(32), 1); // charset list - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(36), 1); // max charset list - WriteCharspec(buf, 48); // logical volume id charset (CS0) - WriteCharspec(buf, 240); // file set charset (CS0) - // Root ICB: long_ad at offset 400 - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(400), (uint)Sector); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(404), (uint)rootIcbLbn); - WriteRegid(buf, 416, "*OSTA UDF Compliant", DomainSuffix); // domain identifier (kernel-checked) - FinalizeTag(buf, 0, FsdBodySize); + WriteTimestamp(buf, 16, RecordingTime); + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(28), 3); // InterchangeLevel + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(30), 3); // MaximumInterchangeLevel + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(32), 1); // CharacterSetList + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(36), 1); // MaximumCharacterSetList + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(40), 0); // FileSetNumber + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(44), 0); // FileSetDescriptorNumber + WriteCharacterSet(buf, 48); // LogicalVolumeIdentifierCharacterSet + OstaCompressedUnicode.WriteDString(buf, 112, 128, this.EffectiveVolumeIdentifier); + WriteCharacterSet(buf, 240); // FileSetCharacterSet + OstaCompressedUnicode.WriteDString(buf, 304, 32, this.EffectiveVolumeIdentifier); + WriteLongAd(buf, 400, Sector, (uint)rootIcbLbn); // RootDirectoryICB + WriteEntityId(buf, 416, "*OSTA UDF Compliant", DomainSuffix); + FinalizeTag(buf, 0, VolumeDescriptorBodySize); output.Write(buf); } /// - /// Writes a directory File Entry (file type 4). The file link count is set - /// to 1 (the parent's reference) plus one per subdirectory child, matching - /// ECMA-167's accounting of incoming directory links. + /// Writes the common part of an ECMA-167 §4/14.9 File Entry and returns the + /// buffer, leaving the caller to append allocation descriptors. /// - private static void WriteDirectoryFe(Stream output, Node dir) { + private static byte[] BeginFileEntry(Node node, byte fileType, uint permissions, ushort linkCount) { var buf = new byte[Sector]; - WriteTag(buf, 0, 261, (uint)dir.FeLbn); - // ICB tag at offset 16: strategy_type(2)@20 must be 4 (the only type the - // kernel supports; 0 → "unsupported strategy type"), max_entries(2)@24 = 1. - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(20), 4); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(24), 1); - buf[27] = 4; // file type = directory - // Permissions at FE offset 44. Left at zero, a mounted volume gives every - // object mode 0000 and the mount cannot even be listed. r-x for owner, - // group and other (ECMA-167 4/14.9.5 bit order). - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(44), DirectoryPermissions); - // File link count is at FE offset 48 (ECMA-167 §14.9, after uid/gid/perms); - // offset 28 falls inside the ICB tag and leaves the kernel seeing link - // count 0 → "Error in udf_iget". Parent link + one per child directory. + WriteTag(buf, 0, 261, (uint)node.FeLbn); + // ICB tag (ECMA-167 §4/14.6). Strategy type 4 is the only one Linux's udf + // driver supports; type 0 makes it refuse the entry outright. + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(20), 4); // StrategyType + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(24), 1); // MaximumNumberOfEntries + buf[27] = fileType; + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(34), 0); // Flags: short allocation descriptors + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(44), permissions); + // FileLinkCount is at offset 48, after uid/gid/permissions. Offset 28 falls + // inside the ICB tag and leaves the driver seeing zero links. + BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(48), linkCount); + BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(56), (ulong)node.DataLength); + BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(64), (ulong)node.DataSectors); // LogicalBlocksRecorded + WriteTimestamp(buf, 72, RecordingTime); // AccessTime + WriteTimestamp(buf, 84, RecordingTime); // ModificationTime + WriteTimestamp(buf, 96, RecordingTime); // AttributeTime + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(108), 1); // Checkpoint + WriteEntityId(buf, 128, ImplementationId, ImplementationSuffix); + BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(160), node.UniqueId); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(168), 0); // LengthOfExtendedAttributes + return buf; + } + + /// + /// Writes a directory File Entry (file type 4). The link count is the + /// parent's reference plus one per subdirectory child, matching ECMA-167's + /// accounting of incoming directory links. + /// + private static void WriteDirectoryFe(Stream output, Node dir) { var subDirCount = dir.Children.Count(c => c.IsDirectory); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(48), (ushort)(1 + subDirCount)); - // icb flags at offset 34: adType=0 (short ADs) - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(34), 0); - // info length at offset 56 — the directory's FID data is block-aligned, so - // this covers whole blocks (one short AD each). - BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(56), (ulong)dir.DataLength); - - // One short AD per logical block. ECMA-167 forbids a FID from crossing a - // block boundary, so BuildFidData already padded the data to a block - // multiple; describe each block with its own descriptor (length always a - // full block). Multiple ADs lift the single-block directory cap. - var blocks = dir.DataSectors; - // Short ADs live inside the FE sector after the 176-byte header, so their - // count is bounded by the sector. ~234 blocks ≈ 478 KiB of FID data, which - // is thousands of small entries; beyond that an AD-continuation extent - // would be required (not yet implemented). - var maxAds = (Sector - 176) / 8; - if (blocks > maxAds) - throw new InvalidOperationException( - $"UDF directory '{dir.Name}' needs {blocks} data blocks but only {maxAds} short " + - "allocation descriptors fit in one File Entry; AD-continuation extents are not supported."); - var lAd = blocks * 8; + var buf = BeginFileEntry(dir, fileType: 4, DirectoryPermissions, (ushort)(1 + subDirCount)); + + // One short allocation descriptor per block, the last one only as long as + // the directory's remaining bytes. OSTA UDF §2.3.10.1 requires every extent + // but the last to be a whole number of blocks; making the last one whole + // too would claim bytes the information length says are not there. + var lAd = WriteExtents(buf, 176, dir.DataLbn, dir.DataLength); BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(172), (uint)lAd); - for (var b = 0; b < blocks; b++) { - var adOff = 176 + b * 8; - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(adOff), (uint)Sector); // extent length - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(adOff + 4), (uint)(dir.DataLbn + b)); // LBN - } - // File Entry body: 176-byte header (minus 16-byte tag) + L_EA(0) + L_AD bytes. - FinalizeTag(buf, 0, FeBodyHeader + 0 + lAd); + FinalizeTag(buf, 0, FeHeaderBodySize + lAd); output.Write(buf); } - private static void WriteFileFe(Stream output, int lbn, int fileSize, int dataLbn) { - var buf = new byte[Sector]; - WriteTag(buf, 0, 261, (uint)lbn); - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(20), 4); // ICB strategy type 4 - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(24), 1); // max entries 1 - // Permissions at FE offset 44 — see WriteDirectoryFe. - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(44), FilePermissions); - buf[27] = 5; // file type = file (regular) - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(48), 1); // file link count (ECMA-167 §14.9 offset) - BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(34), 0); // adType=0 short - BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(56), (ulong)fileSize); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(172), 8); // L_AD = 8 - var allocLen = Math.Max(fileSize, Sector); // at least one sector - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(176), (uint)allocLen); - BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(180), (uint)dataLbn); - FinalizeTag(buf, 0, FeBodyHeader + 0 + 8); + private static void WriteFileFe(Stream output, Node file) { + var buf = BeginFileEntry(file, fileType: 5, FilePermissions, linkCount: 1); + var lAd = WriteExtents(buf, 176, file.DataLbn, file.DataLength); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(172), (uint)lAd); + FinalizeTag(buf, 0, FeHeaderBodySize + lAd); output.Write(buf); } + + /// + /// Writes the short allocation descriptors covering + /// contiguous bytes from and returns how many + /// bytes of descriptors that took. A zero-length object gets none at all. + /// + private static int WriteExtents(byte[] buf, int offset, int firstLbn, long length) { + var written = 0; + var lbn = firstLbn; + var remaining = length; + while (remaining > 0) { + var take = Math.Min(remaining, MaxExtentBytes); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(offset + written), (uint)take); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(offset + written + 4), (uint)lbn); + written += 8; + remaining -= take; + lbn += (int)((take + Sector - 1) / Sector); + } + + return written; + } } diff --git a/Hawkynt.FileFormats.FileSystems/REFERENCE.md b/Hawkynt.FileFormats.FileSystems/REFERENCE.md index 11339d8c4..642ac19d3 100644 --- a/Hawkynt.FileFormats.FileSystems/REFERENCE.md +++ b/Hawkynt.FileFormats.FileSystems/REFERENCE.md @@ -13585,7 +13585,7 @@ Implements `IArchiveCreatable`, `IArchiveDefragmentable`, `IArchiveFormatOperati | `OptionsSchema` | `IReadOnlyList OptionsSchema { get; }` | Tunable knobs for UDF 2.01 creation. The natural per-volume knob is the PVD Volume Identifier (ECMA-167 §7.2.5) — Linux's udf driver surfaces this string as the volume label. Image geometry is auto-sized to fit the file content. | | `TarCompressionFormatId` | `string TarCompressionFormatId { get; }` | Gets the tar compression format id. | | `Add` | `void Add(Stream archive, IReadOnlyList inputs)` | Adds (or replaces) files at the root of an existing UDF image. Uses `UdfModifier` for true random-access I/O — only the Partition Descriptor sector, the root directory's File Entry sector, the FID extent, and the new file's FE + data sectors are touched. The 32 KiB system area, VRS, AVDP, LVD, and FSD are left untouched. | -| `CanAccept` | `bool CanAccept(ArchiveInputInfo input, out string reason)` | Performs the can accept operation. | +| `CanAccept` | `bool CanAccept(ArchiveInputInfo input, out string reason)` | Accepts any file the writer can address. Short allocation descriptors cap an extent below 2^30 bytes (OSTA UDF §2.3.10.1) and only so many of them fit in one File Entry, so a file past that ceiling is declined here rather than written as a volume nothing can read. | | `CreateFromStreams` | `void CreateFromStreams(Stream output, IEnumerable inputs, FormatCreateOptions options)` | Streaming creation. UDF descriptor CRCs (FID / File Entry / VDS tags) cover only the 16-byte tag bodies, NEVER file data, and the writer emits sectors strictly forward in LBN order — so each file's body can be streamed from `OpenStream` in 64 KiB chunks straight into the sequential output when its data block is reached. No buffering of the body is required and the output is byte-identical to `Create`. | | `Create` | `void Create(Stream output, IReadOnlyList inputs, FormatCreateOptions options)` | Performs the create operation. | | `Defragment` | `void Defragment(Stream archive)` | Performs the defragment operation. | @@ -13602,7 +13602,7 @@ Implements `IArchiveCreatable`, `IArchiveDefragmentable`, `IArchiveFormatOperati #### `UdfModifier` -In-place UDF (ECMA-167) modifier — true random-access editing without rebuilding the whole image. Targeted at images produced by `UdfWriter`: 2 KiB sectors, partition starting at LBA 257, short allocation descriptors, flat root directory, plain ECMA-167 (no Metadata partition, no VAT). What it touches per Add: One sector for the new file's File Entry (allocated at the partition tail).N sectors for the new file's data (also at the tail).The root directory's FID extent — appended into trailing sector padding, or extended via a second short_ad on the root File Entry.The root File Entry sector — info length and L_AD updated, tag re-CRC'd.The Partition Descriptor sector — partition length grown.Remove uses the FID Characteristics "deleted" flag (bit 2 = 0x04) per ECMA-167 §14.4.3, which is the canonical UDF tombstone. The dead FID's identifier bytes are zeroed and its tag is re-CRC'd; the file's FE sector and data extent are zero-wiped. The root extent is never compacted (preserving existing pointers and offsets).If the layout doesn't fit — e.g. the FID extent is at the very end of the image so its growth area is occupied — the modifier falls back to allocating fresh sectors at the partition tail and adding a second short_ad to the root FE. +In-place UDF (ECMA-167) modifier — true random-access editing without rebuilding the whole image. Targeted at images produced by `UdfWriter`: 2 KiB sectors, partition starting at LBA 257, short allocation descriptors, flat root directory, plain ECMA-167 (no Metadata partition, no VAT). What it touches per Add: One sector for the new file's File Entry (allocated at the partition tail).N sectors for the new file's data (also at the tail).The root directory's FID extent — appended into trailing sector padding, or extended via a second short_ad on the root File Entry.The root File Entry sector — info length and L_AD updated, tag re-CRC'd.Every Partition Descriptor — the main sequence's and the reserve sequence's copy — grown to the partition's new length.The volume's second anchor, re-recorded in its new last block.The Logical Volume Integrity Descriptor — size, free space and the file count it advertises.Remove uses the FID Characteristics "deleted" flag (bit 2 = 0x04) per ECMA-167 §14.4.3, which is the canonical UDF tombstone. The dead FID's identifier bytes are zeroed and its tag is re-CRC'd; the file's FE sector and data extent are zero-wiped. The root extent is never compacted (preserving existing pointers and offsets).If the layout doesn't fit — e.g. the FID extent is at the very end of the image so its growth area is occupied — the modifier falls back to allocating fresh sectors at the partition tail and adding a second short_ad to the root FE. | Member | Signature | Summary | | --- | --- | --- | @@ -13626,12 +13626,12 @@ Implements `IDisposable`. #### `UdfWriter` -Writes a minimal UDF 1.02 filesystem image (ECMA-167). Builds a real directory tree from slash-separated file paths, short allocation descriptors. Computes ECMA-167 §7.2.1 DescriptorCRC (CRC-16/CCITT, init=0, poly=0x1021, non-reflected) and TagChecksum for every descriptor tag so that strict readers (xorriso, Linux udf.ko, mkudffs fsck) accept the produced images. Layout: A directory's data is a sequence of File Identifier Descriptors (FID, tag 257). The first FID of every directory is the parent entry (Parent flag 0x08, zero-length identifier, ICB pointing at the parent FE). Every directory and file is a File Entry (FE, tag 261); directories carry file type 4, regular files file type 5. A subdirectory FID carries the Directory flag 0x02 and points at the child directory's FE. +Writes a UDF 2.01 volume image (ECMA-167 plus the OSTA UDF profile). Builds a real directory tree from slash-separated file paths using short allocation descriptors, and records every descriptor the standard's own tools look for: both volume descriptor sequences, both anchors, the unallocated space and implementation use descriptors, and a closed logical volume integrity descriptor. Layout (2048-byte logical blocks): A directory's data is a dense sequence of File Identifier Descriptors (FID, tag 257) — dense because ECMA-167 lets a FID span a logical block boundary and Linux's udf driver reads directory bytes as one uninterrupted run, so padding a FID onto the next block makes the directory unreadable. The first FID of every directory is the parent entry (Parent flag 0x08, zero-length identifier). Every directory and file is a File Entry (FE, tag 261); directories carry file type 4, regular files file type 5. | Member | Signature | Summary | | --- | --- | --- | | `UdfWriter` | `UdfWriter()` | | -| `VolumeIdentifier` | `string VolumeIdentifier { get; set; }` | ECMA-167 PVD Volume Identifier (dstring at PVD offset 24, 32 bytes). Linux's udf driver surfaces this as the volume label. Default "UDF Volume". Truncated to 31 bytes (ECMA-167 dstring length byte caps at 31). | +| `VolumeIdentifier` | `string VolumeIdentifier { get; set; }` | ECMA-167 PVD Volume Identifier (dstring at PVD offset 24, 32 bytes). Linux's udf driver surfaces this as the volume label. Default "UDF Volume". | | `AddFile` | `void AddFile(string name, byte[] data)` | Performs the add file operation. | | `AddStreamingFile` | `void AddStreamingFile(string name, long size, Func openStream)` | Adds a streaming file whose body is pulled from `openStream` in 64 KiB chunks while the image is written sequentially, never buffered as a `byte[]`. `size` drives the directory/file layout. | | `WriteTo` | `void WriteTo(Stream output)` | Writes the to to the supplied output. |