The byte order mark doesn't skipped by windmc when reading message text file (MC)

[danil-kondr2016]

I want to report a bug found by me in Binutils' windmc utility
which I have already reported at Sourceware's Bugzilla
(https://sourceware.org/bugzilla/show_bug.cgi?id=34359).
When I save the message file in UTF-8 with BOM or and try
compile the message text file with windmc message compiler,
I receive the following error:

C:/Users/danko/source/repos/PostInstallWizard $ cmake --build build
[0/1] Re-running CMake...-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: C:/Users/danko/source/repos/PostInstallWizard/build

[1/18] Generating message resources
FAILED: [code=1] PostInstallWizard/MSG00409.bin PostInstallWizard/MSG00419.bin PostInstallWizard/Messages.rc C:/Users/danko/source/repos/PostInstallWizard/PostInstallWizard/Messages.h C:/Users/danko/source/repos/PostInstallWizard/build/PostInstallWizard/MSG00409.bin C:/Users/danko/source/repos/PostInstallWizard/build/PostInstallWizard/MSG00419.bin C:/Users/danko/source/repos/PostInstallWizard/build/PostInstallWizard/Messages.rc
C:\WINDOWS\system32\cmd.exe /C "cd /D C:\Users\danko\source\repos\PostInstallWizard\build\PostInstallWizard && D:\w64devkit\bin\windmc.exe -h C:/Users/danko/source/repos/PostInstallWizard/PostInstallWizard -r C:/Users/danko/source/repos/PostInstallWizard/build/PostInstallWizard -c -C65001 C:/Users/danko/source/repos/PostInstallWizard/PostInstallWizard/Messages.mc"
In Messages.mc at line 1: parser: syntax error.
In Messages.mc at line 1: fatal: syntax error.
ninja: build stopped: subcommand failed.

This bug also presents when I try to save file in UTF-16LE and compile it with windmc
since BOM is recognized as invalid symbol.

I propose following patch for this case and if needed, I may create a pull request into
w64devkit tree but I want to fix this bug in Binutils' upstream:

--- a/binutils/mclex.c
+++ b/binutils/mclex.c
@@ -379,7 +379,9 @@
       return MCLINE;
     }

-  while ((ch = input_stream_pos[0]) <= 0x20)
+  /* Skip U+FEFF symbol at start */
+  while (((ch = input_stream_pos[0]) <= 0x20)
+         || ((ch = input_stream_pos[0]) == 0xFEFF))
     {
       if (ch == 0)
        return -1;

[Peter0x44]

That patch doesn't look correct to me, though it probably does fix the issue.

This change modifies the entire loop dealing with whitespace, it doesn't just skip a BOM at the start. It's checking at every single token boundry. The comment is for sure wrong.

I think the file has already been decoded by the time the lexer sees it, so maybe the correct fix isn't there anyway. It should probably be checked when the file is opened, but before it gets decoded.

[skeeto]

The windmc-bom branch (819813a, I don't know why it didn't ping this issue) has a better patch. Build it yourself or try a CI build:
https://github.com/skeeto/w64devkit/actions/runs/30713576535

And let us know if that works for you.