elf2uf2 on s390x: ERROR: Not an ELF file

[josch]

Hi,

I have this CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)
include(pico_sdk_import.cmake)
project(hello_world)
pico_sdk_init()

add_executable(hello_world hello.c)
target_link_libraries(hello_world pico_stdlib)
pico_add_extra_outputs(hello_world)

and this hello-world program (yes this program is not even making use of the pico sdk but the problem is also there if we add that):

#include <stdio.h>

int main() {
    printf("Hello, world!\n");
    return 0;
}

On s390x the build will fail like this:

$ make VERBOSE=1
[...]
/usr/bin/arm-none-eabi-objcopy -Oihex /home/josch/hello_world.elf hello_world.hex
/usr/bin/arm-none-eabi-objcopy -Obinary /home/josch/hello_world.elf hello_world.bin
/usr/bin/arm-none-eabi-objdump -h /home/josch/hello_world.elf > hello_world.dis
/usr/bin/arm-none-eabi-objdump -d /home/josch/hello_world.elf >> hello_world.dis
elf2uf2/elf2uf2 /home/josch/hello_world.elf hello_world.uf2
ERROR: Not an ELF file
make[2]: *** [CMakeFiles/hello_world.dir/build.make:812: hello_world.elf] Error 254
make[2]: *** Deleting file 'hello_world.elf'
make[2]: Leaving directory '/home/josch'
make[1]: *** [CMakeFiles/Makefile2:1491: CMakeFiles/hello_world.dir/all] Error 2
make[1]: Leaving directory '/home/josch'
make: *** [Makefile:91: all] Error 2

I added some printfs into tools/elf2uf2/main.cpp to investigate the values of ELF_MAGIC and eh_out.common.magic in read_and_check_elf32_header and got this:

ELF_MAGIC = 464c457f
eh_out.common.magic = 7f454c46

There seems to be an endian problem somewhere? The file utility seems to think that my hello_world.elf has the correct magic:

$ file hello_world.elf
hello_world.elf: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, with debug_info, not stripped

Any ideas?

[lurch]

I guess this would be fixed by raspberrypi/pico-sdk#329 ?

[josch]

Thank you! In the mean-time I had this patch cooked up:

--- a/tools/elf2uf2/main.cpp	2023-06-13 22:27:33.000000000 +0000
+++ b/tools/elf2uf2/main.cpp	2024-06-05 08:20:56.156138030 +0000
@@ -11,6 +11,7 @@
 #include <cstring>
 #include <cstdarg>
 #include <algorithm>
+#include <endian.h>
 #include "boot/uf2.h"
 #include "elf.h"
 
@@ -101,6 +103,20 @@
     if (1 != fread(&eh_out, sizeof(eh_out), 1, in)) {
         return fail(ERROR_READ_FAILED, "Unable to read ELF header");
     }
+    eh_out.common.magic    = le32toh(eh_out.common.magic);
+    eh_out.common.type     = le16toh(eh_out.common.type);
+    eh_out.common.machine  = le16toh(eh_out.common.machine);
+    eh_out.common.version2 = le32toh(eh_out.common.version2);
+    eh_out.entry           = le32toh(eh_out.entry);
+    eh_out.ph_offset       = le32toh(eh_out.ph_offset);
+    eh_out.sh_offset       = le32toh(eh_out.sh_offset);
+    eh_out.flags           = le32toh(eh_out.flags);
+    eh_out.eh_size         = le16toh(eh_out.eh_size);
+    eh_out.ph_entry_size   = le16toh(eh_out.ph_entry_size);
+    eh_out.ph_num          = le16toh(eh_out.ph_num);
+    eh_out.sh_entry_size   = le16toh(eh_out.sh_entry_size);
+    eh_out.sh_num          = le16toh(eh_out.sh_num);
+    eh_out.sh_str_index    = le16toh(eh_out.sh_str_index);
     if (eh_out.common.magic != ELF_MAGIC) {
         return fail(ERROR_FORMAT, "Not an ELF file");
     }
@@ -154,6 +175,16 @@
         if (eh.ph_num != fread(&entries[0], sizeof(struct elf32_ph_entry), eh.ph_num, in)) {
             return fail_read_error();
         }
+	for (int i = 0; i < eh.ph_num; i++) {
+		entries[i].type = le32toh(entries[i].type);
+		entries[i].offset = le32toh(entries[i].offset);
+		entries[i].vaddr = le32toh(entries[i].vaddr);
+		entries[i].paddr = le32toh(entries[i].paddr);
+		entries[i].filez = le32toh(entries[i].filez);
+		entries[i].memsz = le32toh(entries[i].memsz);
+		entries[i].flags = le32toh(entries[i].flags);
+		entries[i].align = le32toh(entries[i].align);
+	}
     }
     return 0;
 }

I also rebased the patch from raspberrypi/pico-sdk#329 onto my local pico-sdk version and:

[100%] Linking CXX executable hello_world.elf
[100%] Built target hello_world

It works fine, thank you!

[will-v-pi]

Transfering this over to picotool, as that's where the elf2uf2 functionality is now

I believe that I have fixed this (in a different way to raspberrypi/pico-sdk#329) for the picotool uf2 convert command which replaces elf2uf2 - however I haven't tried fixing it for the rest of picotool as I don't have a big-endian system to test it on.

@josch Could you test the new picotool and see if this issue has actually been fixed for big-endian systems?

[josch]

Could you test the new picotool and see if this issue has actually been fixed for big-endian systems?

Thank you for working on this! I tried building the new pico-sdk and picotool 2.0 but there seems to be a build dependency loop here and I'm unsure how I am supposed to resolve it. When building pico-sdk 2.0 I get:

CMake Warning at tools/Findpicotool.cmake:28 (message):
  No installed picotool with version 2.0.0 found

And when trying to build picotool I get:

CMake Error at CMakeLists.txt:23 (message):
  Raspberry Pi Pico SDK version 2.0.0 (or later) required.

How am I as a distribution packager supposed to resolve this in practice? Thank you!

[will-v-pi]

Picotool doesn’t require a built SDK, it just requires some of the SDK header files, so you should be able to build picotool first and then build the SDK. I’m not familiar with packaging, but I hope that should be possible?

[will-v-pi]

Additionally, if you’re looking into packaging pico-sdk and picotool then I’d be happy to answer any other questions you have about them

[josch]

Picotool doesn’t require a built SDK, it just requires some of the SDK header files, so you should be able to build picotool first and then build the SDK. I’m not familiar with packaging, but I hope that should be possible?

But if building picotool requires SDK header files, then i need to build the SDK first to create a package containing these header files. But I cannot build the SDK without picotool.

If there is no way around this, then I will have to merge both source packages into one because either one cannot be build without having the source of the other available.

Additionally, if you’re looking into packaging pico-sdk and picotool then I’d be happy to answer any other questions you have about them

It's already packaged:

• https://tracker.debian.org/pkg/pico-sdk
• https://tracker.debian.org/pkg/picotool

But with version 2.0 there seems to be a circular dependency between the two. Maybe there is some cmake flag i can pass to break this cycle?

[will-v-pi]

You can pass -DPICO_NO_PICOTOOL=1 to the SDK build, and it will skip anything that requires picotool, which for building the SDK is just the UF2s for the test programs. So that flag should be fine for just building the docs.

Also, looking at that package, you don’t currently have a pioasm binary - it might be helpful to build and install that (either as part of the sdk, or a separate package), which can be done by cding into the tools/pioasm directory, then doing a normal CMake build and install from there. Without the pioasm binary, each project using PIO will need to build the pioasm tool separately

[josch]

Thank you, I implemented both your suggestions. That was very helpful, thanks!

I'll upload pico-sdk 2.0 to experimental soon and am now stuck with also updating picotool to 2.0: #139

[josch]

Additionally, if you’re looking into packaging pico-sdk and picotool then I’d be happy to answer any other questions you have about them

If you don't mind I'd like to abuse this issue for another packaging related question. Both pico-sdk as well as picotool are now in Debian unstable in version 2.0:

• https://tracker.debian.org/pkg/pico-sdk
• https://tracker.debian.org/pkg/picotool

Now suppose somebody wants to use that package in their project. Lets look at this minimal reproducer:

cmake_minimum_required(VERSION 3.13)
include(/usr/src/pico-sdk/pico_sdk_init.cmake)
project(hello_world)
pico_sdk_init()

add_executable(hello_world hello.c)
target_link_libraries(hello_world pico_stdlib)
pico_add_extra_outputs(hello_world)

#include <stdio.h>
#include "pico/stdlib.h"

int main() {
    setup_default_uart();
    printf("Hello, world!\n");
    return 0;
}

But trying to build this will fail with:

CMake Warning at /usr/src/pico-sdk/tools/Findpicotool.cmake:28 (message):
  No installed picotool with version 2.0.0 found - building from source

  It is recommended to build and install picotool separately, or to set
  PICOTOOL_FETCH_FROM_GIT_PATH to a common directory for all your SDK
  projects
Call Stack (most recent call first):
  /usr/src/pico-sdk/tools/CMakeLists.txt:138 (find_package)
  /usr/src/pico-sdk/src/cmake/on_device.cmake:33 (pico_init_picotool)
  /usr/src/pico-sdk/src/rp2040/boot_stage2/CMakeLists.txt:57 (pico_add_dis_output)
  /usr/src/pico-sdk/src/rp2040/boot_stage2/CMakeLists.txt:101 (pico_define_boot_stage2)

So it cannot find picotool. The picotool Debian package installs the cmake files:

drwxr-xr-x root/root         0 2024-09-09 22:54 ./usr/lib/aarch64-linux-gnu/cmake/
drwxr-xr-x root/root         0 2024-09-09 22:54 ./usr/lib/aarch64-linux-gnu/cmake/picotool/
-rw-r--r-- root/root        96 2024-09-09 22:54 ./usr/lib/aarch64-linux-gnu/cmake/picotool/picotoolConfig.cmake
-rw-r--r-- root/root      2303 2024-09-09 22:54 ./usr/lib/aarch64-linux-gnu/cmake/picotool/picotoolConfigVersion.cmake
-rw-r--r-- root/root       745 2024-09-09 22:54 ./usr/lib/aarch64-linux-gnu/cmake/picotool/picotoolTargets-none.cmake
-rw-r--r-- root/root      4447 2024-09-09 22:54 ./usr/lib/aarch64-linux-gnu/cmake/picotool/picotoolTargets.cmake

So why does ./tools/Findpicotool.cmake fail to find the installed picotool and tries to build it from a git clone instead?

Thanks!

[will-v-pi]

Ah yes - you'll need commit 6ad9c23 as well, which changes the install dir to /usr/lib/cmake... rather than /usr/lib/aarch64-linux-gnu/cmake... - because CMake find_package which we use to find picotool doesn't search the architecture-specifc libdirs for the picotoolConfig.cmake file (see #117)

You'll probably also want to take raspberrypi/pico-sdk#1865 as well, which applies the same fix to pioasm

[josch]

Aha! raspberrypi/pico-sdk#1865 reminds me of a very similar fix that I already did to pico-sdk to fix the pioasm cmake installation path:

https://sources.debian.org/src/pico-sdk/2.0.0-1/debian/patches/pioasm.cmake.patch/

As you can see from that patch, I chose to let it install to ${CMAKE_INSTALL_DATAROOTDIR} which is share by default instead of hard-coding lib. I'm unsure what is correct. In Debian I see architecture-independent cmake files in both /usr/share/cmake as well as in /usr/lib/cmake. I don't know if there is a document somewhere that prescribes whether $prefix/lib or $prefix/share should be used for arch-indep cmake files?

In any case, I'll apply your patch and report back once everything is built. Thanks!

[josch]

Okay, no that picotool 2.0 and pico-sdk 2.0 are both finally in Debian and building on all architectures, lets get back to the original problem of s390x. Right now, the test script, which compiles a "hello world" program as I posted above, succeeds on all architectures except for s390x: https://ci.debian.net/packages/p/pico-sdk/testing/s390x/51562659/

103s [100%] Linking CXX executable hello_world.elf
103s ERROR: ELF File Read from 0x100000 with size 0x10000 exceeds the file size 0x58fdc
103s make[2]: *** [CMakeFiles/hello_world.dir/build.make:973: hello_world.elf] Error 254
103s make[2]: *** Deleting file 'hello_world.elf'
103s make[1]: *** [CMakeFiles/Makefile2:1713: CMakeFiles/hello_world.dir/all] Error 2
103s make: *** [Makefile:91: all] Error 2

I can have a closer look on real s390x hardware later, but maybe you already have some theories that I could check?

Not sure if this is picotool or pico-sdk related?