Trajectory is silently mirrored (rotation det = −1) in Release builds: -ffast-math miscompiles Eigen::Matrix4d::inverse()

[peitonglee]

Summary

When the project is built in Release mode, the estimated camera trajectory is silently mirrored (handedness flipped, rotation determinant = −1). The bug is caused by -ffast-math / -ffinite-math-only in the CMake Release flags, which miscompiles Eigen::Matrix4d::inverse() under GCC, flipping one axis of the rotation block.

This affects all front-ends (mono, RGB-D, with/without line segments) because every pose goes through a matrix inverse.

Environment

• OS: Ubuntu 22.04
• Compiler: GCC 11 (libstdc++ 6)
• Eigen: 3.4.0 (bundled)
• Dataset used for reproduction: TUM RGB-D fr1_desk
• Build flags (default Release): CMAKE_CXX_FLAGS_RELEASE contains -ffast-math

Symptoms

• Estimated trajectory is a mirror of the ground truth; SE3 alignment with the mirror hypothesis matches (~3 cm) while the proper (non-mirror) hypothesis fails (~30 cm).
• Quaternions in the output frame_trajectory.txt are non-unit (norm ≈ 1/√2 ≈ 0.7071) and contain a flipped sign in one component.
• Tracking still "works" internally because the flip is applied consistently to every pose — the map is self-consistent, only globally mirrored. This makes the bug easy to miss.

Root Cause

The CMake Release flags set -ffast-math. Under GCC, -ffast-math allows unsafe algebraic reassociation that breaks Eigen 3.4.0's hand-rolled 4×4 inverse (cofactor / Laplace expansion). The rotation block of the inverse cam_pose_cw.inverse() ends up with det = −1 instead of +1, i.e. one axis is flipped.

Direct instrumentation (first keyframe, which has an identity cam_pose_cw):

Built WITH -ffast-math:
  det(rot_wc)   = -1.000000
  trace(rot_wc) =  1.000000
  |q|           =  0.7071      <-- non-unit, one component sign-flipped

Built WITHOUT -ffast-math:
  det(rot_wc)   = +1.000000
  trace(rot_wc) =  3.000000
  |q|           =  1.000000     <-- correct

The inverse is used widely, e.g. src/.../util/trajectory_io.cc:

const Mat44_t cam_pose_wc = cam_pose_cw.inverse();  // det = -1 under -ffast-math

Reproduction

cmake -B build -DCMAKE_BUILD_TYPE=Release        # Release flags include -ffast-math
cmake --build build
./build/run_tum_rgbd_slam -v ./orb_vocab/orb_vocab.dbow2 \
    -d /path/to/fr1_desk -c ./example/tum_rgbd/TUM_RGBD_rgbd_1.yaml \
    --no-sleep
# frame_trajectory.txt is a mirror of ground truth

Proposed Fix

Remove -ffast-math / -ffinite-math-only from the Release flags. In CMakeLists.txt:

# BEFORE
set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} -O3 -ffast-math")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3 -ffast-math")

# AFTER
set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} -O3 -DNDEBUG")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3 -DNDEBUG")

-DNDEBUG is kept so the latent assert below does not break compilation (see note).

Verification (after fix)

On TUM fr1_desk (RGB-D), proper rigid SE3 alignment vs ground truth:

• posRMSE = 1.6 cm, scale = 0.9991
• handedness agreement (proper) = 92.6% (mirror hypothesis ≈ 0%)
• baseline mirror fit posRMSE = 29.4 cm (clearly wrong)

Same result for the line-segment binary run_tum_rgbd_slam_with_line.

Related latent bug (separate issue, for info)

src/PLPSLAM/data/map_database.cc line ~814:

assert(id == lm_line->id_);   // `Line` has member `_id`, not `id_` -> typo

In Release (-DNDEBUG) the assert is compiled out, so this never surfaces. It should be lm_line->_id. Keeping -DNDEBUG in Release is what currently hides it.

You can clone my repaired version from: https://github.com/peitonglee/Structure-PLP-SLAM-Reverse-repair

[PeterFWS]

Incredible! Thank you for your valuable investigation!