Weird slash appears when formatting exponential notation

[jagot]

When formatting some numbers (I have only seen this for 1e-10) using exponential notation, a wild slash appears, see below:

julia> using Format

julia> format("{1:.16g}", 1e-10)
"1.000000000000000/e-10"

julia> format("{1:.16g}", 1e-5)
"1.0000000000000000e-05"

julia> format("{1:.17g}", 1e-10)
"1.00000000000000000e-10"

julia> format("{1:.17g}", 1e-5)
"1.00000000000000000e-05"

julia> format("{1:.16e}", 1e-10)
"1.000000000000000/e-10"

julia> format("{1:.16e}", 1e-9)
"1.0000000000000000e-09"

julia> format("{1:.16e}", 1e-10)
"1.000000000000000/e-10"

julia> format("{1:.16e}", 2e-10)
"1.9999999999999996e-10"

(tmp) pkg> st
Status `/private/tmp/Project.toml`
  [1fa38f19] Format v1.3.7

As seen above, I can somehow avoid it by increasing to 17 decimals, but it seems slightly arbitary.

[R3G3N3R4T0R]

I have identified the issue as floating point error when normalizing the number for printing. In "src/fmtcore.jl", the function extract the radix digits by deducting the normalized number with itself truncated to integer. The problem is the clause that if the first digit (the truncated number) is 0 and the radix part is not exactly 0, the first digit is set as 0 and the radix part is reduced by 1.

Since your number normalize to 0.9999999999999999 instead of 1.0, the function tries to print the digit "-1" at the 16th radix digit (-1E-16), as printing is done using ASCII offsets, "/" is printed as it is 1 codepoint less than "0".

Your band aid fix circumvented it because it only bugs out if the floating point error rounds it to slightly less than 1. I don't think it happens to other numbers, but I need more time reading through the code to understand the intent of the clause to deduct instead of setting 0.

[ScottPJones]

I'll look at this this week (at JuliaCon right now), to see where this is coming from. A lot of the code came from Formatting.jl, and some came from Printf.jl, I'll have to see if I messed it up somehow trying to maintain it!

[R3G3N3R4T0R]

I think the core problem is in rounding. If I remove the rax line and adjust it I can get 1E-16 to work, but I am not sure of other implications. It usually isn't useful to have 16 sigdigits anyway since you can see floating point errors. Keep increasing the precision and eventually you can get a negative error large enough that you get InexactError in _digitchar, (030352b
) attempt to fix this but the error is much more fundamental to floating point arithmetic.

julia> format("{1:.18e}",1E-11)
ERROR: InexactError: trunc(UInt32, -174)

The clause I mentioned is added in (f068523), it seems to be trying to force u smaller than 1 due to rounding error to be cast as 1.xxxx since the integer part should never be 0 unless the number is 0. I am not sure if this is the right way to go, but trying to drop the first digit by passing through it again may incur more inaccuracies. Not sure if setting decv=0.0 will have other effects on other numbers.

[jagot]

I know very little about these kind of issues, but maybe it would be worthwhile to look what other, similar libraries have done. E.g. the C++ library {fmt} uses the so-called Dragonbox algorithm.

[R3G3N3R4T0R]

Not an update, but I found JuliaPlots/NumericIO.jl using the binary to decimal converter in Base.Ryu. Since I doubt any of us have the ability to implement Dragonbox or Schubfach, this is probably a better lead.