ENH: float configuration limited to 1e-13 in log space

[sdtemple]

Description

This is a feature request for smaller floating point values. The Float hyperparameter with log=True encounters numerical precision issues when sampling very small positive values, specifically those below approximately 5e-14. This limitation prevents users from properly tuning hyperparameters that have scientifically valid ranges extending to very small values. For instance, the rcond parameter in numpy.linalg.pinv has a default value of 1e-15.

Current Behavior

import ConfigSpace as CS
from ConfigSpace import ConfigurationSpace, Float

cs = CS.ConfigurationSpace(seed=42)

rcond = CS.Float(
        name="rcond",
        bounds=(1e-15, 1e-2),
        log=True)

cs.add(
    [
        rtol,
    ]
)

configs = cs.sample_configuration(size=5000)
# ValueError: Hyperparameter 'rcond' has illegal settings

Version

1.2.2

Interest in helping

Yes.

[eddiebergman]

Hi @sdtemple , out of curiosity, do you care about the floating point precision, or just the exponent? You could in theory sample an integer in the range you care about, for example, (-20, -3) and then convert that to 17 possible values from (1e-20, 1e-3)

[sdtemple]

We do care about the floating point precision, up to a reasonable point, but I agree that you can sample the configuration space in log and transform with existing python ** operation or numpy functions. Sampling in log space has been my solution thus far.

I certainly understand if it would be a pain to add this floating point precision as opposed to using this simple log solution.

[tylerjereddy]

@eddiebergman on the other hand, if we frame this as "we're willing to help" with the angle that we'd really like to be able to explore the full IEEE 754 (double) floating point lines, is that something that ConfigSpace team is open to? For example, when trying to break state of the art records for estimator performance, it may be helpful for the community to be able to leverage algorithms that first do a coarse search over log space, but that still have the ability to "zoom in" at full IEEE 754 spacing once the general region of interest is identified (rather than having the user hardcode several different searches manually).

Is the big blocker related to JSON data storage per #377? Is there a roadmap for this project? Another project to consider if we're really interested in this kind of thing in the future?