Do we need complex modern optimization algorithms?
The "CMA" part in "CMAES" solves badly scaled non-separable cost functions (ill-conditioning), see e.g. Issue 356. However, if one's variables are proper, the ES part is literally this code:
import numpy as np
class CWALK:
def __init__(self, D, x0=None, sigma=1.0, lam=None, rng=None):
self.D = D
self.rng = np.random.default_rng() if rng is None else rng
if x0 is None:
raise ValueError("x0 must be provided by the driver script.")
self.xmean = np.asarray(x0, dtype=float).copy()
self.sigma = float(sigma)
self.lam = 100 * D if lam is None else int(lam)
self.mu = self.lam // 2
self.best_x = self.xmean.copy()
self.best_f = np.inf
# ------------------------------------------------------------
# ASK
# ------------------------------------------------------------
def ask(self):
self.Z = self.rng.standard_normal((self.lam, self.D))
X = self.xmean + self.sigma * self.Z
return X
# ------------------------------------------------------------
# TELL
# ------------------------------------------------------------
def tell(self, X, fitness, sigma):
X = np.asarray(X, dtype=float)
fitness = np.asarray(fitness, dtype=float)
order = np.argsort(fitness)
# best-so-far
if fitness[order[0]] < self.best_f:
self.best_f = float(fitness[order[0]])
self.best_x = X[order[0]].copy()
# update mean
self.xmean = np.mean(X[order[:self.mu]], axis=0)
self.normz = np.linalg.norm(np.mean(self.Z[order[:self.mu]], axis=0))
# update sigma
if sigma is not None:
self.sigma = sigmaBelieve it or not, the code solves the rotated Lunacek bi-Rastrigin (F24 BBOB-2009). This is where all the intricate Newton/Powell methods fail, including the MCS and Nomad.
git clone https://github.com/aabbtree77/cwalk.git
cd cwalk
uv venv
source .venv/bin/activate
uv pip install \
numpy \
scipy \
matplotlib \
cma \
coco-experiment \
ipython \
minionpy
python3 test_bbob2009.py
Problem
----------------------------------------
Backend : BBOB
Name : bbob_f024_i01_d40
D : 40
Bounds : [-5.0, 5.0] for every coordinate
fopt : 102.61
Created : 2026-07-27 00:29:20
Initial sigma : 1
evals= 100000 best_f=5.299968e+02 error=4.273868e+02 sigma=8.017e-01
evals= 200000 best_f=4.000308e+02 error=2.974208e+02 sigma=6.368e-01
evals= 300000 best_f=3.666502e+02 error=2.640402e+02 sigma=5.058e-01
evals= 400000 best_f=3.473649e+02 error=2.447549e+02 sigma=4.018e-01
evals= 500000 best_f=3.265643e+02 error=2.239543e+02 sigma=3.192e-01
evals= 600000 best_f=3.240048e+02 error=2.213948e+02 sigma=2.535e-01
evals= 700000 best_f=3.240048e+02 error=2.213948e+02 sigma=2.014e-01
evals= 800000 best_f=3.240048e+02 error=2.213948e+02 sigma=1.600e-01
evals= 900000 best_f=3.240048e+02 error=2.213948e+02 sigma=1.271e-01
evals= 1000000 best_f=3.240048e+02 error=2.213948e+02 sigma=1.009e-01
evals= 1100000 best_f=3.240048e+02 error=2.213948e+02 sigma=8.017e-02
evals= 1200000 best_f=3.240048e+02 error=2.213948e+02 sigma=6.368e-02
evals= 1300000 best_f=3.121135e+02 error=2.095035e+02 sigma=5.058e-02
evals= 1400000 best_f=3.013288e+02 error=1.987188e+02 sigma=4.018e-02
evals= 1500000 best_f=2.701156e+02 error=1.675056e+02 sigma=3.192e-02
evals= 1600000 best_f=2.091583e+02 error=1.065483e+02 sigma=2.535e-02
evals= 1700000 best_f=1.721276e+02 error=6.951763e+01 sigma=2.014e-02
evals= 1800000 best_f=1.526473e+02 error=5.003727e+01 sigma=1.600e-02
evals= 1900000 best_f=1.316534e+02 error=2.904339e+01 sigma=1.271e-02
evals= 2000000 best_f=1.206309e+02 error=1.802090e+01 sigma=1.009e-02
evals= 2100000 best_f=1.119537e+02 error=9.343675e+00 sigma=8.017e-03
evals= 2200000 best_f=1.104614e+02 error=7.851370e+00 sigma=6.368e-03
evals= 2300000 best_f=1.074570e+02 error=4.847032e+00 sigma=5.058e-03
evals= 2400000 best_f=1.054741e+02 error=2.864093e+00 sigma=4.018e-03
evals= 2500000 best_f=1.047269e+02 error=2.116855e+00 sigma=3.192e-03
evals= 2600000 best_f=1.039641e+02 error=1.354105e+00 sigma=2.535e-03
evals= 2700000 best_f=1.035237e+02 error=9.137222e-01 sigma=2.014e-03
evals= 2800000 best_f=1.032888e+02 error=6.788106e-01 sigma=1.600e-03
evals= 2900000 best_f=1.031278e+02 error=5.178318e-01 sigma=1.271e-03
evals= 3000000 best_f=1.029578e+02 error=3.478413e-01 sigma=1.009e-03
evals= 3100000 best_f=1.028953e+02 error=2.853422e-01 sigma=8.017e-04
evals= 3200000 best_f=1.028682e+02 error=2.582219e-01 sigma=6.368e-04
evals= 3300000 best_f=1.028322e+02 error=2.222083e-01 sigma=5.058e-04
evals= 3400000 best_f=1.028317e+02 error=2.216639e-01 sigma=4.018e-04
evals= 3500000 best_f=1.028190e+02 error=2.090154e-01 sigma=3.192e-04
evals= 3600000 best_f=1.028112e+02 error=2.012098e-01 sigma=2.535e-04
evals= 3700000 best_f=1.028073e+02 error=1.973345e-01 sigma=2.014e-04
evals= 3800000 best_f=1.028049e+02 error=1.948547e-01 sigma=1.600e-04
evals= 3900000 best_f=1.028034e+02 error=1.934189e-01 sigma=1.271e-04
evals= 4000000 best_f=1.028017e+02 error=1.917442e-01 sigma=1.009e-04
Finished
----------------------------------------
Completed evaluations : 4000000
Requested budget : 4000000
Best f : 1.028017441860e+02
fopt : 1.026100000000e+02
Error : 1.917441860190e-01
Progress saved to : progress_bbob2009_f24.csv
It takes 1e4xD evals to reach 0.2% relative error.
- Reduce budget 10x, reduce lambda 10x, relative error will increase 10x.
- Increase budget 10x, increase lambda 10x, relative error will decrease 100x!
For 1e7xD evals the relative error is still O(1e-5).
lambda=D does not reach the global optimum at all. Anything interesting starts with lambda=10D.
Restart to avoid adversarial initial points. Restarting does not improve precision/convergence. However, it is essential: unlike in CMAES, a zero does not lead to the optimum.
Normality is not essential, but other distributions do not improve optmization.
One can reach relative error O(1e-5) with
self.Z = self.rng.laplace(0.0, 1.0, (self.lam, self.D))or even uniform distribution:
self.Z = self.rng.uniform(-3.0, 3.0, (self.lam, self.D))Uniformity within [-5.0, 5.0] will still work, but [-1.0, 1.0] won't. The scale in the Laplace distribution can go up to 3.0..4.0, but no further.
The choice of the final sigma value at the end of the budget, be it 1e-4 or 1e-6, is not too critical. The choice of the initial sigma value is. For very large budgets sigma can be tiny and constant, otherwise we go with 10% of the biggest coordinate range (from box constraints).
Adding random sigma bursts during the optimization does not improve anything.
Expect to solve a good half of the whole BBOB-2009 with the ES (if not everything except F2, F10 - F14, but these can be done with scipy BFGS).
python3 test_cec2022.py
Problem
----------------------------------------
Backend : CEC2022
Name : CEC2022 f12
D : 20
Bounds : [-100.0, 100.0] for every coordinate
fopt : 2700.0
Created : 2026-07-27 00:53:02
Initial sigma : 20
evals= 100000 best_f=3.078020e+03 error=3.780203e+02 sigma=1.274e+01
evals= 200000 best_f=2.995935e+03 error=2.959349e+02 sigma=8.036e+00
evals= 300000 best_f=2.986139e+03 error=2.861390e+02 sigma=5.070e+00
evals= 400000 best_f=2.986139e+03 error=2.861390e+02 sigma=3.199e+00
evals= 500000 best_f=2.986139e+03 error=2.861390e+02 sigma=2.019e+00
evals= 600000 best_f=2.985843e+03 error=2.858433e+02 sigma=1.274e+00
evals= 700000 best_f=2.985738e+03 error=2.857379e+02 sigma=8.036e-01
evals= 800000 best_f=2.985697e+03 error=2.856973e+02 sigma=5.070e-01
evals= 900000 best_f=2.985667e+03 error=2.856670e+02 sigma=3.199e-01
evals= 1000000 best_f=2.985649e+03 error=2.856490e+02 sigma=2.019e-01
evals= 1100000 best_f=2.985646e+03 error=2.856463e+02 sigma=1.274e-01
evals= 1200000 best_f=2.985643e+03 error=2.856425e+02 sigma=8.036e-02
evals= 1300000 best_f=2.985642e+03 error=2.856422e+02 sigma=5.070e-02
evals= 1400000 best_f=2.985642e+03 error=2.856419e+02 sigma=3.199e-02
evals= 1500000 best_f=2.985642e+03 error=2.856417e+02 sigma=2.019e-02
evals= 1600000 best_f=2.985642e+03 error=2.856417e+02 sigma=1.274e-02
evals= 1700000 best_f=2.985642e+03 error=2.856416e+02 sigma=8.036e-03
evals= 1800000 best_f=2.985642e+03 error=2.856416e+02 sigma=5.070e-03
evals= 1900000 best_f=2.985642e+03 error=2.856416e+02 sigma=3.199e-03
evals= 2000000 best_f=2.985642e+03 error=2.856416e+02 sigma=2.019e-03
Finished
----------------------------------------
Completed evaluations : 2000000
Requested budget : 2000000
Best f : 2.985641603543e+03
fopt : 2.700000000000e+03
Error : 2.856416035435e+02
Progress saved to : progress_cec2022_f12.csvMost of the state of the art is within 10% from the optimum. I have not seen any algorithm to go below 2900.
"F10 is the Ellipsoidal Function (a high-conditioning, unimodal function). It is hard to optimize because it features an extreme condition number (around 1e6) combined with non-separability, meaning its axes are rotated and scale at vastly different rates."
The ES becomes brittle with ill-conditioning. It still solves these problems, but one needs to increase the budget 1000x, say to a billion evals.
"A very rough rule of thumb is that without CMA, the number of evaluations are proportional to the condition number..." - Nikolaus Hansen, Issue 356.
That number can be proprotional to the condition number squared... The ES reaches f = -29.5 (when fopt = -54.94) on F10 BBOB-2009 in 1B evals with a constant step size 1e-3. After 1M evals it is still at f = 2.61e+07...
After some more thorough testing, see Minion Issue 11, it is tempting to resort to BIPOP-aCMAES or ARRDE.
So this is all about multimodality and ill-conditioning.
The figure above indicates that a large part of BBOB-2009, if not entirely the whole benchmark, can be covered by running any solid Newton (scipy SLSQP/BFGS) with the ES and choosing the better result.
CEC-2017 is a bigger challenge as there are a lot of functions which are both: multimodal and ill-contioned. Except for F22, F24, and F27, F20 - F30 are beyond any known method if we require an optimizer to get close to the global optimum with, say, 1% relative error in 1B evals.
I propose the following benchmark to compress the whole BBOB-2009 and CEC-2017:
| Algorithm | F10 BBOB-2009 | F24 BBOB-2009 | F24 CEC-2017 |
| ------------ | ------------- | ------------- | -------------- |
| ES | >1B | <10M | >200M (f=2800) |
| BIPOP-aCMAES | <50K | <10M | >200M (f=2500) |
| ARRDE | <500K | >200M | >200M (f=2400) |One could add F7 BBOB-2009 to remove pure Newton/gradient methods.
-
ES: wipes the floor with Newton/Powell on Rastrigin-like multimodals. It works only with mild condition numbers (up to ~1000, solves F18 BBOB-2009). It is sensitive w.r.t. starting points and initial sigma.
-
BIPOP-aCMAES (pycma CMAES), used to be the best, fails on F24 - F30 CEC-2017 when there is no single coordinate system to unrotate.
-
ARRDE: pushes the frontier, but demands C++ and budgets larger than 1e7xD to differentiate itself from pycma CMAES. It completely solves F24 CEC-2017 (!), yet cannot nail F25 CEC-2017. It is still better than CMAESes even on the F25: ARRDE f = 2800, BIPOP-aCMAES f = 2900. Notably, the ARRDE sustains ill-conditioning without matrices.
RDEx-SOP is a winner of CEC-2025, but it is tuned for tiny budgets (2e4xD evals). It already has improvements, alternatives.
-
Sichen Tao et al. (2026) RDEx-SOP: Exploitation-Biased Reconstructed Differential Evolution for Fixed-Budget Bound-Constrained Single-Objective Optimization
-
Dikshit Chauhan (2026) DE-2LS: Differential Evolution with Late-Stage local-search for Unconstrained Single-Objective Numerical Optimization
-
Dikshant et al. (2026) RDEx-CASK: Cauchy Mutation, Archive, and Stagnation Kick for RDEx-CSOP
-
Ryoji Tanabe and Alex Fukunaga (2020) How Far Are We From an Optimal, Adaptive DE?
-
Lots of CMAES complications exist, but I could not get anything from them so far, e.g.
Dimitar Nedanovski et al. (2026) MSC-CMA-ES: Structure-Aware Restarts for CMA-ES via Cyclic Nearest-Better Basin Discovery, Github
It does not reach f = 2400 on F24 CEC-2017 at all and does not look any different than BIPOP-aCMAES, despite the paper hinting that it could be interesting on the CEC-2017 composites. Very slow even with the C++ acceleration.
Default parameters, SEED = 20260825, F24 CEC-2017 D=20 got precisely f = 2500 in 200M evals, which took about 5 hours to run (a single optimization) on i7 gen4 16GB RAM. The C++ acceleration is only for clustering, pycma CMAES runs inside MSC-CMA-ES.
-
Another one bites the dust:
Khoirul Faiq Muzakka et al. (2026) RCMAES: A Robust CMA-ES Variant for CEC2026 Competition
No difference, except that it is much faster to test than pycma and MSC-CMA-ES and is integrated into Minion.
-
LLMs are everywhere now. This one was quite early and used local minimal models to "explain" concrete optimization results after the run. This is not very useful per se, but might stimulate some thinking towards embracing a brave new world:
Jill Baumann and Oliver Kramer (2024) Towards Explainable Evolution Strategies with Large Language Models
-
Endless hopeless theory, e.g. indicating that the population size should be O(sqrt(D)xlog(D)):
Lisa Schönenberger and Hans-Georg Beyer (2023) On a Population Sizing Model for Evolution Strategies Optimizing the Highly Multimodal Rastrigin Function
-
Simplifications exist, but I would not recommend them, e.g.
Zhenhua Li and Qingfu Zhang (2017) A Simple Yet Efficient Rank One Update for Covariance Matrix Adaptation
See pycma's Issue 356 for some of it in action, also consider adjusting the CSA according to pycma Issue 231.
The problem is, for any such simplification, everything starts anew, e.g. the rank one algorithm is too sensitive/unreliable w.r.t. starting points and initial step sizes on F10 BBOB-2009, while pycma brings no such trouble. The rank one update also does not work with larger lambdas as its simplistic CSA blows up the step size.
None of this is valuable as we simply lose years of testing and tuning present in pycma. This is why I would also not recommend any custom implementation of CMAESes including the ones by Minion.
Any simplification should be tested on every BBOB-2009 function one by one, with different step sizes, initial points, lambdas.
scipy includes an algorithm called "dual annealing" (DA) which runs BFGS as local search. Scroll down this code for all the references. DA got visible first in the R community.
I did not get anything from DAs on CEC2017 F24 - F30 in D=20. Also tried this code to no avail.
Minion includes one interesting comparison between the ARRDE, numerous BFGS implementations, and two DA implementations. It turns out that Minion's DA is worse than scipy DA, except on F17 and F26 (CEC-2017). The ARRDE is clearly better than anything on: F10, F12, F17 (somewhat), F21, F22, F24, F26, F28, and F30. However, in the rest of the cases DAs are close and on F25 scipy DA = 2600 (!), the ARRDE and the rest are close and only around 2900. It is the first time I see the problem where the ARRDE could be clearly worse.
Minion's result in D=10 depends on the starting point and D=10 does not generalize to D=20 at all. According to Minion's notebook, the ARRDE solves F26 CEC-2017 in D=10 in fewer than 100K evals (reaching 2600). In my runs, for the zero starting point, SEED = 20260815, the ARRDE reaches only 2800 in 2B evals (F26 CEC-2017 D=20). Night and day.
Nowadays it is much faster to git clone and test an algorithm than to decipher any terse report.
One exception is Baeysian Optimization (BO) as it is complex and annoyingly slow to run. I would avoid this domain entirely as those tiny budgets lack stability, and there is no convergence/critical mass on any existing algorithm since 1970s. Use scipy SLSQP/BFGS with pycma CMAES, it is very frugal with evals and far more developed than any BO.
-
Youssef Diouane et al. (2022) TREGO: a Trust-Region Framework for Efficient Global Optimization
-
Zachary Hoffman and Steve Huntsman (2022) Benchmarking an algorithm for expensive high-dimensional objectives on the bbob and bbob-largescale testbeds
-
Ryoji Tanabe (2022) Benchmarking the Hooke-Jeeves Method, MTS-LS1, and BSrr on the Large-scale BBOB Function Set
-
Nikolaus Hansen (2019) A Global Surrogate Assisted CMA-ES, pycma (github), pycma Issue 356
-
Nikolaus Hansen at al. (2019) Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions
-
Konstantinos Varelas (2019) Benchmarking Large Scale Variants of CMA-ES and L-BFGS-B on the bbob-largescale Testbed
-
Aurore Blelly at al. (2018) Stopping Criteria, Initialization, and Implementations of BFGS and their Effect on the BBOB Test Suite
Early algorithms did not survive the test of time. Analysis, boundary handling did.
-
H. H. Rosenbrock (1960) An Automatic Method for Finding the Greatest or Least Value of a Function
-
R. Fletcher and M.J.D. Powell (1963) A Rapidly Convergent Descent Method for Minimization
-
L.A. Rastrigin (1965) Solution of inverse problems by statistical optimization methods
-
M.J. Box (1966) A Comparison of Several Current Optimization Methods, and the use of Transformations in Constrained Problems
-
M.A. Schumer and K. Steiglitz (1968) Adaptive step size random search
-
L.J. White and R.G. Day (1971) An Evaluation of Adaptive Step-Size Random Search
-
J. Mockus, V. Tiesis, A. Zilinskas (1978) The Application of Bayesian Methods for Seeking the Extremum
-
J. Bernussou and J. Geromel (1981) An easy way to find gradient matrix of composite matricial functions
-
...
-
...
-
Khoirul Faiq Muzakka, Ahsani Hafizhu Shali, Haris Suhendar, Sören Möller, Martin Finsterbusch (2026) Robust Differential Evolution via Nonlinear Population Size Reduction and Adaptive Restart: The ARRDE Algorithm, Minion (github), Minion Issue 11, algolist
Farewell to matrices and convergence proofs.
At the moment I have two variants of my own "improved" ARRDE, call them M1 and M2, but do they improve Minion?
This space of super tuned algorithms is very demanding to test. One needs harder problems to discern the cases, they in turn demand big eval budgets (a lot of computational time), and the algorithms turn out to be very sensitive to random seeds which I simply choose as dates here.
Single optimization is already painfully slow, ~1000s.
In this problem fopt = 2400, f = 2500 is reached by any strong variant of CMAES or DE.
| Seed | M1 | M2 | Minion |
|---|---|---|---|
| 20250306 | 2500 | 2500 | 2500 |
| 20260818 | 2500 | 2500 | 2500 |
| 20260820 | 2500 | 2500 | 2500 |
| 20260821 | 2500 | 2500 | 2500 |
| 20260822 | 2500 | 2500 | 2500 |
| 20260823 | 2500 | 2500 | 2500 |
| 20260824 | 2500 | 2500 | 2500 |
| 20260825 | 2400 | 2500 | 2500 |
| 20260826 | 2400 | 2500 | 2500 |
| 20260827 | 2500 | 2500 | 2500 |
| 20260828 | 2500 | 2500 | 2500 |
| 20260829 | 2500 | 2500 | 2500 |
One may conclude that M1 is indeed an improvement, but the F25 test below will negate that.
Note: Minion is capable of getting into 2400 with some of these seeds, but one needs to add a zero starting point. The test uses default parameters and nothing extra added, same with the F25 below. F24 Seed=20250306 with a zero improves Minion's ARRDE from f=2500 to f=2400. No improvement for F24 Seed=20260821.
These independent runs with different seed numbers show the need for at least 200M evals and O(10) restarts to discern ARRDE from BIPOP-aCMAES, which is at least 2B evals or 1e8xD. This is a lot. On the positive side, restarts are parallelizable. However, the non-parallelizable part is already taking 1000s.
Increasing evals to 500M, adding a zero, may not improve anything:
| Seed | M1 | M2 | Minion |
|---|---|---|---|
| 20260829 | 2500 | 2500 | 2500 |
In this problem fopt = 2500, most of the strong algorithms reach ~2900, ARRDE: f = 2800.
| Seed | M1 | M2 | Minion |
|---|---|---|---|
| 20250306 | 2899 | 2899 | 2800 |
| 20260818 | 2800 | 2899 | 2800 |
| 20260820 | 2800 | 2800 | 2899 |
| 20260821 | 2899 | 2899 | 2899 |
| 20260822 | 2899 | 2899 | 2899 |
| 20260823 | 2899 | 2899 | 2800 |
| 20260824 | 2899 | 2899 | 2899 |
| 20260825 | 2899 | 2899 | 2899 |
| 20260826 | 2800 | 2899 | 2800 |
| 20260827 | 2800 | 2800 | 2800 |
| 20260828 | 2800 | 2899 | 2800 |
| 20260829 | 2899 | 2800 | 2800 |
Minion's ARRDE required the least number of restarts here.
Increasing evals to 500M can be critical:
| Seed | M1 | M2 | Minion |
|---|---|---|---|
| 20260821 | 2899 | 2899 | 2800 |
| 20260824 | 2800 | 2800 | 2800 |
The zero inclusion does not change anything with these two seeds.
F25 CEC-2017 D=20, 1B evals: ARRDE f=2700. Seed=20260829, single run takes 4.68 hours on i7 gen 4 16GB RAM. I have made a modification (call it "M3") which reaches f=2700 in 500M evals.
F28 CEC-2017 D=20, <=200M evals: ARRDE f=3000, BIPOP-aCMAES f=3100; fopt = 2800.
Notice that F24 and F25 are night and day:
F24 CEC-2017: restarts are critical (10 is not enough), longer runs do not improve anything, 200M evals are sufficient to solve the problem completely (f=fopt=2400) when the seed is good.
F25 CEC-2017: restarts are still needed (10 is too much), very long runs are essential, 1B evals still do not solve the problem (f=2700, fopt=2500).
I got sidetracked. The main idea was to share a surprise pulled by the ES on "Rastrigins" (quadrics mixed with harmonics). This superpower did not generalize to ill-conditioned functions.
My recommendations (experience) for problems in D = 20..40:
-
pycma CMAES: replaces Newton and Bayesian Optimization. 10K evals.
-
pycma BIPOP-aCMAES: very solid black-box/baseline, often not much is gained with extreme budgets. 10M evals.
-
ARRDE: better than BIPOP-aCMAES on CEC-2017 composites. More than 200M..1B evals.
ARRDE does not need matrices/linear algebra, but it demands C++.