This book examines the mismatch between discrete programs, which lie at
the center of modern applied mathematics, and the continuous space
phenomena they simulate. The author considers whether we can imagine
continuous spaces of programs, and asks what the structure of such
spaces would be and how they would be constituted. He proposes a
functional analysis of program spaces focused through the lens of
iterative optimization.
The author begins with the observation that optimization methods such as
Genetic Algorithms, Evolution Strategies, and Particle Swarm
Optimization can be analyzed as Estimation of Distributions Algorithms
(EDAs) in that they can be formulated as conditional probability
distributions. The probabilities themselves are mathematical objects
that can be compared and operated on, and thus many methods in
Evolutionary Computation can be placed in a shared vector space and
analyzed using techniques of functional analysis. The core ideas of this
book expand from that concept, eventually incorporating all iterative
stochastic search methods, including gradient-based methods. Inspired by
work on Randomized Search Heuristics, the author covers all iterative
optimization methods and not just evolutionary methods. The No Free
Lunch Theorem is viewed as a useful introduction to the broader field of
analysis that comes from developing a shared mathematical space for
optimization algorithms. The author brings in intuitions from several
branches of mathematics such as topology, probability theory, and
stochastic processes and provides substantial background material to
make the work as self-contained as possible.
The book will be valuable for researchers in the areas of global
optimization, machine learning, evolutionary theory, and control
theory.
