This book presents the first part of a planned two-volume series devoted
to a systematic exposition of some recent developments in the theory of
discrete-time Markov control processes (MCPs). Interest is mainly
confined to MCPs with Borel state and control (or action) spaces, and
possibly unbounded costs and noncompact control constraint sets. MCPs
are a class of stochastic control problems, also known as Markov
decision processes, controlled Markov processes, or stochastic dynamic
pro- grams; sometimes, particularly when the state space is a countable
set, they are also called Markov decision (or controlled Markov) chains.
Regardless of the name used, MCPs appear in many fields, for example,
engineering, economics, operations research, statistics, renewable and
nonrenewable re- source management, (control of) epidemics, etc.
However, most of the lit- erature (say, at least 90%) is concentrated on
MCPs for which (a) the state space is a countable set, and/or (b) the
costs-per-stage are bounded, and/or (c) the control constraint sets are
compact. But curiously enough, the most widely used control model in
engineering and economics--namely the LQ (Linear system/Quadratic cost)
model-satisfies none of these conditions. Moreover, when dealing with
"partially observable" systems) a standard approach is to transform them
into equivalent "completely observable" sys- tems in a larger state
space (in fact, a space of probability measures), which is uncountable
even if the original state process is finite-valued.