Definition
Continuous State Space
A state space is called continuous if the variables that define the state can take on any value within a continuous range. In other words, the state variables are real-valued and are not restricted to a finite set of discrete values.
Search Approaches
Constraint Optimisation (Mathematical Programming)
This is a method for finding the best possible solution (optimising) according to a specific goal (the objective function ) while respecting certain rules or limitations (the constraints, ).
Convex optimisation: It occurs when the set of solutions allowed by the constraints forms convex region (geometrically, a shape where a line drawn between any two points within the shape stays entirely within the shape) and the objective function is convex.
Discretisation
This is the process of converting a continuous problem space (where variables can take any real value within a range) into a discrete one (where variables can only take specific, separate values).
Gridding: Impose a grid with a fixed spacing onto the continuous space, where only points on the grid are considered valid states. This reduces the infinite possibilities to a finite set.
Sampling: Instead of a rigid grid, randomly sample potential “successor” states near the current state.
Empirical Gradient Search: Steepest-ascent hill-climbing for discretised version.
Gradient Methods
This is an analytical approach to find an optimum of . Consider the gradient:
for the local change of . We aim to solve (find the closest form):
, which is often not feasible. To compute the local gradients for steepest-ascent hill-climbing (gradient ascent):
Newton-Raphson method: This another, often effective, method, i.e. iterate:
where is the matrix with (Hessian matrix of second derivatives).