Where are you allowed to look for a minimum?

Source scope: §4.1, PDF pages 27. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

The same curve can have different answers

For $f(x)=x^2$, minimizing over $\mathbb R$ gives $x=0$. Over $[1,3]$ it gives $x=1$. Over $(0,3]$ no minimum exists: moving closer to zero always lowers the value, but zero is excluded. The curve alone does not specify the problem.

A cost function, or objective function, is a scalar function of several variables. In machine learning it often measures how poorly a model fits data. The feasible set $X\subseteq\mathbb R^d$ lists allowed inputs. A problem is unconstrained when $X$ is the function’s entire domain; otherwise it is constrained. Constraints can change both the answer and the difficulty of finding it.

Compare nearby points or all points

Let $f:\mathbb R^d\to\mathbb R$ and $x^*\in X$.

• A local minimum in $X$ satisfies $f(x^*)\le f(y)$ for every $y\in X\cap B_\epsilon(x^*)$ for some $\epsilon>0$.
• A global minimum in $X$ satisfies $f(x^*)\le f(y)$ for every $y\in X$.
• A strict local or global minimum replaces $\le$ by $<$ for all compared $y\ne x^*$.
• Local and global maxima reverse the inequalities.

Strictness excludes equality with other points, not with $x^*$ itself. It makes the minimizer unique within the stated comparison set. A local minimum need not be global. The source assumes optimization over the whole domain when “in $X$” is omitted and the context does not say otherwise.

Source footnote 8 and boundary clarification. A neighborhood of $x$ is an open set containing $x$. For a constrained problem use the relative neighborhood $X\cap B_\epsilon(x)$; it need not be open in the ambient space. This expresses the source’s local comparison while retaining boundary points such as $x=1$ in $[1,3]$.

Turn a maximum into a minimum

Maximizing $f$ is equivalent to minimizing $-f$. Every comparison reverses when multiplied by $-1$. The source therefore discusses minimization without losing generality.

At a boundary minimum the gradient may be nonzero. For $x^2$ on $[1,3]$, $f'(1)=2$. Later gradient-zero conditions apply to unconstrained interior points with differentiability, not to every feasible minimum.

Transfer

For $f(x)=x$ on $(0,1)$, neither a minimum nor a maximum exists. Zero and one are an infimum and a supremum, respectively, but neither is attained. On $[0,1]$ both extrema exist at the endpoints.

Explore the relationship