$\newcommand{\R}{\mathbb R }$ $\newcommand{\N}{\mathbb N }$ $\newcommand{\Z}{\mathbb Z }$ $\newcommand{\bfa}{\mathbf a}$ $\newcommand{\bfb}{\mathbf b}$ $\newcommand{\bff}{\mathbf f}$ $\newcommand{\bfu}{\mathbf u}$ $\newcommand{\bfx}{\mathbf x}$ $\newcommand{\bfy}{\mathbf y}$ $\newcommand{\ep}{\varepsilon}$
Definition. A set $S\subset \R^n$ is said to be compact if every sequence in $S$ has a subsequence that converges to a limit in $S$.
A technical remark, safe to ignore.
In more advanced mathematics courses, what we have defined above
is called sequentially compact, and the word compact is reserved for something a little different.
Our first main theorem about compactness is the following:
Theorem 1: Bolzano-Weierstrass Theorem $\ \ S\subset \R^n$ is compact $ \ \ \iff \ \ $ $S$ is closed and bounded.
Remark 1.We emphasize that although, as the theorem shows, compact
is the same as closed and bounded
for subsets of Euclidean space,
it is not always true that
compact = closed and bounded.
How can this be?
Because there are vast realms of mathematics, none of which we will discuss in this class, that
take place in settings much more general (and much bigger
) than
finite-dimensional Euclidean space.
See here for a vivid illustration of the importance of the fact that compact is not always the same as closed and bounded. In practice, however, this only really becomes important if you study more advanced mathematics. In this class, it will probably always be true that compact is the same as closed and bounded.
Remark 2. We have called this the Bolzano-Weierstrass Theorem. That name is sometimes given to what we called the Bounded Sequence Theorem in the previous section. The two theorems are very closely related.
Below we will present the proof of Theorem 1. One ingredient in the proof is the following proposition, which we present separately, since it is a basic and useful fact.
Proposition 1. If $\{ \bfx_j\}_j$ is a convergent sequence in a $closed$ set $S\subset \R^n$, then the limit of the sequence must belong to $S$.
This is a straightforard deduction from the definitions, and you should be able to do it yourself. But if you prefer to read the details, then here they are:
Proof of Proposition 1.
Let $\{ \bfx_j\}_j$ be a sequence in a closed set $S\subset \R^n$, and assume that $\bfx_j\to \bfx$ as $j\to \infty$. We will show that $\bfx \in S$. Since $S$ is closed, this is the same as proving that $\bfx \in \bar S$, which is the same as showing that \begin{equation}\label{l1} \forall \ep >0, \quad B(\ep, \bfx) \cap S \ne \emptyset. \end{equation} To do this, note that by the definition of limit, and the convergence of the sequence, we know that $$ \forall \ep >0, \quad \exists J>0\mbox{ such that }\bfx_{j}\in B(\ep, \bfx) \quad\mbox{ for all }j\ge J. $$ Since $\bfx_{_j} \in S$, this implies that \eqref{l1} holds. $\quad \Box$
Relying in part on the Proposition, we can now give the
Proof of Theorem 1 (with some details left to you).
Proof of $\Leftarrow$.
Assume that $S$ is closed and bounded, and let $\{ \bfx_j\}_j$ be any sequence in $S$. Since $S$ is bounded, the Bounded Sequence Theorem implies that there is a convergent subsequence, say $\{ \bfx_{k_j}\}_j$. Since $S$ is closed, Proposition 1 above implies that the limit
of this subsequence must belong to $S$. Thus every sequence in $S$ has a a subsequence that converges to a limit in $S$. This is $\Leftarrow$.
(Note that the hardest part of the proof was contained in the Bounded Sequence Theorem from the previous section.)
Proof of $\Rightarrow$. (sketch)
We will show that unless $S$ is closed and bounded, there exists a sequence in $S$ with no subsequence that converges to a limit in $S$.
If $S$ is not closed and bounded, then it is either not closed or not bounded. We will consider these two cases separately:
Case 1: $S$ is not bounded. This means that there does not exist any $R>0$ such that $S \subset B(R, {\bf 0})$.
How can we use this assumption to define a sequence $\{ \bfx_j\}_j$ in $S$ that does not converge to a limit in $S$ (in fact, that does not converge at all)? Figure out the idea, and fill in the details if you wish. Your argument should look roughly like this: For each $j$, let $\bfx_j$ be a point in $S$ such that .... We know that such a point exists because .... This procedure for choosing points $\bfx_j$ defines a sequence $\{ \bfx_j\}_j$. The sequence cannot have a limit because the way that we chose it implies that ...
Case 2: $S$ is not closed. This means (see Theorem 4 in Section 1.1
that there exists $\bfx\in \partial S$ such that $\bfx \not\in S$.
The definition of boundary
then implies for every $\ep>0$
$$
B(\ep , \bfx) \cap S \ne \emptyset.
$$
Since $\bfx\not\in S$, it follows that
$$
\forall \ep >0, \exists \bfy\in S\mbox{ such that }\bfy\ne\bfx
\mbox{ and }|\bfy - \bfx|<\ep.
$$
The idea is to use this assumption to show that we can
find a sequence in $S$ that converges to $\bfx$. Then every subsequence will also converge to
$\bfx$, and so (since $\bfx\not\in S$) no subsequence can converge to a limit in $S$. Try to figure out how to choose the sequence; then
if you wish, you can write down the details, which should look
like the missing details from Case 1 above. $\quad \Box$
Theorem 2: The Extreme Value Theorem . Assume that $K$ is a compact subset of $\R^n$, and that $f:K\to \R$ is continuous. Then \begin{equation}\label{evt0}\mbox{ the set } \{ f(\bfx) : \bfx \in K \} \mbox{ is compact,} \end{equation} and there exists $\bfx^*$ and $\bfx_*$ in $K$ such that \begin{equation}\label{evt1} f(\bfx^*) = \sup \{ f(\bfx) : \bfx \in K \}, \qquad f(\bfx_*) = \inf \{ f(\bfx) : \bfx \in K \}. \end{equation}
When there exists $\bfx^*$ as in \eqref{evt1}, we say that $f$ attains its supremum, and when there exists $\bfx_*$, we say that $f$ attains its infimum.
Notation. We often use the abbreviations $$ \inf_S f := \inf \{ f(\bfx) : \bfx \in S \}, \qquad \sup_S f := \sup \{ f(\bfx) : \bfx \in S \}. $$
We will discuss the proof below, after first seeing what the theorem is good for.
A major application of mathematics is to optimization problems: figure out a way to do something that maximizes good outcomes, minimizes bad outcomes, and respects certain constraints.
The Extreme Value Theorem is useful because it can sometimes guarantee that an optimization problem must have a solution. Its weakness is that it does not given any indication how to find the solution, when it exists; but at least it can potentially spare us from wasting time and effort that we might otherwise spend looking for a nonexistent solution.
Consider the following two problems:
Example 1. Design a rectangular box in a way that maximizes the volume of the box, while respecting the constraint that the $$ \mbox{ depth+width+height } \le 3. $$
Example 2. Design a rectangular box in a way that maximizes depth+width+height, while respecting the constraint that the $$ \mbox{ volume of the box } \le 3. $$
By formulating these as problems of maximizing a continuous function in a set $S\subset \R^3$, one can see that the Extreme Value Theorem guarantees that one of the problems has a solution (without telling us anything about how to find that solution, a question we will address later). It does not tell us anything about the other problem, which in fact does not have a solution.
You are asked to consider these in the Problems below.
Example 3. Consider the problem $$ \mbox{ minimize } f(x,y) = x^2+y^2 + \cos(x^3 + e^y) \mbox{ on }\R^2. $$ This is a short way of writing: find $(x_*, y_*)\in \R^2$ such that $f(x_*, y_*) = \inf_{\R^2} f$.
Does a solution exist?
Here the Extreme Value Theorem does not seem to help us, since $\R^2$ is not compact. Nonetheless, we can use it as follows:
Proposition 2. Assume that $\{ {\bf z}_j\}_j$ is a sequence in a set $S\subset \R^k$ for some $k\ge 1$, and that ${\bf z}_j \to {\bf z}\in S$ as $j\to \infty$. Assume also that $f$ is a continuous function on $S$. Then $f({\bf z}_j ) \to f({\bf z})$ as $j\to \infty$.
The proof of this is an exercise in the practice problems.
We will also need
Proposition 3. If $S$ is a compact subset of $\R$, then $\sup S \in S$ and $\inf S\in S$.
This may be familiar from MAT137. If not, the proof is outlined below.
Incidentally, you may also remember that when $\sup S$ belongs to $S$, then we say that the supremum is actually a maximum, and we have the right to refer to it as $\max S$ istead of $\sup S$, if we want to.
Proof of Proposition 3 (sketch)
We will prove only that $\sup S\in S$, since the proof for $\inf$ is
almost identical.
First, since $S$ is compact, $S$ is bounded, so $S$ has a (finite) supremum. Let's call it $x = \sup S$. We have to show that
$x \in S$.
Since $S$ is compact and hence closed, this is the same as proving that $x \in \bar S$,
which is the same as showing that
\begin{equation}\label{p3}
\forall \ep >0, \quad (x-\ep, x+\ep)\cap S \ne \emptyset.
\end{equation}
(Recall that in $1$-dimension, the ball
$B(\ep, x)$ is just the interval $(x-\ep, x+\ep)$.) So our task is to prove \eqref{p3}.
To do this, we must use the definition of supremum: the least upper bound, and make some kind of argument. This can
probably be done in 1-2 sentences.
Using Proposition 3 (among other ingredients), we can now present the
Proof of the Extreme Value Theorem
Assume that $K$ is compact that that $f:K\to \R$ is continuous.
Let $S := \{ f(\bfx) : \bfx\in K\}$.
Proof that $S$ is compact. To show that $S$ is compact, we will prove that any sequence in $S$ has a subsequence that converges to a limit in $S$. Toward this end, consider a sequence $\{y_j\}_j$ in $S$. By definition of $S$, for each such point there is some $\bfx_j$ in $K$ such that $y_j = f(\bfx_j)$. Because $K$ is compact, there exists a subsequence $\{ \bfx_{k_j}\}_j$ that converges to a limit $\bfx_*\in K$. According to Proposition 2 above, since $f$ is continuous, $$ y_{k_j} = f(\bfx_{k_j}) \to f(\bfx_*) \quad\mbox{ as }j\to \infty. $$ Since $f(\bfx_*)\in S$, this proves that $\{y_j\}_j$ has a subsequence that converges to a limit in $S$. Since this argument applies equally well to any sequence in $S$, it follows that $S$ is compact.
Proof that $f$ attains its supremum. Since $S$ is compact, it follows from Proposition 3 that $\sup S \in S$. Let $y^* = \sup S = \sup\{ f(\bfx) : \bfx\in K\}$. Then the statement that $y^*\in S$ says exactly that $y^* = f(\bfx^*)$ for some $\bfx^*\in K$. In other words, $\sup_K f$ is attained at $\bfx^*\in K$.
Proof that $f$ attains its infimum. This is exactly parallel to the proof that $f$ attains its supremum.
Remark. The first part of the proof of the Extreme Value Theorem can be easily modified to show that if $K$ is a compact subset of $\R^n$ and $\bff:K\to \R^k$ is continuous, then $\{ \bff(\bfx) : \bfx\in K\}$ is a compact subset of $\R^k$.
Definition. a function $\bff: S\to \R^k$ (for some $S\subset \R^n$) is uniformly continuous if $$ \forall \ep >0, \ \exists \delta>0 \mbox{ such that } \ \ \left. \begin{array}{r} \bfx, \bfy\in S\mbox{ and } \\ |\bfx - \bfy|< \delta \end{array} \right\} \Rightarrow |\bff(\bfx) - \bff(\bfy)|<\ep. $$
Uniform continuity will be important for us when we begin to look at integration of functions of several variables, some months from now. (Until then we may not say much about it.)
Compare this to the definition of continuous, which is that $\bff$ is continuous if
$$
\forall \ep >0, \forall \bfx\in S, \ \exists \delta>0 \mbox{ such that } \ \
\left.
\begin{array}{r} \bfy\in S\mbox{ and } \\ |\bfx - \bfy|< \delta
\end{array}
\right\}
\Rightarrow |\bff(\bfx) - \bff(\bfy)|<\ep.
$$
The difference is that if a function is continuous, then the choice of $\delta$ may depend on $\bfx$, whereas if it is uniformly continuous, then given $\ep>0$ we can find a $\delta>0$ that works
for all $\bfx \in S$ simultaneously.
Example 4. Let $S := [0,100]\subset \R$, and let $f(x) = x^2$. Then $f$ is uniformly continuous on $S$.
Proof
Suppose we are given $\ep>0$. Then for any $x,y\in S$,
$$
|f(x) - f(y)| = |x^2-y^2| = |(x-y)(x+y)| = |x-y| \ |x+y| \le 200 |x-y|,
$$
since $|x+y| \le |x| + |y|\le 200$, using the assumption the $S = [0,100]$.
It follows that
$$
\mbox{ if }x,y\in S\mbox{ and } |x-y|\le \ep/200, \qquad\mbox{ then }
|f(x) - f(y)|<\ep.
$$
This shows that $f:S\to \R$ is uniformly continuous, as claimed.
Example 5. Let $S := [0,\infty)\subset \R$, and let $f(x) = x^2$. Then $f$ is not uniformly continuous on $S$.
Proof
Negating the definition of uniform continuity, we see that we have to check that
$$
\exists \ep>0 \mbox{ such that }\forall \delta>0, \exists x,y\in S\mbox{ such that }|x-y|<\delta \mbox{ but }|f(x)-f(y)|\ge\ep.
$$
Let's try to prove this for $\ep=1$. Thus, given $\delta>0$, we have to find $x,y\in S$ such that $|x-y|<\delta$ but $|f(x)-f(y)|\ge 1$.
Again we use the fact that
$$
|f(x)-f(y)| = |x-y|\ |x+y|.
$$
After staring at this, it seems like a good idea to choose
$x = \delta^{-1}$ and
$y= \delta^{-1} + \delta/2$.
Then $|x-y| = \delta/2 <\delta$, and
$$
|f(x)-f(y)| = |x-y|\,|x+y| = \frac \delta 2 \, (\frac 2 \delta + \frac \delta 2) = 1+ \frac1 4 \delta^2.
$$
Thus $f$ is not uniformly continuous. (You can easily check for yourself that in this example we could have chosen any $\ep>0$. This is not always the case however.)
The next example shows that we can sometimes use calculus to check that a function is uniformly continuous. (But so far we do not know any multi-variable calculus, so at this point we can only do this for functions of a single variable.)
Example 6. Let $f$ be a function that is continuous on a closed interval $[a,b]\subset \R$ and differentiable on the open interval $(a,b)$. If there exists some number $M>0$ such that $|f'(x)|\le M$ for all $x\in (a,b)$, then $f$ is uniformly continuous on $[a,b]$.
Proof
Consider any two points $x,y\in [a,b]$. Then according to the Mean Value Theorem, there exists $z$ between $x$ and $y$ such that $$ f(y)-f(x) = f'(z)(y-x). $$ Since $|f'(z)| \le M$ everywhere, by assumption, it follows that $$ |f(y)-f(x)|\le M |y-x|. $$ So, given $\ep >0$, it follows that for any $x,y\in S$, $$ |x-y|< \frac \ep M \qquad \Rightarrow \qquad |f(y)-f(x)|\le M |y-x| <\ep. $$ Thus $f$ is uniformly continuous.
Example 7. The sine and cosine functions are both uniformly continuous on all of $\R$.
This follows easily from Example 6.
Example 8. Let $S = [0,1]\subset \R$, and define $f:S\to \R$ by $f(x) = \sqrt x$. Then $f$ is uniformly continuous on $S$.
There are two things to note about this example.
We will see an easy way to prove this below.
In this example, $f$ is uniformly continuous, even though $f'$ is unbounded on $S$. (right?) So the condition $|f'|\le M$ in Example 3 is sufficient, but not necessary, for uniform continuity.
the rather clever proof, which we can avoid by just using Theorem 3 below.
Note that for any $x,z\ge 0$, $$ (\sqrt x+ \sqrt z)^2 = x + z + 2\sqrt{xz} \ge x+z = (\sqrt{x+z})^2. $$ It follows that $ \sqrt{x} + \sqrt{z} \ge \sqrt{x+z}$, or equivalently, that $\sqrt{x+z} - \sqrt{x} \le \sqrt{z}$.
Now suppose that $x,y\in S$, and that $x\le y$. If we let $z = y-x$, we find from the above inequality that $$ |\sqrt{y} - \sqrt{x}| = \sqrt{y} - \sqrt{x} \le \sqrt{y-x} = \sqrt{|y-x|}. $$ The reasoning is almost identical if $x>y$. Either way, we conclude that $$ |f(y)-f(x)| \le \sqrt{|y-x|} \quad\mbox{ for }x,y\in S. $$ So, given $\ep>0$ it follows that for any $x,y\in S$, if $|x-y|<\ep^2$, then $|f(y)-f(x)| \le \sqrt {|x-y|} <\ep$. Hence $f$ is uniformly continuous on $S$. $\quad \Box$
So far the only methods we have for checking that a function is uniformly continuous are
use calculus, which for now restricts us to functions of a single variable, or
think of some clever tricks, which can be hard to do.
The following theorem is valid in $\R^n$ for any $n\in \N$, and using it does not require us to have any special insight. (Proving it, on the other hand, does.)
Theorem 3. If $K$ is a compact subset of $\R^n$ and $f:K\to \R^k$ is continuous, then $f$ is uniformly continuous.
A proof of this theorem can be found in Section 1.8 of Folland's Advanced Calculus. A different proof can be found in the problems below.
Example 8 revisited. Clearly $[0,1]$ is compact, and we know from basic properties of continuity that $f$ is continuous. So $f:S\to \R$ is uniformly continuous. Easy!
More problems may be added later.
Give an example of
Determine whether the set $S = \ \ldots\ $ is compact; or Determine whether a sequence $\ \ldots \ $ has a subsequence that converges to a limit in a set $S = \ \ldots \ $.
Does the Extreme Value Theorem guarantee that the problem of minimizing/maximizing $f = \ldots$ with the constraints $\ldots$ must have a solution? For example:
Consider the problem of designing a rectangular box in a way that maximizes the volume while respecting the constraints height+width+depth $\le 3$. Does the Extreme Value Theorem guarantee that this problem must have a solution? (Clearly depth, width and height are all $\ge 0$.)
Consider the problem of designing a rectangular box in a way that maximizes height+width+depth while respecting the constraint that the volume $\le 3$. Does the Extreme Value Theorem guarantee that this problem must have a solution?
See Example 3 above for a model of how to address this kind of problem
Let $f(x,y) = x+y + e^{x^2-5x -y^2 -2y}$. Prove that the problem $$ \mbox{ minimize }f\mbox{ in the set } S := \{(x,y)\in \R^2: x\ge 0, y\ge 0\} $$ has a solution.
Let $f(x,y,z ) = x^2 + y^2 + z^2 +\frac{\sin yx}{1+z^2}$. Prove that the problem $$ \mbox{ minimize }f\mbox{ on }\R^3 $$ has a solution.
etc. Among the Basic Skills in this section, this is the one about which we can most easily think of diverse qusetions that are not too easy and not impossibly hard.
Give a different proof of Proposition 1, by showing that
if $S$ is a closed subset of $\R^n$, and $\{ \bfx_j\}_j$ is
any sequence in $\R^n$ that converges to a limit $\bfx \not\in S$, then
$\{\bfx_j\}_j$ is not a sequence in $S$ (in other words, that
$\bfx_j\not \in S$ for some $j$.)
Hint. $\bfx\not\in S$ means that $\bfx\in S^c$. If $S$ is closed, what do we know about $S^c$?
Fill in the details in the proof in Theorem 1 (sketched above) that in any unbounded set $S$, there exists a sequence with no convergent subsequence.
Fill in the details in the proof in Theorem 1 (sketched above) that in any set $S$ that is not closed, there exists a sequence with no convergent subsequence.
Prove Proposition 2.
Fill in the last missing details in the proof of Proposition 3.
Assume that $K_1$ is a compact subset of $\R^n$ and $K_2$ is a compact subset of $\R^m$. Define
$$
K_1\times K_2 := \{(\bfx, \bfy)\in \R^n\times \R^m = \R^{n+m} : \bfx \in K_1, \bfy\in K_2 \}.
$$
Prove that $K_1\times K_2$ is a compact subset of $\R^{m+n}$.
(In short, prove that a Cartesian Product of two compact sets is compact.)
There are two different possible proofs, using two different characterizations of compact sets.
Prove that a closed subset of a compact set is compact.
(a proof of Theorem 3) Assume that $K$ is a compact subset of $\R^n$ and that $\bff:K\to \R^k$ is continuous. Fix $\ep>0$ and consider the set $$ S := \{(\bfx, \bfy)\in K\times K : |\bff(\bfx) - \bff(\bfy)|\ge \ep\}. $$ Also, let $g:S\to \R$ by $g(\bfx, \bfy) = |\bfx - \bfy|$, and define $\delta = \inf \{ g(\bfx, \bfy) : (\bfx, \bfy)\in S\}$.